Data Vault Modeling & Methodology. Technical Side and Introduction Dan Linstedt, 2010,

Transcription

1 Data Vault Modeling & Methodology Technical Side and Introduction Dan Linstedt, 2010,

2 Technical Definition The Data Vault is a detail oriented, historical tracking and uniquely linked set of normalized tables that support one or more functional areas of business. It is a hybrid approach encompassing the best of breed between 3 rd normal form (3NF) and star schema. The design is flexible, scalable, consistent and adaptable to the needs of the enterprise. Architected specifically to meet the needs of today s enterprise data warehouses 2

3 Customer What Does One Look Like? Records a history of the interaction Product Elements: Hub Link Satellite Sat Sat Customer Sat F(x) Sat Link F(x) F(x) Sat Product Sat Sat Sat Hub = List of Unique Business Keys Link = List of Relationships, Associations Satellites = Descriptive Data Order F(x) Order Sat Sat 3

4 Excel As A Source Level A Level B Level C Item Item Item Staging Table Hub Grouping Link Acct To Group Hierarchical Link of Groups Sat Group Type User Grouping Structures Flattened Structure Hub Account Raw Source Data in DV Do you have a power executive who is technically inclined, who runs the business off a rogue spreadsheet? 4

5 Data Vault Basic Elements CORE ARCHITECTURE 5

6 Data Vault Core Architecture Hubs, Links, Satellites Hubs = Unique List of Business Keys Links = Unique List of Relationships across keys Satellites = Descriptive Data Satellites have 1 and only one parent table Satellites cannot be Parents to other tables Hubs cannot be child tables Last Seen Dates, Load Dates, Record Sources, and Surrogate keys are not part of the core architecture. They exists to help models and key migration. 6

7 Hub Entity A Hub is a list of unique business keys Hub Structure Primary Key <Business Key> Load DTS Last Seen DTS Record Source Unique Index (Primary Index) Hub Product Product Sequence ID Product Number Product Load DTS Product Last Seen DTS Prod Record Source A Hub s business key is a unique index A Hub s load date represents the FIRST TIME the EDW saw the data A Hub s record source represents: First the Master data source (on collisions), if not available it holds the origination source of the actual key 7

8 Link Entity A Link is an intersection of two or more business keys It can contain Hub keys and other Link keys Link Structure Primary Key Link Line-Item Link Line Item Sequence ID {Hub/Lnk Surrogate Keys 2..N} Load DTS Last Seen DTS Record Source Unique Index (Primary Index) Hub Product Sequence ID Hub Order Sequence ID **Line Item Number Load DTS Last Seen DTS A Link s business key is a composite unique index Record Source A Link may or may not have a **Item Numbering attribute A Link s load date represents the FIRST TIME the EDW saw the data A Link s record source represents: first the Master data source (on collisions), if not available, it holds the origination source of the actual key 8

9 Satellite Entity A Satellite is a time-dimensional table housing detailed information about the Hub s or Link s business keys Primary Key Load DTS Extract DTS **Load End Date Detail Business Data {Update User} {Update DTS} Record Source Unique Index (Primary Index) Customer # Load DTS Extract DTS **Load End Date Customer Name Customer Addr1 Customer Addr2 {Update User} {Update DTS} Record Source Satellites are defined by TYPE of data and RATE OF CHANGE Mathematically this reduces redundancy and decreases storage requirements over time (compared to a star schema) 9

10 Rules and Standards GOVERN your deployment THINKING OF BREAKING RULES 10

11 Some Rules For You NO Foreign Keys in the Satellites! NO Hub to Hub (Parent Child relationships) NO Enforcement of relationships in the data model NO Date Time attributes in HUB or LINK Primary Keys Why?? It breaks flexibility It breaks auditability / accountability It breaks Scalability It breaks Performance It introduces Decisions in the architecture, which breaks Patterns! Up Next Links and the Unit Of Work 11

12 Business Key Definitions The contracts system is responsible for creating customer account numbers. The EDW will never see other systems creating customer account numbers. (Requirement #101) Sales is clearly creating customer numbers, how do we detect the issue and alert the business? Point: Not all business keys are created EQUAL! 12

13 Link: Unit of Work Hub Category Link Prod-Cat Hub Product Sat Effectivity Link Line Item Unit Of Work Link: Product by Supplier by Category Link Prod-Supp Sat Effectivity Hub Supplier Link Product by Category Link Product by Supplier These links are Optional, used For exploration only 13

14 What Happens When: We Break the Unit of Work Source System UOW Product_ID Category_ID Supplier_ID Link Product by Supplier Product_ID Supplier_ID Link Product by Category Product_ID Category_ID Model Normalization Question: After normalizing, how can you reconstruct the source image EXACLTY as it stands? 14

15 What Happens When: Trying to Rebuild from Two Links Source System UOW Product_ID Category_ID Supplier_ID Model Normalization Link Product by Supplier Product_ID Supplier_ID Link Product by Category Product_ID Category_ID Re-joining the data, creates a record that does not exist in the original source system, this is the same problem that BI engines will have when putting together Data Mart results. 15

16 Link: Unit of Work Kept Together Source System Source Table UOW Product_ID Category_ID Supplier_ID Data Vault Link: Product by Category by Supplier Product_ID Category_ID Supplier_ID Commutative Property: Enable reproduction of the source exactly as it stands UOW is properly represented by a single Link in the Data Vault 16

17 What keeps you up at night? CURRENT LOADING PAIN 17

18 Problems with EDW Loads Today Technical Issues: 2am Wakeup Calls because data won t fit the business rules Emergency Fixes to Production Speed, Speed, Speed (shrinking load window + more data) Can t load real-time data (business rules in the way!!) Business won t buy better, faster, hardware! Business Issues: Maintenance cycles take too long Maintenance costs continue to increase Fixes to existing mappings break working logic Complexity of existing systems become unsustainable to business IT isn t using 80%+ of the hardware resources given to them (their jobs are running at 40% utilization when they are full-bore ) 18

19 Solutions! Technical Solutions All Parallel Job Streams As much as possible 1 Target Per Map, Per Action reduces complexity Generate Data Flows based on patterns (then focus on the real work) Get some SLEEP at night!! (no more production modifications) Business Solutions Decrease turn-around time Increase Performance Handle Real-Time Data!! Reduce Complexity = Reduce Costs, Reduce Time to Implement Get the power back for decision making, discovering and building your own marts 19

20 How? 20

21 Some standards to follow BASIC LOADING CONCEPTS 21

22 Loading: A Golden Rule 100% of the Data Loaded to the EDW 100% of the time! It s all about Auditability 22

23 Load Date / End Date Geology Batch Load Real-Time Loading 23

24 Real Time Loading - DV Stock Trade ACCOUNT= TRADE="Buy" STOCK= DAN" SHARES=100.0 CURRENCY="USD" PRICE= DATE="Feb 20, 2002 Comment="Buy Order to Execute" = Inserts Only, no Updates Acct Hub Trade Link DAN Stock Hub TRADE="Buy" SHARES=100.0 CURRENCY="USD" PRICE= DATE="Feb 20, 2002 Comment="Buy Order to Execute" Transactional Link # of Inserts 75M 50M 25M 10M First Data Set Loaded New Systems Data Added Months in Production As critical mass of current business keys is reached, the insert rates decrease rapidly. New systems add new keys, quickly and efficiently to an existing Hub. 24

25 Batch Load Date Time Stamp Stage Load Stage Load Staging Area CNTRL_DTE LOAD_DTS STAGING TABLE Sequence_ID. Load_DTS Record_Source STAGING TABLE Sequence_ID. Load_DTS Record_Source EDW Data Vault Load Date Is exactly the same For All rows 25

26 Parallel Load Architecture - Batch Staging Loads Data Vault Loads Data Mart Loads Sources Stage Hubs Hub Satellites Link Satellites Dimensions Facts Links Major Synchronization Points Processing: All loads are done in parallel Sets of processes wait for the previous set to complete Processes are run as soon as data is ready No other waiting time is required Load dependencies are greatly reduced 26

27 Mathematics of Batch Loading Its all about SPEED SPEED SPEED 10 Million Incoming Rows 60% - 80% Inserts (Never Seen Before) 10%-20% Updates Matched By KEY 5% Deletes EDW: 1 Billion Rows And growing Inserts are the single fastest operation in the Database! Updates are the single slowest operation in the Database! Q: Why push 80% of your Insert data through the heaviest/slowest transformation logic? 27

28 Simple Loading Patterns Rule: 1 Target Per Data Flow (map/graph) Per Action Source SQ LKP Target Filter If Exists Target Insert Source (Stage) Insert View: Select ALL that do not exist By PK in target Update View: Select ALL that exist By PK in target ONLY those with DELTA Source SQ Target Insert Target 28

29 Results of Pattern Tuning FROM THIS.. 5M 600 RPS = 2.31 hrs OR: 7k rps = 11.9 mins No parallelism This map must run at a minimum of 10k rps to beat the parallel times 10k rps = 8.33 mins TO THIS! Pass 1: 33k RPS = 2.52 mins Pass 2: 33k RPS = 2.52 mins 25k RPS = 3.33 mins Pass 3: 50k RPS = 1.66 mins 33k RPS = 2.52 mins 40k RPS = 2.03 mins 23k RPS = 3.61 mins Total Time: = 9.46 mins 29

30 Patterns Take the Cake! LOADING THE DATA VAULT 30

31 Loading Templates: Hubs Staging Data Distinct List BK Keys Exists In Target? No Insert Into Target (Gen Surrogate) Hub Yes Drop Row From Feed Select a Master system, and a hierarchy of importance for sub-systems to annotate arrival location of data Purpose of the loading template: Find out if the business key exists in the hub, if not insert it Use a distinct list (unique) of business keys coming from the staging area 31

32 Loading Templates: Links Staging Data Distinct List Busn Keys Lookup EACH Hubs Surrogate Keys Exists In Target? No Insert Into Target (gen surrogate) Link Yes Drop Row From Feed Select a Master system, and a hierarchy of importance for sub-systems to annotate arrival location of data Purpose of the loading template: Find all relationships between business keys, then, is the relationship already recorded in the Link, if not insert it Use a distinct list of related business keys 32

33 Loading Templates: Satellites Staging Data Distinct List Sat Rows Lookup EACH Hub s or Link s Surrogate Keys All Columns Match? No Insert Into Target Satellite Find Latest Sat Row Yes Drop Row From Feed Select a Master system, and a hierarchy of importance for sub-systems to annotate arrival location of data Purpose of the loading template: Gather descriptive data, compare to most recent copy of information in satellite, and if there are any deltas load, if not, don t load Use a distinct list of descriptive fields from the source systems 33

34 How to build your Data Vault GETTING STARTED HOW TO 34

35 Step 1: Establish Scope (Build Business Case Model) 35

36 Step 1: Define Business Keys Hub Invoice Hub Campaign Hub Customer Hub Products 36

37 Step 2: Define Associations Hub Invoice Hub Campaign Link Campaign by Invoice by Customer Hub Customer Link Product on Campaign Hub Products Link Invoice Line Items 37

38 Step 3: Define Descriptive Data Hub Invoice Hub Campaign Link Campaign by Invoice by Customer Hub Customer Sat Effectiveness Ratings Sat Effectiveness Dates Sat Dates and Amounts Sat Address Sat Details Sat Contacts Link Product on Campaign Hub Products Link Invoice Line Items Sat Availability Dates Sat Defect Reasons Sat Descriptions Sat Stock Quantities Sat Amounts Sat Quantities 38

39 Step 4: Build Source Model (PK/FK) (No Pictures, Sorry) Ensure the source model (DDL Only) has Primary and Foreign Keys defined Normalize the source model (if not normalized) Capture and integrate all source systems involved (if not already captured) Add Comments to the DDL (tables and fields) 39

40 Step 5: Build Cross-Reference The purpose of such an exercise is not to identify all the elements, but specifically to identify the target Hubs, (ie: the business keys), target Links, and at LEAST a single Satellite for at least 1 source column The engine (SaaS) will automatically assign all other descriptive elements to the first Satellite identified. SOURCE TABLE SOURCE COLUMN GROUP TARGET TABLE TARGET COLUMN AHLTAT_DIAGNOSIS DOC_REF 1 SAT_AHLTAT_DIAGNOSIS DOC_REF DATAID 1 HUB_DIAGNOSIS DIAGNOSIS_DATAID FACILITYNCID 1 HUB_FACILITY FAC_ID DIAGNOSISNCID 1 SAT_AHLTAT_DIAGNOSIS DIAGNOSISNCID ENCOUNTERNUMBER 1 HUB_EVENT EVNT_ID CLINICIANNCID 1 HUB_CLINICIAN CLINICIAN_NCID UNIT_NUMBER 1 HUB_UNIT UNIT_ID MEDCINID 1 HUB_MEDCIN MEDCIN_ID CREATETIME 1 SAT_AHLTAT_DIAGNOSIS CREATETIME CREATEUSERNCID 1 SAT_AHLTAT_DIAGNOSIS CREATEUSERNCID MODIFYUSERNCID 1 SAT_AHLTAT_DIAGNOSIS MODIFYUSERNCID MODIFYTIME 1 SAT_AHLTAT_DIAGNOSIS MODIFYTIME PRIORITY 1 SAT_AHLTAT_DIAGNOSIS PRIORITY DIAGNOSESCOMMENT 1 SAT_AHLTAT_DIAGNOSIS DIAGNOSESCOMMENT 40

41 Step 6: Generate Baseline ETL/ELT Source DDL Cross-Ref Mapping XLS Target DDL Generate Code, Reports, Documentation Data Flows (Mappings / Graphs) 41

42 What did we learn? CONCLUSIONS / SUMMARY 42

43 Data Vault Modeling Is Made up of Hubs, Links, and Satellites Easy to create and build Hardest thing is to find/locate and define the Business Keys Consistent, Scalable, Repeatable, Pattern Based RULES BASED / STANDARDS DRIVEN Loading Is. Scalable, Fault-Tolerant, Parallelizable, Pattern Based Generatable Performance Based 100% Restartable Set Based Devoid of Soft Business Rules!! 43

44 Still - Lots To Learn We didn t cover: Joins point-in-time tables building marts business logic components SQL extraction bridge tables what to do when dealing with bad data architecting security, managing governance, handling metadata Contact me for Workshops (training), and Mentoring 44

45 Questions? Dan Linstedt President, Empowered Holdings, LLC Tel: SERVICES: Consulting Assessments Product Selection Scorecards Architecture / Design Mentoring and Workshops (training) 45