Modeling the. Agile. with Data Vault. Data Warehouse. Hans Hultgren

Transcription

1 Agile Modeling the Data Warehouse with Data Vault Hans Hultgren

2 Contents FORWARD 4 ABOUT THE AUTHOR 7 ACKNOWLEDGEMENTS 8 CHAPTER 1 DATA VA ULT DEF IN ED data Vault is a Data Modeling Approach The Underlying Premise of Data Vault SingleGrainofKey Concept Constellations Where Data Vault ensemble fits In DATA VAULT CORE CONSTRUCTS DATA VAULT METHODOLOGY WHY DATA VAULT WHEN TO USE DATA VAULTMODELING 29 CHAPTER2 ENSEMBLE MODELING ensemble modeling ensemble defined data vault ensemble Applying the DataVault ensemble entityversus data vault ensemble 37 CHAPTER 3 DATA MODELING modeling and design data modeling approaches the normalized database & 3 normal form dimensional modeling and the star schema Common factorsand Data Vault Modeling integration & the enterprise data warehouse Applying modeling Techniques to EDW optimal mod eling for dl fferent layers Addressing the Issues we Experience today modeling approach Summary 54 CHAPTER 4 MODELING WITH DATA VAULT Data Vault Modeling THE business Key the modeling process adata vault model 65 CHAPTER 5 COLORS OF DATA VAULT Data Modeling Color Category Analysis colors of 3rd Normal Form Colors of star Schema colors analysis for data vault comparing colors for the edw summary ofthe colors 80 Page 10

3 CHAPTER 6 CORE CONSTRUCTS: HUB THE HUB Applying Hubs in data Vaultmodeling Identifying and Modeling Hubs hub Attributes 87 CHAPTER 7 CORE CONSTRUCTS: LINK THE LINK Applying Links in data Vaultmodeling relationsh i p I ntegrity / Conti nuity links and Cardinality Common Applications of a Link relative Cardinalityof w-way Relationships identifying and Modeling Links Modeling Recursive Relationships Link Attributes 107 CHAPTER 8 CORE CONSTRUCTS: SATELLITE The Satellite Applying Satellites in data vault modeling Designing and Modeling Satellites Satellite Attributes 117 CHAPTER 9 THINKING DIFFERENTLY WITH DATA VAULT Thinking differentlywith Data Vault changing Modeling Paradigms Implications of the New Paradigms Satellite Keys links and Cardinality Foreign Keys in asatellite Summaryof thinking Differently CHAPTER 10 MODELING A CORE BUSINESS CONCEPT The Core Business Concept Aboutthe EnterpriseView Whereto find the enterprise View Modeling the Core business Concept core Concept Super-types and Sub-types 143 CHAPTER 11 ENTERPRISE DATA WAREHOUSING Introduction to Enterprise Data Warehousing Definition and Characteristics 149 CHAPTER 12 DWBI DWBI Bl in the Organization DWBI VISUALIZATION SUMMARY 165 CHAPTER 13 DATA INTEG RATION DATA INTEGRATION 168 Page 11

4 13.2 The Semantic Gap Integration is notbased on Data ETLand data visualization ETLMechanicsand the Pipeline 172 CHAPTER 14 MOVING FROM LOGICAL TO PHYSICAL DESIGN MOVING FROM LOGICAL TO PHYSICAL THE REALITY OF SOURCES ABSTRACTIONS AND GENERIC FORMS SUMMARY OF GENERIC CONSTRUCTS 185 CHAPTER 15 BIG DATA BIG DATA HUGE DATA VOLUMES N-STRUCTURED & VERY COMPLEX DATA STREAMING AND SHAPE-SHIFTING DATA BIG DATA AND DATA VAULT 192 CHAPTER 16 MASTER DATA MANAGEMENT MASTER DATA MANAGEMENT MDM and Data Warehousing THE DATA VAULT MDM INITIATIVES MDM AND THE DATA VAULT 203 CHAPTER 17 DATA WAREHOUSE ARCHITECTURE data warehouse architecture architectural states of data aboutthe layers architectural layers and semantics data vault architecture the raw versus b dw layers bellyup to the bar from Upstream to downstream architecting the flow of data persisting data in the architecture reference models and taxonomies Views, data Virealization, and More architecture summary 231 CHAPTER 18 KEY ALIGNMENT whykey alignment? level of existing alignment keyalignmentinthe data Warehouse aligning the keys in the edw architecture 239 CHAPTER 19 CODES & REFERENCE TABLES Codesand Reference Tables Codes and Reference Tables Defined Using Reference Tables 247 Page 12 ~"

5 19.4 REFERENCE TABLES AND DATAWAREHOUSING REFERENCE TABLES IN THE DATA VAULT EDW 249 CHAPTER 20 INFORMATION MODELS, TAXONOMIES & INDUSTRY REFERENCE MODELS Chapter Forward Scope ofthe various models taxonomiesand the edw buckets of data domain values and reference models taxonomy models in the edw industry Reference models information Models more models 269 CHAPTER 21 SATELLITE DESIGN TOPICS satellite keys and grain satellite grain modeling issues multi-valued satellite toggling between 3nf and data vault ConceptConstellations Satellite Design and the Number ofsatellites Satellite Design Variations 286 CHAPTER 22 LINK DESIGN TOPICS link Design topics overview business driven versus source driven an eventalways requires a hub relationship consistency & unit of work what drives the unitof work Relative Cardinality Link Analysis and Design Process 300 CHAPTER 23 HUB DESIGN TOPICS alternate ids for business keys duplicate business keys Using date/time as partof Hub Key the importance of hub business keys 321 CHAPTER 24 CORE BUSINESS CONCEPT DESIGN TOPICS CORE BUSINESS CONCEPT THE CENTRAL, ENTERPRISE-WIDE VIEW THE CORE CONCEPT LEVEL ATTRIBUTE SWARMING THE BUZZAROUND CORE CONCEPTS 330 CHAPTER 25 ATTRI BUTE DESIGN Core Data Warehouse Attributes Compound Attributes Context Attributes becoming Entities Attribute Clusters 343 Page 13

6 25.5 Split apart and draw together 345 CHAPTER 26 ARCHITECTURE DESIGN CONSIDERATIONS ARCHITECTURAL LAYERS ANALYSIS LAYERS BETWEEN SOURCES AND MARTS FLOW OF DATAAND THE EDW PIPE 354 CHAPTER 27 LOADING THE EDW loading the edw Technical benefits of unified decomposition Thetechnical benefits of data vault Loading Process fundamentals loading Automation loading the edw summary 364 CHAPTER 28 SOURCING THE EDW, sourcing the edw no business access to data warehouse layer the roleof data marts Setting Your Data Marts Free Building Dimensions from the data Vault Building Facts from the data Vault automation sourcing the edw summary 378 CHAPTER 29 HELPFUL ANCILLARY TABLES TRANSFORMATION ENABLING STRUCTURES BRIDGE TABLES POINT IN TIME (PIT) TABLE 385 CHAPTER 30 ADVANCED CONCEPTS all data is unstructured enterprise data integration is impossible the transactional link satellites and name/value pair Data Driven versus Model Driven satellite design and nvp summary anchor modeling 406 CHAPTER31 COMPARISON EXAMPLE COMPARING MODELING APPROACHES. 412 TABLE OF FIGURES 428 Page 14