DATA WAREHOUSE 03 COMMON DWH ARCHITECTURES ANDREAS BUCKENHOFER, DAIMLER TSS

Transcription

1 A company of Daimler AG DATA WAREHOUSE 03 COMMON DWH ARCHITECTURES ANDREAS BUCKENHOFER, DAIMLER TSS

2 ABOUT ME Andreas Buckenhofer Senior DB Professional Since 2009 at Daimler TSS Department: Big Data Business Unit: Analytics

3 ANDREAS BUCKENHOFER, DAIMLER TSS GMBH Forming good abstractions and avoiding complexity is an essential part of a successful data architecture Data has always been my main focus during my long-time occupation in the area of data integration. I work for Daimler TSS as Database Professional and Data Architect with over 20 years of experience in Data Warehouse projects. I am working with Hadoop and NoSQL since I keep my knowledge up-to-date - and I learn new things, experiment, and program every day. I share my knowledge in internal presentations or as a speaker at international conferences. I'm regularly giving a full lecture on Data Warehousing and a seminar on modern data architectures at Baden-Wuerttemberg Cooperative State University DHBW. I also gained international experience through a two-year project in Greater London and several business trips to Asia. I m responsible for In-Memory DB Computing at the independent German Oracle User Group (DOAG) and was honored by Oracle as ACE Associate. I hold current certifications such as "Certified Data Vault 2.0 Practitioner (CDVP2)", "Big Data Architect, Oracle Database 12c Administrator Certified Professional, IBM InfoSphere Change Data Capture Technical Professional, etc. Contact/Connect Daimler TSS Data Warehouse / DHBW 3 DOAG DHBW xing

4 NOT JUST AVERAGE: OUTSTANDING. As a 100% Daimler subsidiary, we give 100 percent, always and never less. We love IT and pull out all the stops to aid Daimler's development with our expertise on its journey into the future. Our objective: We make Daimler the most innovative and digital mobility company. Daimler TSS

5 INTERNAL IT PARTNER FOR DAIMLER + Holistic solutions according to the Daimler guidelines + IT strategy + Security + Architecture + Developing and securing know-how + TSS is a partner who can be trusted with sensitive data As subsidiary: maximum added value for Daimler + Market closeness + Independence + Flexibility (short decision making process, ability to react quickly) Daimler TSS 5

6 LOCATIONS Daimler TSS Germany 7 locations 1000 employees* Ulm (Headquarters) Stuttgart Berlin Karlsruhe * as of August 2017 Daimler TSS India Hub Bangalore 22 employees Daimler TSS China Hub Beijing 10 employees Daimler TSS Malaysia Hub Kuala Lumpur 42 employees Daimler TSS Data Warehouse / DHBW 6

7 DWH LECTURE - LEARNING TARGETS Describe different DWH architectures Explain DWH data modeling methods and design logical models Name DB techniques that are well-suited for DWHs Explain ETL processes Specify reporting & project management & meta data requirements Name current DWH trends Daimler TSS Data Warehouse / DHBW 7

8 EXERCISE: CLASSICAL DWH ARCHITECTURES The article compares THE two classic DWH architectures. Read the paper and complete the table / questions on the next slide. (Caution: The paper is biased / favors one approach; you may want to read other/more papers for a neutral view.) Daimler TSS Data Warehouse / DHBW 8

9 EXERCISE: CLASSICAL DWH ARCHITECTURES How are the approaches called? Who invented the approach? How many layers are used and how are the layers called? Which data modeling approaches are used in which layer? In which layer are atomic detail data stored? In which layer are aggregated / summary data stored? List at least 2 advantages List at least 2 disadvantages Daimler TSS Data Warehouse / DHBW 9

10 EXERCISE: CLASSICAL DWH ARCHITECTURES How are the approaches called? Who invented the approach? How many layers are used and how are the layers called? Which data modeling approaches are used in which layer? In which layer are atomic detail data stored? In which layer are aggregated / summary data stored? Kimball Bus Architecture Ralph Kimball Bill Inmon Data Staging Dimensional Data Warehouse Data Staging: variable, corresponds to source system Dimensional Data Warehouse: Dimensional Model Corporate Information Factory Data Acquisition Normalized Data Warehouse Data Delivery / Dimensional Mart Data Acquisition: variable, corresponds to source system Normalized Data Warehouse: 3NF Data Delivery: Dimensional Model Dimensional Data Warehouse Normalized Data Warehouse Dimensional Data Warehouse Data Delivery / Dimensional Mart Daimler TSS Data Warehouse / DHBW 10

11 EXERCISE: CLASSICAL DWH ARCHITECTURES Kimball Bus Architecture Advantages Two layers only mean faster development and less work Rather simple approach to make data fast and easily accessible Lower startup costs (but higher subsequent development costs) Corporate Information Factory Separation of concerns: long-term enterprise data storage separated from data presentation Changes in requirements and scope are easier to manage Lower subsequent development costs (but higher startup costs) Disadvantages If table structures change (instable source systems), high effort to implement the changes and reload data, especially conformed dimensions ( Dimensionitis desease) Non-metric data not optimal for dimensional model Dimensional model (esp. Star Schema) contains data redundancy Data model transformations from 3NF to Dimensional model required More complex as two different data models are required Larger team(s) of specialists required Daimler TSS Data Warehouse / DHBW 11

12 OTHER ARCHITECTURES Kimball Bus Architecture (Central data warehouse based on data marts) Inmon Corporate Information Factory Data Vault 2.0 Architecture (Dan Linstedt) DW 2.0: The Architecture for the Next Generation of Data Warehousing Virtual Data Warehouse Operational Data Store (ODS) Daimler TSS Data Warehouse / DHBW 12

13 KIMBALL BUS ARCHITECTURE (CENTRAL DATA WAREHOUSE BASED ON DATA MARTS) Source: Daimler TSS Data Warehouse / DHBW 13

14 KIMBALL BUS ARCHITECTURE (CENTRAL DATA WAREHOUSE BASED ON DATA MARTS) Internal data sources Data Warehouse OLTP OLTP External data sources Staging Layer (Input Layer) Backend Core Warehouse Layer = Mart Layer Data Mart 1 Data Mart 2 Data Mart 3 More Businessprocess oriented than subjectoriented, integrated, time-variant, non-volatile Frontend Metadata Management Security DWH Manager incl. Monitor Daimler TSS Data Warehouse / DHBW 14

15 KIMBALL BUS ARCHITECTURE (CENTRAL DATA WAREHOUSE BASED ON DATA MARTS) Bottom-up approach Dimensional model with denormalized data Sum of the data marts constitute the Enterprise DWH Enterprise Service Bus / conformed dimensions for integration purposes (don t confuse with ESB as middleware/communication system between applications) Kimball describes that agreeing on conformed dimensions is a hard job and it s expected that the team will get stuck from time to time trying to align the incompatible original vocabularies of different groups Data marts need to be redesigned if incompatibilities exist Daimler TSS Data Warehouse / DHBW 15

16 Core Warehouse Layer DATA INTEGRATION WITH AND WITHOUT CORE WAREHOUSE LAYER Daimler TSS Data Warehouse / DHBW 16

17 INMON CORPORATE INFORMATION FACTORY Source: Daimler TSS Data Warehouse / DHBW 17

18 INMON CORPORATE INFORMATION FACTORY Internal data sources Data Warehouse Backend Frontend OLTP OLTP Staging Layer (Input Layer) Core Warehouse Layer (Storage Layer) subjectoriented, integrated, timevariant, nonvolatile Mart Layer (Output Layer) (Reporting Layer) External data sources Metadata Management Security DWH Manager incl. Monitor Daimler TSS Data Warehouse / DHBW 18

19 INMON CORPORATE INFORMATION FACTORY Top-down approach (Normalized) Core Warehouse is essential for subject-oriented, integrated, time-variant and nonvolatile data storage Create (departmental) Data Marts as subsets of Core Enterprise DWH as needed Data Marts can be designed with Dimensional model The logical standard architecture is more general compared to CIF, but was mainly influenced by CIF Daimler TSS Data Warehouse / DHBW 19

20 DATA VAULT 2.0 ARCHITECTURE TODAY S WORLD (DAN LINSTEDT) Daimler TSS Data Warehouse / DHBW

21 DATA VAULT 2.0 ARCHITECTURE (DAN LINSTEDT) Michael Olschimke, Dan Linstedt: Building a Scalable Data Warehouse with Data Vault 2.0, Morgan Kaufmann, 2015, Chapter 2.2 Daimler TSS Data Warehouse / DHBW 21

22 DATA VAULT 2.0 ARCHITECTURE (DAN LINSTEDT) Internal data sources Data Warehouse Backend Frontend OLTP OLTP External data sources Staging Layer (Input Layer) Hard Rules only Raw Data Vault Metadata Management Security DWH Manager incl. Monitor Business Data Vault Soft Rules Mart Layer (Output Layer) (Reporting Layer) Daimler TSS Data Warehouse / DHBW 22

23 DATA VAULT 2.0 ARCHITECTURE (DAN LINSTEDT) Core Warehouse Layer is modeled with Data Vault and integrates data by BK (business key) only (Data Vault modeling is a separate lecture) Business rules (Soft Rules) are applied from Raw Data Vault Layer to Mart Layer and not earlier Alternatively from Raw Data Vault to additional layer called Business Data Vault Hard Rules don t change data Data is fully auditable Real-time capable architecture Architecture got very popular recently; also applicable to BigData, NoSQL Daimler TSS Data Warehouse / DHBW 23

24 DATA VAULT 2.0 ARCHITECTURE (DAN LINSTEDT) In the classical DWHs, the Core Warehouse Layer is regarded as single version of the truth Integrates + cleanses data from different sources and eliminates contradiction Produces consistent results/reports across Data Marts But: cleansing is (still) objective, Enterprises change regularly, paradigm does not scale as more and more systems exist Data in Raw Data Vault Layer is regarded as Single version of the facts 100% of data is loaded 100% of time Data is not cleansed and bad data is not removed in the Core Layer (Raw Vault) Daimler TSS Data Warehouse / DHBW 24

25 DATA VAULT 2.0 ARCHITECTURE (DAN LINSTEDT) Data Vault is optimized for the following requirements: Flexibility Agility Data historization Data integration Auditability Bill Inmon wrote in 2008: Data Vault is the optimal approach for modeling the EDW in the DW2.0 framework. (DW2.0) Daimler TSS Data Warehouse / DHBW 25

26 Data Lifecycle DW 2.0: THE ARCHITECTURE FOR THE NEXT GENERATION OF DATA WAREHOUSING Operational application data model Integrated corporate data model Integrated corporate data model Archival data model Source: W.H. Inmon, Dan Linstedt: Data Architecture: A Primer for the Data Scientist, Morgan Kaufmann, 2014, chapter 3.1 Daimler TSS Data Warehouse / DHBW 26

27 DW 2.0: THE ARCHITECTURE FOR THE NEXT GENERATION OF DATA WAREHOUSING Main characteristics: Structured and unstructured data, not just metrics Life Cycle of data with different storage areas Hot data: High speed, expensive storage (RAM, SSDs) for most recent data Cold data: Low speed, inexpensive storage (e.g. hard disks) for old data; archival data model with high compression Metadata is an integral part of the DWH and not an afterthought Daimler TSS Data Warehouse / DHBW 27

28 VIRTUAL DATA WAREHOUSE Internal data sources Data Warehouse Backend Frontend OLTP OLTP External data sources Query Management Weakly+partly subject-oriented, Weakly+partly integrated, Not time-variant, Not non-volatile Daimler TSS Data Warehouse / DHBW 28

29 VIRTUAL DATA WAREHOUSE Data not extracted from operational systems and stored separately Standardized interface for all operational data sources One "GUI" for all existing data Generates combined queries Query Processor joins query result data from different sources Can also access data in Hadoop (Polybase, Big SQL, BigData SQL, etc) Daimler TSS Data Warehouse / DHBW 29

30 VIRTUAL DATA WAREHOUSE Query Management manages metadata about all operational systems (physical) location of data and algorithms for extracting data from OLTP system Implementation easier Low cost: can use existing hardware infrastructure Queries cause significant performance problems in operational systems Known problems when analyzing operational data directly Same query is processed multiple times (if queried multiple times) Same query delivers different results when processed at different times Daimler TSS Data Warehouse / DHBW 30

31 OPERATIONAL DATA STORE (ODS) Internal data sources Data Warehouse Backend Frontend OLTP OLTP Staging Layer (Input Layer) Core Warehouse Layer (Storage Layer) subjectoriented, integrated, timevariant, nonvolatile Mart Layer (Output Layer) (Reporting Layer) External data sources Operational Data Store Metadata Management Security DWH Manager incl. Monitor Daimler TSS Data Warehouse / DHBW 31

32 OPERATIONAL DATA STORE (ODS) ODS: Real-time/Right-time layer Replication techniques used to transport data from source database to ODS layer with minimal impact on source system Data in the ODS has no history and is stored without any cleansing and without any integration (1:1 copy from single source) DWH performance not optimal as data model is suited for OLTP and not for reporting requirements ODS normally additionally to Staging / Core DWH / Mart Layer but can exist alone without other layers Daimler TSS Data Warehouse / DHBW 32

33 EXAMPLE DWH FOR STATE OF CONSTRUCTION DOCU Daimler TSS Data Warehouse / DHBW 33

34 ARCHITECTURE FROM AN ACTUAL PROJECT IN THE AUTOMOTIVE INDUSTRY ETL Engine Datastore Source IIDR ReplEngine Source OLTP DB Logs Backend DWH DB Datastore DWH IIDR ReplEngine DWH Mart Layer Raw + Business Data Vault Staging Layer Standard Reports Frontend AdHoc Reports TSM Datastore Mirror IIDR ReplEngine Mirror Mirror DB (Operational Data Store) Daimler TSS Data Warehouse / DHBW 34

35 END USER SAMPLE QUESTIONS Which vehicles or aggregates are documented incompletely? (Data quality) Which vehicles / which control units require SW updates? Which interiors are most common by region? Supply data for external simulations, customs clearance, spare part planning, etc. Daimler TSS Data Warehouse / DHBW 35

36 EXERCISE: RECOMMEND AN ARCHITECTURE Review the presented data warehouse architectures. Which architecture would you recommend for A holding of 3 telecommunication companies An online store with real/right-time data integration needs Marketing department of a bank List advantages and drawbacks of your proposal. Daimler TSS Data Warehouse / DHBW 36

37 EXERCISE: RECOMMEND AN ARCHITECTURE A holding of 3 telecommunication companies Architecture: Virtual Data Warehouse + Companies may not want to provide their data to a new storage + Can easily be extended if new companies join the holding or reduced if a company leaves the holding - Bad performance - Not really data integration achieved, low Data Quality - Firewalls have to be opened Daimler TSS Data Warehouse / DHBW 37

38 EXERCISE: RECOMMEND AN ARCHITECTURE An online store with real-time/right-time data integration needs Architecture: Data Vault Integration of many internal and external source systems (e.g. integrate social media data about the online store) + Fast data delivery in Raw Vault Layer (Real-time/Right-time data integration). Complex data cleansing / transformation / soft rules are delayed downstream towards Mart Layer - Transformation overhead (Source system data model to Data Vault data model to Dimensional data model) Daimler TSS Data Warehouse / DHBW 38

39 EXERCISE: RECOMMEND AN ARCHITECTURE Marketing department of a bank Architecture: Kimball Bus architecture + Start small for a department. If other departments are interested, new data and new Marts can be added on demand - High risk to loose the Enterprise view and several DWHs are built That s still quite a common scenario nowadays. A single Enterprise DWH is often not achieved (e.g. Mergers & Acquisitions, inflexibility due to a single centralized DWH, rapidly changing conditions, etc.) and therefore very often several DWHs with different architectures exist in parallel within a company. Daimler TSS Data Warehouse / DHBW 39

40 EXERCISE - INTRODUCTION AND DWH ARCHITECTURE GROUP TASK Now imagine that you prepare an exam. Identify 1-3 questions about DWH architecture (and/or DWH introduction) that you would ask in an exam. Write down the questions on stick-it cards. Daimler TSS Data Warehouse / DHBW 40

41 SUMMARY Which layers does the logical standard architecture have? Staging (Input), Integration (Cleansing), Core Warehouse (Storage), Aggregation, Mart (Reporting, Output) and additionally Metadata, Security, DWH Manager, Monitor Which other architectures exist? Kimball Bus Architecture (Central data warehouse based on data marts) Inmon Corporate Information Factory Data Vault 2.0 Architecture (Dan Linstedt) DW 2.0: The Architecture for the Next Generation of Data Warehousing Virtual Data Warehouse Operational Data Store (ODS) Daimler TSS Data Warehouse / DHBW 41

42 THANK YOU Daimler TSS GmbH Wilhelm-Runge-Straße 11, Ulm / Telefon / Fax tss@daimler.com / Internet: Domicile and Court of Registry: Ulm / HRB-Nr.: 3844 / Management: Christoph Röger (CEO), Steffen Bäuerle Daimler TSS Data Warehouse / DHBW 42