Proposal of a new Data Warehouse Architecture Reference Model

Transcription

1 Proposal of a new Data Warehouse Architecture Reference Model Dariusz Dymek Wojciech Komnata Piotr Szwed AGH University of Science and Technology Department of Applied Computer Science eidymek@kinga.cyf-kr.edu.pl, {wkomnata,pszwed}@agh.edu.pl BDAS 2015 Ustroń, Poland, May 26-29, 2015

2 Agenda 1. Motivation 2. Overview of DW architecturs 3. Data Warehouse Architecture Reference Model: description 4. Mapping and examples 5. Conclusions 2

3 Data Warehouse Architecture Reference Model - definition Software Architecture Definitions The software architecture of a program or computer system is the structure or structures of the system, which comprise software components, the externally visible properties of those components, and the relationships among them [Bass et al. CMU/SEI-2006-TR-012 ESC-TR ]. Architecture is the fundamental organization of a system embodied in its components, their relationships to each other and to the environment and the principles guiding its design and evolution. [ISO 2000 standard] Data Warehouse Architecture Reference Model (DWARM) description of components appearing in DW architectures (and their properties) guidelines defining how to use them 3

4 Motivation 1 Long experience with industrial Data Warehouse (DW) implementations Scientific projects, which used DW components for data analysis Design of BPWIP platform data integration and exchange between security agencies. (DW collected facts related to communication events) 4

5 Motivation 2 DW architectures are presented in various ways: layered models five basic architecture patterns Problems with mixed architectures: evolution of DWs integration of existent systems (company mergers and acquisitions) Data Warehouse Architecture Reference Model (DWARM) goals: a systematic description of DW components and architectural patterns, which are developed to meet very diverse technical and business needs comprehensive set of components allowing to model standard architectures guidelines on how to arrange them to form hybrid architectures 5

6 Overview layered classification 1 Single-layer [Golfarelli, Rizzi, Data Warehouse Design, Modern Principles and Methodologies, 2009] Operational data Source layer DW is a multidimensional view of operational data created by middleware Middleware Data Warehouse Analysis Reporting tools OLAP tools 6

7 Overview layered classification 2 Two-layer [Golfarelli, Rizzi, Data Warehouse Design, Modern Principles and Methodologies, 2009] Operational data ETL tools External data Source data Data Staging Data stored in central repository Data marts replicate it contents (selection of data relevant for business area, department or user category) Data Warehouse Metadata Data Warehouse layer Data Marts Analysis Reporting tools OLAP tools Data Mining tools What-if analysis 7

8 Overview layered classification 3 Operational data ETL tools External data Source layer Data staging Three-layer [Golfarelli, Rizzi, Data Warehouse Design, Modern Principles and Methodologies, 2009] Reconciled data Metadata Reconciled layer Reconciled layer materializes operational data after clensing and integrating. ETL tools Loading introduces additional redundancy sometimes used for daily reports Data Warehouse Data Warehouse layer Data Marts Analysis Reporting tools OLAP tools Data Mining tools What if analysis tools 8

9 Functional classification Five architectures identified based on roles of central Data Warehouse and Data Marts [Watson, Ariyachandra, 2005; Connolly, Begg, 2005; Golfarelli, Rizzi, 2009; Ponniah, 2010] Data Data Data Storage Data & Information Source Staging Delivery Centralized Independent Data Marts Federated Hub-and- Spoke Data-Mart Bus 9

10 Five architecture models 1 Centralized (Enterprise DW) Data Staging Data Storage Data & Information Delivery Central data repository: fine-granular data in 3NF materialized aggregates logical dimensional views No data marts. Hub and Spoke (Inmon) Data Source Data Staging Data Storage Data & Information Delivery Central data repository: fine-granular data in 3NF Dependent data marts fed from CDR: summarized data selected elementary data no constraints on data models: normalized, denormalized, multidimensional 10

11 Five architecture models 2 Independent Data Marts Data Source Data Mart Bus (Kimball) Data Source Data Staging Data Staging Data Storage Data Storage Data & Information Delivery Data & Information Delivery No Central data repository Data marts: fed separetely independent data models numerous versions of truth hinders making enterprise-wide queries No Central data repository Data marts: conformed enterprise-wide dimensional model provided by a supermart star (snowflake) data models used in marts arranged into a constelation by the supermart model logical data integration and entrprise-wide view 11

12 Five architecture models 3 Federated Data Source Data Staging Data Storage Data & Information Delivery Integration of legacy systems No changes in flows from source systems to data storages (multiple CDRs and DMs) Data integrated logically based on shared keys, metadata and distributed queries Materialized or virtual 12

13 Popularity of architectures in Poland Centralized Based on: Moh d Alsqour, Kamal Matouk Mieczysław L. Owoc A survey of data warehouse architectures - preliminary results, Proceedings of the Federated Conference on Computer Science and Information Systems, 2013, pp

14 Architecture selection factors Central Hub and Spoke Data Mart Bus Independent Data Mart Information inter-dependence (data exchange level) Senior management informational needs Urgency Task routiness Resource constraints (availability) Strategic view of the data warehouse (DM as enabler for strategic transformations within an enterprise) Experts opinions Compatibility with running systems Perceived ability (IT in-house staff skills) Sponsorship level (influence of the sponsor) Technical issues According to [Watson, Ariyachandra, 2005/2010]. Data collected from 400 large organizations, mean revenue 660M$, mean number of emplyees correlation between a factor and architectures no influence 14

15 Data Warehouse Reference Model 11. Consumer Applications 10. Data Delivery App Data Analysis & Visualization Simple language borrowed from Structured Anaysis. 9. Federated Data Repository 8. Data Integration FDR 11 layers comprising data stores and processes 7. Data Mart 6. Data Mart Feeding 5. Central Data Repository DM CDR Granularity level: tradeof between generality and flexibility at describing various components and connections 4. Loading 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 15

16 DWARM: Source layer 11. Consumer Applications 10. Data Delivery 9. Federated Data Repository App FDR Data Analysis & Visualization Source - entities that can deliver data to the data warehouse: enterprise information systems registering devices. 8. Data Integration 7. Data Mart 6. Data Mart Feeding DM Data: structured or unstructured synchronous or asynchronous 5. Central Data Repository CDR 4. Loading 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 16

17 DWARM: Extraction layer 11. Consumer Applications 10. Data Delivery 9. Federated Data Repository 8. Data Integration App FDR Data Analysis & Visualization Extraction processes data acquisition, transforming to transport formats sending them to a data warehouse (DW) 7. Data Mart 6. Data Mart Feeding 5. Central Data Repository 4. Loading DM CDR Processes: Triggered manually or automatically Partly or fully controlled 3. Temporary Staging Area 2. Extraction 1. Source TSA Src ETL Transformation Data Collection Data: complete or incremental metadata (timestamps, source identification, checksums) 17

18 DWARM: Temporary Staging Area (TSA) 11. Consumer Applications App Data Analysis & Visualization Data storage and transformation before loading. 10. Data Delivery 9. Federated Data Repository FDR Fully controlled by DW. 8. Data Integration 7. Data Mart 6. Data Mart Feeding 5. Central Data Repository 4. Loading DM CDR Metadata describing: data transformations (model, structure, values) quality assurance (completness, consistency, compliance with control rules) 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 18

19 DWARM: Loading layer 11. Consumer Applications 10. Data Delivery 9. Federated Data Repository 8. Data Integration App FDR Data Analysis & Visualization Loading processes Transferring data to Central Data Repository (CDR) Data Marts Automatic data processing (integration, aggregation) 7. Data Mart 6. Data Mart Feeding DM Processes started automatically (periodically). 5. Central Data Repository 4. Loading CDR Primary data after loading are immutable 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 19

20 DWARM: Central Data Repository 11. Consumer Applications App Data Analysis & Visualization Appears in Centralized and Hub and Spoke architectures. 10. Data Delivery 9. Federated Data Repository 8. Data Integration 7. Data Mart FDR DM Data structured according to common schema. Depending on approach: multi dimensional models or 3NF. 6. Data Mart Feeding 5. Central Data Repository CDR Implementations: typically, relational, rarely flattened models. 4. Loading 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 20

21 DWARM: Data Mart feeding layer 11. Consumer Applications App Data Analysis & Visualization Appears in Hub and Spoke architecture. 10. Data Delivery 9. Federated Data Repository 8. Data Integration 7. Data Mart 6. Data Mart Feeding FDR DM Processes selecting data from CDR applying transformations and loading data to separate Data Marts 5. Central Data Repository CDR 4. Loading 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 21

22 DWARM: Data Mart layer 11. Consumer Applications 10. Data Delivery 9. Federated Data Repository 8. Data Integration App FDR Data Analysis & Visualization Primary data storage in: Independed Data Mart Data Mart bus Local data storage (materialized view) for Hub and Spoke. 7. Data Mart 6. Data Mart Feeding 5. Central Data Repository 4. Loading 3. Temporary Staging Area DM CDR TSA ETL Transformation Data models similar to CDR. Smaller data volumes and subject orientation favor redundancy. Often multidimensional DB (data cubes) used for storing data. 2. Extraction 1. Source Src Data Collection 22

23 DWARM: Data integration layer 11. Consumer Applications 10. Data Delivery 9. Federated Data Repository 8. Data Integration App FDR Data Analysis & Visualization Processes responsible for loading data originating from different data marts / data warehouses into Federated Data Repository. 7. Data Mart 6. Data Mart Feeding 5. Central Data Repository 4. Loading DM CDR Two cases: shared common data model: joins from databases different data models: similar to ETL processes 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 23

24 DWARM: Federated Data Repository layer 11. Consumer Applications 10. Data Delivery App Data Analysis & Visualization Additional source of data satisfying various needs of Consumer Applications. 9. Federated Data Repository 8. Data Integration 7. Data Mart 6. Data Mart Feeding FDR DM Implementations: virtual (middleware, closely connected to data integration) materialized 5. Central Data Repository 4. Loading CDR Materialized FDR shares characteristics with CDR (although does not contain primary data). 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 24

25 DWARM: Data Delivery 11. Consumer Applications 10. Data Delivery App Data Analysis & Visualization A link between Consumer Applications and all needed sources of data (CDR, DM, FDR). 9. Federated Data Repository 8. Data Integration FDR Main goal: delivering demanded data to consumer applications. 7. Data Mart 6. Data Mart Feeding 5. Central Data Repository DM CDR Auxiliary functions: security and access control logging 4. Loading 3. Temporary Staging Area 2. Extraction 1. Source TSA Src ETL Transformation Data Collection Data delivery processes synchronized with processes of other layers (e.g. mutual exclusion with loading and aggregation). 25

26 DWARM: Customer Applications 11. Consumer Applications 10. Data Delivery 9. Federated Data Repository 8. Data Integration 7. Data Mart 6. Data Mart Feeding App FDR DM Data Analysis & Visualization End-user applications: controlled by metadata, e.g. reporting systems with specfied data formats, periodicity and recipients. not described by metadata (e.g. Excel spreadsheet using exported data). 5. Central Data Repository CDR 4. Loading 3. Temporary Staging Area TSA ETL Transformation 2. Extraction 1. Source Src Data Collection 26

27 Mapping between DWARM and five architectures 1 exactly one layer many several layer instances (including data stores and processes) 27

28 Examples Consumer applications Data delivery Consumer applications Data delivery Consumer applications Data delivery Consumer applications Data delivery Consumer applications Data delivery Mixed architecture: Hub and Spoke Data Mart Bus 7 DM 7 DM 7 DM 7 DM Data Mart Data Mart Data Mart Data Mart 6 Data Mart Loading 5 CDR Central Data Repository Loading Loading Loading 3 TSA 3 TSA 3 TSA Temporary Staging Area Temporary Staging Area Temporary Staging Area Extraction Extraction Extraction 1 Source 28

29 Examples 2 11 Aplikacje analityczno-prezentacyjne 10 Udostępnianie danych 11 Aplikacje analityczno-prezentacyjne 10 Udostępnianie danych 9 FRD 11 Aplikacje analityczno-prezentacyjne 10 Udostępnianie danych Aplikacje analityczno-prezentacyjne 10 Udostępnianie danych Three architectures: Centralized Independent Data Mart Federated Federacyjne Repozytorium Danych 8 Integracja danych 5 CRD Centralne Repozytorium Danych 7 HT Hurtownia Tematyczna... 7 HT Hurtownia Tematyczna Ładowanie danych Ładowanie danych Ładowanie danych 3 DT 3 DT Dane Tymczasowe Dane Tymczasowe 2 2 Pozyskanie danych Pozyskanie danych 1 1 Dane źródłowe Dane źródłowe Organization 1 Ogranization 2 29

30 Conclusions The goal of our work was to define a comprehensive model that can be used to express various data warehouse architectures define components and guidelines how to arrange them DWARM Model: 11 layers (elements having common properties and roles) processes containers (data stores) and data An instance of data warehouse architecture model can be obtained by: tailoring (removing selected elements or layers) and creating instances of appropriately connected components Architectures of desired complexity can be obtained by multiplying layers and their elements. 30

31 Further information Discussion on DW achitectures, data modeling, development process and metadata. Presentation of the reference model including characterisctics of data and processes. Ontological model (DWARM). Concise information on BPWIP (secure platform for data integration) Application of graph grammars to data transformations. 31

32 Thank you 32