Integrating evolving MDM and EDW systems by Data Vault based System Catalog

Transcription

1 Integrating evolving MDM and EDW systems by Data Vault based System Catalog D. Jakšić *, V. Jovanović ** and P. Poščić * * Department of informatics-university of Rijeka/ Rijeka, Croatia ** Georgia Southern University/ Statesboro, GA, USA dsubotic@inf.uniri.hr, vladan@georgiasouthern.edu, patrizia@inf.uniri.hr Abstract - The paper presents results of a research on integration of enterprise data warehouse (EDW) and a master data management (MDM) system. The primary goal was solving a schema evolution problem, and the corner stone of our approach was utilization of a data vault modeling of an integrated meta-model of EDW and MDM as an expansion of a traditional relational database system catalog. The main contributions of this paper are: a) common integration architecture, b) new system catalog based on a meta-model for DW and MDM integration, and c) research prototype used for empirical validation of the effectiveness of the proposed solution. I. INTRODUCTION A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process [15]. This means that a data warehouse (DW) can be used to analyze a particular subject area (such as sales, marketing, finance, etc.), it integrates data from multiple heterogeneous data sources, it keeps the history of data, and it never alters the data once it enters a DW. A simpler form of a DW is a data mart (DM). DM is focused on a single subject area and it draws its data not from all the DW data sources, but from a limited number of them (such as retail sales applications with daily transactions). For the logical representation of a DW (or DM), we traditionally use a denormalized dimensional model [12][20] in a combination with a 3NF model for modelling a central and integrated enterprise data warehouse (also called operational data store) [12][15][16]. DW environment (namely its data sources) nowadays is in a state of constant structural (schema) change. Master data management (MDM) comprises the processes, governance, policies, standards and tools that consistently define and manage the critical data of an organization to provide a single point of reference [3][23]. Some of the fundamental tasks of MDM system are duplicate removal, data standardization, and rule implementation and incorporation - all with a goal to eliminate incorrect data from entering the system and to create an authoritative source of master data for further distribution. It is a method of enabling a business organization to link all of its critical data to a common point of reference, which is further shared throughout all the departments and relevant employees. This way, a data quality of an organization greatly improves and the organization can better serve its clients, as well as improve their business by running a more accurate and efficient business analysis and reporting (based on a single version of the truth data pool [15]). Dimensional model can also be used here - for the logical representation of MDM data (master data are represented as dimensions here). Master data are the key business entities and their descriptive attributes (e.g., the customer has a name, address, etc., the product has a name, color, weight, category, etc.), which are used by multiple systems, applications, and business processes of the organization as a unique source of data. Reference data can be internal or external and are used for the validation of other data. MDM environment is also in a state of constant change data sources nowadays often change their structure and content. In this work our focus was on an enterprise size data warehouse (DW) and a master data management (MDM) system integration. We approached the problem through a development of a common system catalog meta-model, with the goal of solving a common schema evolution problem. Schema evolution occurs in both of these systems and is traditionally resolved separately. DW integrates current and historical data from many data sources and serves for business reporting and data analysis. MDM is traditionally used (by other business systems, applications, databases and data warehouses) as a physically independent database of master and reference data (also collected from many data sources). Data sources are often the same ones for DW and MDM and they often change their structure. These changes have to be implemented into both systems, so that they could accurately reflect the current (and historical) state of the real world - so that the DW could provide for effective business analysis and the MDM could achieve the optimal data quality throughout all the systems involved. We aim to resolve this problem on a common level - we state that the schema evolution problem can be viewed as a double issue: at a DW level and at the MDM level. From the DW perspective every event (fact) that is associated with dimensions is monitored, but from the MDM perspective the master data (dimensions) are monitored and the events (facts) give them context. And in both of these cases schema evolution problem to solve exists. The paper is organized as follows: section II gives a brief overview of a related work, section III describes our DW/MDM integration research (general research idea, our common integration architecture and integration part of a system catalog meta-model), section IV presents our research prototype and some relevant test results, and in section V we conclude our work with a brief summary and some plans and guidelines for the future. MIPRO 2017/miproBIS 1633

2 II. RELATED WORK With respect to the literature, the DW evolution can generally be traced through three approaches - schema evolution [5][10][29], schema versioning [1][9][25] and view maintenance [2][7][14]. The first two approaches are more interesting to our research because they are based on a DW defined as a multidimensional schema, and the third is based on a DW that is defined as a set of materialized views. Our extensive and comprehensive state-of-the-art on this problem can be found in [28], but from analyzing the related work we can generally conclude that the process of schema evolution and versioning is still demanding in terms of invested time and resources. It is necessary to balance the resource requirements and the quality of schema evolution process. Perhaps the biggest problem here is the preservation of schema consistency and data integrity (there is still a lack of an integrated system-of-records), as well as the simultaneous performance of temporal queries against multiple versions of the schema. Also, migration and transformation of data is still slow and expensive, the loss of information during these processes is still present and there is a lack of effective integration, organization and management of metadata. On the other hand, the view maintenance process can cause network saturation, depending on the amount of updated views and the amount of information they contain. The problems of anomalies and inconsistent changes in the views are still unresolved, the proposed approaches for view maintenance are still limited in terms of efficiency and performance, and the ETL processes, which are an integral part of most of today's DWs are completely ignored. Different approaches to solving the DW schema evolution problem are presented in literature (including a variety of techniques, algorithms, algebras, models, prototypes, methodologies and frameworks), but there is still no widely accepted solution and general framework for managing DW schema changes. More importantly, previous research does not emphasize the fact that the DW requirements, in this day and age, are increasing in the data and meta-data scope and structure (the growing number of data sources and more new and different types of data), which additionally requires developing some new approaches and solutions to the schema evolution problem. III. MDM AND DW INTEGRATION RESEARCH A. General research idea As we already mentioned, the DW needs to preserve the history of data and metadata changes, as well as the history of schema and scope changes, for a very long time period [19][26]. On the other hand, the MDM needs to preserve a data quality of an organization by achieving and maintaining a single version of the truth data pool as a basis for accurate and efficient business analysis [3][23]. Seeing that the DW and MDM both integrate basically the same data sources and have the same schema evolution problem, we will integrate those two systems into one and address the evolution problem at the common level. Our main research question was: Can our new system catalog model serve to successfully integrate DW and MDM systems?. In order to answer this question, we developed a new, integrated architecture for these systems, as well as a system catalog model based on a Data Vault modelling methodology [21][22]. The data vault (DV) is a data modeling method designed for supporting the long-term storage of historical data collected from various data sources and tracking the origin of data contained in the database [21][22]. The DV model (due to structural separation, usage of empiric meta-data and addition-only policy, which relation model does not implement) is able to track the data source values and the history of changes, which is a vital function of a DW system-of-records [19][26][27]. Furthermore, we developed and tested a research prototype based on said architecture and system catalog model. B. Common integration architecture Fig. 1 shows our common integration architecture. Central and integrated enterprise DW/MDM (purple section in Fig. 1) is in focus of our research and consists of two parts - the raw copy of the source data (SDV) and the synchronized master and business data (PMDV). Both parts are based on the Data Vault (DV) method, in contrast to traditional approaches based on the relational model [12][15][20]. SDV is focused on obtaining and preserving the original and unchanged copy of data sources for the purpose of governance and auditing, and PMDV is focused on obtaining and preserving the data that has been modified according to business and master rules the data that later feeds DMs and master dimensions (MDMs) and is oriented to user requirements. The integration of SDV and PMDV is carried out over an extended DBMS system catalog (i.e. our new meta-data repository, MDV), based on the DV model. MDV repository serves to integrate the two parts and to monitor history of changes of meta-data and their schemas, of the business rules and transformations and of mappings between the two systems. MDV contains the history of data sources meta-data (their domains and schemas), of central DW/MDM integration and schema changes, of DMs and MDMs schema changes and of security schema changes (user access rights). We can say that MDV repository represents the DW on DW and is a key part of our new architecture. In materialized DMs and MDMs (blue section in Fig. 1) master data and business analysis and reporting data is stored. Data is stored according to the user requirements and a dimensional model is used for representation (data are summarized, aggregated and calculated). A traditional DMs are integrated here with the traditional MDMs the PMDV forwards the data to MDMs so the MDMs could then forward the data to DMs and data sources. The end-user can then access the DM/MDMs and analyze the data through the selected data analysis and reporting tools. Additionally, the master data is returned from enterprise DW/MDM to the data sources in order to harmonize and refine the data within business organization. Master data collected from PMDV is maintained in one central MDM location and all source systems and applications, as well as DMs, are using this data. This directly affects the quality of data, which later re-enters the raw SDV and passes again through the described layers of the architecture. Also, by reducing the amount of data over which was necessary to make "heavy" transformations we thus speed up the process of 1634 MIPRO 2017/miproBIS

3 Figure 1. Common integration DW/MDM system architecture data integration [19]. However, these issues are out of the scope of this paper. C. Meta Data Vault (MDV) model for SDV/PMDV integration Fig. 2 shows a simplified meta-data-vault (MDV) model for DW/MDM integration. Due to complexity and size of the full MDV system catalog model and the scope of this work, we will show here one small, but relevant part of the MDV model the one for SDV/PMDV integration. The whole model is much more extensive because it serves for schema evolution purpose - it incorporates data sources, data marts, materialized views and issues of security - but those are out of scope of this paper. The model (as well as other models in this work) is made in IDEF1X method [17] with the use of the CA Erwin Data Modeler 9.5 modelling software [8]. In our MDV model, SDV/PMDV integration is managed through the same-as-links on four main hubs (H_HUB, H_LINK, H_SATELLITE, and H_ATTRIBUTE) which store metadata about three main DV concepts in a data model (hubs, links and satellites respectively additionally H_ATTRIBUTE stores data about satellite attributes; column meta-data). For example, if we have two different relational data sources which both have the table CUSTOMER, we integrate these sources into a DV based central EDW (EDV). This means that the relational table CUSTOMER from both sources becomes one hub H_CUSTOMER (stores business keys for CUSTOMER) with its satellites S_CUSTOMER1 (stores columns from CUSTOMER in Source1) and S_CUSTOMER2 (stores columns from CUSTOMER in Source2). This is considered as a copy of the original data sources (SDV), Figure 2. MDV system catalog model for SDV/PMDV integration MIPRO 2017/miproBIS 1635

4 which is then copied into a PMDV where the business and master rules are applied. For this example it means that hub H_CUSTOMER gets only one satellite S_CUSTOMER by the rules those two satellites are integrated into common one and the data is cleansed. Also, data in the H_CUSTOMER is integrated and deduplicated. The corresponding meta-data is then stored in our MDV system catalog from Fig. 2. In the H_HUB and its satellites S_BUSINESS_KEY and S_HUB_DEF the meta-data about hub H_CUSTOMER (and generally about all the hubs in a enterprise DW/MDM from both SDV and PMDV) is stored, including the data about names of the hubs, names of their keys, hub types and their general descriptions, as well as their load dates and record sources. Also, in the H_SATELLITE and its satellite S_SAT_DEF the meta-data about satellites S_CUSTOMER1, S_CUSTOMER2 and S_CUSTOMER is stored. The same is true for all other MDV structures (links, attributes, ). However, at this point the DW and MDM (SDV and PMDV) hubs are not yet integrated (list of hubs with their descriptive meta-data is just stored in a corresponding meta-hub and its meta-satellites - H_HUB, S_BUSINESS_KEY, S_HUB_DEF). For the integration we use the same-as link SAL_MASTER_HUB which relates a single hub from a list of hubs in H_HUB to another hub from that list, in a base-master relation (SDV hub is base one, PMDV hub is master). In the example it would mean that the hub H_CUSTOMER form PMDV becomes master hub to hub H_CUSTOMER in SDV. For the satellites, their integration is stored in the SAL_MASTER_SATELLITE link for example, the S_CUSTOMER1 and S_CUSTOMER2 meta-data records stored in H_SATELLITE can have (separate) base-master relationship with S_CUSTOMER record. This relationship is stored as a record in SAL_MASTER_SATELLITE where S_CUSTOMER is master structure on S_CUSTOMER1 and S_CUSTOMER2. Additionally, SAL_MASTER_HUB (as well as other master links) relates to H_RULE and H_TRANSFORMATION, the two hub meta-concepts that store the data about business rules that can be applied to data and (ETL) transformations needed to apply those rules, respectively. This way, we get the complete history of SDV/PMDV integration we can keep track of all the business rules and transformations applied to the specific integration, with SAL_MASTER_HUB storing the load date and record source data about that specific integration. In that way we can create and manage master hubs and store the history of their changes. SAL_MASTER_HUB has its satellite S_BDV_CODE_HUB, with descriptive data about the code (ETL or other) used for that specific integration. This satellite will later serve as a basis for automating the process of integration. The same can be applied to the other three main hubs in a MDV model (SAL_MASTER_LINK for H_LINK, SAL_MASTER_SATELLITE for H_SATELLITE and SAL_MASTER_ATTRIBUTE for H_ATTRIBUTE), so we can create complete and historicized golden copy of business and master data in PMDV, from the simple SDV raw copy of data. Regarding the reference table concept, as it exists only in a data model (it represents external or internal reference data and as such it is already consolidated and organized by internal business rules or external data standards), we have no need to further make the master concepts in the meta-model. We will simply store its meta-data in a H_REFERENCE concept in MDV. IV. RESEARCH PROTOTYPE AND RESULTS A. Business case example For the purpose of theoretical validation of the proposed architecture and DW/MDM integration, as well as building a prototype, we developed a simple business case (in a similar way as in [18]), which monitors the work of the employees on projects and their participation in job trainings and educations. We have two data sources (JobDB and TrainingDB) which we integrate into a common SDV and PMDV database. This integrated DW/MDM further feeds the MDM and local DMs with relevant data. JobDB data source schema is describing employees, their bosses and projects they are currently working. TrainingDB data source schema also describes employees, but this time in the context of business trainings in which they participate and competences they achieve through them. SdvDB and PmdvDB databases integrate these two data sources into a single schema, through the use of Data Vault model. However, due to size restrictions, business case data models will not be shown in this paper. B. Prototype description Research prototype has been developed according to the business case examples and has been tested by running a set of queries against SDV, PMDV and MDV databases, as well as the original relational system catalog. Fig. 3 shows the architecture of a prototype. Corresponding to business case models shown in the previous section, there are two source databases (JobDB and TrainingDB) which are then integrated into a raw copy of the source data (SdvDB). A business and master PmdvDB is built and loaded from the SdvDB and it keeps "purified" data. The MdmDB is loaded from PmdvDB and it practically represents the MDM system which keeps the master data. In MdmDB the same data as in PmdvDB is stored, but organized according to a dimensional model - in the dimensional database. MdmDB is then the basis for loading the DmDB, which is also based on a dimensional model and represents a local DM. This way, DW and MDM are integrated - MdmDB serves as the employee master data and the basis of loading the local DMs, and Figure 3. Prototype s architecture diagram 1636 MIPRO 2017/miproBIS

5 has the ability to return 'golden copy' of employee master data back to data sources. The prototype (with all of its separate databases) is developed and implemented on the same Windows 10 Education x64 operating system and is made in Microsoft SQL Server 2012 [24] database management system using Microsoft SQL Server 2012 Integration Services (SSIS) and SQL Server data Tool for Visual Studio 2012 for extracting, loading and transforming the data between specific databases. To generate the data with which we initially loaded the source databases JobDB and TrainingDB, we used the web tool FreeDataGenerator [11]. C. Testing the sustainability of the integration As we already stated, our main research question was Can our new system catalog model serve to successfully integrate DW and MDM systems?. We presume that the successful integration is a sustainabile one if a MDV model for integration is developed (which it is, shown in Fig. 2) and if: a) a new system catalog (MdvDB in the prototype) collects and stores historical meta-data about SdvDB and PmdvDB schema mappings, and b) queries defined over MdmDB and DmDB return the same results. In order to prove the integration was successful we developed and conducted two simple tests on a prototype: a) run a query on MdvDB system catalog that can return information about SdvDB and PmdvDB schema mappings, and b) run equivalent queries over MdmDB and DmDB that return the same results. For the test A we developed 8 queries on the new MdvDB system catalog that return information on SdvDB and PmdvDB structures (hubs, links, satellites and attributes) and their mappings, as well as the mappings between data sources and SdvDB structures, or mappings between PmdvDB and MdmDB structures. Fig. 4 shows one of those queries a query which serves to monitor the mappings and integration between base hubs in SdvDB and their corresponding master hubs in PmdvDB. This way we can monitor the transformations and the origin of master data, as well as structural changes (Change Type and LEDTS columns in Fig. 4). Fig. 5 shows the query that returns a similar information on the history of mappings between data sources tables and SdvDB hubs. All the queries that we have created and run by MdvDB system catalog successfully returned needed information on creation of PmdvDB structures from SdvDB structures, as well as the historical state of integration thus we conclude the test A was successfully conducted. For the test B we made a data change in the JobDB Figure 4. SdvDB and PmdvDB hub mappings stored in MdvDB system catalog Figure 5. SdvDB and data sources hub mappings stored in MdvDB system catalog data source, but only after loading with start data all the databases in the prototype (SdvDB, PmdvDB, MdmDB and DmDB). An employee Desiree Farmer became Desiree Farmer-Matthews we changed her last name. Then we propagated this data change to all the levels in the prototype architecture. Fig. 6 shows a pair of equivalent queries on both MdmDB and DmDB queries that return the number of rows in Employee master table after the data change. We can see they have returned the same results. Fig. 7 shows the data state in the MdmDB and DmDB after the change. Two equivalent queries have been made on those databases and we can also see that the results are identical. From the results from test B (as well as test A) we have gotten the answer to our research Figure 6. Number of rows in DmDB and MdmDB after a data change in data sources Figure 7. DmDB and MdmDB state after a data change in data sources MIPRO 2017/miproBIS 1637

6 question yes, it is possible to successfully integrate DW and MDM systems through the usage of a new system catalog and new DW/MDM system architecture. V. CONCLUSION AND FUTURE WORK In this paper we presented our research on a DW/MDM integration into one common business intelligence system. Seeing that the DW and MDM both integrate basically the same data sources and have the same evolution problem (preserving the history of data and structure), our goal was to integrate those two systems into one so we could in the future address the evolution problem at the common level. We have presented our MDV meta-model which serves to integrate those two systems and to provide the means for managing their evolution at a mutual level. In our proposed common solution, the data repository preserves the history of raw (DW) and master (MDM) data, as well as their schemas. Meta-data repository (i.e. system catalog) preserves the history of meta-data for the data repository. This way, the problem of the DW and the MDM evolution could be addressed at the general level and a permanent general solution on a meta-level could be developed. The end result could be a flexible, modular solution which will be able to track and manage changes in both data and metadata, as well as their schemas. In the paper we described our common integration architecture, which includes the new system catalog based on a Data Vault modelling method. We further described the integration part of underlying data model for the new system catalog and we presented our research prototype used for empirical validation of the sustainability of the proposed solution. Also, we described a couple of tests that we conducted in order to prove this kind of integration is possible and can be successfully deployed. These are also the main contributions of this paper. The benefits of this approach could be various, the least of all a development of a simpler common solution which can effectively manage data and schema evolution in both of those systems. The next step of our research is the main one - proving that the schema evolution problem can be solved more effortlessly and efficiently through the usage of our solution. In order to do so, we plan to define a final set of structural change cases, together with evolution operations and test them against the prototype, as well as the traditional RDBMS system catalog. REFERENCES [1] B. Bebel, J. Eder, C. Koncilia, T. Morzy and R. Wrembel, Creation and Management of Versions in Multiversion Data Warehouse, 19th ACM Symposium on Applied Computing, Nicosia, Cyprus, [2] Z. Bellahsene, Schema Evolution in Data Warehouses, Knowledge and Information Systems, pp , [3] A. Berson and L. Dubbov, Master data management and data governance, 2nd ed. New York: McGraw Hill, [4] P. Chen, "The Entity-Relationship Model - Toward a Unified View of Data", ACM Transactions on Database Systems, vol. 1 pp. 9 36, [5] J. Chen, S. Chen and E. Rundensteiner, A transactional model for data warehouse maintenance, In: Spaccapietra, S., March, S.T., Kambayashi, Y. (eds.) ER LNCS, vol. 2503, pp Springer, Heidelberg, [6] E. F. Codd, "Relational database: a practical foundation for productivity", Communications of the ACM, vol.25, pp , [7] Y. Cui and J. Widom, Practical Lineage Tracing in Data Warehouses, Proceedings of the 16th International Conference on Data Engineering, San Diego, California, [8] (2015) The CA ERwin Data Modeler Site [Online]. Available: [9] J. Eder and C. Koncilla, Evolution of Dimension Data in Temporal Data Warehouses, Technical Report, [10] H. Fan and A. Poulovassilis, Schema Evolution in Data Warehousing Environments A Schema Transformation-based Approach, Proceedings of 23rd International Conference on Conceptual Modeling, Shanghai, China [11] FreeDataGenerator (June 2016). Available at: [12] M. Golfarelli and S. Rizzi, Data warehouse design, New York: McGraw Hill, [13] M. Golfarelli, J. Lechtenbörger,S. Rizzi and G.Vossen, Schema Versioning in Data Warehouses, In: ER Workshops, LNCS Springer, vol. 3289, pp , [14] A. Gupta and I. Mumick, Maintenance of Materialized Views: Problems, Techniques, and Applications, Data Engineering Bulletin, [15] W. H. Inmon, Building the data warehouse, 4th ed., Indianapolis: Wiley Publishing, [16] W. H. Inmon, D. Strauss and G. Neushloss, DW 2.0: The Architecture for the Next Generation of Data Warehousing, Burlington: Morgan Kaufmann Publishers, [17] Integration Definition for Information Modeling (IDEF1X) Standard, FIPS Publication 184, Computer Systems Laboratory of the National Institute of Standards and Technology (NIST), [18] D. Jakšić and P. Poščić, Data Warehouse Models in Higher Education Courses, International Conference on Advanced Technology & Sciences, Antalya, Turkey, [19] V. Jovanovic, S. Bojicic, C. Knowles and M. Pavlic, Persistent staging area models for data warehouses, Issues in Information Systems, vol.13, pp , [20] R. Kimball and M. Ross, The data warehouse toolkit, 3rd ed., Indianapolis: Wiley Publishing, [21] D. Linstedt, SuperCharge Your Data Warehouse: Invaluable Data Modeling Rules to Implement Your Data Vault, USA: CreateSpace Independent Publishing Platform, [22] D. Linstedt, and M. Olschimke, Building a Scalable Data Warehouse with Data Vault 2.0: Implementation Guide for Microsoft SQL Server Morgan Kaufmann, [23] D. Loshin, Master Data Management, San Francisco: Morgan Kaufmann, [24] Microsoft SQL Server 2012 (October 2016). Available at: [25] T. Morzy and R. Wrembel, On Querying Versions of Multiversion Data Warehouse, In Proceedings of the International Workshop on Data Warehousing and OLAP, DOLAP 04, Washington, USA, [26] D. Subotić, Data Warehouse Schema Evolution Perspectives, Advances in Intelligent Systems and Computing, Springer, vol. 312, pp , [27] D. Subotić, V. Jovanović, and P. Poščić, Data Warehouse and Master Data Management Evolution-A Meta-Data-Vault Approach, Issues in Information Systems, vol.15, pp , [28] D. Subotić, V. Jovanovic, P. Poščić, "Data Warehouse Schema Evolution: State of the Art". 25th Central European Conference on Information and Intelligent Systems CECIIS, Varaždin, Croatia, [29] C. Quix, Repository Support for Data Warehouse Evolution, Proceedings of the Workshop DMDW, Germany, MIPRO 2017/miproBIS