Multidimensional modeling using MIDEA

Transcription

1 Multidimensional modeling using MIDEA JOSÉ MARÍA CAVERO 1, MARIO PIATTINI 2, ESPERANZA MARCOS 1 1 Kybele Research Group, Escuela Superior de Ciencias Experimentales y Tecnología Universidad Rey Juan Carlos C/ Tulipán s/n, Móstoles (Madrid) 2 Escuela Superior de Informática Universidad de Castilla-La Mancha Ronda de Calatrava 5, Ciudad Real SPAIN Abstract: - Developing a Data Warehouse has become a critical factor for the success of many companies. Specific issues, such as conceptual modeling, scheme translation from operational systems, physical design, etc..., have been widely treated. Unfortunately, there is still not a general accepted complete methodology for data warehouse design. In this work we present MIDEA, a multidimensional data warehouse development methodology based on a multidimensional data model, and an application example of its conceptual modeling activity. Key-Words: - Data base design, Multidimensional modeling, Data warehouse design 1 Introduction A data warehouse is a subject oriented, integrated, non-volatile, and time variant collection of data in support of management's decisions [6]. It is a concept very related to the OLAP technology, first introduced by Dr. E.F. Codd in 1993 to characterize the requirements of aggregation, consolidation, view production, formulae application and data synthesis in many dimensions [2]. A data warehouse is a repository of information mainly coming from online transactional processing (OLTP) systems that provides data for analytical processing and decision support. Multidimensional view of data is a very old concept: managers observe the evolution of interesting data organized in dimensions, such as products, clients, promotions, sell points, and, of course, time. The need of having simply and rapidly every historical information of the operational systems has pushed to companies to look for new ways of structuring and accessing their data, for having advantage to their competitors. There is an agreement in that traditional data base systems are not appropriate for multidimensional data analysis. Traditional OLTP systems are optimized for providing high performance in processing a lot of concurrent transactions. These transactions usually affect to a very few records. Meanwhile, multidimensional systems have to answer to complex queries (sometimes unpredictable) that need a huge number of records [1]. In fact, OLTP is profoundly different from dimensional data warehousing in their users, their data content and structures, their hardware and software, their administration and management, and their daily rhythms [7]. 2 Data warehouse design OLTP and OLAP environments are profoundly different Therefore, the techniques used for operational database design are inappropriate for data warehouse design [7,8]. The development of a data warehouse needs the integration of data mainly proceeding from legacy systems. The process of developing a data warehouse is, like any other task that implies some kind of preexisting resources integration, profoundly complex. This process is labor-intensive, errorprone, and generally frustrating, leading a number o warehousing projects to be abandoned mid-way through development [13]. To this respect, in latest years, there have been many proposals restricted to some of the particular aspects involved in the data warehouse design process. However, although many solutions have been developed for interesting sub-problems like handling multidimensional data as typical requirement for data warehouses, view maintenance for aggregated data, data integration etc., combining these partial and often very abstract and formal solutions to an overall design methodology and warehousing strategy is still left over to the practitioners [4]. Despite the obvious importance of having a methodological support for the development of

2 OLAP systems, the design process has received very little attention of the scientific community and the product providers. Models usually utilized for operational data base design (like E/R model) shouldn t be used without further ado for analytical environments design. Attending only to technical reasons, databases obtained from E/R models are inappropriate for decision support systems, in which query performance and data loading (including incremental loading) are important [7]. Multidimensional paradigm should be used not only in data base queries, but also during its design and maintenance. To use the multidimensional paradigm during all development phases it is necessary to define dedicated conceptual, logical and physical data models for the paradigm and to develop a sound methodology which gives guidelines how to create and transform these models during the development process [3]. In [14] authors claim for data warehouse design methodologies and tools with the appropriate support for aggregation hierarchies, mapping between the multidimensional and the relational models, and cost models for partitioning and aggregation that can be used from the early design stages. There are a few proposals for data warehouse design [1, 8, 15], most of them incomplete or very focused from its initial phases on the relational model. There are also many partial proposals, focused on issues such as models translations, view materialization, indexing, etc. For example, in [12] using data mining techniques in data warehouse design phases is proposed (for example, using data mining algorithms for discovering implicit information on data, for conflicts resolution in schemes integration, for recovering lost values and incorrect data, etc.). The problem with all these works is that they propose to use a new different methodology for data warehouse design, so organizations must use at least two totally different methodologies: one for OLTP environments and one for OLAP environments. We think that it is better to integrate data warehouse design in the existing methodologies, modifying and adding new activities, so that training and learning curve for data warehouse design should be less difficult. 3 MIDEA methodology In this paper we propose a methodology integrated into a preexisting traditional methodology. It has been developed in the EINSTEIN project. EINSTEIN is a research and development project that applies the experience and knowledge obtained in relational data base system development in the last decade (SQL, ER modeling, CASE tools, methodologies...) to MultiDimensional DataBase (MDDB) design. The project is based on the following three points: - IDEA, a multidimensional conceptual model used to understand and represent analytical users requirements in a similar manner than ER model is used to interact with micro-data users [11]. - IDEA-DWCASE, a CASE tool that supports multidimensional modeling using IDEA [10]. IDEA- DWCASE incorporates a graphical interface and allows the translation of a conceptual IDEA scheme into a logical scheme based on a model supported by some multidimensional or relational products. Figure 1 shows a tool prototype window, including an IDEA schema, whose graphical notation is based on [5]. - MIDEA, a data warehouse development methodology. In the following subsection, we outlines the main characteristics of the methodology. Figure 1. IDEA-DWCASE 3.1 Methodology overview MIDEA methodology uses as reference framework the Spanish Public Methodology METRICA version 3 proposal (MV3), which is similar to British SSADM or French Merise [9]. MV3 processes considered are those on which the data warehouse development has more influence, that is, Information System Analysis, Design and Construction (ASI, DSI and CSI). The new processes, modified from the MV3 proposal, have been named as ASI-MD (MultiDimensional), DSI-MD and CSI-MD. Of course, considering only these three processes doesn t mean that the others processes shouldn t be taken into account on a data warehouse development, but we have considered that the differences shouldn t be significant with respect to any other information system development. Every process of the methodology is divided into activities and every activity is divided into tasks. The

3 order of the activities doesn t mean a necessary sequential order. The activities can be developed in a different order, or in parallel, overlapping tasks of different activities. However, a process should not be considered finished until completing every of its activities. In every process, a graphic emphasizing its most important activities is included. In the following, we offer a general overview of the three process of the methodology, graphically outlined in figure 2. Expert users (Business analysts, specialists,...) Translation into a multidimensional model (MOLAP) Pure Multidimensional Logical Scheme Data Warehouse (MOLAP) Conceptual Scheme (using IDEA model) Construction ExistingDB (ER Conceptual schemes) Conceptual Modeling Translation into a relational model (ROLAP) Relational with Multidimensional issues Logical Scheme Data Warehouse (ROLAP) Figure 2. MIDEA processes ASI-MD DSI-MD CSI-MD Analysis of the Information System (ASI- MD) The basic purpose of ASI-MD process is to obtain a detailed specification of the data warehouse. This specification has to satisfy the information needs of users (business analysts, specialists,...) and serve as a basis for the design. Information gathering is mainly done in ASI-MD 2 activity, Obtaining Detailed Requirements. The General Requirements Catalogue and high level schemes obtained during the Feasibility Study are used as starting points in this phase. If the Feasibility Study weren t done, such Catalogue should be done in the first activity of this process. That Catalogue consists of a set of generic and user oriented requirements. These products should be refined with users by means of work sessions. In this way, data warehouse requirements should be in more detail specified. In addition, data warehouse non-functional requirements have to be identified, that is, constraints that have to be accomplished related to performance, security, etc. The purpose of activity ASI-MD 2 is to define a detailed and validated Requirements Catalogue, which serve as a basis to test correctness of schemes obtained in activity ASI-MD 3, Data Warehouse Conceptual Modeling. This activity contains a verification and validation task in which the schema must be reviewed to guarantee that it is complete, complied with the Requirements Catalogue, and met some predetermined quality criteria. Participation of users is essential to this process, because it constitutes a warranty that requirements initially identified have been understood and incorporated into the system and, therefore, that it will be accepted. As an example, next we outline ASI-MD 3 activity, Datawarehouse Conceptual Modeling. We also briefly explain the construction of a schema example, in which we model information about the products sold by a company, the sales obtained, and the average price of each product. Such example is graphically represented in figure 1. Datawarehouse Conceptual Modeling activity has seven tasks. Due to space restrictions, input and output artifacts, techniques and participants of every task haven t been specified in this paper. The purpose of this activity is to obtain, using IDEA model, the datawarehouse data multidimensional conceptual schema. We use as inputs the requirements catalog and existing ER schemes. The first step to obtain the multidimensional conceptual scheme is to obtain a preliminary scheme. This preliminary scheme is obtained during first and second tasks. The purpose of Task 1, Obtaining Preliminary Sub-cell structures is to obtain preliminary sub-cell structures. This preliminary structures represent events occurring dynamically in the enterprise world [5], such as sales of a company, movements in a bank account, etc. At this moment it is not mandatory to detail attributes and synthesis functions (sum, average,...) that comprises every sub-cell structure. We are still only interested in preliminary, generic structures. Necessary information for modeling preliminary sub-cell structures can come from different sources: on the one hand, and more important, expert users opinion. They know which are their problems, and which are the data they need for their daily work. Usually these correspond with numeric, continuously valued, and additive data [7]. On the other hand, if we have a data base ER conceptual scheme, these preliminary sub-cell

4 structures use to correspond with some of the entities or N:M relationships attributes. We can also use the previous multidimensional conceptual scheme made in activity ASI-MD 1. Preliminary sub-cell structures detected so far represent company interesting variables. The next step (Task 2, Obtaining Preliminary dimensions ) is to detect dimensions that should take part on them, that is, how the values detected in task 1 can be aggregated. In this moment, users have to think about dimensions in a very general manner. They don t need to detail dimensions hierarchy attributes. For example, users must think in dimensions such as time, space, etc..., but no in attributes such as daysmonths-years, delegation-province-country. This descending way of working can be complemented with the study of operational data bases conceptual schemes observing the attributes of entities and relationships connected (directly or by means of others) to those identified as facts in the ER scheme. Those attributes could give us clues about hidden dimensions not detected by users. If we have a general multidimensional scheme as output of ASI- MD 1 activity, then it can be used as another information source. At this point we have a preliminary scheme with a set of preliminary sub-cell structures defined over some dimensions. In our example, our preliminary sub-cell structures could be the items sold, and the sales (in dollars). Our preliminary dimensions should be the dimensions along which we can define our facts, that is, time, stores, and products: each sale is made in a point of time (or during an interval) in a store and corresponds with some product. The purpose of Task 3, Obtaining Preliminary Hierarchies, is to identify in a more precise manner dimensions and their hierarchies. We have to identify every dimension, describing (if exists) its subhierarchy and sub-hierarchy aggregations. It is not necessary at this moment to detail dimension domains of every aggregation, nor aggregation functions. At this point new dimensions could be detected. A typical example is time, which sometimes is not in the elementary databases, but is essential in every datawarehouse. Next step, done in task 4 ( Obtaining Detailed Hierarchies ) is to refine hierarchies obtained in previous step. This refinement consists in a detailed enumeration of sub-hierarchy dimension attributes of each dimension. New attributes could be detected, useless attributes eliminated, and hierarchies attributes properties detected and converted into description attributes. For example, telephone or address attributes used to be only dimension attributes properties (description attributes). For every attribute its domain must be defined, or assigned a previously defined domain. Domain aggregation hierarchy must be specified, detailing aggregation functions. At this point we have completely defined dimensions and its hierarchies. In our example, with the help of the users and the ER schemes available, we can refine our preliminary dimensions, as shown in figure 3. Each dimension consists of a set of attributes organized in hierarchies. A dimension attribute can have description attributes. The purpose of a description attribute is to describe some dimension element (for example, the address of an store). Aggregations between dimension attributes must also be specified (for example, the city of every store). Month Week Week Manufacturer Product Type Product Size Figure 3. Dimensions City Address Store Store Now we have to study the detail of sub-cell structures. It is done in Task 5, Obtaining Detailed Sub-cell Structures. For each sub-cell structure its attribute and the synthesis functions must be specified. Every synthesis function should be studied with respect to the dimensions that affect it. Perhaps some of them couldn t be applied (for example, it doesn t have sense adding temperatures along time). Usually, synthesis functions are sums, but also can be taken into consideration average, maximum, minimum, etc. At this point we have sub-cell structures with associated dimensions. In Task 6 Obtaining Fact Schemes, we have to group these sub-cell structures into cell structures, to form fact schemes. Every cell structure belongs to a fact scheme, and every fact scheme has associated dimensions. Therefore, subcell structures that could be joined must be detected. This join can be one of the following: - Joining two sub-cell structures into one, because they both represent the same fact (they are duplicated). Its synthesis attributes and dimensions should be the same. The resulting synthesis functions will be the union of the synthesis functions of both sub-cell structures, deleting those duplicated. Synthesis functions applicability to dimensions and dimension attributes should be reviewed.

5 - Aggregation of two sub-cell structures into one, or one sub-cell structure into one previously detected. In this case its dimensions should be similar. We can be interested in joining substructures whose dimensions are not the same. In this case, aggregations must be studied with respect to the new dimensions (to be or not aggregable). Of course, we always have the possibility of join every sub-cell structures into one (fact scheme), but in this case many of the synthesis attributes could be not aggregable, and the resulting scheme should be unreadable. After these last tasks, we have our example almost completed. Our sub-cell structures should be the following: one for the quantity of products sold, and one more for the sales made (in dollars). The synthesis functions for both of them should be the sum, that is, to obtain the aggregated data we have to sum the elementary data from the transactional data bases. We do not need any other sub-cell structure to obtain the average price of the products sold, because we can calculate it by means of a formula (we call it a method): Figure 4 shows the complete schema of the example, which graphical representation corresponds with figure 1. are not available, perhaps could be planned to modify the elementary DB Design of the Information System (DSI- MD) In this process are described the necessary activities to obtain the data warehouse design starting from the Software Requirements Specification obtained in ASI-MD process. The design process describes how to implement the elements detected in the analysis process. In this process the following tests are designed: query tests, query consistency tests and data warehouse acceptance tests. Due to the non-existence of a standard or commonly accepted multidimensional logical model, the data design process is done in one step, from conceptual to logical specific (that is, product dependent) model. This one-step logical design is carried out in activity DSI-MD 2 in case of MOLAP systems, or in DSI-MD 3 for ROLAP systems. Previously (in the activity DSI-MD 1), the appropriate technology (ROLAP or MOLAP) and product must be chosen. In this process there are three activities focused in the Physical Design. The purpose of them is to carry out and tune the physical design starting from the logical design obtained in previous activities (DSI- MD 2 and DSI-MD 3) Construction of the Information System (CSI-MD) The main purposes of this process are codification and test of data warehouse starting from the design specification obtained in DSI-MD process. Tests made during this process are focused into query and consistency query. Acceptance tests will be carried out during system implantation. Figure 4. Final IDEA conceptual schema Finally, in task 7, Multidimensional scheme verification and validation, the conceptual multidimensional scheme has to be verified and validated, assuring that it is complete, adjusted to requirements catalogue, and to some predefined quality criteria. Some other verifications should be done, such as availability of data from elementary DB. If those data 4 Conclusion The development of a Data warehouse has turned into a critical success factor for many companies, and there is a need of methodologies that considers the special characteristics of this kind of systems. MIDEA, a general multidimensional methodology based on a public Spanish methodology proposal has been developed. MIDEA integrates OLTP and OLAP database design in a unique methodology. A CASE tool (IDEA-DWCASE) supports part of the methodology. A first prototype of the tool is available and was presented in [10]. It allows the creation of IDEA multidimensional conceptual schemes, and its

6 translation into different logical schemes directly supported by MOLAP or ROLAP products. At this moment, IDEA-CASE tool translates IDEA schemes into EXPRESS and ORACLE. Acknowledgement: This work is being carried out as part of the MIDAS project. MIDAS is partially financed by the Spanish Government and the European Community (reference number: 2FD ). References: [1] L. Cabibbo and R. Torlone. "A Logical Approach to Multidimensional Databases" In Sixth International Conference on Extending Database Technology (EDBT'98), Valencia, España, Lecture Notes in Computer Science 1377, Springer-Verlag, , [2] E. F. Codd, S. B. Codd, and C. T. Salley, "Providing OLAP (On-Line Analytical Processing) to User-Analyst: An IT Mandate". Technical Report, E. F. Codd and Associates, 1993 [3] B. Dinter, C. Sapia, M. Blaschka, G. Höfling. "OLAP Market and Research: Initiating the Cooperation". Journal of Computer Science and Information Management, Vol 2, N. 3, [4] S. Gatziu, M. A. Jeusfeld, M. Staudt y Y. Vassiliou. "Design and Management of Data Warehouses - Report on the DMDW'99 Workshop". SIGMOD Record 28(4), Dec [5] Golfarelli, M., Maio, D. and Rizzi, S., Conceptual design of data warehouses from E/R schemes en: 31st Hawaii International Conference on System Sciences, [6] W. H. Inmon. Building the Data Warehouse, John Wiley & Sons, New York, 1993 [7] R. Kimball. The Data Warehouse Toolkit: Practical techniques for building dimensional data warehouses. John Wiley & Sons, 1996 [8] R. Kimball, L. Reeves, M. Ross, W. y Thornthwaite. The Data Warehouse Lifecycle Toolkit., John Wiley & Sons, Inc., 1998 [9] A. de Miguel et al., METRICA Version 3: Planning and Development Methodology of Information Systems. Designing a Methodology: A practical experience, In Proceedings of the CIICC`98, Aguascalientes, México, Nov [10] A. de Miguel et al. "IDEA-DWCASE: Modeling mutidimensional databases" EDBT 2000 Software Demonstrations track. Konstanz, Alemania, March [11] A. Sánchez, J.M. Cavero and A. de Miguel. "IDEA: A conceptual multidimensional data model and some methodological implications". Proceedings of the CIICC'99, Cancún, Méjico [12] C. Sapia, G. Höfling, M. Müller, C. Hausdorf, H. Stoyan and U. Grimmer. "On Supporting the Data Warehouse Design by Data Mining Techniques" To appear in GI-Workshop: Data Mining and Data Warehousing, September , 1999, Magdeburg, Germany. [13] J. Srivastava y P-Y. Chen. "Warehouse Creation - A Potential Roadblock to Data Warehousing". IEEE Transactions on Knowledge and Data Engineering. Vol 11, Num. 1, Ene/Feb 1999 [14] M.C. Wu and A.P. Buchmann, "Research Issues in Data Warehousing". BTW'97, Ulm, March, [15] M.Golfarelli and S. Rizzi Designing the data warehouse: key steps and crucial issues. Journal of computer science and information management, Vol. 2, N. 3, 1999