CHAPTER 3 Implementation of Data warehouse in Data Mining

Transcription

1 CHAPTER 3 Implementation of Data warehouse in Data Mining 3.1 Introduction to Data Warehousing A data warehouse is storage of convenient, consistent, complete and consolidated data, which is collected for the purpose of making quick analysis for the end users who take place in Decision Support Systems (DSS). These data is obtained from different operational sources and kept in separate physical store. A data warehouse is not only a relational database that contains historical data derived from transactional data but also it is an environment that includes all the operations and applications to manage the process of gathering data, and delivering it to business users such as extraction, transportation, transformation, and loading (ETL) solution, an online analytical processing (OLAP) engine, client analysis tools. Data warehouses have no standard definition and the people who work on data warehouse subject have defined it in many ways as follows: [1] The basic data warehouse architecture interposes between end-user desktops and production data sources a warehouse that we usually think of as a single, large system maintaining an approximation of an enterprise data model. [2] A data warehouse is a copy of transaction data specifically structured for querying and reporting. [3] A data warehouse as a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management s decision making process. Subject-Oriented: Data warehouses are designed to aid in decision making for a specific subject. For example, sales data for applications contains specific sales of specific products to specific customers. In contrast, sales data for decision support contains a historical record of sales over specific time intervals. If designed well, subject-oriented data provides a stable image of business processes, independent of legacy systems. In other words, it captures the basic nature of the business environment. Integrated: Data warehouse consists of different kind of data which are collected from separate legacy systems and this can create conflicts and inconsistencies among units of measure. 38

2 Because of this, they have to be put in a consistent format and by this way they become integrated. Nonvolatile: Nonvolatile means that, once entered into the warehouse, data should not change. This is logical because the purpose of a warehouse is to enable a user to analyze what has occurred. New data is always appended to the database, rather than replaced. The database continually absorbs new data, integrating it with the previous data. Time variant: There is difference between operational data and informational data from the point of time valiancy. Operational data is valid only at the moment of access-capturing a moment in time. When performance requirements are demanded, historical data is needed. Due to the data warehouse data represents data over a long time horizon; historical analysis can be easily performed The Goals of a Data Warehouse The fundamental goals of the data warehouse are: 1- Makes an organization s information accessible. The contents of the data warehouse are correctly labeled and obvious. It is very easy to reach to data because they are one click away and there is no need to wait for this. These properties are called as same in the above order; understandable, navigable and fast performance. 2- Makes the organization s information consistent. Consistent information has a key importance for the data warehouses since they get data from different parts of an organization. They have to be matched properly. If two measures of the organization have the same name then they must mean the same thing. Conversely, in two measures don t mean the same thing, they are labeled differently. 3- To be an adaptive and resilient source of information. It enables to add new data and ask new questions without any change in existing data and the technologies due to it are designed for continuous change. 4- To be a secure bastion that protects owner s information asset. The data warehouse not only controls access to the data effectively, but also gives its owners great visibility into the uses and abuses of that data, even after it has left the data warehouse. 39

3 5- To be the foundation for decision-making. The data warehouse provides the right data for the decision makers. The decisions are output of the data warehouses Basic Elements of the Data Warehouse Source System A source system is called as legacy system that captures business data and transactions. Source system has to be uptime and available and it gives a chance to share basic dimensions as product and customer with other legacy system in the organization. It is the largest source of data for analysis systems therefore it is a burden to create queries and management reports directly from these systems Data Staging Area A data staging area is an initial storage area where set of processes- that clean, transform, combine, de-duplicate, household, archive- are performed on the data in order to use them in the data warehouse. The data staging area acts as a bridge between the source system and presentation server. Data staging area can be spread over a number of machines and does not need to be based on relational technology. Unlike the presentation service, which will be describe below, the main restriction of data staging area is that it never provides query and presentation services Presentation Server A presentation server is a physical machine that stores the processed data for the end user s querying and reporting requirements. It is fed from data staging area. If the query able presentation resource for an enterprise s data organizes around an entity-relation model, understandability and performance will be lost. Also the tables will be organized as star schema if the presentation server presents and stores data in a dimensional framework Dimensional Model Dimensional model, which is designed to provide higher query performance, resilience to change and to be more understandable, is an alternative model to entity relation model. The dimensional model consists of fact table and dimension tables. A fact table contains measurement of the business that is preferred to be numeric and additive. There has to be a set of two or more foreign keys that helps to join dimension tables 40

4 to fact table. A dimension table is complementary to the fact table. Most of them have many textual attributes. It also has primary key enables to make a relation with the fact table Data Mart Data mart is a logical subset of the complete data warehouse and prepared for a single business process in an organization. When they come together, an integrated enterprise data warehouse is formed. Data marts must be built from shared dimensions and fact. By this way they can be combined and used together OLAP (On-Line Analytic Processing) OLAP enables querying and presenting text and number data from data warehouses for end users. OLAP technology is based on multidimensional cube of data and OLAP databases have multidimensional structure End User Application These applications help end users to prepare queries, make analysis and perform other activities which are targeted to support business needs such as end user data access tool and ad hoc query tool. End user data access tool works with SQL session and provides to the user a report, a screen of data or another forms of analysis. Ad hoc query tool facilitates preparing queries by given an opportunity to the user to use pre-built query templates Modeling Application Modeling applications enable to transform or make a summary from the data warehouse by forecasting models, behavior scoring models allocation models and data mining tools Metadata Metadata contains information and definitions about the data, which is stored. The basic elements of the data warehouse are given in Figure

5 Source Systems (Legacy) Data Staging Area The Data Warehouse Presentation Services End User Data Access Storage: Flat files(fastest); Data Mart#1: Extract RDBMS; Others; Processing; OLAP(ROLAP and /or MOLAP) Query Services; Dimensional subject Oriented; AD Hoc Query Clean; Locally implemented; Prune; User group driven; Report Writers Extract Combine; Remove duplications; May store atomic data; May be frequently refreshed; End User Household; Conform to DW Bus Applications Standardize; Extract Conform dimensions; Store awaiting; Replication; Export to data marts; No User Query Services; DW Bus Data Mart#2 Data Mart#3 Models: Forecasting; Scoring; Allocating; Data mining; Other downstream system; Other parameters special UI Figure The Basic Elements of the Data Warehouse 3.4 Differences between Operational Database Systems and Data Warehouses The major task of online operational database systems is to perform online transaction and query processing. These systems are called online transaction processing (OLTP) systems. They cover most of the day-to-day operations of an organization. Data warehouse systems, on the other hand, users or knowledge workers in the role of data analysis and decision making. Such systems can organize and present data in various formats in order to accommodate the diverse needs of different users. These systems are known as online analytical processing (OLAP) systems. The main differences of OLTP and OLAP are given below. 42

6 (1) Users and System Orientation: An OLTP system is used for transaction and query processing by clerk, clients and information technology professionals. An OLAP system is used for data analysis by knowledge workers, analysts, managers and executives. (2) Data Contents: An OLTP system manages current data that typically are too detailed to be easily used for decision making. An OLAP system manages large amounts of historic data, provides facilities for summarization and aggregation and stores and manages information at different levels of granularity. These features make the data easier to use for informed decision making. (3) Database Design: An OLTP systems use the entity-relationship(er) data model and an application-oriented database design. An OLAP systems use a star or snowflake model and subject-oriented database design. (4) View: An OLTP system focuses mainly on the current data within an enterprise or department, without referring to historic data or data in different organization. In contrast, an OLAP system often spans multiple versions of a database schema, due to the evolutionary process of an organization. OLAP systems also deal with information that originates from different organizations, integrating information from many data stores. Because of their huge volume, OLAP data are stored on multiple storage media. (5) Access patterns: The access patterns of an OLTP system consist mainly of short, atomic transactions. Such a system requires concurrency control and recovery mechanisms. However, accesses to OLAP systems are mostly-read only operations, although many could be complex queries. 3.5 Comparison of OLTP and OLAP Systems. Feature OLTP OLAP Characteristic Operational processing Informational processing Orientation Transaction Analysis User Clerk, DBA, database professional Knowledge worker (Manager,analyst,executive) Function Day-to-day operations long-term informational 43

7 requirements decision support DB design ER-based, applicationoriented Star/snowflake, subject-oriented Data Summarization Current, guaranteed Up-to-date Primitive, Highly detailed Historic, accuracy maintained over time Summarized, consolidated View Detailed, flat relational Summarized, multidimensional Unit of work Short, simple Transaction Complex query Access Read/write Mostly read Focus Data in Information out Operations Index/ hash on primary key Lots of scans Number of records accessed Tens Millions Number of Users Thousands Hundreds DB size GB to high- order GB >=TB Priority High performance, High availability High flexibility, end-user autonomy 3.6 Data Warehouse Architectures Data warehouses and their architectures vary depending upon the specifics of an organization's situation. Three common architectures are: Data Warehouse Architecture (Basic) Data Warehouse Architecture (with a Staging Area) 44

8 Data Warehouse Architecture (with a Staging Area and Data Marts) Data Warehouse Architecture (Basic) By this simple architecture for a data warehouse seen in Figure 3.6.1, end users directly access data derived from several source systems through the data warehouse. Data Sources Warehouses Users Operational System Meta Data Analysis Operational System Summary Data Raw Data Reporting Mining Flat files Figure Architecture of a Data Warehouse(Basic) An additional type of data, summary data is very valuable in data warehouses because they pre-compute long operations in advance. For example, the result of the query that is about sales of last year is retrieved by adding sales data Data Warehouse Architecture (with a Staging Area) The most data warehouses use a staging area in order to clean and process the operational data before putting it into the warehouse. A staging area simplifies building summaries and general warehouse management. The quite common architecture is shown in Figure

9 Data Sources Staging Warehouses Users Area Operational Sys. Meta Data Analysis Reporting Operational System Summary Data Raw Data Mining Flat files Figure Architecture of a Data Warehouse with a Staging Area Data Warehouse Architecture (with a Staging Area and Data Marts) A warehouse s architecture can be customized for different groups within the organization by adding data marts, which are systems designed for specific parts of business. The following Figure shows an example. In this example, there are three data marts which are designed separately for purchasing, sales, and inventories. This architecture gives an opportunity to analyze historical data for purchases and sales. 46

10 Data Sources Staging Area Warehouses Data Marts Users Operational Sys Meta Data Purchasing Analysis Sales Reporting Operational System Summary Data Raw Data Inventory Mining Flat files Figure Architecture of a Data Warehouse with a Staging Area and Data Marts 3.7 Data Warehousing in Multitier Architecture Generally Data warehouses often adopt a three-tier architecture, Following are the three tiers of the data warehouse architecture. Bottom tier: The bottom tier is a Data warehouse server that is almost always a relational database system. Back-end tools and utilities are used to feed data into the bottom tier from operational databases or other external sources. These tools and utilities perform data extraction, cleaning and transformation and as well as load and refresh functions to update the data warehouse. The data are extracted using application program interface known as gateways. This tier also contains a metadata repository, which stores information about the data warehouse and its contents. Middle tier: The middle tier is an OLAP server that is typically implemented in two ways. (1) Relational OLAP (ROLAP) model which is an extended relational DBMS that maps operations on multidimensional data to standard relational operations. 47

11 (2) A multidimensional OLAP(MOLAP) model which is a special-purpose server that directly implements multidimensional data and different operations. Top tier: The top tier is a front-end client layer, which contains query and reporting tools, analysis tools and data mining tools. 48

12 Query/Report Analysis Data Mining Top tier: Front- end tools ROLAP Server O/P MOLAP Server Middle tier: OLAP Server Monitoring Administration Metadata Repository Bottom tier: Data warehouse System Extract, Clean, Transform, Load Refresh Operational Database Extract Data Flat files Figure 3.7 A Three tier data warehousing Architecture 49

13 3.8 Define Extraction, Transformation and Loading Data warehouse systems use back-end tools and utilities to populate and refresh their data. These tools and utilities include the following functions. Data Extraction which typically gathers data from multiple, heterogeneous and external sources. Data Cleaning which detects errors in the data and rectifies them when possible. Data Transformation which converts data from legacy or host format to warehouse format. Load, which sorts, summarizes, consolidates, computes views, checks integrity and builds indexes and partitions. Refresh, which propagates the updates from data source to the data warehouse. 3.9 Data warehouse Metadata Given the complexity of information in an ODS and data warehouse, it is essential that there be a mechanism for users to easily find out what data is there and how it can be used to meet their needs. Providing metadata about the ODS or the data warehouse achieves this. Metadata is data about data or documentation about the data that is needed by the users. It is not the actual data warehouse, but answers who, what, where, when, why and how questions about the data warehouse. Another thing is that of Metadata is that it is structured data which describes the characteristics of resource. Metadata is stored in the system itself and can be queried using tools that are available on the system. Examples: (1) The table of contents and index in a book may be considered metadata for the book. (2) A library catalogue may be considered metadata. The catalogue metadata consists of a number of predefined elements representing specific attributes of a resource, and each element can have one or more values. These elements could be the name of the author, 50

14 the name of the document, the publisher s name, the publication date and the category to which it belongs. They could even include an abstract of the data. (3) Suppose we say that a data element about a person is 80. This must be described by nothing that it is the person s weight and the unit is kilograms. Therefore (weight, kilogram) is the metadata about the data is 80. A metadata repository is a database of data about data (metadata). The purpose of the metadata repository is to provide consistent and reliable access to data. The metadata repository itself may be stored in a physical location in which metadata is drawn from separate sources. Metadata may include information about how to access specific data or more details about the data Role of Metadata Metadata has a very important role in a data warehouse. The role of metadata in a warehouse is different from the warehouse data, and it plays an important role. The various roles of metadata are explained below. Metadata acts as a directory. This directory helps the decision support system to locate the contents of the data warehouse. Metadata helps in decision support system for mapping of data when data is transformed from operational environment to data warehouse environment. Metadata helps in summarization between current detailed data and highly summarized data. Metadata is used for query tools. Metadata is used in extraction and cleansing tools. Metadata is used in reporting tools. Metadata is used in transformation tools. Metadata plays an importing role in loading functions. 51

15 Metadata plays a very different role than data warehouse and it is important for many reasons. Example: A metadata are used as a directory to help the decision support system analyst locate the contents of the data warehouse, and as a guide to the data mapping when data are transformed from the operational environment to the data warehouse environment. Metadata also serve as a guide to the algorithms used for summarization between the current detailed data and the highly summarized data, and between the lightly summarized data and the highly summarized data. Metadata should be stored and managed persistently. The following diagrams show the role of Metadata. Transformation tools Data mining tools Data load function Extraction tools Data Metadata warehouse Query Tools OLAP tools Source systems Application Figure Role of Metadata Chart Metadata Repository: Metadata repository is an integral part of a data warehouse system. A Metadata repository should contain the following: (1) Definition of data warehouse: It includes the description of the structure of data warehouse. The description is defined by schema, view, hierarchies, derived data definitions, and data mart location and contents. (2) Business Metadata: It includes the business terms and definitions, data ownership information and changing policies. 52

16 (3) Operational Metadata: It includes currency of data and data lineage. Currency of data means whether the data is active, archived or purged. Lineage of data means the history of data migrated and transformation applied on it. (4) Data for mapping from operational environment to data warehouse: It includes source databases and their contents, data partitions, data extraction, cleaning, transformation rules, data refresh and purging rules and security (user authorization and access control). (5) The algorithms used for summarization: It includes measure and dimension definition algorithms, data on granularity, partitions, subject areas, aggregation, summarization, and predefined queries and reports. (6) Data related to system performance: It includes indices and profiles that improve data access and retrieval performance, in addition to rules for the timing and scheduling or refresh, update and replication cycles Types of Metadata in Data Warehouse Architecture: The two most common approaches to building Meta data repository architecture are: (1) Centralized (2) Decentralized Generally small to medium sized organizations, a single metadata repository (the centralized approach) is sufficient for handling all of the metadata required by the various groups in the corporation. This architecture offers a single and centralized approach to administering and sharing metadata. On the Other hand most large enterprises that have multiple and disparate divisions will require several metadata repository for handling all of the corporation s various types of metadata content and applications Centralized Metadata Repository Architecture: This approach is the most common one that corporations have implemented. 53

17 The concept of a centralized Metadata architecture, consistent Meta model that mandates the schema for defining and organizing the various metadata be stored in a global metadata repository. The strength of this approach is that it integrates all of the metadata and stores it in the Meta model schema that can be easily accessed. Meta Data Repository Process to build the metadata repository Metadata sources Figure Centralized Metadata Repository Architecture Decentralized Metadata Repository Architecture: Decentralized Metadata architecture creates a uniform and consistent Meta model that mandates the schema for defining and organizing the various Metadata to be stored in a global metadata repository and in the shared metadata elements that appear in the local meta data repository. All the Metadata that is shared and reused among the various repositories must first go through the central global repository, but sharing and access to the local metadata is independent of the central repository. 54

18 Global Metadata Repository Local Metadata Local Metadata Local Metadata MetaData sources MetaData sources MetaData sources Figure Decentralized Metadata Repository Architecture 3.10 Mapping A basic part of the data warehouse environment is that of mapping from the operational environment into the data warehouse. The mapping includes a wide variety of feature include some here. Mapping from one attribute to another Conversions Changes in mapping conventions. Changes in physical characteristics of data. Filtering of data, etc. Example: Consider the Vice president of marketing who has just asked for a new report of product selling and purchasing. The manager turns to the data warehouse for the data for report. Upon inspection, the vice president proclaims the report to be fiction. Than manager who can prove that data in the report to be valid. The manager first looks to the validity of the data in the 55

19 warehouse. If the data warehouse, data has not been reported properly then the reports are adjusted. However, if the reports have been made properly from the data warehouse, the manage having to go back to the operational sources. At this point, if the mapping data has been carefully stored, then the manager can quickly and easily go to the operational source. However, if the mapping has not been stored properly, then manager has a difficult time defending conclusion to the vice president. The metadata store for the data warehouse then is natural place for the storing of mapping information. Metadata Mapping Operational Environment Data warehouse Figure Functionality chart of Mapping Data Mart The data mart is a model, which represents the same data structure with the data warehouse. They are prepared for specific requirements of the whole organization or a part of it. The data mart contains less data that gives to users some advantages. Firstly it enables to work with faster queries. Another advantage is mobility due to it requires less hard disk space so the user can carry the data mart with the laptop. During the designing process of the data marts, it is possible to follow up two different methods in order to collect the data. One option is to collect the granular data from the enterprise data warehouse and then process it according to the needs around which the data mart was prepared. The second option is to collect shaped data directly to the data mart. The data, which is designed up to the requirements of data mart, then is kept in the central repository of all enterprise data. In Figure 3.11 the options can be seen. 56

20 Data Marts Applications (OLAP Systems) Staging (transformation & integration) Enterprise Data Warehouse ODS Figure Enterprise Information Architecture Data marts can have dependent or independent structure. If the characteristic of the data marts dimensions are defined at the beginning, as they would be compliant to each other then these data marts will have dependent characteristic. In some situations it is better to have independent data marts. This time the characteristic of the other data marts will not take in the consideration during the preparation of the data mart. However, this can prevent future integration and add development cost if there will be an interest in sharing information across departments Reason for creating a Data Mart To give users more flexible access to the data they need to analyze most often. To provide data in a form that matches the collective view of a group of users. To improve end uses response time. Potential users of a data mart are clearly defined and can be targeted for support to retrieve the data. 57

21 To provide appropriately structured data as dictated by the requirements of the enduser access tools. Building a data mart is simpler compared with establishing a corporate data warehouse. The cost of implementing data marts is far less than that required to establish a data warehouse. Data mart is the access larger of the data warehouse environment. That means we create data mart to retrieve the data to the users faster. The Data mart is the subset of warehouse that means all the data available in the data mart will be available in database. This Data mart will be created for the purpose of specific business. It is easy to access frequently needed data from the database when required by the client. We can give access to group of users to view the Data mart when it is required. Of course performance will be good. It is easy to maintain and to create the data mart. It will be related to specific business. And it is low cost to create a data mart rather than creating data warehouse with a huge space. Flat files Operational Systems External Data Marketing Sales, Finance Human Resource Marketing Sales Finance External Data Figure Functionality chart of Data Mart Data Mart 58

22 Data Marts Development Approaches There are three main approaches for building data marts; top-down approach, bottomup approach and federated approach Top-Down Approach As shown in the Figure below the data firstly comes to the data staging area from the operational sources and in this area some of the processes are performed to the data. After this it is transferred to the data warehouse which then feeds it to the dependent data mart. Data Marts Applications (OLAP Systems) Staging (transformation & integration) Enterprise Data Warehouse (EDW) ODS Figure Top-Down Approach to Data Mart Development Bottom-Up Approach In this approach, the data, which comes from legacy systems to the staging area, flows directly into the independent data marts and then these data marts feed the enterprise data warehouse as it is illustrated in Figure

23 Data Marts Applications (OLAP Systems) Staging (transformation & integration) Enterprise Data Warehouse (EDW) ODS Figure Bottom-Up Approach to Data Mart Development Federated Approach With the federated approach, there are both dependent and independent data marts as shown in Figure The independent data mart gathers the data from the staging area directly, and the dependent data marts get the data from the data marts. 60

24 Independent Data Marts DSS Tools Meta Data Transform Extract Enterprise Data Dependent Data Marts Applications (OLTP Systems) Staging Area Warehouse (EDW) Transform ODS Figure Federated Approach to Data Mart Development The Differences between Data Mart and Data Warehouse When the data mart is compared with the data warehouse, two fundamental distinctions can easily be noticed. One of them is that data mart is a subset of the data warehouse and it is requirement-oriented. Against this data warehouse holds the enterprise data without taking care about any specific requirements. But of course, during the design of data mart the structure of the whole warehouse has to be considered, if not it will be very hard to integrate the data marts later. The implementation of the data mart is much faster and costs cheaper, since a data mart contains only a specific part of the data warehouse whose implementation is more timeconsuming and costs much more. There are some data mart solutions that are developed by the many decision support systems (DSS) vendors. But using them to design a data mart for the specific requirements needs to spend much more effort to customize them; due to this solutions are produced for general purposes. The other main difference of the data mart from the data warehouse is that the data in the data mart can be more granular than the data warehouse. Since the requirements of the 61

25 data mart are more defined than those of the data warehouse, preaggregation can be afforded to the data along the requirements. So the extraction of the data can be done faster and more efficient. 62