A Proposal of Integrating Data Mining and On-Line Analytical Processing in Data Warehouse

Transcription

1 A Proposal of ntegrating Data Mining and On-Line Analytical Processing in Data Warehouse Zhen LU Faculty of Human Environment, Nagasaki nstitute of Applied Science, 536 Aba-rnachi, Nagasaki , Japan Tel.: , Fax: t , Minyi GUO Department of Computer Software, The University of Aizu, Aim Wakamatsu City, Fukushima %S8580, Japan Tel.: , Fax: , Abat ract As an analysis tool either of On-Line Analytical Processing (OW) or Data Mining has its own strong points and weaknesses. t is an effective way to enhance the power and flexibility of data mining in data warehouse and large databases by intepating data mining with O W to offset their weaknesses. n this paper, a proposal of integrating data mining and O W is put forward. The mechanism of the on-line analytical data mining in data warehouse is described. Aspects which is necessary to develop successful on-line analytical data mining system is also discussed. Key words: Data Mining, On-Line Analytical Processing, Data Warehouse 1. ntroduction n recent years, with the development of the technologies of Data Warehouse, Data Mining and OnLine Analytical Processing (OLAP), Decision Support System (DSS) research and implementation entered a completely new stage [1][2]. OLAP and data mining have become integral parts of decision support process. Although Data mining and O M are all analytical tools, obvious differences exist between each other. The analysis process of data mining is completed automatically. t is only needed to extract hidden patterns, and predict the future trends and behaviors without giving exact query by user. t is ofbenefit to finding unknown facts. While OLAP depends on user s queries and propositions to complete analysis process. t restricted the scope of queries and propositions, and affects the final results. From the view of data analysis, OLAP lies in a lower leve1,while data mining can find more complex and detail information which can not be found by OM. On the other hand, to data, most O W systems have focused on providing access to multi-dimensional data, while data mining systems have deal with influence analysis of data along a single dimension. n the application of data mining in data warehouse, the problem is that data mining is difficult to be realized. There are enormous data and hundreds or thousands attributes in data warehouse. As the process of minng analysis is in progress automatically, and user only points mining tasks but not provide search path, it will lead to the search space too large and the patterns generated too many. And, most of the generated patterns may be general knowledge or non-meaning ones. On the other hand, also OLAF can provide views in different viewpoints anddifferent abstract levels, as user s query could not be understand completely beforehand, it will lack dimensions which should be included in the view, and the results obtained from different view may not be consistent. Therefore, it is easy to bring about erroneous leading. Basing on data warehouse, an integration of data mining with OLAP will improve the efficiency and effectiveness of data warehouse based DSS. n this paper, an approach of integrating O M and data mining architecture is put forward. t integrates data mining with online analytical processing organically to accomplish /01/$ EEE. 146

2 knowledge discovery in data warehouse. The mechanism of the orrline analytical data mining in data warehouse is described. 2. Data Warehouse, Data Mining and OLAP 2.1 Data Warehouse Data Warehouse has been defined as a subject oriented, timevariant, nonvolatile collection of data in support of management s decision needs by W. H. nmon [3][4] t provides tools to satisfy the information needs of users at all organizational levels- not just for complex data queries, but as a general facility for getting quick, accurate, and often insightful information. A Data Warehouse is designed so that its users can recognize the information they want access that information using simple tools. Data Warehouse is physically separated from operational systems, and hold both aggregated and atomic data for management separate from the databases used for OW. t should have two basic functions. One is that it should provide subject-oriented summarized data for supporting decision. The data warehouse provides a clear and unambiguous definition of every key data entity, describing the way each is used, as well as defining derivation formulas, aggregation categories, and refreshment time periods. Another function is that it should take on the task of capturing data from operational databases and other different external data sources. The schematic drawing of the data warehouse is shown in Figure , n order to obtain consistent data for mining, data mining tools often require the raw data to be first integrated and cleaned. t requires costly preprocessing steps of data cleaning, data transforming, and data integration. Since a data warehouse normally goes through these preprocessing steps for O W operations, it serves as a valuable data source for data mining. 2.2 Knowledge Discovery in Database and Data Mining Knowledge Discovery in Database (KDD) is a process of applying technologies mostly from artificial intelligence to discover new information from data warehouse [5][6]. t is a noctrivial process of identifying valid, novel, potentially useful, ultimately understandable patterns in data. There are five steps in KDD process (Figure 2): Selection; Preprocessing; Transformation; Data mining; nterpretation/ evaluation. 147

3 Figure 2. Outlining of the Steps of the KDD Process Data mining is one step of the KDD process. t extracts hidden predictive information from data warehouse or large database. Data mining tools predict future trends and behaviors, allowing businesses to make proactive, knowledge-driven decisions. The automated, prospective analyses offered by data mining move beyond the analyses of past events provided by retrospective tools typical of DSS. 23 On-Line Analytical Processing On-line analytical processing was introduced by E. F.Wd [7][8], the father of Relational Databases in 1993 in a major article in Computerworld. Codd came to the conclusion that relational databases for Transaction Processing) had reached the maximum of their capabilities in terms of the viewsof the data they provided the user. The problem stemmed principally from the massive computing required when relational databases were asked to answer relatively simple SQL quires. He also came to the view that operational data are not adequate for answering managerial questions. He therefore advocated the use of multidimensional databases. His convetsion to the DSS/ES viewpoint gave legitimacy to the data warehouse based concepts. The basic idea in OLAP is that managers should be able to manipulate enterprise data across many dimensions to understand changes that are occurring. As the facility of powerful multidimensional analysis for data warehouse, it is necessary to adopt online analytical processing technology in data warehouse and large database. O W provides such facilities as drilling, pivoting, filtering, dicing and slicing so the user can traverse the data flexibly, define the set ofrelevant data, analyze data at different granularities. and visualize the results in different forms. These operations can also be applied to data mining to make it an exploratory and effective process. Together with OW, data mining functions can provide an overview of the discovered knowledge such that the user can investigate further on any interesting pattens or anomalies. Because with O W operations, the size of the data set is relatively more compact. So that, The mining integrated with OLAP technology can do insure faster response than mining in the raw data directly. 3. On-Line Analytical Data Mining 3.1 The Architecture The integrated data mining and on-line analytical processing architecture is suggested as shown in Figure 3. t mainly consists of 7 components. (1) Data Warehouse: the platform of the on-line analytical data mining; (2) Data Mining Agent: performing analytical mining in data cubes aided by OLAP engine; (3) OLAP Engine: providing fast access to summarized data along multiple dimensions; (4) Applications Programming nterface: aggregation of instructions, functions, regulations and rules for on-line data mining in the data warehouse platform; (5) Data Cube: aggregation of data warehouse information; (6) Meta Data: data for managing and controlling data warehouse creation and maintenance. 148

4 e Applications Programming nterface iil Data Warehouse L_ Figure 3. The ntegrated Data Mining and O M Architecture 3.2 The mechanism Data cube is a core of on-line analytical data mining. t provides aggregated information that can be used to analyze the contents of databases and data warehouses. t is constructed from a subset of attibutes in the databases and data warehouses. Data mining agent performs analytical mining in data cubes with the aid of OLAP engine. Data mining agent and the OLAP engine both accept user's on-line queries through the user interface and work with the data cube through the applications programming interface in the analysis. Furthermore, data mining agent may perform multiple data mining tasks, such as concept description, association, classification, prediction, clustering, time-series analysis, etc. Therefore, data mining agent is more sophisticated than the O W engine since it usually consists of multiple mining modules which may interact with each other for effective mining. Since some requirements in data mining agent, such as the construction of numerical dimensions, may not be readily available in the commercial O W products, particular mining modules should be built in model base. With many OLAP products available on the market, it is important to develop online analytical mining mechanisms directly on top of constructed data cubes and OLAP engines. Although, data mining agent analysis may often involve the analysis of a large number of dimensions the finer granularities and thus require more powerful data cube construction and accessing tools than O W analysis, there is no fundamental difference between the data cube required for OLAP engine and that for data mining agent. Since data mining agent is constructed either on customized data cubes which often work with relational database systems, or on bp of data cubes provided by the O W products, it is suggested to build on-line analytical mining systems on top of the existing OLAP and relational database systems, rather than from the group up. As data warehouse provides subject oriented summarized data (Figure l), data warehouse data is advantageous to improve the efficiency of data mining. When data mining is camed on data warehouse, the first two steps of selection and preprocessing are already completed roughly. The KDD process begins from data tansformation for the purpose of determining useable dimensions for special data mining problem and using particular algorithm to carry out data mining. n data transformation, particular analysis algorithms are used for determining the dimension which is useful to special data mining task or affects the special data mining task greatly. As data mining functions usually cost more than simple O W operations, efficient implementation and fast response are the keys in the realization of on-line analytical data mining. t is very important to develop the online analytical data mining, and test and share data mining module that achieve modularization design and standard 149

5 application programming interface. n order to achieve fast response for data mining queries, efficient and constraint-based data mining algorithms are needed to be applied. 4. Discussion There have been many studies on on-line analytical data mining recently [9][10]. A thoughtful design will help systematic development of owline analytical data mining mechanisms in data warehouse. But, there are many difficulties to develop a thoughtful on-line analytical data mining system in practice. Efficient support of multi-feature cubes and cubes with complex dimensions and measures are necessary. Many data mining tasks need discovery-driven exploration of multi-feature cubes, which are complex sub queries involving multiple dependent queries at multiple granularities. Moreover, traditional data cubes support only dimensions of categorical data and measures of numerical data. n practice, the dimensions of a data cube can be of. numerical, spatial and multimedia data. The measures of a cube can also be of spatial and multimedia aggregations or the collections of such object pointers. Support of such non-traditional data cubes will enhance the power of data mining. On-line analytical data mining must have the ability of mining anywhere. With a multidimensional database and an O W engine, it is easy to carve and portions of data sets at multiple levds of abstraction using OLAP operations, such as drilling, dicing/slicing, pivoting, filtering, etc. This greatly facilities the online analytical data mining process since such a process should be exploratory in nature, that is, mining should be performd at different portions of data at multiple levels of abstraction. By interaction with O W operations, one can perform drilling, dicinghlicing, and pivoting during data mining as well. Moreover, some data mining process may need to explore at least some of the data in great detail. An OLAP engine often provides facilities to drill through the data cube down to the primitiveflow level data store in the database. The interaction of multiple data mining modules with an O W engine will ensure that mining can be easily performed anywhere in a data warehouse. On-line analytical data mining must have the ability of interaction among multiple data mining functions. The strength of on-line analytical data mining should be not only at the selection of a set of datamining functions but also at the interaction among multiple data mining and OLAP functions. Fast response and high performance mining is necessary. t is highly desirable and productive to interact with the mining process and dynamically explore data spaces. However, fast response is critical for interactive mining. Sometimes one may even like to trade mining accuracy for fast response since interactive mining may progressively lead miners to focus the search space and find more and more important pattern. Once a user can identify a small search space, more sophisticated but slower mining algorithm can be called up for careful examination. An on-line analytical data mining system is a system which will communicate with users and knowledge visualization packages at the top and data cubeddatabases at the bottom. Thus, it should be highly modularized with careful design and systematic development. Moreover, an online analytical data mining system should be designed with extensibility in consideration since an on-line analytical data mining system will be expected to be integrated with many subsystems or be extended in many ways. For example, an O W data mining system may be integrated with a statistical data analysis package, or be extended for spatial dab mining, text mining, financial data analysis, multimedia data mining, Web mining, and so on. A modularized design may lead to easy extension towards new domains. To develop a successful on-line data mining system, visualization tools are indispensable. Since an O W data mining system will integrate OLAP and data mining and mine various kinds of knowledge from data warehouse, it i's important to develop a variety of knowledge and data visualization tools. Charts, curves, decision trees, rule graphs, cube views, boxplot graphs, etc. are effective tools to describe data mining results and help users monitor the process of data mining and interact with the mining process. 150

6 5. References [l] S. Anahory and D. Murray, Data Warehousing in the Real World: A Practical Guide for Building Decision Support Systems, Harlow, UK Addison Wesley Longman, [2] P. Gray and H. J. Watson, Decision Support in the Data Warehouse, Upper Saddle River, NJ, PrenticsHall, PTR [3] W. H. nmon, Building the Data Warehouse, New York: John Wiley & Sons, [4] W. H. nmon and R. H. Terdeman, Claudia mhoff, Exploration Warehousing: Turning Business nformation into Business Opportunity, Wiley, 2000 [5] U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, Advances in Knowledge Discovery and Data Mining, AAA/MT Press, 1996 [6] M.S. Chen, J. Han, and P.S. Yu. Data mining: An overview from a database perspective, EEE Transactions on Knowledge and Data Engineering, [7] E. F. Codd, E. S. Codd and C. T. Salley, Beyond Decision Support, Computerworld, Vo1.27, No.30, July [8] Qing Chen, Mining Exceptions And Quantitative Association Rules n Olap Data Cube, EEE Transactions on Knowledge and Data Engineering, J. W. Han, Towards On-Line Analytical Mining in Large Databases, ACM SGMOD Record, [lo] K. Parsaye, OLAP and Data Mining: Bridging the Gap, Database Programming and Design,