Making Data Warehouse Usable and Useful

Transcription

1 Making Data Warehouse Usable and Useful Rogerio DePaula CSCI 5817 Database Systems. If computers are to be helpful to us at all, it must not be in producing more information we already have enough to occupy us from dawn to dusk but to help us to attend to the information that is the most useful or interesting or, by whatever criteria you use, the most valuable information Herbert Simon Table of Content MAKING DATA WAREHOUSE USABLE AND USEFUL INTRODUCTION... 2 WHY DATA WAREHOUSE?... 3 INITIAL CONSIDERATIONS FOR THE DESIGN OF DATA WAREHOUSE SYSTEMS... 5 ENHANCING DATA QUALITY... 7 BEING BUSINESS EXPERT CREATING EFFECTIVE DATA WAREHOUSING SOLUTIONS... 8 MAKING DATA WAREHOUSE USABLE AND USEFUL...10 CONCLUSION...12 REFERENCES...13

2 Introduction It is fairly accurate to argue how significant a technology is at a certain time by looking at the number of references is available. On the other hand, for such a technology to be widely adopted, it has to show its usability and usefulness. After an initial excitement, users usually realize that there are more complex issues that may affect the leverage of the adoption and use of the technology than just technical solutions. Certain technologies fall into this category, for instance, in the middle eight's expert systems was considered the solution for all business problems (overstating the actual facts), however, after an initial excitement from business in general, this technology never took off for a broader audience. In reality, expert system lends itself to very domain oriented applications (for which it has been useful), however, due to this fact, the integration of those applications became very complex, preventing then organizations to take advantage of it. The integration of their current information technologies and other knowledge base systems showed to be very hard to be achieved. That is, this technology has not proven to be useful and usable for addressing the majority of business needs. Yet, Data warehouse has proven the worthwhile attention as a database technology for integrating all sort of information, distributed within organizations whose database systems usually range from legacy applications (non-relational database) to advanced relational database systems. Significantly, data warehouse promise to provide a technological framework for supporting decision-making processes by providing informational data. However, organizations that invested in such a technology usually do not derive the return on investments they expected [1]. Two complementary variables, which are indirectly related to the actual data warehouse technology but inherent in the nature of the problems that it attempts to address, have shown to add a level of complexity to the process of making such a technology useful and usable. This complexity in turn inhibits, on one hand, the access of information that users

3 really need, and, on the other hand, data managers to design the right application for their users' needs. These two variables are data quality and business processes. In this respect, this paper states that in addition to technical considerations (such as, data warehouse architecture, data-mart solutions, and centralized or distributed data warehouse), managers have to take into account those two factors, while designing data warehousing systems. For instance, data quality may be inappropriate, misunderstood or ignored and developers may not have a clear understanding of what kind of information users want to acquire from the system, when and where. That is, they risk designing the "right" technical solution that addresses the "wrong" problem, or even worst, they may provide wrong answers for a problem. In arguing so, this paper starts defining important technical considerations with respect to data warehouse architecture in order to situate the upcoming discussion. It presents then an analysis of both factors. Finally, it discusses some approaches for making data warehouse useful and usable, and presents how the database and information management marketing have tackled on those two problems. Why Data Warehouse? Recently, due to the rapid development of computer technologies, organizations realized the need for understanding their processes in-depth. In so doing, they created the need for collecting informational data for supporting planning, management and decision-making processes. However, when trying to derive useful information from their operational data sources, they experienced the following problems [2]: - The operational model that was designed for short, and predictable update transactions

4 - They run across historical, soft data and point-in-time data (usually not saved in OLTP databases) - They had to operationalize legacy systems and pre-relational systems - Finally, they encountered heterogeneous data, inhibiting rapid integration of data necessary for decision-making processes. Initially, managers attempted to employ traditional relational database management systems. Even though RDBMSs provides substantial data independence and a query language, supporting then both operational and decision-making processes, using OLTP as support, this approach was marginally successful, for the following reasons [2]: - Complexity of the query schema - Requirement for more interesting information, which involves increasingly complex data analysis - Lock content necessary for mixed decision-support and operational workloads degrades the performances of database systems. In this regard, data warehouse was developed to enables those organizations to integrate all sort of information spread across their sites. In summary, data warehouse contains summary, historical and detail data to support decisions making activities. Data is extracted from operational sources (RDBMSs or otherwise), transformed, cleansed, reconciled, aggregated, and summarized for being used by a data warehouse application. In data warehouse there is an important different between informational data and operational data. Basically, informational data provides information for decision-support activities, whereas operational data is organized around business operations. This difference clearly shows what data warehouse is essentially about. Gardner [3, p.54] presents an interesting definition of data warehouse supports this papers' argument:

5 "Data warehousing is a process, not a product, for assembling and managing data from various sources for the purpose of gaining a single detailed view of part or all of a business." Data warehouse system developers have then to understand business processes and users' needs, in addition to hardware, software and development tools. Essentially, information highly depends on business needs and objectives, organizational structures, and finally time and costs constraints. Therefore, in order for a data warehouse to effectively provide the right information for the right question at the right time, it has to be developed taking into account those facts. In this respect, this paper argues that it is fundamental to understand data quality and business processes in addition to technical details of the system, otherwise, this technology may fall into the category of useless and unusable great ideas. Initial Considerations for the Design of Data Warehouse Systems Businesses today are overloaded with all sort of data, but have little information available. In the information age, data is not any longer a scarce resource, whereas it became a problem because we have it but we do not know exactly what to do with it. In addition, such a data is stored in legacy systems, and its quality usually compromised. In fact, it has no value if it cannot be turned into information [3]. In this respect, it has been argued that data warehouse provides the solution for this problem, not just allowing users to find the answers to their questions but also to understand how and why specific answers were received [3]. However, it cannot guarantee such results if the quality of the data is compromised, and more importantly, if the design of a data warehouse system does not address the

6 actual question of those users. "Nothing is more likely to undermine the performance and business value than inappropriate, misunderstood, and ignored data quality" [4, p.73]. In order to facilitate the design process of designing data warehouse systems, different frameworks have been development. They basically address the following issues [3]: - Planning. Information discovery services, which attempts to better understand the actual business processes and what problems have to be solves - Design and Implementation. When the information about the business is identified and well understood, developers begin the first data-warehousing project. - Support and enhancement. It comprises a series of operations: day-to-day support of data warehouse running operations; assistance of expanding the use of the solution; expansion of the system to address new problems; and helping keep the system continually update and growing and supporting better business decisions. This guideline provides a necessary blueprint for developers to think about and eventually develop a data warehouse systems. However, it is not sufficient to address the complexity of real scenarios. Users usually do not know exactly what kind of information they need and even if they in fact need it. There is not a clear picture of the organization as a whole in order to enable them to create a system that effectively integrates all this information and to allow them to know the quality of the data available. Finally, there is not a standard use of technology that facilitates designers to come up with a solution that addresses the majority of needs in an organization. This paper then follows presenting two issues that should be also addressed during the designing of data warehouse systems.

7 Enhancing Data Quality As it was advised before, inappropriate, misunderstood, or ignored data quality has a negative effect on business decisions, performance and value of data warehouse systems. In other words, a company would be better off not using such a system, than using it wrongly. Since, there is recently a very high need for information in supporting decision-making processes, data warehouse systems neatly provides the necessary technological infrastructure. Subsequently, they have to provide means for developers as well as users to better understand their business processes, and how effectively they use the information being provided by those systems. To do so, they have first to evaluate the quality of the data that they are using. The problem is then how to measure the quality of data. Firstly, informational data is the most appropriate for decision making process, and therefore, more effective for data warehousing applications. Data can then be characterized via multiple attributions: accuracy, completeness, consistency, timeless, believability, value-added, interpretability and accessibility [4]. Those attributes can be then grouped into broader categories: intrinsic, contextual, representational, and accessibility. For instance, accuracy belongs to intrinsic; completeness and timeless to contextual; consistency to representational; and availability to accessibility. This scheme lends itself to help developer and users to evaluate the quality of a data set (clearly distinct kind of data). Therefore, they have to define the data sets that will be needed to support a data warehousing effort. However, the problem is to identify those data sets in the context of an organization. In this respect, those attributes guide this determination. For instance, if a data does not exist, its quality is deficient on the availability attribute. In addition, Ballou and Tayi [4] present different ways in which data quality can be improved. For instance, one can resolve the difference among data formats. Another might be to obtain data in a more timely basis. Another could be to

8 identify and enforce a common definition of key concepts in the organization (i.e., "sales," as far as data warehouse is concerned). Another might be to obtain external data. Each one influences the quality of a data set, which in turn influences data quality available for the business decisions. Being Business Expert Creating effective Data Warehousing Solutions To understand data quality is to understand the context in which information derived from such a data will be used. Therefore, users and developers alike (people involved in a data warehousing effort) should have a clear picture of the actual business contexts in which this information is necessary and will be used. In this respect, they have to think systematically about what they want to achieve in deploying a data warehousing system. To do so, they have to acquire needed expertise with respect to their business. In addition, each group has to understand the needs of the other. Data managers (developers) know what is going behind a data warehousing project SQL optimization, data modeling, legacy extract and data integrity, schema design, indexes, and load performance. On the other hand, users know business needs, and what kind of information is mostly relevant for certain decision making processes, although they might see a data warehouse only as schemas, and tools provided by data managers that allow them to query data, create reports, and analyze information. Therefore, both groups have to at least acknowledge the other group in order to realistically design a data warehousing system that addresses users actual needs, and on the other hand to realistically set the requirements for the system. In this respect, Glassey [1] outlines key design criteria that aim to support a more user-centric design: - Organize and structure the database that is more legible for users by using commonly used business terminology

9 - Cleanse the data on the warehouse rather than using complex transformation rules in client-side tools - Store and reuse the metadata in the warehouse - Look for optimized RDBMSs to function as backbone of the warehouse; performance, responsiveness, and scalability are critical - Apply different strategies to make information available for users, i.e., the use of data warehouse over the Web. On the other hand, developers need to have a clear picture of the kind of information that they have make available for the users. In order to provide tools that allows users to obtain the right information that they need for supporting their decision-making processes, developers have to understand business processes and structures. That is, they have to be able to see the enterprise as a whole in order to be able to discern with respect to the data to be used, and equally important, the type of data warehousing architecture they have to employ to better support this organization to date and hopefully in the future. One important aspect of decision-making processes is that today's decisions are likely to be different from tomorrow's decisions. Therefore, developers have also to be able to foresee, to some extent, the directions that the organization is heading toward. In addition, there are other questions that developers as well as users have to jointly look for answers, when designing a data warehousing system. There are numerous vendors and each of them offering a specific kind of solution that may or may not be the most appropriate for the project, therefore developers and users have to carefully weight the following categories [3]: Costs. How much can and should the company spend for the project? Time. How long will it take?

10 Users. What are the profiles of the end users, that eventually will take advantages of the system? Maintenance. Who will build the system? And who will eventually maintain it? Hardware, software, and tools. What do the company have now? What and how to get it? Services. How and what can this systems improve the company's work processes? In summary, both users and developers have to be not just experts in their work field, but also they have to be able to extend this expertise toward the other's area of expertise. In so doing, they will be able to obtain the highest benefits that data warehousing systems could offer, because they will be able to balance their needs with technological advantages and constraints. Making Data Warehouse Usable and Useful Different data warehouse solution providers have already recognized the issues presented in this article. They have basically integrated consulting support in order to help their customers to better understand their actual needs, and therefore to be able to obtain higher benefits from those technologies. In addition, they have understood the importance of providing easy-to-use systems as they realized the wide range of users that would eventually manipulate data from those systems in order to obtain the information that they need. In contrast to previous database systems on which data was usually manipulated by data managers, who in turn provided the reports to the actual information consumers, in data warehousing scenarios, the information consumers are more likely to directly manipulate data in order to analyze it, and therefore obtain the needed information.

11 In order to improve the use of data warehousing systems, IBM developed a DataGuide, which is an end-user-oriented tool that facilitates the process of gathering business previous database metadata information. The IBM developed, for instance, a visual warehouse tool that aims to facilitate what usually are very complex tasks, such as data mapping, extraction from heterogeneous data sources, process scheduling, and warehouse operation monitoring. In addition, they have formed partnerships with other corporations in order to expand the functionalities of their warehouse system, and services offered. For instance, they joint with Evolutionary Technologies International of Austin, whose Extract tool generates 3GL programs to transform, consolidate, and extract data from virtually any data source. They also formed partnership with Vality Technology of Boston, whose Integrity data reengineering environment performs data cleansing [2] Even though Oracle Corporation is coming up short as its delays shipment of a new data warehousing tool, they are doing so because they want to improve customers' access to their data warehousing solution. They are developing the Warehouse Builder, a complex package for extracting, transforming and loading data from legacy systems, packaged applications and other sources into a data warehouse for analysis. Their problem is to improve the system that handles metadata in distributed environments. In so doing, they want to facilitate the extraction of data from any data source [5]. On the other hand, NCR, who is a recognized world leader in data warehousing solutions, announced a partnership with ETI Extract, who provides a complete solution to data integration management by automating the process of consolidating data between incompatible systems, thereby helping customers complete projects faster and more cost effectively, they argue. This is basically a joint effort to offer not just the development and implementation of technical

12 solutions, but also marketing and consulting solutions. They claim then that they are now able to provide robust solutions and superior customer support [6]. Finally, Brio Technology tackles on the problem user-friendly interfaces. That is, they aim to reduce IT department's overload by providing tools that facilitate end-users' interaction with data warehousing systems, by firstly reducing the complexity of end-user support, in addition supporting a wide range of platforms such as Unix, Macintosh, and Windows. They tightly integrated the process of gathering information with data warehousing tools. In so doing, they lowered the IT's costs by enabling end-users to build reports, determine access, schedule, print and ship reports while automating installation and maintenance [1]. Conclusion Information consumes human attention, so a wealth of information creates a poverty of human attention. Design approaches suitable for a world in which the scarce factor is information may be exactly the wrong ones for a world in which the scarce factor is attention. Herbert Simon This last quote shows that we are living in a world that data became a problem simply because we do not know exactly what to do with it. In addition, old technological paradigms are likely to not work in this current scenario. Therefore, as software developers, we have to able to shift our previously accepted technocentrism to a more user-centric approach in developing technical solutions. That is, we have to perceive that old great solutions may not be the right ones for today's reality. In this respect, Data warehousing efforts may not succeed for various reasons, but nothing is more certain to yield failure than the lack of concern for the quality of the data and for understanding users' needs.

13 Finally, data warehouse promises to provide the right solution for current business' needs of information of high quality at the right format at the right time. To do so, developers and users as well have to be caution in order not to overestimate the values of the technology, without however taking into account the complexity of the problem, that is, it has to be useful and usable. In cautiously taking this fact into account, this technology has the potential to payoff all investments and efforts that it requires, without risking to be another great idea that did not take off. References [1] K. Glassey, Seducing the End User, Communications of the ACM, vol. 41, pp , [2] C. Bontempo and G. Zagelow, The IBM Data Warehouse Architecture, Communications of the ACM, vol. 41, pp , [3] S. R. Gardner, Building the Data Warehouse, Communications of the ACM, vol. 41, pp , [4] D. P. Ballou and G. K. Tayi, Enhancing Data Quality in Data Warehouse Environments, Communications of the ACM, vol. 42, pp , [5] M. Hammond, Oracle Slow to Deliver Data Warehouse Tool,, [6] J. E. Brawner, ETI and NCR Annouce Two-Pronged Partnership Bringing Enhanced Data Warehouse Solutions and Stronger Support to Customers,, vol. 1999: ZDNew, ml