COMMUNICATION COMPUTER ENGINEERING Discovery Engineering, Volume 2, Number 8, November Engineering

Transcription

1 COMMUNICATION COMPUTER ENGINEERING Discovery Engineering, Volume 2, Number 8, November 2013 ISSN EISSN discovery Engineering Data warehousing & Olap: a short note Pooja Dang 1, Parul Singh 2, Swati Sharma 3 Dronacharya College of Engineering, Khentawas, Farrrukh Nagar, Gurgaon, India Correspondence: Dronacharya College of Engineering, Khentawas, Farrrukh Nagar, Gurgaon, India, Mail: pooja.dang07@gmail.com Publication History Received: 12 September 2013 Accepted: 26 October 2013 Published: 1 November 2013 Citation Pooja Dang, Parul Singh, Swati Sharma. Vikrant Dewan, Nipun Yadav, Neha Yadav. Data warehousing & Olap: a short note. Discovery Engineering, 2013, 2(8), INTRODUCTION The paper introduces the data warehouse and the online analysis process. Data warehouse provides an effective way for analysis and statistic to the mass data, and helps to do the decision-making. Data warehousing and on-line analytical processing (OLAP) are essential elements of decision support, which has increasingly become a focus of the database industry. As companies over the last couple decades have done more logging and data capture with the advent of computers with database capabilities. Many have found that these data are quite useful to augment or focus market groups if only the information were available for statistical analyses. The increasing popularity of intranets and the Internet, itself, has given rise to repositories of data and engines that can search for correlates for internal uses and "sellable" information. The warehouse database is a special component in the federation, made up of historical data, external data and materialized views of the operational data. This approach enables users to access historical and current data when required, and provides a method of maintaining the warehouse as an integrated view of the underlying operational data sources. 2. THE CASE FOR DATA WAREHOUSING The following is a list of the basic reasons why organizations implement data warehousing. This list was put together because too much of the data warehousing literature confuses "next order" benefits with these basic reasons. For example, spend a little time reading data warehouse trade material and you will read about using a data warehouse to "convert data into business intelligence", "make management decision making based on facts not intuition", "get closer to the customers", and the seemingly ubiquitously used phrase "gain competitive advantage". In probably 99% Page74

2 of the data warehousing implementations, data warehousing is only one step out of many in the long road toward the ultimate goal of accomplishing these highfalutin objectives. The basic reasons organizations implement data warehouses are: To perform server/disk bound tasks associated with querying and reporting on servers/disks not used by transaction processing systems. To use data models and/or server technologies that speed up querying and reporting and that are not appropriate for transaction processing. To provide an environment where a relatively small amount of knowledge of the technical aspects of database technology is required to write and maintain queries and reports and/or to provide a means to speed up the writing and maintaining of queries and reports by technical personnel. To provide a repository of "cleaned up" transaction processing systems data that can be reported against and that does not necessarily require fixing the transaction processing systems. To make it easier, on a regular basis, to query and report data from multiple transaction processing systems and/or from external data sources and/or from data that must be stored for query/report purposes only. To provide a repository of transaction processing system data that contains data from a longer span of time than can efficiently be held in a transaction processing system and/or to be able to generate reports "as was" as of a previous point in time. To prevent persons who only need to query and report transaction processing system data from having any access whatsoever to transaction processing system databases and logic used to maintain those databases. 3. DATA WAREHOUSE DESIGN After the tools and team personnel selections are made, the data warehouse design can begin. The following are the typical steps involved in the data warehousing project cycle. Requirement Gathering Physical Environment Setup Data Modeling ETL OLAP Cube Design Front End Development Report Development Performance Tuning Query Optimization Quality Assurance Rolling out to Production Production Maintenance Incremental Enhancements Each point listed above represents a typical data warehouse design phase, and has several sections: Task Description: This section describes what typically needs to be accomplished during this particular data warehouse design phase. Time Requirement: A rough estimate of the amount of time this particular data warehouse task takes. Deliverables: Typically at the end of each data warehouse task, one or more documents are produced that fully describe the steps and results of that particular task. This is especially important for consultants to communicate their results to the clients. Possible Pitfalls: Things to watch out for. Some of them obvious, some of them not so obvious. All of them are real. 4. DATA WAREHOUSE ARCHITECTURE Different data warehousing systems have different structures. Some may have an ODS (operational data store), while some may have multiple data marts. Some may have a small number of data sources, while some may have dozens of data sources. In view of this, it is far more reasonable to present the different layers of data warehouse architecture rather than discussing the specifics of any one system. There are two border areas in data warehouse architecture - the single-layer architecture and the N-layer architecture. Single-layer architecture Page75

3 In the simplest form, sometimes referred to as messaging in a box, all the components of the messaging server run on a single system and no proxy is involved. A simple architecture is the single-layer architecture. There is no physical data warehouse or datamart between the operation data and the analytic tools. The middleware in this type of system should be considered a virtual data warehouse, which consists of a software layer and not a data based layer. The single-layer model is light weight as it minimises redundancies and thereby the amount of data stored. It has, however, no separation between analytical and operational processing. Three-layer architecture The three-layer architecture consists of the source layer (containing multiple source systems), the reconciled layer and the data warehouse layer (containing both data warehouses and data marts). The reconciled layer sits between the source data and data warehouse. It is populated with data from the source systems through an ETL process and the data stored in it is published further through another ETL process. In the reconciled layer the data has been cleaned up once and integrated to a common standardised form from multiple different source systems. The ETL process that feeds the data warehouse then only gets already integrated data that has less need for transformation. 5. MULTIDIMENSIONAL MODEL Over the last few years, multidimensional databases have generated much research and market interest because they are fundamental for many decision-making support applications, such as data warehouse systems. The reason why the multidimensional model is used as a paradigm of data warehouse data representation is fundamentally connected to its ease of use and intuitiveness even for IT newbies. The multidimensional model's success is also linked to the widespread use of productivity tools, such as spreadsheets, that adopt the multidimensional model as a visualization paradigm. The multidimensional model begins with the observation that the factors affecting decisionmaking processes are enterprise-specific facts, such as sales, shipments, hospital admissions, surgeries, and so on. Instances of a fact correspond to events that occurred. For example, every single sale or shipment carried out is an event. Each fact is described by the values of a set of relevant measures that provide a quantitative description of events. For example, sales receipts, amounts shipped, hospital admission costs, and surgery time are measures. Meta-data The term meta-data can be applied to the data used to define other data. In the scope of data warehousing, metadata plays an essential role because it specifies source, values, usage, and features of data warehouse data and defines how data can be changed and processed at every architecture layer. According to Kelly's approach, you can classify meta-data into two partially overlapping categories. This classification is based on the ways system administrators and end users exploit meta-data. System administrators are interested in internal meta-data because it defines data sources, transformation processes, population policies, logical and physical schemata, constraints, and user profiles. External meta-data is relevant to end users. For example, it is about definitions, quality standards, units of measure, relevant aggregations. Meta-data is stored in a meta-data repository which all the other architecture components can access. According to Kelly, a tool for meta-data management should allow administrators to perform system administration operations, and in particular manage security; allow end users to navigate and query meta-data; use a GUI; allow end users to extend meta-data; allow meta-data to be imported/exported into/from other standard tools and formats. 6. OLAP (ONLINE ANALYTICAL PROCESSING) OLAP stands for On-Line Analytical Processing. The first attempt to provide a definition to OLAP was by Dr. Codd, who proposed 12 rules for OLAP. Later, it was discovered that this particular white paper was sponsored by one of the OLAP tool vendors, thus causing it to lose objectivity. OLAP (online analytical processing) is computer processing that enables a user to easily and selectively extract and view data from different points of view. For example, a user can request that data be analyzed to display a spreadsheet showing all of a company's beach ball products sold in Florida in the month of July, compare revenue figures with those for the same products in September, and then see a comparison of other product sales in Florida in the same time period. To facilitate this kind of analysis, OLAP data is stored in a multidimensional database. Whereas a relational database can be thought of as two-dimensional, a multidimensional database considers each data attribute (such as product, geographic sales region, and time period) as a separate "dimension." OLAP software can locate the intersection of dimensions (all products sold in the Eastern region above a certain price during a certain time period) and display them. Attributes such as time periods can be Page76

4 broken down into sub attributes. OLAP can be used for data mining or the discovery of previously undiscerned relationships between data items. An OLAP database does not need to be as large as a data warehouse, since not all transactional data is needed for trend analysis. Using Open Database Connectivity (ODBC), data can be imported from existing relational databases to create a multidimensional database for OLAP. Two leading OLAP products are Oracle's Express Server. Hyperion Solution's Essbase 7. OLAP CUBE An OLAP cube is an array of data understood in terms of its 0 or more dimensions. OLAP is an acronym for online analytical processing. OLAP is a computer-based technique for analyzing business data in the search for business intelligence. OLAP cubes are data structures that the Data Warehouse uses to contain the data that you import. The cubes divide the data into subsets that are defined by dimensions.a dimension is the descriptive attribute of a measure. For example, a dimension might describe the number of transactions on your site. The number of transactions is a measure. The transaction and the products purchased are dimensions. The following illustration shows the relationship between cubes, dimensions and measures. 10. WHAT IS SPATIAL OLAP? "Spatial OLAP can be defined as a visual platform built especially to support rapid and easy spatiotemporal analysis and exploration of data following a multidimensional approach comprised of aggregation levels available in cartographic displays as well as in tabular and diagram displays." A SOLAP system supports three types of spatial dimensions: the non-geometric spatial dimensions, the geometric spatial dimensions and the mixed spatial dimensions. The non-geometric spatial dimensions use nominal spatial reference, i.e. only the name of places or objects such as Canada, Province of Quebec, Quebec City and St.John Street. This type of spatial dimension is the only one supported by conventional (non-spatial) OLAP tools. When used with SOLAP tools, the non-geometric spatial dimension is treated like the other descriptive dimensions and the geometric data allowing for the representation of the dimension members on maps is not used. In this case, the spatio-temporal analysis can be incomplete and certain spatial relations or correlations between the phenomena under study can be missed by the analyst. The two other types of spatial dimensions aim at minimizing this potential problem. To do so, the geometric spatial dimensions comprise, for all dimension members, at all levels of details, geometric shapes (ex. polygons to represent country boundaries) that are spatially referenced to allow their dimension members (ex. Canada) to be visualized and queried cartographically. The mixed spatial dimensions comprise geometric shapes for a subset of the members or the levels of details. 11. PRESENT SCENARIO The globalization of business, the liberalization of the economy and the rapid strides in technology make strategic and operational plan (micro level) virtually outdated, by the time they are generated and ready for implementation. ". In the present economic scenario, for large number of organizations, making unbiased and faster decisions can make the difference between surviving and thriving, more so in an increasingly competitive market. Buried in the huge databases assembled by large organizations is the information useful for generating new facts and relationships that can provide significant competitive advantage. Organizations have lately realized that just processing transactions and/or information s faster and more efficiently, no longer provides them with a competitive advantage vis-à-vis their competitors for achieving business excellence. The increasing competitive pressures and the desire to leverage viz. "Data Warehousing and Data Mining". The increasing competitive pressures and the desire to leverage viz. "Data Warehousing and Data Mining". The increasing competitive pressures and the desire to leverage viz. "Data Warehousing and Data Mining". The data mining technology is like extracting gold, parallel to gold extraction technology. Data mining is based on filtration and assaying of a mountain of data "ore", in order to get data "nuggets" and is designed to help corporate organization(s) to discover hidden patterns and to delve deeper to establish hidden connections in organization's data patterns that can help planner & decision makers to understand the behavior of key users, detect likely trends-growth pattern, predict change(s) in the financial sector etc. Thus, managing the business effectively and gaining competitive edge. Page77

5 RELATED RESOURCES 1. Inmon W. The Operational Data Store. Designing the Operational Data Store. Information Management Magazine, July Marchand P, Brisebois A, Bédard Y, Edwards G. Implementation and evaluation of a hypercube-based method for spatio-temporal exploration and analysis. Journal of the International Society of Photogrammetry and Remote Sensing (ISPRS) 2004, 59(1-2), Rivest S, Bédard Y, Proulx MJ, Nadeau M, Hubert F, Pastor J. SOLAP: Merging Business Intelligence with Geospatial Technology for Interactive Spatio-Temporal Exploration and Analysis of Data. Journal of the International Society for Photogrammetry and Remote Sensing (ISPRS) (1), Page78