Adnan YAZICI Computer Engineering Department

Transcription

1 Data Warehouse Adnan YAZICI Computer Engineering Department Middle East Technical University, A.Yazici, 2010

2 Definition A data warehouse is a subject-oriented integrated time-variant nonvolatile collection of data in support of management s decision-making process.

3 Definition Other definitions: An integrated and time varying collection of data derived from operational data and primarily il used in strategic t decision i making by means of online analytical processing (OLAP) techniques. A process, not a product, for assembling and managing data from various sources for the purpose of gaining a single, detailed view of part or all of a business. A single, complete and consistent store of data obtained from a variety of different sources made available to end users in a what they can understand and use in a business context.

4 Motivation In modern advanced information systems the data layer is enormous. Dimension number, along with data size, is one of the most critical factors infuencing the effectiveness of such applications. Data sources are often highly heterogeneous. Consequently, the use of a Data Warehouse Systems (DWS) for collecting, integrating, and making business-critical information available in a transparent for-user way has become mandatory for DSSs. Recently, systems efficiently performing OnLine Analytical Processing (OLAP) and Data Mining (DM) against a DWS have gained a leading role. A.Yazici, 2010

5 Decision Support Systems Decision-support systems are used to make business decisions, often based on data collected by on-line transaction-processing systems. Examples of business decisions: What items to stock? What insurance premium to change? To whom to send advertisements? Examples of data used for making decisions Retail sales transaction details Call detail repositories Customer profiles (income, age, gender, etc.) A.Yazici, 2010

6 Data Warehousing Motivation: Data sources often store only current data, not historical data Corporate decision making requires a unified view of all organizational data, including historical data Definition: i i A data warehouse is a repository (archive) of information gathered from multiple sources, stored under a unified schema, at a single site, Greatly simplifies querying, permits study of historical trends Used by knowledge workers to view the big picture in data, not the specific details. Shifts decision support query load away from transaction processing systems. A.Yazici, 2010

7 Data Warehouses (cont.) Subject oriented, integrated, time varying, non- volatile collection of data that is used primarily in organizational decision making Maintained separately from operational databases of organizations Useful for information & analytical processing, data mining Numerical data is summarized and aggregated in a multidimensional fashion A.Yazici, 2010

8 A.Yazici, 2010 Data Warehousing (cont.)

9 Data Warehouses (cont.) Source: S. Chaudhuri, U. Dayal, An Overview of Data Warehousing and OLAP Technology, ACM SIGMOD Record, vol 26, p66, 1997 A.Yazici,2010

10 Architecture With a Staging Area

11 Architecture With a Staging Area and Data Marts

12 OLTP vs. Data Warehousing Environments

13 OLAP - OnLine Analytical Processing Interactive analysis of data, allowing data to be summarized and viewed in different ways in an online fashion (with negligible ibl delay) Data that can be modeled as dimension attributes t and measure attributes t are called multidimensional data. A.Yazici, 2010 Measure attributes t measure some value can be aggregated upon e.g. the attribute number of the Sales relation Dimension attributes define the dimensions on which measure attributes (or aggregates thereof) are viewed e.g., the attributes item_ name, color, and size of the Sales relation

14 OLAP (cont.) OLAP data model is multidimensional array of numeric measures, i.e., data cube. A.Yazici, SDM

15 Dimensional Fact Model Database Designs Star Schema A fact table in the middle connected to a set of dimension tables Snowflake Schema A refinement of star schema where hierarchy h is normalized into a set of smaller dimension tables, forming a shape similar to snowflake

16 OLAP (cont.) Star schema is the most common schema type used in relational databases for OLAP Data Warehouse and OLAP Technology 16

17 Snowflake Schema In the snowflake schema some dimensioni tables are normalized, thereby further splitting the data into additional tables. Example: The single dimension table for location in the star schema can be normalized into two new tables: location and city. The city_key in the new location table links to the city dimension. Further normalization can be performed on province_or_state and country in the snowflake schema, when desirable. 17

18 Snowflake Schema Data Warehouse and OLAP Technology 18

19 OLAP (cont.) OLAP operations rollup (increase the level of aggregation) drill-down (decrease the level of aggregation) slice_and_dice dice (selection and projection) pivot(reorient multidimensional view of data) OLAP Servers ROLAP (relational) MOLAP (multidimensional) HOLAP (hybrid) DOLAP (directory) JOLAP (J2EE) A.Yazici, 2010

20 Drill-down Sales amount measure with product dimension Drill-down is the process of viewing data at a level of increased detail.

21 Drill Up Quantity and Amount dimensions with time dimension Drill up is exploring a higher level of summarization.

22 Slice and Dice (projection and selection) Slice and dice are the operations for browsing the data through the visualized cube perspectives. Sli i t th h th b th t Slicing cuts through the cube so that users can focus on some specific perspectives.

23 Dice : Before dicing, analyze the sales amount report of a specific month by terriority and product category. After dicing, get the quarterly view of sales amount by terriority. Dicing is changing the dimension from product to time.

24 Slice : Slicing is cutting of the cube only for Accesories and focus on the on the Accessories only, rather than the other categories.

25 Cross Tabulation of sales by item-name and color The table above is an example of a cross-tabulation (cross-tab), also referred to as a pivot-table. Values for one of the dimension attributes form the row headers Values for another dimension attribute form the column headers Other dimension attributes are listed on top Values in individual cells are (aggregates of) the values of the dimensi on attributes that specify the cell. A.Yazici, 2010

26 Relational Representation of Cross-tabs * Cross-tabs can be represented as relations * The value all is used to represent aggregates g * The SQL:1999 standard actually uses null values in place of all despite confusion with regular null values A.Yazici, 2010

27 Data Cube * A data cube is a multidimensional generalization of a cross-tab * Can have n dimensions; we show 3 below * Cross-tabs can be used as views on a data cube A.Yazici, 2010

28 Data Cube Multidimensional data model Relational data model Data cube Table Dimensions Fields Measures/cells Records Data is modeled and viewed in multiple dimensions Hierarchies and aggregations are handled Easy for data mining A.Yazici, 2010

29 Data Cube (cont.) Time By production&location By production Sum Base data Production By location&time By time&production By time Location A.Yazici, 2010

30 Association Rules Rules have an associated support, as well as an associated confidence. Body->Head [support, confidence] Support is a measure of what fraction of the population satisfies both the antecedent and the consequent of the rule. E.g. suppose only 0.01 percent of all purchases include milk and screwdrivers. The support for the rule is milk screwdrivers is [0.01, conf]. Confidence is a measure of how often the consequent is true when the antecedent is true. E.g. the rule bread milk has a confidence of 80 percent if 80 percent of the purchases that include bread also include milk. bread milk [sup, 08] 0.8] A.Yazici, 2010

31 Association Rules (cont.) A fuzzy association rule generated from fuzzy spatial data cube is: mild(%83), wet(%94) small(%96) [0.78,0.96] Given that an area is mild (0.83) and has wet (0.94) precipitation, it can be concluded that it covers small piece of land (with 0.96) with the certainty 0.96, while the significance of that area being both mild and wet is A.Yazici, 2010

32 Conclusion Data warehousing is a unique and rapidly accepted business application class. Data warehouses greatly increases capabilities of information users in Data Mining, Analysis and Reporting. It is constantly affected by the software and hardware developments.

33 References An overview of Data Warehousing and OLAP Technology: ACM SIGMOD Record Vol 26, Issue 2 The Data Warehouse and Data Mining: Communications of the ACM-1996-Vol. 39, No. 11 The IBM Data Warehouse Architecture: Communications of the ACM-1998-Vol. 41, No. 9 Building The Data Warehouse: Communications of the ACM-1998-Vol. 41, No. 9 How Good is that Data in the Warehouse?: The DATA BASE for Advances in Information Systems Vol. 28, No. 3 Data Warehousing and Data Mining:S. Sudarshan, Krithi Ramamritham-IIT Bombay Conceptual Design of Data Warehouses from E/R Schemes Matteo Golfarelli, DEIS - Univ. of Bologna DEIS Dario Maio CSITE - Univ. of Bologna DEIS Stefano Rizzi, i Univ. of Bologna Using Design Guidelines to Improve Data Warehouse Logical Design Verónika Peralta, Raúl Ruggia, Instituto de Computación, Universidad de la República. Uruguay. Data Warehouse Schema Design Jens Lechtenbörger, Dept. of Information Systems, University of Münster