Dta Mining and Data Warehousing

Transcription

1 CSCI6405 Fall 2003 Dta Mining and Data Warehousing Instructor: Qigang Gao, Office: CS219, Tel: , Teaching Assistant: Christopher Jordan, Office Hours: TR, 1:30-3:00 PM 26 September

2 Lectures Outline Pat I: Overview on DM and DW 1. Introduction (ch1) Ass1 Due: Sep 23 Tue 2. Data preprocessing (ch3) Part II: DW and OLAP 3. Data warehousing and OLAP (Ch2) Ass2: Sep 23 Oct 7 Part III: Data Mining Methods/Algorithms 4. Data mining primitives (ch4) 5. Classification data mining (ch7) Ass3: Oct 7 Oct Association data mining (ch6) Ass4: Oct 21 Nov 5 7. Characterization data mining (ch5) 8. Clustering data mining (ch8) Part IV: Mining Complex Types of Data 9. Mining the Web (Ch9) 10. Mining spatial data (Ch9) Project Presentations Project Due: Dec 8 26 September

3 2. DATA WAREHOUSING AND OLAP (Ch2) Objectives of DW/OLAP What is a DW? Multidimensional Data Model DW Schemas Aggregations OLAP Operations DW Architecture From data warehousing to data mining 26 September

4 Multidimensional Data Model (MDM) OLAP applications are dominated by ad hoc, complex queries. In SQL terms, these are queries that involved group-by and aggregation operations. The natural way to think about typical OLAP queries, however, is in terms of a multidimensional data model 26 September

5 Multidimensional Data Model (cont) - What is a MDM? MDM is a data model using logical dimensions to define a space of business events. This logical space is also called a hypercube (data cube). Each dimension of the cube represents an aspect of the possible business events which is divided into discrete values representing attribute domain of the dimension. Both DW schemas and OLAP operations are based on MDM. 26 September

6 MDM (cont): From physical space to logical space 26 September

7 MDM (cont): Manipulating a subspace 26 September

8 How to make business events (views) to be generated and measured easily? 26 September

9 26 September

10 Data cube: sales (time, product, location) TV PC VCR sum Product Time 1Qtr 2Qtr 3Qtr 4Qtr sum U.S.A Canada Mexico Total annual sales of TV in U.S.A. Location sum 26 September

11 E.g. AllElectronics data cube (Book: pp45-47) 1. For the cube Sales (time, location, item), in the text Fig Define an OLAP query (a business event/fact) "Sales in $ for different types of items sold per quarter in the city Vancouver OLAP query: time(quarter), location="vancouver", item(type) Report: Table The data cube visualization of Sales (time, location, item, supplier), Fig 2.2. The business events which can be formed from the data cube are the possible combinations of the logical dimensions, i.e., various cuboids (data sets). How many cuboids a N dimension MDM has: the cuboid lattice: Fig September

12 Cube: A Lattice of Cuboids E.g., sale (time, item, location, supplier) all 0-D(apex) cuboid time item location supplier 1-D cuboids time,item time,location item,location location,supplier time,supplier item,supplier 2-D cuboids time,item,location time,location,supplier 3-D cuboids time,item,supplier item,location,supplier 4-D(base) cuboid time, item, location, supplier 26 September

13 Cube: A Lattice of Cuboids (cont) In data warehousing literature, a n-d base cube is called a base cuboid. The top most 0-D cuboid, which holds the highest-level of summarization, is called the apex cuboid. The lattice of cuboids forms a data cube. 26 September

14 26 September

15 26 September

16 How to represent a hypercube on a computer screen? E.g., How to map multiple logical dimensions onto a single computer screen? The goal: we want to use 2-D screen to see the space. The solution: to combine multiple logical dimensions within the same display dimensions (Row, Column, Page). - physical dimension metaphor: Virtual Camera. - logical dimensions metaphor: Multidimensional Domain Structures (MDS). Data display for a 6D cube. The following figures shows each dimension of of the vertical bar connected to either a row, column, or page. The following figures show two different ways that the same model dimensions can be mapped onto row, column, and page axes. * The ability to easily view the same data by reconfiguring how dimensions are displayed is one of the great benefits of multidimensional systems. The reason is due to the separation of data structure, as represented in the MDS, from data display, as represented in the multidimensional grid. 26 September

17 26 September

18 26 September

19 Make optimal use of Analytical Screen The more screen space is consumed displaying dimension members, the less space is left for displaying data. The less space left for displaying data, the more scrolling you need to do between screens to see the same data. The more scrolling you need to perform, the harder it is to understand what you are looking for. * To maximize the degree to which everything on the screen is relevant, try keeping dimensions along pages unless you know you need to see more than one member at a time. * Ask yourself "What do I want to look at?", or "What am I trying to compare?" before deciding how to display information on the screen. 26 September

20 26 September

21 Summary of logical dimensions As distinguished from physical dimensions, which are based on angles and limited to three, logical dimensions have no such limits. Two types of dimensions of a data cube * Identifier Dimensions: Dimensions are logical factors or identifying attributes of measurable events or things that we track. * Variable dimension: Dimension that identifies what we track in a situation. Multidimensional software enables multiple dimensions of information to be combined onto each row, column, and page axis of a display device, thus making it possible to visualize and understand a multidimensional data set in terms of information presented on flat screen. The ability of multidimensional software to model multidimensional information and to handle the user representation of the information makes it better suited for working with complex datasets than either SQL databases or traditional spreadsheets. 26 September

22 Conceptual Modeling of DW: from MDS to DW Schema DW schema is the conceptual model (i.e. description, or meta-data) of a DW which converted from MDS and corresponds to a hypercube. A data cube is defined by identifier dimensions and variable dimension Dimension tables Fact table Fact table: contains variables which need to be measured according to defined event on the subject, such as Sales, Cost,..., and keys to each of the related dimension tables Dimension tables: contain data for forming various business events, such as item (item_name, brand, type), or time(day, week, month, quarter, year), 26 September

23 Conceptual Modeling of Data Warehouses Modeling data warehouses: dimensions & measures Star schema: A fact table in the middle connected to a set of dimension tables Snowflake schema: A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation 26 September

24 time time_key day day_of_the_week month quarter year branch branch_key branch_name branch_type Example of Star Schema Measures Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales item item_key item_name brand type supplier_type location location_key street city state_or_province country 26 September

25 Example of Star Schema (cont) Star schema is the most common used schema for OLAP applications, and used as Data mart for department-level DW. The star schema is simple but some redundancy may occur. E.g., Location {location_key, street, city, province, country} (.., Vancouver, British Columbia, Canada) (.., Victoria, British Columbia, Canada) 26 September

26 Example of Snowflake Schema time time_key day day_of_the_week month quarter year Sales Fact Table time_key item_key item item_key item_name brand type supplier_key supplier supplier_key supplier_type branch branch_key branch_name branch_type Measures branch_key location_key units_sold dollars_sold avg_sales location location_key street city_key city city_key city state_or_province country 26 September

27 Example of Snowflake Schema (cont) A variation of star schema, in which the dimension tables are normalized. Main purpose: saving space and for easier maintenance - Normalizing large dimension tables for saving storage space - Keeping small dimension tables as it is for reducing the cost and performance degradation of join operation on multiple tables. 26 September

28 Example of Fact Constellation time time_key day day_of_the_week month quarter year Sales Fact Table time_key item_key branch_key item item_key item_name brand type supplier_type Shipping Fact Table time_key item_key shipper_key from_location branch branch_key branch_name branch_type Measures location_key units_sold dollars_sold avg_sales location location_key street city province_or_state country to_location dollars_cost units_shipped shipper shipper_key shipper_name location_key 26 September 2003 shipper_type 28

29 Example of Fact Constellation (cont) Fact constellation schema is for sophisticate DWs: For the DWs which need to define multiple subjects, such as for large corporations which need information for quickly updating the picture of entire organization. 26 September

30 Classification of DWs Enterprise warehouse collects all of the information about subjects spanning the entire organization Data Mart a subset of corporate-wide data that is of value to a specific groups of users. Its scope is confined to specific, selected groups, such as marketing data mart Independent vs. dependent (directly from warehouse) data mart Virtual warehouse A set of views over operational databases 26 September

31 Concept Hierarchy -A concept hierarchy of dimension is a sequence of ordered concepts of the dimension. Time: day-week-month-quarter-year. Location: city-province-regions-country. (-> many to one, <- one to many) - Why we need hierarchies? * In business, as in most types of activity, hierarchies are a necessity of life: Managers and analysts spend most of their time thinking about groups of things, and many of such groupings are along various hierarchies. * Hierarchies are the backbone of aggregating 26 September

32 Hierarchies: the backbone of aggregating - The ability of multidimensional software to reference things according to their position along a hierarchy is incredibly useful for managing real-world applications. Hierarchies are the foundation for aggregating data and for navigating between levels of detail within a hypercube. - A basic difference between OLAP and OLTP query data styles OLTP: transaction queries which return, most often, the same data that was input as data originally supplied by the application, such as the customers. OLAP: derived data by aggregating. Dimensional hierarchies are part of the structure. 26 September

33 Hierarchy structure: - Hierarchy tree: Root: the top node (has not parent). Members: individual elements or nodes. Leafs: the termination nodes (have no children). E.g. Student Grades / / \ \ 1st 2nd 3 rd 4 th Fail / \ \ cs1000 cs2000 cs3000 cs Navigations: Many-to-one connection: roll up. One-to-many connection: drill down. 26 September

34 E.g. Dimension location hierarchy. all all region Europe... North_America country Germany... Spain Canada... Mexico city Frankfurt... Vancouver... Toronto office L. Chan... M. Wind 26 September

35 E.g., More illustration. Sales volume as a function of product, month, and region Region Dimensions: Product, Location, Time Hierarchical summarization paths Industry Region Year Category Country Quarter Product Product City Month Week Branch Day Month 26 September

36 How data is materialized in a data warehouse? 26 September