Data mining: Hmm, what is it?

Transcription

1 Data mining: Hmm, what is it? Data warehousing Examples Discussions The extraction of implicit, previously unknown and potentially useful information from large bodies of data often accumulated for other purposes. Mining: large volumes of low grade data sifted through to find something interesting September 9, 2003 UMASSD, CIS, Iren Valova 2 Data mining vs. database query Now that we are in the course, perhaps she will really tell us what is data mining all about?!?! Database query languages used when: finding and reporting information within the database (found in terms of factual entry); Provides data with no analysis. Data mining used when: looking for patterns and associations in the database/data warehouse, that are not explicitly present/coded in the database. Analysis needed to unveil hidden patterns and relationships. September 9, 2003 UMASSD, CIS, Iren Valova 3 As if it is not clear already DB can answer: get a list of all dept.store customers who used credit card to buy gas grill get a list of all patients who have had at least one heart attack and whose cholesterol < 20 get a list of all credit card holders who used their credit card to purchase more than $300 in groceries during January DM can answer: develop a general profile for credit card customers who take advantage of promotions offered with their credit card billing determine which patient is likely to come back to work after heart attack differentiate individuals who are poor credit risks from those who are likely to make their loan payment on time. September 9, 2003 UMASSD, CIS, Iren Valova 4 As usual, a picture is worth mlns. Patient ID S. Throat Fever Sw. Glands Congest. Headache Diagnosis R C C C C C C U I I I I I O 1 YES YES YES YES YES ST 2 NO NO NO YES YES AL 3 YES YES NO YES NO COLD 4 YES NO YES NO NO ST 5NO YES NO YES NO COLD 6NO NO NO YES NO AL 7NO NO YES NO NO ST 8 YES NO NO YES YES AL 9 NO YES NO YES YES COLD 10 YES YES NO YES YES COLD Database query: List all patients with Swollen Glands and Diagnosis of ST. Database query: List all patients with Diagnosis of Cold, or Diagnosis of Allergy and Swollen Glands. September 9, 2003 UMASSD, CIS, Iren Valova 5 One more picture for a good measure Income RangMagazine ProWatch Promo Life Ins PromCredit Car Sex Age C C C C C C R I I I O I I I 40-50,000 Yes No No No Male ,000 Yes Yes Yes No Fema ,000 No No No No Male ,000 Yes Yes Yes Yes Male ,000 Yes No Yes No Fema ,000 No No No No Fema ,000 Yes No Yes Yes Male ,000 No Yes No No Male ,000 Yes No No No Male ,000 Yes Yes Yes No Fema ,000 No Yes Yes No Fema ,000 No Yes Yes No Male ,000 Yes Yes Yes No Fema ,000 No Yes No No Male ,000 No No Yes Yes Fema 19 September 9, 2003 UMASSD, CIS, Iren Valova 6

2 It becomes claer I meant clear? Well, then we move on with Data Warehouses. Suppose that AllElectronics is a successful international company, with branches around the world. Each branch has its own set of databases. The president of AllElectronics has asked you to provide an analysis of the company's sales per item type per branch for the third quarter. This is a difficult task, particularly since the relevant data are spread out over several databases, physically located at numerous sites. If AllElectronics had a data warehouse, this task would be easy. A data warehouse is a repository of information collected from multiple sources, stored under a unified schema, and which usually resides at a single site. Data warehouses are constructed via a process of data cleaning, data transformation, data integration, data loading, and periodic data refreshing. September 9, 2003 UMASSD, CIS, Iren Valova 7 Data warehousing In order to facilitate decision making, the data in a data warehouse are organized around major subjects, such as customer, item, supplier, and activity. The data are stored to provide information from a historical perspective (such as from the past 5-10 years) and are typically summarized. For example, rather than storing the details of each sales transaction, the data warehouse may store a summary of the transactions per item type for each store or, summarized to a higher level, for each sales region. A data warehouse is usually modeled by a multidimensional database structure, where each dimension corresponds to an attribute or a set of attributes in the schema, and each cell stores the value of some aggregate measure, such as count or sales-amount. September 9, 2003 UMASSD, CIS, Iren Valova 8 Data warehousing Multi-dimensional Data Cube September 9, 2003 UMASSD, CIS, Iren Valova 9 September 9, 2003 UMASSD, CIS, Iren Valova 10 Multi-dimensional Data Cube Data warehouse vs. Data Mart Data warehouse: collects information about subjects that span an entire organization. Data mart: department subset of a data warehouse. Drill-down on time data Roll-up on address OLAP By providing multidimensional data views and the precomputation of summarized data, data warehouse systems are well suited for On-Line Analytical Processing, or OLAP. OLAP operations make use of background knowledge regarding the domain of the data being studied in order to allow the presentation of data at different levels of abstraction. Such operations accommodate different user viewpoints. September 9, 2003 UMASSD, CIS, Iren Valova 11 September 9, 2003 UMASSD, CIS, Iren Valova 12

3 DW and OLAP for DM Data warehousing provides architectures and tools for business executives to systematically organize, understand, and use their data to make strategic decisions. In the last several years, many firms have spent millions of dollars in building enterprise-wide data warehouses. Many people feel that with competition mounting in every industry, data warehousing is the latest must-have marketing weapon - a way to keep customers by learning more about their needs. DW and major characteristics Loosely speaking, a data warehouse refers to a database that is maintained separately from an organization's operational databases. Data warehouse systems allow for the integration of a variety of application systems. They support information processing by providing a solid platform of consolidated historical data for analysis. data warehouse is organized around major subjects constructed by integrating multiple heterogeneous sources every key structure contains an element of time physically separate store of data transformed from the application data found in the operational environment. September 9, 2003 UMASSD, CIS, Iren Valova 13 September 9, 2003 UMASSD, CIS, Iren Valova 14 Data Warehouse vs. Operational DBMS OLTP (on-line transaction processing) Major task of traditional relational DBMS Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc. OLAP (on-line analytical processing) Major task of data warehouse system Data analysis and decision making Distinct features (OLTP vs. OLAP): User and system orientation: customer vs. market Data contents: current, detailed vs. historical, consolidated Database design: ER + application vs. star + subject View: current, local vs. evolutionary, integrated Access patterns: update vs. read-only but complex queries OLTP vs. OLAP OLTP OLAP users clerk, IT professional knowledge worker function day to day operations decision support DB design application-oriented subject-oriented data current, up-to-date detailed, flat relational isolated usage repetitive ad-hoc historical, summarized, multidimensional integrated, consolidated access read/write lots of scans index/hash on prim. key unit of work short, simple transaction complex query # records accessed tens millions #users thousands hundreds DB size 100MB-GB 100GB-TB metric transaction throughput query throughput, response September 9, 2003 UMASSD, CIS, Iren Valova 15 September 9, 2003 UMASSD, CIS, Iren Valova 16 Why Separate Data Warehouse? From Tables and Spreadsheets to Data Cubes High performance for both systems DBMS tuned for OLTP: access methods, indexing, concurrency control, recovery Warehouse tuned for OLAP: complex OLAP queries, multidimensional view, consolidation. Different functions and different data: missing data: Decision support requires historical data which operational DBs do not typically maintain data consolidation: DS requires consolidation (aggregation, summarization) of data from heterogeneous sources data quality: different sources typically use inconsistent data representations, codes and formats which have to be reconciled A data warehouse is based on a multidimensional data model which views data in the form of a data cube A data cube, such as sales, allows data to be modeled and viewed in multiple dimensions Dimension tables, such as item (item_name, brand, type), or time(day, week, month, quarter, year) Fact table contains measures (such as dollars_sold) and keys to each of the related dimension tables September 9, 2003 UMASSD, CIS, Iren Valova 17 September 9, 2003 UMASSD, CIS, Iren Valova 18

4 Cube formation September 9, 2003 UMASSD, CIS, Iren Valova 19 September 9, 2003 UMASSD, CIS, Iren Valova 20 Cube: A Lattice of Cuboids all 0-D(apex) cuboid time item location supplier 1-D cuboids time,item time,location item,location location,supplier 2-D cuboids time,supplier item,supplier time,item,location time,location,supplier 3-D cuboids time,item,supplier item,location,supplier 4-D(base) cuboid time, item, location, supplier September 9, 2003 UMASSD, CIS, Iren Valova 21 September 9, 2003 UMASSD, CIS, Iren Valova 22 Conceptual Modeling of Data Warehouses Start schema example Modeling data warehouses: dimensions & measures Star schema: A fact table in the middle connected to a set of dimension tables Snowflake schema: A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation September 9, 2003 UMASSD, CIS, Iren Valova 23 September 9, 2003 UMASSD, CIS, Iren Valova 24

5 Snowflake schema example Fact constellation schema September 9, 2003 UMASSD, CIS, Iren Valova 25 September 9, 2003 UMASSD, CIS, Iren Valova 26 A Data Mining Query Language: DMQL Defining a Star Schema in DMQL Cube Definition (Fact Table) define cube <cube_name> [<dimension_list>]: <measure_list> Dimension Definition ( Dimension Table ) define dimension <dimension_name> as (<attribute_or_subdimension_list>) Special Case (Shared Dimension Tables) First time as cube definition define dimension <dimension_name> as <dimension_name_first_time> in cube <cube_name_first_time> define cube sales_star [time, item, branch, location]: dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold = count(*) define dimension time as (time_key, day, day_of_week, month, quarter, year) define dimension item as (item_key, item_name, brand, type, supplier_type) define dimension branch as (branch_key, branch_name, branch_type) define dimension location as (location_key, street, city, province_or_state, country) September 9, 2003 UMASSD, CIS, Iren Valova 27 September 9, 2003 UMASSD, CIS, Iren Valova 28 Defining a Snowflake Schema in DMQL define cube sales_snowflake [time, item, branch, location]: dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold = count(*) define dimension time as (time_key, day, day_of_week, month, quarter, year) define dimension item as (item_key, item_name, brand, type, supplier(supplier_key, supplier_type)) define dimension branch as (branch_key, branch_name, branch_type) define dimension location as (location_key, street, city(city_key, province_or_state, country)) Defining a Fact Constellation in DMQL define cube sales [time, item, branch, location]: dollars_sold = sum(sales_in_dollars), avg_sales = avg(sales_in_dollars), units_sold = count(*) define dimension time as (time_key, day, day_of_week, month, quarter, year) define dimension item as (item_key, item_name, brand, type, supplier_type) define dimension branch as (branch_key, branch_name, branch_type) define dimension location as (location_key, street, city, province_or_state, country) define cube shipping [time, item, shipper, from_location, to_location]: dollar_cost = sum(cost_in_dollars), unit_shipped = count(*) define dimension time as time in cube sales define dimension item as item in cube sales define dimension shipper as (shipper_key, shipper_name, location as location in cube sales, shipper_type) define dimension from_location as location in cube sales define dimension to_location as location in cube sales September 9, 2003 UMASSD, CIS, Iren Valova 29 September 9, 2003 UMASSD, CIS, Iren Valova 30

6 Measures: Three Categories A Concept Hierarchy: Dimension (location) distributive: if the result derived by applying the function to n aggregate values is the same as that derived by applying the function on all the data without partitioning. E.g., count(), sum(), min(), max(). algebraic: if it can be computed by an algebraic function with M arguments (where M is a bounded integer), each of which is obtained by applying a distributive aggregate function. E.g., avg(), min_n(), standard_deviation(). holistic: if there is no constant bound on the storage size needed to describe a subaggregate. E.g., median(), mode(), rank(). all region country city office all Europe... North_America Germany... Spain Canada... Mexico Frankfurt... Vancouver... Toronto L. Chan... M. Wind September 9, 2003 UMASSD, CIS, Iren Valova 31 September 9, 2003 UMASSD, CIS, Iren Valova 32 View of Warehouses and Hierarchies Multidimensional Data Sales volume as a function of product, month, and region Specification of hierarchies Schema hierarchy day < {month < quarter; week} < year Set_grouping hierarchy {1..10} < inexpensive Product Region Dimensions: Product, Location, Time Hierarchical summarization paths Industry Region Year Category Country Quarter Product City Month Week Office Day Month September 9, 2003 UMASSD, CIS, Iren Valova 33 September 9, 2003 UMASSD, CIS, Iren Valova 34 A Sample Data Cube Cuboids Corresponding to the Cube TV PC VCR sum Product Date 1Qtr 2Qtr 3Qtr 4Qtr sum Total annual sales of TV in U.S.A. U.S.A Canada Mexico Country all 0-D(apex) cuboid product date country 1-D cuboids product,date product,country date, country 2-D cuboids sum product, date, country 3-D(base) cuboid September 9, 2003 UMASSD, CIS, Iren Valova 35 September 9, 2003 UMASSD, CIS, Iren Valova 36

7 Browsing a Data Cube Visualization OLAP capabilities Interactive manipulation September 9, 2003 UMASSD, CIS, Iren Valova 37 Typical OLAP Operations Roll up (drill-up): summarize data by climbing up hierarchy or by dimension reduction Drill down (roll down): reverse of roll-up from higher level summary to lower level summary or detailed data, or introducing new dimensions Slice and dice: project and select Pivot (rotate): reorient the cube, visualization, 3D to series of 2D planes. Other operations drill across: involving (across) more than one fact table drill through: through the bottom level of the cube to its back-end relational tables (using SQL) September 9, 2003 UMASSD, CIS, Iren Valova 38 A Star-Net Query Model Data Warehouse Design Process Shipping Method Customer Orders Customer CONTRACTS AIR-EXPRESS TRUCK ORDER PRODUCT LINE Time Product ANNUALY QTRLY DAILY PRODUCT ITEM PRODUCT GROUP CITY SALES PERSON COUNTRY DISTRICT REGION DIVISION Location Each circle is called a footprint Promotion Organization September 9, UMASSD, CIS, Iren 2003 Valova 39 Top-down, bottom-up approaches or a combination of both Top-down: Starts with overall design and planning (mature) Bottom-up: Starts with experiments and prototypes (rapid) From software engineering point of view Waterfall: structured and systematic analysis at each step before proceeding to the next Spiral: rapid generation of increasingly functional systems, short turn around time, quick turn around Typical data warehouse design process Choose a business process to model, e.g., orders, invoices, etc. Choose the grain (atomic level of data) of the business process Choose the dimensions that will apply to each fact table record Choose the measure that will populate each fact table record September 9, 2003 UMASSD, CIS, Iren Valova 40 Multi-Tiered Architecture Three Data Warehouse Models other sources Operational DBs Metadata Extract Transform Load Refresh Monitor & Integrator Data Warehouse OLAP Server Serve Analysis Query Reports Data mining Enterprise warehouse collects all of the information about subjects spanning the entire organization Data Mart a subset of corporate-wide data that is of value to a specific groups of users. Its scope is confined to specific, selected groups, such as marketing data mart Independent vs. dependent (directly from warehouse) data mart Virtual warehouse A set of views over operational databases Only some of the possible summary views may be materialized Data Marts Data Sources Data Storage OLAP Engine Front-End Tools September 9, 2003 UMASSD, CIS, Iren Valova 41 September 9, 2003 UMASSD, CIS, Iren Valova 42

8 Data warehouse usage Business executives in almost every industry use the data collected, integrated, preprocessed, and stored in data warehouses and data marts to perform data analysis and make strategic decisions. In many firms, data warehouses are used as an integral part of a plan-executeassess "closed-loop" feedback system for enterprise management. Data warehouses are used extensively in banking and financial services, consumer goods and retail distribution sectors, and controlled manufacturing, such as demand-based production. Typically, the longer a data warehouse has been in use, the more it will have evolved. This evolution takes place throughout a number of phases. Initially, the data warehouse is mainly used for generating reports and answering predefined queries. Progressively, it is used to analyze summarized and detailed data, where the results are presented in the form of reports and charts. Later, the data warehouse is used for strategic purposes, performing multidimensional analysis and sophisticated slice-and-dice operations. Finally, the data warehouse may be employed for knowledge discovery and strategic decision making using data mining tools. DW and DM applications DW: Information processing supports querying, basic statistical analysis, and reporting; Analytical processing supports basic OLAP operations, including slice-and-dice, drill-down, roll-up, and pivoting. Operates on historical data. DM: knowledge discovery by finding hidden patterns and associations, constructing analytical models, performing classification and prediction. September 9, 2003 UMASSD, CIS, Iren Valova 43 September 9, 2003 UMASSD, CIS, Iren Valova 44 So, now DW? What about DM? "How does data mining relate to information processing and on-line analytical processing?" Information processing, based on queries, can find useful information. However, answers to such queries reflect the information directly stored in databases or computable by aggregate functions. They do not reflect sophisticated patterns or regularities buried in the database. Therefore, information processing is not data mining. "Do OLAP systems perform data mining? Are OLAP systems actually data mining systems?" The functionalities of OLAP and data mining can be viewed as disjoint: OLAP is a data summarization/aggregation tool that helps simplify data analysis, while data mining allows the automated discovery of implicit patterns and interesting knowledge hidden in large amounts of data. Say what? OLAP functions are essentially for user-directed data summary and comparison (by drilling, pivoting, slicing, dicing, and other operations). Data mining covers a much broader spectrum than simple OLAP operations because it not only performs data summary and comparison, but also performs association, classification, prediction, clustering, time-series analysis, and other data analysis tasks. September 9, 2003 UMASSD, CIS, Iren Valova 45 September 9, 2003 UMASSD, CIS, Iren Valova 46 One last attempt: DM rules! Data mining can help business managers find and reach more suitable customers, as well as gain critical business insights that may help to drive market share and raise profits. In addition, data mining can help managers understand customer group characteristics and develop optimal pricing strategies accordingly, correct item bundling based not on intuition but on actual item groups derived from customer purchase patterns, reduce promotional spending, and at the same time increase the overall net effectiveness of promotions. September 9, 2003 UMASSD, CIS, Iren Valova 47 Mining query OLAM Engine Filtering&Integration An OLAM Architecture User GUI API Data Cube API MDDB Database API Mining result OLAP Engine Meta Data Filtering Data Warehouse Layer4 User Interface Layer3 OLAP/OLAM Layer2 MDDB Layer1 Databases Data cleaning Data Data integration Repository September 9, 2003 Iren Valova UMASSD, CIS, 48

9 References (I) References (II) S. Agarwal, R. Agrawal, P. M. Deshpande, A. Gupta, J. F. Naughton, R. Ramakrishnan, and S. Sarawagi. On the computation of multidimensional aggregates. VLDB 96 D. Agrawal, A. E. Abbadi, A. Singh, and T. Yurek. Efficient view maintenance in data warehouses. SIGMOD 97. R. Agrawal, A. Gupta, and S. Sarawagi. Modeling multidimensional databases. ICDE 97 K. Beyer and R. Ramakrishnan. Bottom-Up Computation of Sparse and Iceberg CUBEs.. SIGMOD 99. S. Chaudhuri and U. Dayal. An overview of data warehousing and OLAP technology. ACM SIGMOD Record, 26:65-74, OLAP council. MDAPI specification version 2.0. In G. Dong, J. Han, J. Lam, J. Pei, K. Wang. Mining Multi-dimensional Constrained Gradients in Data Cubes. VLDB 2001 J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow, and H. Pirahesh. Data cube: A relational aggregation operator generalizing group-by, cross-tab and subtotals. Data Mining and Knowledge Discovery, 1:29-54, J. Han, J. Pei, G. Dong, K. Wang. Efficient Computation of Iceberg Cubes With Complex Measures. SIGMOD 01 V. Harinarayan, A. Rajaraman, and J. D. Ullman. Implementing data cubes efficiently. SIGMOD 96 Microsoft. OLEDB for OLAP programmer's reference version 1.0. In K. Ross and D. Srivastava. Fast computation of sparse datacubes. VLDB 97. K. A. Ross, D. Srivastava, and D. Chatziantoniou. Complex aggregation at multiple granularities. EDBT'98. S. Sarawagi, R. Agrawal, and N. Megiddo. Discovery-driven exploration of OLAP data cubes. EDBT'98. E. Thomsen. OLAP Solutions: Building Multidimensional Information Systems. John Wiley & Sons, W. Wang, H. Lu, J. Feng, J. X. Yu, Condensed Cube: An Effective Approach to Reducing Data Cube Size. ICDE 02. Y. Zhao, P. M. Deshpande, and J. F. Naughton. An array-based algorithm for simultaneous multidimensional aggregates. SIGMOD 97. September 9, 2003 UMASSD, CIS, Iren Valova 49 September 9, 2003 UMASSD, CIS, Iren Valova 50 Thank you!!! September 9, 2003 UMASSD, CIS, Iren Valova 51