MSCIT 5210/MSCBD 5002: Knowledge Discovery and Data Mining
|
|
- Bernard O’Brien’
- 5 years ago
- Views:
Transcription
1 MSCIT 5210/MSCBD 5002: Knowledge Discovery and Data Mining Acknowledgement: Slides modified by Dr. Lei Chen based on the slides provided by Jiawei Han, Micheline Kamber, and Jian Pei 2012 Han, Kamber & Pei. All rights reserved. 1 1
2 Chapter 4: Data Warehousing, On-line Analytical Processing and Data Cube Data Warehouse: Basic Concepts Data Warehouse Modeling: Data Cube and OLAP Data Cube Computation: Preliminary Concepts Data Cube Computation Methods Summary 2
3 DATA WAREHOUSES and OLAP On-Line Transaction Processing (OLTP) Systems manipulate operational data, necessary for day-today operations. Most existing database systems belong this category. On-Line Analytical Processing (OLAP) Systems support specific types of queries (based on groupbys and aggregation operators) useful for decision making. Data Mining tools discover interesting patterns in the data
4 Data Warehouse Query Databases Users Need to wait for a long time (e.g., 1 day to 1 week) Databases Data Warehouse Users Pre-computed results 4
5 Why OLTP is not sufficient for Decision Making Lets say that Welcome supermarket uses a relational database to keep track of sales in all of stores simultaneously product id... store id SALES table quantity sold date/time of sale :35: :35: :35:17
6 Example (cont.) PRODUCTS table Prod. product id name 567 Colgate Gel Pump 6.4 oz. 219 Diet Coke 12 oz. can... product category toothpast e Manufac t. id 68 soda 5 stor e id cit y id STORES table store location 510 Main Street 13 Maple Avenue phone number CITIES table id name state Popul. 34 San Francisco 58 East Fishkill California 700,000 New York 30,000
7 Example (cont.) An executive, asks "I noticed that there was a Colgate promotion recently, directed at people who live in small towns. How much Colgate toothpaste did we sell in those towns yesterday? And how much on the same day a month ago?" select sum(sales.quantity_sold) from sales, products, stores, cities where products.manufacturer_id = restrict to Colgateand products.product_category = 'toothpaste and cities.population < and sales.datetime_of_sale::date = 'yesterday'::date and sales.product_id = products.product_id and sales.store_id = stores.store_id and stores.city_id = cities.city_id
8 Example (cont.) You have to do a 4-way JOIN of some large tables. Moreover, these tables are being updated as the query is executed. Need for a separate RDBMS installation (i.e., a Data Warehouse) to support queries like the previous one. The Warehouse can be tailor-made for specific types of queries: if you know that the toothpaste query will occur every day then you can denormalize the data model.
9 Example (cont.) Suppose Welcome acquires ParknShop which is using a different set of OLTP data models and a different brand of RDBMS to support them. But you want to run the toothpaste queries for both divisions. Solution: Also copy data from the ParknShop Database into the Welcome Data Warehouse (data integration problems). One of the more important functions of a data warehouse in a company that has disparate computing systems is to provide a view for management as though the company were in fact integrated.
10 Motivation In most organizations, data about specific parts of business is there -- lots and lots of data, somewhere, in some form. Data is available but not information -- and not the right information at the right time. To bring together information from multiple sources as to provide a consistent database source for decision support queries. To off-load decision support applications from the on-line transaction system.
11 What is a Data Warehouse? Defined in many different ways, but not rigorously. A decision support database that is maintained separately from the organization s operational database Support information processing by providing a solid platform of consolidated, historical data for analysis. A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management s decision-making process. W. H. Inmon Data warehousing: The process of constructing and using data warehouses 11
12 Data Warehouse Subject-Oriented Organized around major subjects, such as customer, product, sales Focusing on the modeling and analysis of data for decision makers, not on daily operations or transaction processing Provide a simple and concise view around particular subject issues by excluding data that are not useful in the decision support process 12
13 Data Warehouse Integrated Constructed by integrating multiple, heterogeneous data sources relational databases, flat files, on-line transaction records Data cleaning and data integration techniques are applied. Ensure consistency in naming conventions, encoding structures, attribute measures, etc. among different data sources E.g., Hotel price: currency, tax, breakfast covered, etc. When data is moved to the warehouse, it is converted. 13
14 Data Warehouse Time Variant The time horizon for the data warehouse is significantly longer than that of operational systems Operational database: current value data Data warehouse data: provide information from a historical perspective (e.g., past 5-10 years) Every key structure in the data warehouse Contains an element of time, explicitly or implicitly But the key of operational data may or may not contain time element 14
15 Data Warehouse Nonvolatile A physically separate store of data transformed from the operational environment Operational update of data does not occur in the data warehouse environment Does not require transaction processing, recovery, and concurrency control mechanisms Requires only two operations in data accessing: initial loading of data and access of data 15
16 OLTP vs. OLAP O LTP O LA P users clerk, IT professional know ledge w orker function day to day operations decision support D B design application-oriented subject-oriented data current, up-to-date detailed, flat relational isolated usage repetitive ad-hoc access read/w rite index/hash on prim. key historical, summarized, multidimensional integrated, consolidated lots of scans unit of w ork short, simple transaction complex query # records accessed tens millions #users thousands hundreds D B size 100M B -GB 100G B -TB m etric transaction throughput query throughput, response 16
17 Why a Separate Data Warehouse? High performance for both systems DBMS tuned for OLTP: access methods, indexing, concurrency control, recovery Warehouse tuned for OLAP: complex OLAP queries, multidimensional view, consolidation Different functions and different data: missing data: Decision support requires historical data which operational DBs do not typically maintain data consolidation: DS requires consolidation (aggregation, summarization) of data from heterogeneous sources data quality: different sources typically use inconsistent data representations, codes and formats which have to be reconciled Note: There are more and more systems which perform OLAP analysis directly on relational databases 17
18 Data Warehouse: A Multi-Tiered Architecture Other sources Operational DBs Metadata Extract Transform Load Refresh Monitor & Integrator Data Warehouse OLAP Server Serve Analysis Query Reports Data mining Data Marts Data Sources Data Storage OLAP Engine Front-End Tools 18
19 Review of SQL 19
20 Basic Structure SQL is based on set and relational operations with certain modifications and enhancements A typical SQL query has the form: select A 1, A 2,, A n from R 1, R 2,, R m where P - A i represent attributes - R i represent relations - P is a predicate. This query is equivalent to the relational algebra expression: A1, A2,, An ( P (R 1 R 2 R m )) The result of an SQL query is a relation (but may contain duplicates). SQL statements can be nested. COMP3311 Fall 2017 CSE, HKUST Slide 20
21 Projection The select clause corresponds to the projection operation of the relational algebra. It is used to list the attributes desired in the result of a query. Find the names of all branches in the loan relation select branch-name from loan Equivalent to: branch-name (loan) An asterisk in the select clause denotes all attributes select * from loan Note: for our examples we use the tables: Branch (branch-name, branch-city, assets) Customer (customer-name, customer-street, customer-city) Loan (loan-number, amount, branch-name) Account (account-number, balance, branch-name) Borrower (customer-name, loan-number) Depositor (customer-name, account-number) COMP3311 Fall 2017 CSE, HKUST Slide 21
22 Duplicate Removal SQL allows duplicates in relations as well as in query results. Use select distinct to force the elimination of duplicates. Find the names of all branches in the loan relation, and remove duplicates select distinct branch-name from loan The keyword all specifies that duplicates are not removed. select all branch-name from loan force the DBMS to remove duplicates force the DBMS not to remove duplicates COMP3311 Fall 2017 CSE, HKUST Slide 22
23 Arithmetic Operations on Retrieved Results The select clause can contain arithmetic expressions involving the operators,,, and, and operating on constants or attributes of tuples. The query: select branch-name, loan-number, amount * 100 from loan would return a relation which is the same as the loan relations, except that the attribute amount is multiplied by 100 COMP3311 Fall 2017 CSE, HKUST Slide 23
24 The where Clause The where clause specifies conditions that tuples in the relations in the from clause must satisfy. Find all loan numbers for loans made at the Perryridge branch with loan amounts greater than $1200. select loan-number from loan where branch-name= Perryridge and amount >1200 SQL allows logical connectives and, or, and not. Arithmetic expressions can be used in the comparison operators. Note: attributes used in a query (both select and where parts) must be defined in the relations in the from clause. COMP3311 Fall 2017 CSE, HKUST Slide 24
25 The where Clause (Cont.) SQL includes the between operator for convenience. Find the loan number of those loans with loan amounts between $90,000 and $100,000 (that is, $90,000 and $100,000) select loan-number from loan where amount between and COMP3311 Fall 2017 CSE, HKUST Slide 25
26 The from Clause The from clause corresponds to the Cartesian product operation of the relational algebra. Find the Cartesian product borrower loan select * from borrower, loan It is rarely used without a where clause. Find the name and loan number of all customers having a loan at the Perryridge branch. select distinct customer-name, borrower.loan-number from borrower, loan where borrower.loan-number = loan.loan-number and branch-name = Perryridge COMP3311 Fall 2017 CSE, HKUST Slide 26
27 Tuple Variables/Alias Tuple variables are defined in the from clause via the use of the as clause. Find the customer names and their loan numbers for all customers having a loan at some branch. select distinct customer-name, T.loan-number from borrower as T, loan as S where T.loan-number = S.loan-number Tuple variable/alias can be used as short hand, but it is more than just a short hand (see next slide) Loan (loan-number, amount, branch-name) Borrower (customer-name, loan-number) COMP3311 Fall 2017 CSE, HKUST Slide 27
28 Tuple Variables/Alias Find the names of all branches that have greater assets than some branch located in Brooklyn. select distinct T.branch-name from branch as T, branch as S where T.assets > S.assets and S.branch-city= Brooklyn branch T branch S: a branch in Brooklyn branches in Brooklyn Does it returns branches within Brooklyn? COMP3311 Fall 2017 CSE, HKUST Slide 28
29 Ordering the Display of Tuples List in alphabetic order the names of all customers having a loan at Perryridge branch select distinct customer-name from borrower, loan where borrower.loan-number = loan.loan-number and branch-name = Perryridge order by customer-name order by customer-name desc, amount asc desc for descending order; asc for ascending order (default) SQL must perform a sort to fulfill an order by request. Since sorting a large number of tuples may be costly, it is desirable to sort only when necessary. Loan (loan-number, amount, branch-name) Borrower (customer-name, loan-number) COMP3311 Fall 2017 CSE, HKUST Slide 29
30 SQL - Nested Subqueries Every SQL statement returns a relation/set in the result; remember a relation could be null or merely contain a single atomic value You can replace a value or set of values with a SQL statement (ie., a subquery) select * select * from loan from loan where amount > 1200 where amount > select avg(amount) from loan Illegal if the subquery returns the wrong type for the comparison COMP3311 Fall 2017 CSE, HKUST Slide 30
31 Aggregate Functions Operate on a column of a relation, and return a value avg: average value min: minimum value max: maximum value sum: sum of values count: number of values Note: for our examples we use the tables: Branch (branch-name, branch-city, assets) Customer (customer-name, customer-street, customer-city) Loan (loan-number, amount, branch-name) Account (account-number, balance, branch-name) Borrower (customer-name, loan-number) Depositor (customer-name, account-number) COMP3311 Fall 2017 CSE, HKUST Slide 31
32 Aggregate Function Computation Find the average account balance at the Perryridge branch. select avg(balance) from account where branch-name= Perryridge account select balance from account where branch-name = Perryridge Balances of Perryridge accounts Avg() 120,000 COMP3311 Fall 2017 CSE, HKUST Slide 32
33 Examples of Aggregate Functions Find the numbers of tuples in the customer relation. select count(*) from customer remember * stands for all attributes Same as: select count(customer-city) from customer Different from: select count(distinct customer-city) from customer Because customer-city is not a key COMP3311 Fall 2017 CSE, HKUST Slide 33
34 Group by Find the number of accounts for each branch. select branch-name, count(account-number) from account group by branch-name For each group of tuples with the same branch-name, apply aggregate function count and distinct to account-number branch-nam e account-num ber balance Perryridge B righton Perryridge B righton R edw ood a-102 a- 217 a-201 a-215 a branch-name account-number balance Perryridge Perryridge Brighton Brighton Redwood a-102 a-201 a-217 a-215 a account table branch-name Perryridge Brighton Redwood count-account-no COMP331 Fall 2017 CSE, HKUST Slide 34
35 Group by Attributes Attributes in select clause outside of aggregate functions must appear in group by list, why? select branch-name, balance, count( distinct account-number) from account group by branch-name correct select from account group by branch-name, balance OR select branch-name, sum(balance), count( ) from account group by branch-name branchnam e Perryridge Perryridge B righton B righton R edw ood accountnum ber a-102 a-201 a-217 a-215 a-222 balance COMP3311 Fall 2017 CSE, HKUST Slide 35
36 Group by with Join Find the number of depositors for each branch. select branch-name, count( distinct customer-name) from depositor, account where depositor.account-number = account.account-number group by branch-name Perform Join then group by then count ( distinct () ) depositor (customer-name, account-number) account (branch-name, account-number, balance) Join (customer-name, account-number, branch-name, balance) Group by and aggregate functions apply to the Join result COMP3311 Fall 2017 CSE, HKUST Slide 36
37 Group by Evaluation select branch-name, customer-name from depositor, account where depositor.account-number = account.account-number branch-name Perryridge Perryridge Uptown Uptown Downtown Downtown Downtown cust-name John Wong Jacky Chan John Wong Mary Kwan John Wong Pat Lee May Cheung distinct count branch-name Perryridge Uptown Downtown count branch-name Perryridge Downtown Uptown Perryridge Uptown Downtown Perryridge Downtown cust-name John Wong Pat Lee John Wong Jacky Chan Mary Kwan John Wong John Wong May Cheung group by branch-name Perryridge Perryridge Perryridge Uptown Uptown Downtown Downtown Downtown cust-name John Wong Jacky Chan John Wong John Wong Mary Kwan John Wong Pat Lee May Cheung COMP3311 Fall 2017 CSE, HKUST Slide 37
38 Having Clause Find the names of all branches where the average account balance is more than $700 select branch-name, avg(balance) from account group by branch-name having avg (balance) >700 predicates in the having clause are applied to each group after the formation of groups branchnam e Perryridge Perryridge B righton B righton R edw ood accountnum ber a-102 a-201 a-217 a-215 a-222 balance COMP3311 Fall 2017 CSE, HKUST Slide 38
39 Derived Relations Find the name(s) of branches with the maximum average account balance. select branch-name from (select branch-name, avg(balance) from account group by branch-name) as result (branch-name, avg-balance) where avg-balance = (select max(avg-balance) from result) Return avg balance of each branch COMP3311 Fall 2017 CSE, HKUST Slide 39
40 Group-by Motivation: Group-by permits us to display aggregate results (e.g., max, min, sum) for groups. For instance, if we have GROUP-BY X, we will get a result for every different value of X. Recall that aggregate queries without group-by return just a single number. If we put an attribute in SELECT, the attribute must also appear in GROUP- BY. The opposite is not true: there may be attributes in GROUP-BY that do not appear in SELECT. Any condition that appears in WHERE, is applied before the formation of groups in other words, records that do not pass the WHERE condition are eliminated before the formation of groups. Any condition that appears in HAVING refers to the groups and is applied after the formation of the groups. The condition must involve aggregate functions, or attributes that appear in the SELECT or GROUP-BY lines. COMP3311 Fall 2011 CSE, HKUST Slide 40
41 Query 1: Find the total number of copies in stock for each poet poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet sum Douglas Livingstone 23 Mongane Wally 13 SELECT poet, SUM (copies_in_stock) as sum FROM writer GROUP BY poet COMP3311 Fall 2011 CSE, HKUST Slide 41
42 Query 2: For each poet, find the max, min, avg and total number of copies in stock poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet max min avg sum Douglas Livingstone Mongane Wally SELECT poet, MAX(copies_in_stock) AS max, MIN(copies_in_stock) AS min, AVG(copies_in_stock) AS avg, SUM(copies_in_stock) AS sum FROM writer GROUP BY poet COMP3311 Fall 2011 CSE, HKUST Slide 42
43 Query 3: For each poet, find the max, min, avg and total number of copies in stock take into account only books that have > 5 copies in stock poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet max min avg sum Douglas Livingstone Mongane Wally SELECT poet, MAX(copies_in_stock) AS max, MIN(copies_in_stock) AS min, AVG(copies_in_stock) AS avg, SUM(copies_in_stock) AS sum FROM writer WHERE copies_in_stock > 5 GROUP BY poet COMP3311 Fall 2011 CSE, HKUST Slide 43
44 Query 4: Find the total number of copies in stock for each poet who has a total of more than 20 copies in stock poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet sum Douglas Livingstone 23 Mongane Wally 13 SELECT poet, SUM(copies_in_stock) AS sum FROM writer GROUP BY poet HAVING sum>20 COMP3311 Fall 2011 CSE, HKUST Slide 44
45 Query 5: Find the total number of copies in stock for each poet who has a total of more than 20 copies in stock take into account only books that have more than 5 copies in stock poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet sum Douglas Livingstone 21 Mongane Wally 8 SELECT poet, SUM(copies_in_stock) AS sum FROM writer WHERE copies_in_stock > 5 GROUP BY poet HAVING sum>20 COMP3311 Fall 2011 CSE, HKUST Slide 45
46 Query 6: Find the total number of copies in stock for each poet whose name starts with any letter after E poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet sum Mongane Wally 13 SELECT poet, SUM(copies_in_stock) as sum FROM writer WHERE poet > E GROUP BY poet SELECT poet, SUM(copies_in_stock) as sum FROM writer GROUP BY poet HAVING poet > E COMP3311 Fall 2011 CSE, HKUST Slide 46
47 Query 7: Find the total number of copies in stock for each poet who has more than 2 books poet book copies_in_stock Douglas Livingstone The Skull 21 Douglas Livingstone A Littoral Zone 2 Mongane Wally Tstetlo 3 Mongane Wally Must Weep 8 Mongane Wally A Tough Tale 2 poet sum Mongane Wally 13 SELECT poet, SUM(copies_in_stock) as sum FROM writer GROUP BY poet HAVING count(*)>2 COMP3311 Fall 2011 CSE, HKUST Slide 47
48 Chapter 4: Data Warehousing and On-line Analytical Processing Data Warehouse: Basic Concepts Data Warehouse Modeling: Data Cube and OLAP Data Cube Computation: Preliminary Concepts Data Cube Computation Methods Summary 48
49 What is OLAP? The term OLAP ( online analytical processing ) was coined in a white paper written for Arbor Software Corp. in 1993 Interactive process of creating, managing, analyzing and reporting on data Analyzing large quantities of data in real-time
50 OLAP Data is perceived and manipulated as though it were stored in a multi-dimensional array Ideas are explained in terms of conventional SQL-styled tables
51 Data aggregation Data aggregation in many different ways The number of possible groupings quickly becomes large The user has to consider all groupings Analytical processing problem
52 Queries for supplier-and-parts database 1) Get the total shipment quantity 2) Get total shipment quantities by supplier 3) Get total shipment quantities by part 4) Get the shipment by supplier and part
53 SP S# P# QTY S1 P1 300 S1 P2 200 S2 P1 300 S2 P2 400 S3 P2 200 S4 P2 200
54 1. SELECT SUM(QTY) AS TOTQTY FROM SP GROUP BY () ; TOTQTY 1600
55 2. SELECT S#, FROM SP SUM(QTY) AS TOTQTY GROUP BY (S#) ; S# TOTQTY S1 500 S2 700 S3 200 S4 200
56 3. SELECT P#, SUM(QTY) AS TOTQTY FROM SP GROUP BY (P#) ; P# TOTQTY P1 600 P2 1000
57 4. SELECT S#, P#, SUM(QTY) AS TOTQTY FROM SP GROUP BY (S#,P#), S# P# TOTQTY S1 P1 300 S1 P2 200 S2 P1 300 S2 P2 400 S3 P2 200 S4 P2 200
58 Drawbacks Formulation so many similar but distinct queries is tedious Executing the queries is expensive Make life easier, more efficient computation Single query GROUPING SETS, ROLLUP, CUBE options Added to SQL standard 1999
59 GROUPING SETS Execute several queries simultaneously SELECT S#, P#, SUM (QTY) AS TOTQTY FROM SP GROUP BY GROUPING SETS ( (S#), (P#) ) ; Single results table Not a relation!! null missing information S# P# TOTQTY S1 null 500 S2 null 700 S3 null 200 S4 null 200 null P1 600 null P2 1000
60 SELECT CASE GROUPING ( S# ) WHEN 1 THEN?? FROM SP AS S#, ELSE S# CASE GROUPING ( P# ) WHEN 1 THEN!! AS P#, ELSE P# SUM ( QTY ) AS TOTQTY GROUP BY GROUPING SETS ( ( S# ), ( P# ) ); S# P# TOTQTY S1!! 500 S2!! 700 S3!! 200 S4!! 200?? P1 600?? P2 1000
61 ROLLUP SELECT S#,P#, SUM ( QTY ) AS TOTQTY FROM SP GROUP BY ROLLUP (S#, P#) ; S# P# TOTQTY S1 P1 300 S1 P2 200 S2 P1 300 S2 P2 400 S3 P2 200 S4 P2 200 S1 null 500 S2 null 700 S3 null 200 S4 null 200 null null 1600 GROUP BY GROUPING SETS ( ( S#, P# ), ( S# ), ( ) )
62 ROLLUP The quantities have been roll up for each supplier Rolled up along supplier dimension GROUP BY ROLLUP (A,B,...,Z) (A,B,...,Z) (A,B,...) (A,B) (A) () GROUP BY ROLLUP (A,B) is not symmetric in A and B!
63 SELECT S#, P#, SUM ( QTY ) AS TOTQTY FROM SP GROUP BY CUBE ( S#, P#) ; CUBE S# P# TOTQTY S1 P1 300 S1 P2 200 S2 P1 300 S2 P2 400 S3 P2 200 S4 P2 200 S1 null 500 S2 null 700 S3 null 200 S4 null 200 null P1 600 null P null null 1600 GROUP BY GROUPING SETS ( (S#, P#), ( S# ), ( P# ), ( ) )
64 CUBE Confusing term CUBE (?) Derived from the fact that in multidimensional terminology, data values are stored in cells of a multidimensional array or a hypercube The actual physical storage may differ In our example cube has just two dimensions (supplier, part) The two dimensions are unequal (no square rectangle..) Means group by all possible subsets of the set {A, B,..., Z }
65 CUBE Means group by all possible subsets of the set {A, B,..., Z } M={A, B,..., Z }, M =N Power Set (Algebra) P(M):={N N M}, P(M) =2 N induction..proof by Subset represent different grade of summarization Data Mining: such a subset is called a Cuboid
66 Cross Tabulations Display query results as cross tabulations More readable way Formatted as a simple array Example: two dimensions (supplier and parts) P1 P2 Total S S S S
67 What is a Data Cube? Data Mining definition A data cube, such as sales, allows data to be modeled and viewed in multiple dimensions Dimension tables, such as item(item_name, brand, type) time(day, week, month, quarter, year)...hierarchy Fact table contains measures (numerical values, such as dollars_sold) and keys to each of the related dimension tables
68 Data cube A data cube, such as sales, allows data to be modeled and viewed in multiple dimensions Suppose ALLELETRONICS create a sales data warehouse with respect to dimensions Time Item Location
69 Country A Sample Data Cube TV PC VCR sum Date 1Qtr 2Qtr 3Qtr 4Qtr sum Total annual sales of TV in Portugal Portugal Spain Germany sum
70 3D Data cube Example
71 Data cube A data cube, such as sales, allows data to be modeled and viewed in multiple dimensions Suppose ALLELETRONICS create a sales data warehouse with respect to dimensions Time Item Location Supplier
72 4D Data cube Example
73 Cuboid (Data Mining Definition) Names in data warehousing literature: The n-d cuboid, which holds the lowest level of summarization, is called a base cuboid.. {{A},{B},..} The top most 0-D cuboid, which holds the highestlevel of summarization, is called the apex cuboid.. { } The lattice of cuboids forms a data cube
74 Cube: A Lattice of Cuboids...(Power Set) all time item location supplier 0-D(apex) cuboid 1-D cuboids time,item time,location item,location location,supplier time,supplier item,supplier 2-D cuboids time,location,supplier time,item,location time,item,supplier item,location,supplier 3-D cuboids 4-D(base) cuboid time, item, location, supplier 2 4 =16
75 Conceptual Modeling of Data Warehouses Modeling data warehouses: dimensions & measures instead of relational model Subject, facilitates on-line data analysis oriented Most popular model is the multidimensional model Most common modeling paradigm: Star schema Data warehouse contains a large central table (fact table) Contains the data without redundancy A set of dimension tables (each for each dimension)
76 time time_key day day_of_the_week month quarter year branch branch_key branch_name branch_type Measures Example of Star Schema Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales item item_key item_name brand type supplier_type location location_key street city state_or_province country
77 Snowflake schema Snowflake schema: A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake
78 time time_key day day_of_the_week month quarter year branch branch_key branch_name branch_type Measures Example of Snowflake Schema Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales item item_key item_name brand type supplier_key location location_key street city_key supplier supplier_key supplier_type city city_key city state_or_province country
79 Fact constellations Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation
80 time time_key day day_of_the_week month quarter year Example of Fact Constellation Sales Fact Table time_key item_key branch_key item item_key item_name brand type supplier_type Shipping Fact Table time_key item_key shipper_key from_location branch branch_key branch_name branch_type Measures location_key units_sold dollars_sold avg_sales location location_key street city province_or_state country to_location dollars_cost units_shipped shipper shipper_key shipper_name location_key shipper_type
81 Hierarchies Independent variables are often related in hierarchies (taxonomy) Determine ways in which dependent data can be aggregated Temporal hierarchy Seconds, minutes, hours, days, weeks, months, years Same data can be aggregated in many different ways Same independent variable can belong to different hierarchies
82 all Hierarchy - Location all region Europe... North_America country Germany... Spain Canada... Mexico city Frankfurt... Vancouver... Toronto office L. Chan... M. Wind
83 View of Warehouses and Hierarchies Specification of hierarchies Schema hierarchy day < {month < quarter; week} < year Set_grouping hierarchy {1..10} < inexpensive
84 Product Multidimensional Data Sales volume as a function of product, month, and region Dimensions: Product, Location, Time Hierarchical summarization paths Industry Region Year Category Country Quarter Product City Month Week Month Office Day
85 Measures of Data Cube: Three Categories (Depending on the aggregate functions) Distributive: if the result derived by applying the function to n aggregate values is the same as that derived by applying the function on all the data without partitioning E.g., count(), sum(), min(), max() Algebraic: if it can be computed by an algebraic function with M arguments (where M is a bounded integer), each of which is obtained by applying a distributive aggregate function E.g., avg(), min_n(), standard_deviation() Holistic: if there is no constant bound on the storage size needed to describe a subaggregate. E.g., median(), mode(), rank()
86 Drill up and down Drill up: going from a lower level of aggregation to a higher Drill down: means the opposite Difference between drill up and roll up Roll up: creating the desired groupings or aggregations Drill up: accessing the aggregations Example for drill down: Given the total shipment quantity, get the total quantities for each individual supplier
87 Typical OLAP Operations Roll up (drill-up): summarize data by climbing up hierarchy or by dimension reduction Drill down (roll down): reverse of roll-up from higher level summary to lower level summary or detailed data, or introducing new dimensions Slice and dice: project and select Pivot (rotate): reorient the cube, visualization, 3D to series of 2D planes Other operations drill across: involving (across) more than one fact table drill through: through the bottom level of the cube to its back-end relational tables (using SQL) 87
88 Fig Typical OLAP Operations 88
89 Cube Operators for Roll-up day 2 day 1 s1 s2 s3 p p2 s1 s2 s3 p p sale(s1,*,*) s1 s2 s3 p p sale(s2,p2,*) s1 s2 s3 sum sum p1 110 p sale(*,*,*) 89
90 Extended Cube day 1 * s1 s2 s3 * p p * day 2 s1 s2 s3 * p p2 s1 s2 s3 * * 44 4 p p * sale(*,p2,*) 90
91 Aggregation Using Hierarchies day 2 day 1 s1 s2 s3 p p2 s1 s2 s3 p p store region country re g io n A re g io n B p p (store s1 in Region A; stores s2, s3 in Region B) CS
92 Slicing day 2 day 1 s1 s2 s3 p p2 s1 s2 s3 p p TIME = day 1 s1 s2 s3 p p CS
93 Slicing & Pivoting Products Store s1 Electronics $5.2 Toys $1.9 Clothing $2.3 Cosmetics $1.1 d1 Store s2 Electronics $8.9 Toys $0.75 Clothing $4.6 Cosmetics $1.5 Sales ($ millions) Time d2 Products Store s1 d1 Store s2 Store s1 Electronics $5.2 $8.9 Toys $1.9 $0.75 Clothing $2.3 $4.6 Cosmetics $1.1 $1.5 Store s2 Electronics Toys Clothing Sales ($ millions) 93
94 Attribute-Based Data Warehouse Problem Data Warehouse NP-hardness Algorithm Performance Study 94
95 Data Warehouse Parts are bought from suppliers and then sold to customers at a sale price SP Table T part supplier customer SP p1 s1 c1 4 p3 s1 c2 3 p2 s3 c1 7 95
96 Table T Data Warehouse part supplier customer SP p1 s1 c1 4 p3 s1 c2 3 p2 s3 c1 7 Parts are bought from suppliers and then sold to customers at a sale price SP c4 customer c3 c2 3 Data cube c1 4 s2 s3s4 p1 p2 p3 p4 p5 s1 supplier part 96
97 Table T Data Warehouse part supplier customer SP p1 s1 c1 4 p3 s1 c2 3 p2 s3 c1 7 Parts are bought from suppliers and then sold to customers at a sale price SP e.g., select part, customer, SUM(SP) from table T group by part, customer part customer SUM(SP) p1 c1 4 p3 c2 3 p2 c1 7 e.g., select customer, SUM(SP) from table T group by customer customer SUM(SP) c1 11 c2 3 pc 3 c 2 AVG(SP), MAX(SP), MIN(SP), 97
98 Table T Data Warehouse part supplier customer SP p1 s1 c1 4 p3 s1 c2 3 p2 s3 c1 7 Parts are bought from suppliers and then sold to customers at a sale price SP psc 6M pc 4M ps 0.8M sc 2M p 0.2M s 0.01M c 0.1M none 1 98
99 Data Warehouse Suppose we materialize all views. This wastes a lot of space. Cost for accessing pc = 4M psc 6M Cost for accessing ps = 0.8M pc 4M ps 0.8M sc 2M p 0.2M s 0.01M c 0.1M Cost for accessing p = 0.2M none 1 Cost for accessing sc = 2M Cost for accessing s = 0.01M Cost for accessing c = 0.1M 99
100 Data Warehouse Suppose we materialize the top view only. Cost for accessing pc = 6M (not 4M) psc 6M Cost for accessing ps = 6M (not 0.8M) pc 4M ps 0.8M sc 2M p 0.2M s 0.01M c 0.1M Cost for accessing p = 6M (not 0.2M) none 1 Cost for accessing sc = 6M (not 2M) Cost for accessing s = 6M (not 0.01M) Cost for accessing c = 6M (not 0.1M) COMP
101 Data Warehouse Suppose we materialize the top view and the view for ps only. Cost for accessing pc = 6M (still 6M) psc 6M Cost for accessing ps = 0.8M (not 6M previously) pc 4M ps 0.8M sc 2M Cost for accessing p = 0.8M (not 6M previously) Cost for accessing s = 0.8M (not 6M previously) p 0.2M s 0.01M c 0.1M none 1 Cost for accessing c = 6M (still 6M) Cost for accessing sc = 6M (still 6M) COMP
102 Gain({view for ps, top view}, {top view}) = 5.2*3 = 15.6 Data Warehouse Selective Materialization Problem: We can select a set V of k views such that Gain(V U {top view}, {top view}) is maximized. Suppose we materialize the top view and the view for ps only. Cost for accessing pc = 6M (still 6M) Gain = 0 psc 6M Cost for accessing ps = 0.8M (not 6M previously) Gain = 5.2M pc 4M ps 0.8M sc 2M p 0.2M s 0.01M c 0.1M Cost for accessing p = 0.8M (not 6M Gain previously) = 5.2M none 1 Cost for accessing sc = 6M (still 6M) Gain = 0 Cost for accessing s = 0.8M Cost for accessing c = 6M (not 6M previously) Gain = 5.2M (still 6M) Gain = 0 COMP
103 Attribute-Based Data Warehouse Problem Date Warehouse Algorithm 103
104 Greedy Algorithm k = number of views to be materialized Given v is a view S is a set of views which are selected to be materialized Define the benefit of selecting v for materialization as B(v, S) = Gain(S U v, S) 104
105 Greedy Algorithm S {top view}; For i = 1 to k do Select that view v not in S such that B(v, S) is maximized; S S U {v} Resulting S is the greedy selection 105
106 psc 6M Benefit from pc = 6M-6M = 0 k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = 0 106
107 psc 6M Benefit from ps = 6M-0.8M = 5.2M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 =
108 psc 6M Benefit from sc = 6M-6M = 0 k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = 0 108
109 psc 6M Benefit from p = 6M-0.2M = 5.8M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 =
110 psc 6M Benefit from s = 6M-0.01M = 5.99M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1=
111 psc 6M Benefit from c = 6M-0.1M = 5.9M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1= x 1 =
112 psc 6M Benefit from pc = 6M-6M = 0 k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1= x 1 = x 2 = 0 112
113 psc 6M Benefit from sc = 6M-6M = 0 k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1= x 1 = x 2 = 0 0 x 2 = 0 113
114 psc 6M Benefit from p = 0.8M-0.2M = 0.6M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1= x 1 = x 2 = 0 0 x 2 = x 1=
115 psc 6M Benefit from s = 0.8M-0.01M = 0.79M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1= x 1 = x 2 = 0 0 x 2 = x 1= x 1=
116 psc 6M Benefit from c = 6M-0.1M = 5.9M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit 1st Choice (M) 2nd Choice (M) pc ps sc p s c 0 x 3 = x 3 = x 3 = x 1 = x 1= x 1 = x 2 = 0 0 x 2 = x 1= x 1= x 1 =
117 psc 6M k = Data Cube pc 6M ps 0.8M sc 6M p 0.2M s 0.01M c 0.1M none 1 Benefit pc ps 1st Choice (M) 0 x 3 = x 3 = nd Choice (M) 0 x 2 = 0 Two views to be materialized are 1. ps 2. c sc 0 x 3 = 0 0 x 2 = 0 p 5.8 x 1 = x 1= 0.6 s c 5.99 x 1= x 1 = x 1= x 1 = 5.9 V = {ps, c} Gain(V U {top view}, {top view}) = =
118 Performance Study How bad does the Greedy Algorithm perform? COMP
119 k = 2 a 200 Benefit from b = = Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes p q r s Benefit 1st Choice (M) none 1 2nd Choice (M) b c d 41 x 100= 4100 COMP
120 k = 2 a 200 Benefit from c = = Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes p q r s Benefit 1st Choice (M) none 1 2nd Choice (M) b c d 41 x 100= x 101= 4141 COMP
121 k = 2 a Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes p q r s Benefit 1st Choice (M) none 1 2nd Choice (M) b c d 41 x 100= x 101= x 100= 4100 COMP
122 k = 2 a 200 Benefit from b = = Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes p q r s Benefit 1st Choice (M) none 1 2nd Choice (M) b c d 41 x 100= x 100= x 101= x 100= 4100 COMP
123 k = 2 a Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes p q r s Benefit 1st Choice (M) none 1 2nd Choice (M) Greedy: V = {b, c} Gain(V U {top view}, {top view}) = = 6241 b c d 41 x 100= x 100= x 101= x 100= x 100= 2100 COMP
124 k = 2 a Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes p q r s Benefit 1st Choice (M) none 1 2nd Choice (M) Greedy: V = {b, c} Gain(V U {top view}, {top view}) = = 6241 b c d 41 x 100= x 101= x 100= x 100= x x 1 = 2141 Optimal: V = {b, d} Gain(V U {top view}, {top view}) = = 8200 COMP
125 k = 2 a Data Cube b 100 c 99 d 100 p 1 97 q 1 97 r 1 97 s nodes If this ratio = 1, Greedy can give an optimal solution. If this ratio p , Greedy q 20 may 97 give a bad r solution. s Greedy Optimal = = none 1 Greedy: V = {b, c} Gain(V U {top view}, {top view}) = = 6241 Does this ratio has a lower bound? It is proved that this ratio is at least Optimal: V = {b, d} Gain(V U {top view}, {top view}) = = 8200 COMP
126 Performance Study This is just an example to show that this greedy algorithm can perform badly. A complete proof of the lower bound can be found in the paper. COMP
127 Chapter 5: Data Cube Technology Data Warehouse: Basic Concepts Data Warehouse Modeling: Data Cube and OLAP Data Cube Computation: Preliminary Concepts Data Cube Computation Methods 127
128 What the hell is that?!! 128
129 Cuboids in Data Cube all 0-D(apex) cuboid time item location supplier 1-D cuboids time,item time,location item,location location,supplier time,supplier item,supplier 2-D cuboids time,location,supplier time,item,location time,item,supplier item,location,supplier 3-D cuboids time, item, location, supplier 4-D(base) cuboid 129
130 time,item Cells in Data Cube across Different Dimensional Cuboids all time item location supplier time,location item,location Base vs. aggregate cells (9/15, cheese, Urbana, KRAFT) (9/15, cheese, Urbana, *) (*, cheese, Urbana, *) (*, cheese, *, *) (*, *, *, *) location,supplier time,item,location time,supplier time,item,supplier time,location,supplier item,supplier item,location,supplier Parent vs. child cells (9/15, *, Urbana, *) (9/15, cheese, Urbana, *) OR (9/15, cheese, Urbana, *) (9/15, cheese, Urbana, KRAFT) time, item, location, supplier Ancestor vs. descendant cells (9/15, *, Urbana., *) 130 (9/15, cheese, Urbana, KRAFT)
131 Cells in Cuboids of the Same Dimensions Parent vs. child cells (9/15, cheese, Illinois, *) (9/15, cheese, Urbana, *) Sibling cells (9/15, cheese, Urbana, *) (9/15, cheese, Chicago, *) Reference [see page 77] Multi-Dimensional, Phrase-Based Summarization in Text Cubes Each parent cell is found by changing exactly one non- dimension value in cell c into its parent value. Each child cell is found by either changing one value into non- or by replacing it by one of the child values. Each sibling cell must share one parent with cell c. 131
132 Cube Materialization: Full Cube vs. Iceberg Cube Full cube vs. iceberg cube compute cube sales iceberg as select date, product, city, department, count(*) from salesinfo cube by date, product, city having count(*) >= min support Compute only the cells whose measure satisfies the iceberg condition Only a small portion of cells may be above the water in a sparse cube Ex.: Show only those cells whose count is no less than
133 Why Iceberg Cube? Advantages of computing iceberg cubes No need to save nor show those cells whose value is below the threshold (iceberg condition) Efficient methods may even avoid computing the un-needed, intermediate cells Avoid explosive growth Example: A cube with 100 dimensions Suppose it contains only 2 base cells: {(a 1, a 2, a 3,., a 100 ), (a 1, a 2, b 3,, b 100 )} How many aggregate cells if having count >= 1? 133
134 Why Iceberg Cube? Advantages of computing iceberg cubes No need to save nor show those cells whose value is below the threshold (iceberg condition) Efficient methods may even avoid computing the un-needed, intermediate cells Avoid explosive growth Example: A cube with 100 dimensions Suppose it contains only 2 base cells: {(a 1, a 2, a 3,., a 100 ), (a 1, a 2, b 3,, b 100 )} How many aggregate cells if having count >= 1? Answer: ( ) 4 134
135 Suppose it contains only 2 base cells: {(a 1, a 2, a 3,., a 100 ), (a 1, a 2, b 3,, b 100 )} How many aggregate cells if having count >= 1? For {(a 1, a 2, a 3..., a 100 ), (a 1, a 2, b 3,..., b 100 )}, the total # of non-base cells should be 2 * (2^{100} 1) 4. This is calculated as follows: (a1, a2, a3..., a100) will generate 2^{100} - 1 non-base cells (a1, a2, b3,..., b100) will generate 2^{100} - 1 non-base cells Among these, 4 cells are overlapped and thus minus 4 so we get: 2*2^{100} These 4 cells are: (a1, a2, *,..., *): 2 (a1, *, *,..., *): 2 (*, a2, *,..., *): 2 (*, *, *,..., *): 2 135
136 Why Iceberg Cube? Advantages of computing iceberg cubes No need to save nor show those cells whose value is below the threshold (iceberg condition) Efficient methods may even avoid computing the un-needed, intermediate cells Avoid explosive growth Example: A cube with 100 dimensions Suppose it contains only 2 base cells: {(a 1, a 2, a 3,., a 100 ), (a 1, a 2, b 3,, b 100 )} How many aggregate cells if having count >= 1? Answer: ( ) 4 What about the iceberg cells, (i.e., with condition: having count >= 2 )? 136
137 Suppose it contains only 2 base cells: {(a 1, a 2, a 3,., a 100 ), (a 1, a 2, b 3,, b 100 )} How many iceberg cells if having count >= 2? For {(a 1, a 2, a 3..., a 100 ), (a 1, a 2, b 3,..., b 100 )}, the total # of non-base cells should be 2 * (2^{100} 1) 4. This is calculated as follows: (a1, a2, a3..., a100) will generate 2^{100} - 1 non-base cells (a1, a2, b3,..., b100) will generate 2^{100} - 1 non-base cells Among these, 4 cells are overlapped and thus minus 4 so we get: 2*2^{100} These 4 cells are: (a1, a2, *,..., *): 2 (a1, *, *,..., *): 2 (*, a2, *,..., *): 2 (*, *, *,..., *): 2 137
138 Why Iceberg Cube? Advantages of computing iceberg cubes No need to save nor show those cells whose value is below the threshold (iceberg condition) Efficient methods may even avoid computing the un-needed, intermediate cells Avoid explosive growth Example: A cube with 100 dimensions Suppose it contains only 2 base cells: {(a 1, a 2, a 3,., a 100 ), (a 1, a 2, b 3,, b 100 )} How many aggregate cells if having count >= 1? Answer: ( ) 4 What about the iceberg cells, (i.e., with condition: having count >= 2 )? Answer: 4 138
139 Is Iceberg Cube Good Enough? Closed Cube & Cube Shell Let cube P have only 2 base cells: {(a 1, a 2, a 3..., a 100 ):10, (a 1, a 2, b 3,..., b 100 ):10} How many cells will the iceberg cube contain if having count(*) 10? Answer: (still too big!) 139
140 Is Iceberg Cube Good Enough? Closed Cube & Cube Shell Let cube P have only 2 base cells: {(a 1, a 2, a 3..., a 100 ):10, (a 1, a 2, b 3,..., b 100 ):10} How many cells will the iceberg cube contain if having count(*) 10? Answer: (still too big!) Close cube: A cell c is closed if there exists no cell d, such that d is a descendant of c, and d has the same measure value as c Ex. The same cube P has only 3 closed cells: {(a 1, a 2, *,, *): 20, (a 1, a 2, a 3..., a 100 ): 10, (a 1, a 2, b 3,..., b 100 ): 10} A closed cube is a cube consisting of only closed cells 140
141 Is Iceberg Cube Good Enough? Closed Cube & Cube Shell Let cube P have only 2 base cells: {(a 1, a 2, a 3..., a 100 ):10, (a 1, a 2, b 3,..., b 100 ):10} How many cells will the iceberg cube contain if having count(*) 10? Answer: (still too big!) Close cube: A cell c is closed if there exists no cell d, such that d is a descendant of c, and d has the same measure value as c Ex. The same cube P has only 3 closed cells: {(a 1, a 2, *,, *): 20, (a 1, a 2, a 3..., a 100 ): 10, (a 1, a 2, b 3,..., b 100 ): 10} A closed cube is a cube consisting of only closed cells Cube Shell: The cuboids involving only a small # of dimensions, e.g., 2 Idea: Only compute cube shells, other dimension combinations can be computed on the fly 141
142 Data Cube Technology Data Cube Computation: Basic Concepts Data Cube Computation Methods 142
143 Roadmap for Efficient Computation General computation heuristics (Agarwal et al. 96) Computing full/iceberg cubes: 3 methodologies Bottom-Up: Multi-way array aggregation (Zhao, Deshpande & Naughton, SIGMOD 97) Top-down: BUC (Beyer & Ramarkrishnan, SIGMOD 99) Integrating Top-Down and Bottom-Up: Star-cubing algorithm (Xin, Han, Li & Wah: VLDB 03) High-dimensional OLAP: A shell-fragment approach (Li, et al. VLDB 04) Computing alternative kinds of cubes: Partial cube, closed cube, approximate cube, 143
144 Efficient Data Cube Computation: General Heuristics Sorting, hashing, and grouping operations are applied to the dimension attributes in order to reorder and cluster related tuples Aggregates may be computed from previously computed aggregates, rather than from the base fact table Smallest-child: computing a cuboid from the smallest, previously computed cuboid Cache-results: caching results of a cuboid from which other cuboids are computed to reduce disk I/Os Amortize-scans: computing as many as possible cuboids at the same time to amortize disk reads Share-sorts: sharing sorting costs cross multiple cuboids when sortbased method is used Share-partitions: sharing the partitioning cost across multiple cuboids when hash-based algorithms are used S. Agarwal, R. Agrawal, P. M. Deshpande, A. Gupta, J. F. Naughton, R. Ramakrishnan, S. Sarawagi. On the computation of multidimensional aggregates. VLDB
145 Cube Computation: Multi-Way Array Aggregation (MOLAP) B Bottom-up: Partition a huge sparse array into chunks (a small subcube which fits in memory) and aggregation. Data addressing: Compressed sparse array addressing (chunk_id, offset) Compute aggregates in multiway by visiting cube cells in the order which minimizes the # of times to visit each cell, and reduces memory access and storage cost C c What is the best traversing order to c do multi-way aggregation? c1 c b3 B b b1 b a0 a1 A - a2 - a ABC AB, BC and AC A: 40 (location), B: 400 (item), C: 4,000 (time) 145
146 Multi-way Array Aggregation (3-D to 2-D) How much memory cost of computation (aggregation for AB, AC, BC planes) can we save? One chunk of BC plane: (400/4) * (4,000/4) = 100,000 4x4x4 chunks Entire AB plane: 40 * 400 = 16,000 A: 40 (location), B: 400 (item), C: 4,000 (time) Min memory size:? < 40* *4, * 4,000 = 1,776,000 One column of AC plane: 40 * (4,000/4) = 40,
147 Multi-way Array Aggregation How to minimizes the memory requirement and reduced I/Os? Keep the smallest plane in main memory Fetch and compute only one chunk at a time for the largest plane The planes should be sorted and computed according to their size in ascending order One chunk of BC plane: (400/4) * (4,000/4) = 100,000 (3-D to 2-D) 4x4x4 chunks Entire AB plane: 40 * 400 = 16,000 A: 40 (location), B: 400 (item), C: 4,000 (time) Min memory size:? < 40* *4, * 4,000 = 1,776,000 One column of AC plane: 40 * (4,000/4) = 40,
148 BC planes: 400 * 4,000 C B b 0 c 0 b 1 c 0 One chunk 148
149 AC planes: 40 * 4,000 c 1 c 0 One column 4 * (40/4) * (4,000/4) = 40 * (4,000/4) c 0 C A a 0 a 3 149
150 Multi-way Array Aggregation How to minimizes the memory requirement and reduced I/Os? Keep the smallest plane in main memory Fetch and compute only one chunk at a time for the largest plane The planes should be sorted and computed according to their size in ascending order One chunk of BC plane: (400/4) * (4,000/4) = 100,000 (3-D to 2-D) 4x4x4 chunks Entire AB plane: 40 * 400 = 16,000 A: 40 (location), B: 400 (item), C: 4,000 (time) Min memory size: 156,000 < 40* *4, * 4,000 = 1,776,000 One column of AC plane: 40 * (4,000/4) = 40,
151 Multi-Way Array Aggregation Array-based bottom-up algorithm (from ABC to AB, ) Using multi-dimensional chunks Simultaneous aggregation on multiple dimensions Intermediate aggregate values are re-used for computing ancestor cuboids Cannot do Apriori pruning: No iceberg optimization Comments on the method Efficient for computing the full cube for a small number of dimensions If there are a large number of dimensions, top-down computation and iceberg cube computation methods (e.g., BUC) should be used Zhao, Deshpande & Naughton. An Array-Based Algorithm for Simultaneous Multidimensional Aggregates. SIGMOD
152 Cube Computation: Computing in Reverse Order BUC (Beyer & Ramakrishnan, SIGMOD 99) BUC: acronym of Bottom-Up (cube) Computation (Note: It is top-down in our view since we put Apex cuboid on the top!) Divides dimensions into partitions and facilitates iceberg pruning (it works now!) If a partition does not satisfy min_sup, its descendants can be pruned If min_sup = 1 Þ compute full CUBE! No simultaneous aggregation AB all A B C AC AD BC BD CD ABC ABD ACD BCD D ABCD 152
153 BUC: Partitioning Usually, entire data set can t fit in main memory Sort distinct values partition into blocks that fit Continue processing Optimizations Partitioning External Sorting, Hashing, Counting Sort Ordering dimensions to encourage pruning Cardinality, Skew, Correlation Collapsing duplicates Can t do holistic aggregates anymore! Ideally the dimension with most discriminative, higher cardinality and having less skew is processed first. 153
154 BUC: Example Tokyo Hong Kong B1 B2 B3 B A A2 8 1 A3 8 8 A4 1 8 Question: Compute cube iceberg_cube as Select A,B,C, count(*) From R Cube by A, B, C Having count(*) >= 5 Hong Kong Tokyo B B B2 B B B A1 A2 A3 A4 A1 A2 A3 A4 154
155 BUC: Example All 77 A A1 A2 A3 A4 (A1, B3, *) does not satisfy the min_sup and it will be pruned. (A2, *, *) does not satisfy the min_sup also it will be pruned.. B1 B2 B3 A, B A1 A2 A3 A4 155
156 Star-Cubing: An Integrating Method D. Xin, J. Han, X. Li, B. W. Wah, Star-Cubing: Computing Iceberg Cubes by Top-Down and Bottom-Up Integration, VLDB'03 Explore shared dimensions E.g., dimension A is the shared dimension of ACD and AD ABD/AB means cuboid ABD has shared dimensions AB Allows for shared computations e.g., cuboid AB is computed simultaneously as ABD Aggregate in a top-down manner but with the bottom-up sub-layer underneath which will allow Apriori pruning Shared dimensions grow in bottom-up fashion C/C D AC/AC AD/A BC/BC BD/B CD ABC/ABC ABD/AB ACD/A BCD ABCD/all 156
Data Warehousing & OLAP
Data Warehousing & OLAP Data Mining: Concepts and Techniques Chapter 3 Jiawei Han and An Introduction to Database Systems C.J.Date, Eighth Eddition, Addidon Wesley, 4 1 What is Data Warehousing? What is
More informationWhat is a Data Warehouse?
What is a Data Warehouse? COMP 465 Data Mining Data Warehousing Slides Adapted From : Jiawei Han, Micheline Kamber & Jian Pei Data Mining: Concepts and Techniques, 3 rd ed. Defined in many different ways,
More informationInformation Management course
Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 07 : 06/11/2012 Data Mining: Concepts and Techniques (3 rd ed.) Chapter
More informationMSCIT 5210/MSCBD 5002: Knowledge Discovery and Data Mining
MSCIT 5210/MSCBD 5002: Knowledge Discovery and Data Mining Acknowledgement: Slides modified by Dr. Lei Chen based on the slides provided by Jiawei Han, Micheline Kamber, and Jian Pei 2012 Han, Kamber &
More informationInformation Management course
Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 14 : 18/11/2014 Data Mining: Concepts and Techniques (3 rd ed.) Chapter
More informationDta Mining and Data Warehousing
CSCI6405 Fall 2003 Dta Mining and Data Warehousing Instructor: Qigang Gao, Office: CS219, Tel:494-3356, Email: q.gao@dal.ca Teaching Assistant: Christopher Jordan, Email: cjordan@cs.dal.ca Office Hours:
More informationChapter 4, Data Warehouse and OLAP Operations
CSI 4352, Introduction to Data Mining Chapter 4, Data Warehouse and OLAP Operations Young-Rae Cho Associate Professor Department of Computer Science Baylor University CSI 4352, Introduction to Data Mining
More informationJarek Szlichta Acknowledgments: Jiawei Han, Micheline Kamber and Jian Pei, Data Mining - Concepts and Techniques
Jarek Szlichta http://data.science.uoit.ca/ Acknowledgments: Jiawei Han, Micheline Kamber and Jian Pei, Data Mining - Concepts and Techniques Frequent Itemset Mining Methods Apriori Which Patterns Are
More informationCS490D: Introduction to Data Mining Chris Clifton
CS490D: Introduction to Data Mining Chris Clifton January 16, 2004 Data Warehousing Data Warehousing and OLAP Technology for Data Mining What is a data warehouse? A multi-dimensional data model Data warehouse
More informationData Warehousing & On-Line Analytical Processing
Data Warehousing & On-Line Analytical Processing Erwin M. Bakker & Stefan Manegold https://homepages.cwi.nl/~manegold/dbdm/ http://liacs.leidenuniv.nl/~bakkerem2/dbdm/ s.manegold@liacs.leidenuniv.nl e.m.bakker@liacs.leidenuniv.nl
More informationECT7110 Introduction to Data Warehousing
ECT7110 Introduction to Data Warehousing Prof. Wai Lam ECT7110 Introduction to Data Warehousing 1 What is Data Warehouse? Defined in many different ways, but not rigorously. A decision support database
More informationECLT 5810 Introduction to Data Warehousing
ECLT 5810 Introduction to Data Warehousing Prof. Wai Lam ECLT 5810 Introduction to Data Warehousing 1 What is Data Warehouse? Provides tools for business executives Systematically organize and understand
More informationData Mining. Data warehousing. Hamid Beigy. Sharif University of Technology. Fall 1394
Data Mining Data warehousing Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1394 1 / 22 Table of contents 1 Introduction 2 Data warehousing
More informationCS 412 Intro. to Data Mining
CS 412 Intro. to Data Mining Chapter 4. Data Warehousing and On-line Analytical Processing Jiawei Han, Computer Science, Univ. Illinois at Urbana -Champaign, 2017 1 2 3 Chapter 4: Data Warehousing and
More informationAnalyse des Données. Master 2 IMAFA. Andrea G. B. Tettamanzi
Analyse des Données Master 2 IMAFA Andrea G. B. Tettamanzi Université Nice Sophia Antipolis UFR Sciences - Département Informatique andrea.tettamanzi@unice.fr Andrea G. B. Tettamanzi, 2016 1 CM - Séance
More informationIntroduction to Data Warehousing
ICS 321 Spring 2012 Introduction to Data Warehousing Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 4/23/2012 Lipyeow Lim -- University of Hawaii at Manoa
More informationData Warehousing & On-line Analytical Processing
Data Warehousing & On-line Analytical Processing Erwin M. Bakker & Stefan Manegold https://homepages.cwi.nl/~manegold/dbdm/ http://liacs.leidenuniv.nl/~bakkerem2/dbdm/ Chapter 4: Data Warehousing and On-line
More informationData Warehouse. Concepts and Techniques. Chapter 3. SS Chung. April 5, 2013 Data Mining: Concepts and Techniques 1
Data Warehouse Concepts and Techniques Chapter 3 SS Chung April 5, 2013 Data Mining: Concepts and Techniques 1 Chapter 3: Data Warehousing and OLAP Technology: An Overview What is a data warehouse? A multi-dimensional
More informationData Mining. Data warehousing. Hamid Beigy. Sharif University of Technology. Fall 1396
Data Mining Data warehousing Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1396 1 / 31 Table of contents 1 Introduction 2 Data warehousing
More informationData Mining. Data warehousing. Hamid Beigy. Sharif University of Technology. Fall 1394
Data Mining Data warehousing Hamid Beigy Sharif University of Technology Fall 1394 Hamid Beigy (Sharif University of Technology) Data Mining Fall 1394 1 / 31 Table of contents 1 Introduction 2 Data warehousing
More informationData Warehousing 2. ICS 421 Spring Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa
ICS 421 Spring 2010 Data Warehousing 2 Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 3/30/2010 Lipyeow Lim -- University of Hawaii at Manoa 1 Data Warehousing
More informationA Multi-Dimensional Data Model
A Multi-Dimensional Data Model A Data Warehouse is based on a Multidimensional data model which views data in the form of a data cube A data cube, such as sales, allows data to be modeled and viewed in
More informationInformation Management course
Università degli Studi di Milano Master Degree in Computer Science Information Management course Teacher: Alberto Ceselli Lecture 05(b) : 23/10/2012 Data Mining: Concepts and Techniques (3 rd ed.) Chapter
More informationData Cube Technology
Data Cube Technology Erwin M. Bakker & Stefan Manegold https://homepages.cwi.nl/~manegold/dbdm/ http://liacs.leidenuniv.nl/~bakkerem2/dbdm/ s.manegold@liacs.leidenuniv.nl e.m.bakker@liacs.leidenuniv.nl
More informationBy Mahesh R. Sanghavi Associate professor, SNJB s KBJ CoE, Chandwad
By Mahesh R. Sanghavi Associate professor, SNJB s KBJ CoE, Chandwad All the content of these PPTs were taken from PPTS of renown author via internet. These PPTs are only mean to share the knowledge among
More informationDatabase design View Access patterns Need for separate data warehouse:- A multidimensional data model:-
UNIT III: Data Warehouse and OLAP Technology: An Overview : What Is a Data Warehouse? A Multidimensional Data Model, Data Warehouse Architecture, Data Warehouse Implementation, From Data Warehousing to
More informationChapter 5, Data Cube Computation
CSI 4352, Introduction to Data Mining Chapter 5, Data Cube Computation Young-Rae Cho Associate Professor Department of Computer Science Baylor University A Roadmap for Data Cube Computation Full Cube Full
More informationData Mining & Data Warehouse
Data Mining & Data Warehouse Associate Professor Dr. Raed Ibraheem Hamed University of Human Development, College of Science and Technology (1) 2016 2017 1 Points to Cover Why Do We Need Data Warehouses?
More informationData Cube Technology. Chapter 5: Data Cube Technology. Data Cube: A Lattice of Cuboids. Data Cube: A Lattice of Cuboids
Chapter 5: Data Cube Technology Data Cube Technology Data Cube Computation: Basic Concepts Data Cube Computation Methods Erwin M. Bakker & Stefan Manegold https://homepages.cwi.nl/~manegold/dbdm/ http://liacs.leidenuniv.nl/~bakkerem2/dbdm/
More informationData mining: Hmm, what is it?
Data mining: Hmm, what is it? Data warehousing Examples Discussions The extraction of implicit, previously unknown and potentially useful information from large bodies of data often accumulated for other
More informationData Warehousing (1)
ICS 421 Spring 2010 Data Warehousing (1) Asst. Prof. Lipyeow Lim Information & Computer Science Department University of Hawaii at Manoa 3/18/2010 Lipyeow Lim -- University of Hawaii at Manoa 1 Motivation
More informationData Mining. Associate Professor Dr. Raed Ibraheem Hamed. University of Human Development, College of Science and Technology
Data Mining Associate Professor Dr. Raed Ibraheem Hamed University of Human Development, College of Science and Technology (1) 2016 2017 Department of CS- DM - UHD 1 Points to Cover Why Do We Need Data
More informationData Warehousing and Data Mining
Data Warehousing and Data Mining Lecture 3 Efficient Cube Computation CITS3401 CITS5504 Wei Liu School of Computer Science and Software Engineering Faculty of Engineering, Computing and Mathematics Acknowledgement:
More informationData Mining Concepts & Techniques
Data Mining Concepts & Techniques Lecture No. 01 Databases, Data warehouse Naeem Ahmed Email: naeemmahoto@gmail.com Department of Software Engineering Mehran Univeristy of Engineering and Technology Jamshoro
More informationCSIT5300: Advanced Database Systems
CSIT5300: Advanced Database Systems L03: Structured Query Language (SQL) Part 2 Dr. Kenneth LEUNG Department of Computer Science and Engineering The Hong Kong University of Science and Technology Hong
More informationDATA WAREHOUSING & DATA MINING. by: Prof. Asha Ambhaikar
DATA WAREHOUSING & DATA MINING by: Prof. Asha Ambhaikar 1 UNIT-I Overview and Concepts 2 Contents of Unit-I Need for data warehousing, Basic elements of data warehousing, Trends in data warehousing. Planning
More informationSQL. Lecture 4 SQL. Basic Structure. The select Clause. The select Clause (Cont.) The select Clause (Cont.) Basic Structure.
SL Lecture 4 SL Chapter 4 (Sections 4.1, 4.2, 4.3, 4.4, 4.5, 4., 4.8, 4.9, 4.11) Basic Structure Set Operations Aggregate Functions Null Values Nested Subqueries Derived Relations Modification of the Database
More informationAcknowledgment. MTAT Data Mining. Week 7: Online Analytical Processing and Data Warehouses. Typical Data Analysis Process.
MTAT.03.183 Data Mining Week 7: Online Analytical Processing and Data Warehouses Marlon Dumas marlon.dumas ät ut. ee Acknowledgment This slide deck is a mashup of the following publicly available slide
More informationSyllabus. Syllabus. Motivation Decision Support. Syllabus
Presentation: Sophia Discussion: Tianyu Metadata Requirements and Conclusion 3 4 Decision Support Decision Making: Everyday, Everywhere Decision Support System: a class of computerized information systems
More informationEvolution of Database Systems
Evolution of Database Systems Krzysztof Dembczyński Intelligent Decision Support Systems Laboratory (IDSS) Poznań University of Technology, Poland Intelligent Decision Support Systems Master studies, second
More informationLecture 3 SQL. Shuigeng Zhou. September 23, 2008 School of Computer Science Fudan University
Lecture 3 SQL Shuigeng Zhou September 23, 2008 School of Computer Science Fudan University Outline Basic Structure Set Operations Aggregate Functions Null Values Nested Subqueries Derived Relations Views
More informationBasic Structure Set Operations Aggregate Functions Null Values Nested Subqueries Derived Relations Views Modification of the Database Data Definition
Chapter 4: SQL Basic Structure Set Operations Aggregate Functions Null Values Nested Subqueries Derived Relations Views Modification of the Database Data Definition Language 4.1 Schema Used in Examples
More informationA Novel Approach of Data Warehouse OLTP and OLAP Technology for Supporting Management prospective
A Novel Approach of Data Warehouse OLTP and OLAP Technology for Supporting Management prospective B.Manivannan Research Scholar, Dept. Computer Science, Dravidian University, Kuppam, Andhra Pradesh, India
More informationSummary of Last Chapter. Course Content. Chapter 2 Objectives. Data Warehouse and OLAP Outline. Incentive for a Data Warehouse
Principles of Knowledge Discovery in bases Fall 1999 Chapter 2: Warehousing and Dr. Osmar R. Zaïane University of Alberta Dr. Osmar R. Zaïane, 1999 Principles of Knowledge Discovery in bases University
More informationChapter 4: SQL. Basic Structure
Chapter 4: SQL Basic Structure Set Operations Aggregate Functions Null Values Nested Subqueries Derived Relations Views Modification of the Database Joined Relations Data Definition Language Embedded SQL
More informationOn-Line Analytical Processing (OLAP) Traditional OLTP
On-Line Analytical Processing (OLAP) CSE 6331 / CSE 6362 Data Mining Fall 1999 Diane J. Cook Traditional OLTP DBMS used for on-line transaction processing (OLTP) order entry: pull up order xx-yy-zz and
More informationDATABASE TECHNOLOGY - 1MB025
1 DATABASE TECHNOLOGY - 1MB025 Fall 2005 An introductury course on database systems http://user.it.uu.se/~udbl/dbt-ht2005/ alt. http://www.it.uu.se/edu/course/homepage/dbastekn/ht05/ Kjell Orsborn Uppsala
More informationDATABASE TECHNOLOGY - 1MB025
1 DATABASE TECHNOLOGY - 1MB025 Fall 2004 An introductory course on database systems http://user.it.uu.se/~udbl/dbt-ht2004/ alt. http://www.it.uu.se/edu/course/homepage/dbastekn/ht04/ Kjell Orsborn Uppsala
More informationReminds on Data Warehousing
BUSINESS INTELLIGENCE Reminds on Data Warehousing (details at the Decision Support Database course) Business Informatics Degree BI Architecture 2 Business Intelligence Lab Star-schema datawarehouse 3 time
More informationAn Overview of Data Warehousing and OLAP Technology
An Overview of Data Warehousing and OLAP Technology What is a data warehouse? A multi-dimensional data model Data warehouse architecture Data warehouse implementation lecture 2 1 What is Data Warehouse?
More informationThis tutorial will help computer science graduates to understand the basic-to-advanced concepts related to data warehousing.
About the Tutorial A data warehouse is constructed by integrating data from multiple heterogeneous sources. It supports analytical reporting, structured and/or ad hoc queries and decision making. This
More informationDATABASTEKNIK - 1DL116
1 DATABASTEKNIK - 1DL116 Spring 2004 An introductury course on database systems http://user.it.uu.se/~udbl/dbt-vt2004/ Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala
More informationManaging Information Resources
Managing Information Resources 1 Managing Data 2 Managing Information 3 Managing Contents Concepts & Definitions Data Facts devoid of meaning or intent e.g. structured data in DB Information Data that
More informationAn Overview of Data Warehousing and OLAP Technology
An Overview of Data Warehousing and OLAP Technology CMPT 843 Karanjit Singh Tiwana 1 Intro and Architecture 2 What is Data Warehouse? Subject-oriented, integrated, time varying, non-volatile collection
More informationDecision Support Systems
Decision Support Systems 2011/2012 Week 3. Lecture 6 Previous Class Dimensions & Measures Dimensions: Item Time Loca0on Measures: Quan0ty Sales TransID ItemName ItemID Date Store Qty T0001 Computer I23
More informationThis tutorial will help computer science graduates to understand the basic-to-advanced concepts related to data warehousing.
About the Tutorial A data warehouse is constructed by integrating data from multiple heterogeneous sources. It supports analytical reporting, structured and/or ad hoc queries and decision making. This
More informationDatabase Systems SQL SL03
Inf4Oec10, SL03 1/52 M. Böhlen, ifi@uzh Informatik für Ökonomen II Fall 2010 Database Systems SQL SL03 Data Definition Language Table Expressions, Query Specifications, Query Expressions Subqueries, Duplicates,
More informationETL and OLAP Systems
ETL and OLAP Systems Krzysztof Dembczyński Intelligent Decision Support Systems Laboratory (IDSS) Poznań University of Technology, Poland Software Development Technologies Master studies, first semester
More informationIT DATA WAREHOUSING AND DATA MINING UNIT-2 BUSINESS ANALYSIS
PART A 1. What are production reporting tools? Give examples. (May/June 2013) Production reporting tools will let companies generate regular operational reports or support high-volume batch jobs. Such
More informationSQL QUERIES. CS121: Relational Databases Fall 2017 Lecture 5
SQL QUERIES CS121: Relational Databases Fall 2017 Lecture 5 SQL Queries 2 SQL queries use the SELECT statement General form is: SELECT A 1, A 2,... FROM r 1, r 2,... WHERE P; r i are the relations (tables)
More informationData Mining: Concepts and Techniques. (3 rd ed.) Chapter 5
Data Mining: Concepts and Techniques (3 rd ed.) Chapter 5 Jiawei Han, Micheline Kamber, and Jian Pei University of Illinois at Urbana-Champaign & Simon Fraser University 2013 Han, Kamber & Pei. All rights
More informationDatabase Systems SQL SL03
Checking... Informatik für Ökonomen II Fall 2010 Data Definition Language Database Systems SQL SL03 Table Expressions, Query Specifications, Query Expressions Subqueries, Duplicates, Null Values Modification
More informationOLAP2 outline. Multi Dimensional Data Model. A Sample Data Cube
OLAP2 outline Multi Dimensional Data Model Need for Multi Dimensional Analysis OLAP Operators Data Cube Demonstration Using SQL Multi Dimensional Data Model Multi dimensional analysis is a popular approach
More informationDecision Support, Data Warehousing, and OLAP
Decision Support, Data Warehousing, and OLAP : Contents Terminology : OLAP vs. OLTP Data Warehousing Architecture Technologies References 1 Decision Support and OLAP Information technology to help knowledge
More informationBasics of Dimensional Modeling
Basics of Dimensional Modeling Data warehouse and OLAP tools are based on a dimensional data model. A dimensional model is based on dimensions, facts, cubes, and schemas such as star and snowflake. Dimension
More informationBusiness Intelligence
Business Intelligence Data Warehouse drives the corporate information supply chain to support Corporate Business Intelligence process. Business Intelligence introduced by Howard Dresner of the Gartner
More informationCognos also provides you an option to export the report in XML or PDF format or you can view the reports in XML format.
About the Tutorial IBM Cognos Business intelligence is a web based reporting and analytic tool. It is used to perform data aggregation and create user friendly detailed reports. IBM Cognos provides a wide
More informationData Quality. Data Cleaning and Integration. Data Cleaning. Data Preprocessing. Handling Missing Values. Disguised Missing Data?
2014-05-06 Data Quality Data Cleaning and Integration Accuracy Completeness Consistency Timeliness Believability Interpretability J. Pei: Big Data Analytics -- Data Cleaning and Integration 2 Data Preprocessing
More informationData warehouses Decision support The multidimensional model OLAP queries
Data warehouses Decision support The multidimensional model OLAP queries Traditional DBMSs are used by organizations for maintaining data to record day to day operations On-line Transaction Processing
More informationcollection of data that is used primarily in organizational decision making.
Data Warehousing A data warehouse is a special purpose database. Classic databases are generally used to model some enterprise. Most often they are used to support transactions, a process that is referred
More informationCSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 8 - Data Warehousing and Column Stores
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 8 - Data Warehousing and Column Stores Announcements Shumo office hours change See website for details HW2 due next Thurs
More informationDecision Support Systems aka Analytical Systems
Decision Support Systems aka Analytical Systems Decision Support Systems Systems that are used to transform data into information, to manage the organization: OLAP vs OLTP OLTP vs OLAP Transactions Analysis
More informationData Warehousing and Decision Support. Introduction. Three Complementary Trends. [R&G] Chapter 23, Part A
Data Warehousing and Decision Support [R&G] Chapter 23, Part A CS 432 1 Introduction Increasingly, organizations are analyzing current and historical data to identify useful patterns and support business
More informationData Warehousing and Decision Support
Data Warehousing and Decision Support Chapter 23, Part A Database Management Systems, 2 nd Edition. R. Ramakrishnan and J. Gehrke 1 Introduction Increasingly, organizations are analyzing current and historical
More informationData Warehousing ETL. Esteban Zimányi Slides by Toon Calders
Data Warehousing ETL Esteban Zimányi ezimanyi@ulb.ac.be Slides by Toon Calders 1 Overview Picture other sources Metadata Monitor & Integrator OLAP Server Analysis Operational DBs Extract Transform Load
More informationData Warehousing Conclusion. Esteban Zimányi Slides by Toon Calders
Data Warehousing Conclusion Esteban Zimányi ezimanyi@ulb.ac.be Slides by Toon Calders Motivation for the Course Database = a piece of software to handle data: Store, maintain, and query Most ideal system
More informationSilberschatz, Korth and Sudarshan See for conditions on re-use
Chapter 3: SQL Database System Concepts, 5th Ed. See www.db-book.com for conditions on re-use Chapter 3: SQL Data Definition Basic Query Structure Set Operations Aggregate Functions Null Values Nested
More informationChapter 3: SQL. Database System Concepts, 5th Ed. Silberschatz, Korth and Sudarshan See for conditions on re-use
Chapter 3: SQL Database System Concepts, 5th Ed. See www.db-book.com for conditions on re-use Chapter 3: SQL Data Definition Basic Query Structure Set Operations Aggregate Functions Null Values Nested
More informationChapter 3: SQL. Chapter 3: SQL
Chapter 3: SQL Database System Concepts, 5th Ed. See www.db-book.com for conditions on re-use Chapter 3: SQL Data Definition Basic Query Structure Set Operations Aggregate Functions Null Values Nested
More informationCHAPTER 3 Implementation of Data warehouse in Data Mining
CHAPTER 3 Implementation of Data warehouse in Data Mining 3.1 Introduction to Data Warehousing A data warehouse is storage of convenient, consistent, complete and consolidated data, which is collected
More informationReporting and Query tools and Applications
BUSINESS ANALYSIS Reporting and Query tools and Applications Five categories of tools Reporting Managed Query Executive information systems On-line analytical processing Data mining Reporting tools Production
More information2 CONTENTS
Contents 4 Data Cube Computation and Data Generalization 3 4.1 Efficient Methods for Data Cube Computation............................. 3 4.1.1 A Road Map for Materialization of Different Kinds of Cubes.................
More informationData Warehousing and OLAP
Data Warehousing and OLAP INFO 330 Slides courtesy of Mirek Riedewald Motivation Large retailer Several databases: inventory, personnel, sales etc. High volume of updates Management requirements Efficient
More informationCHAPTER 8: ONLINE ANALYTICAL PROCESSING(OLAP)
CHAPTER 8: ONLINE ANALYTICAL PROCESSING(OLAP) INTRODUCTION A dimension is an attribute within a multidimensional model consisting of a list of values (called members). A fact is defined by a combination
More informationData Warehousing and Decision Support
Data Warehousing and Decision Support [R&G] Chapter 23, Part A CS 4320 1 Introduction Increasingly, organizations are analyzing current and historical data to identify useful patterns and support business
More informationCS 245: Database System Principles. Warehousing. Outline. What is a Warehouse? What is a Warehouse? Notes 13: Data Warehousing
Recall : Database System Principles Notes 3: Data Warehousing Three approaches to information integration: Federated databases did teaser Data warehousing next Mediation Hector Garcia-Molina (Some modifications
More informationThe SQL database language Parts of the SQL language
DATABASE DESIGN I - 1DL300 Fall 2011 Introduction to SQL Elmasri/Navathe ch 4,5 Padron-McCarthy/Risch ch 7,8,9 An introductory course on database systems http://www.it.uu.se/edu/course/homepage/dbastekn/ht11
More informationDATABASE DESIGN I - 1DL300
DATABASE DESIGN I - 1DL300 Fall 2010 An introductory course on database systems http://www.it.uu.se/edu/course/homepage/dbastekn/ht10/ Manivasakan Sabesan Uppsala Database Laboratory Department of Information
More informationUNIT 4. DATA WAREHOUSING
UNIT 4. DATA WAREHOUSING Data Warehousing Components -Multi Dimensional Data Model- Data Warehouse Architecture-Data Warehouse Implementation- -Mapping the Data Warehouse to Multiprocessor Architecture-
More informationData Warehousing Introduction. Toon Calders
Data Warehousing Introduction Toon Calders toon.calders@ulb.ac.be Course Organization Lectures on Tuesday 14:00 and Friday 16:00 Check http://gehol.ulb.ac.be/ for room Most exercises in computer class
More informationAdvanced Data Management Technologies Written Exam
Advanced Data Management Technologies Written Exam 02.02.2016 First name Student number Last name Signature Instructions for Students Write your name, student number, and signature on the exam sheet. This
More informationDATABASE TECHNOLOGY. Spring An introduction to database systems
1 DATABASE TECHNOLOGY Spring 2007 An introduction to database systems Kjell Orsborn Uppsala Database Laboratory Department of Information Technology, Uppsala University, Uppsala, Sweden 2 Introduction
More informationDatabase System Concepts, 5th Ed.! Silberschatz, Korth and Sudarshan See for conditions on re-use "
Database System Concepts, 5th Ed.! Silberschatz, Korth and Sudarshan See www.db-book.com for conditions on re-use " Data Definition! Basic Query Structure! Set Operations! Aggregate Functions! Null Values!
More informationDATA WAREHOUSE EGCO321 DATABASE SYSTEMS KANAT POOLSAWASD DEPARTMENT OF COMPUTER ENGINEERING MAHIDOL UNIVERSITY
DATA WAREHOUSE EGCO321 DATABASE SYSTEMS KANAT POOLSAWASD DEPARTMENT OF COMPUTER ENGINEERING MAHIDOL UNIVERSITY CHARACTERISTICS Data warehouse is a central repository for summarized and integrated data
More informationQQ Group
QQ Group: 617230453 1 Extended Relational-Algebra-Operations Generalized Projection Aggregate Functions Outer Join 2 Generalized Projection Extends the projection operation by allowing arithmetic functions
More informationData Mining & Analytics Data Mining Reference Model Data Warehouse Legal and Ethical Issues. Slides by Michael Hahsler
Data Mining & Analytics Data Mining Reference Model Data Warehouse Legal and Ethical Issues Slides by Michael Hahsler Data Mining & Analytics Analytics is the discovery and communication of meaningful
More informationSimple SQL Queries (2)
Simple SQL Queries (2) Review SQL the structured query language for relational databases DDL: data definition language DML: data manipulation language Create and maintain tables CMPT 354: Database I --
More informationData Warehouse and Data Mining
Data Warehouse and Data Mining Lecture No. 07 Terminologies Naeem Ahmed Email: naeemmahoto@gmail.com Department of Software Engineering Mehran Univeristy of Engineering and Technology Jamshoro Database
More informationAdnan YAZICI Computer Engineering Department
Data Warehouse Adnan YAZICI Computer Engineering Department Middle East Technical University, A.Yazici, 2010 Definition A data warehouse is a subject-oriented integrated time-variant nonvolatile collection
More informationImproving the Performance of OLAP Queries Using Families of Statistics Trees
Improving the Performance of OLAP Queries Using Families of Statistics Trees Joachim Hammer Dept. of Computer and Information Science University of Florida Lixin Fu Dept. of Mathematical Sciences University
More information