UNIT 4. DATA WAREHOUSING

Transcription

1 UNIT 4. DATA WAREHOUSING Data Warehousing Components -Multi Dimensional Data Model- Data Warehouse Architecture-Data Warehouse Implementation- -Mapping the Data Warehouse to Multiprocessor Architecture- OLAP.-Need- Categorization of OLAP Tools. 4.1 Data Warehousing Components Definition: A data warehouse is a subject- oriented, integrated, time- Variant and non-volatile collection of data in support of management decision making process. Subject-Oriented: Stored data targets specific subjects. Example:It may store data regarding total Sales, Number of Customers, etc. and not general data on everyday operations. Integrated:Data may be distributed across heterogeneous sources which haveto be integrated. Example: Sales data may be on RDB, Customer information on Flatfiles, etc. Time Variant:Data stored may not be current but varies with time and data have an element of time. Example: Data of sales in last 5 years, etc. Non-Volatile:It is separate from the Enterprise Operational Database and hence is not subject to frequent modification. It generally has only 2operations performed on it: Loading of dataand Access of data Features of a Warehouse: It is separate from Operational Database. Integrates data from heterogeneous systems. Stores HUGE amount of data, more historical than current data. Does not require data to be highly accurate. Queries are generally complex.

2 Goal is to execute statistical queries and provide results whichcan influence decision making in favor of the Enterprise. These systems are thus called Online Analytical Processing Systems (OLAP) Three Main reasons for having a separate data base 1. OLTP systems require high concurrency, reliability, locking which provide good performance for short and simple OLTP queries. An OLAP query is very complex and does not require these properties. Use of OLAP query on OLTP system degrades its performance. 2. An OLAP query reads HUGE amount of data and generates the required result. The query is very complex too. Thus special primitives have to provide to support this kind of data access. 3. OLAP systems access historical data and not current volatile data while OLTP systems access current up-to-date data and do not need historical data Difference between OLTP and OLAP Feature OLTP OLAP users clerk, IT professional knowledge worker function day to day operations decision support DB design application-oriented subject-oriented data current, up-to-date detailed, flat relational isolated historical, summarized, multidimensional integrated, consolidated usage repetitive ad-hoc access read/write lots of scans index/hash on prim. key unit of work short, simple transaction complex query # records accessed tens millions #users thousands hundreds DB size 100MB-GB 100GB-TB metric transaction throughput query throughput, response

3 4.1.5 How do we represent this data??? This multi-dimensional data can be represented using a data cube as shown below. Or This figure shows a 3-Dimensional datamodel. X Dimension : Item type Y Dimension : Time/Period Z Dimension : Location Each cell represents the items sold of type x, in location z during the quarter y. Data cube is thus a n -dimensional data model

4 4.1.6 Schemas used for Multidimensional model Star schema There is a central large Fact table with no redundancy Each tuple in the fact table has a foreign key to a dimension table which describes the details of that dimension Problem: Redundancy Values of city, province_or_state and country would be repeated for two streets in the same city.thus we can normalize the table by splitting location into sub tables. (Snowflake Schema) Advantage: PerformanceAs less number of joins required Snowflake schema Some of the dimension tables are normalized thus splitting data into additional tables.

5 Problem: Performance Too many joins required to form the result. Thus Snowflake schema is not as popular as the Star schema. Fact Constellation schema Two or more fact tables share dimension tables. In the figure below the Sales fact table and Shipping fact table Share the dimension tables As As the multiple fact tables are linked to each other by dimension tables, its called as fact Constellation schema Data Mining Query Language (DMQL) Syntax: define cube <cube name>[<dimension list>]:<measure list> define dimension <dimension name> as (<attribute list>) Star Schema Example: Fact Table: define cube sales_star [time,item,branch,location]: dollars_sold=sum(sales_in_dollars),units_sold=count(*) Dimensions: define dimension time as (time_key,day,day_of_week,month,quarter,year)

6 define dimension item as (item_key,item_name,brand,type) define dimension branch as (branch_key,branch_name,branch_type) define dimension location as (location_key,street,city,province,country) Defining a hierarchy of dimension tables for snowflake schema. define dimension location as (location_key,street,city(city_key,city,province,country)) Defining a shared dimension table for Fact Constellation Schema define dimension time as (time_key,day,day_of_week,month,quarter,year) define dimension time as time in cube sales 4.2 Data warehousing architecture Approaches for data warehousing 1. Top-Down Approach Starts with overall design and planning Useful where the technology is mature and well known Useful where the business problems to be solved are clear and well understood 2. Bottom-Up Approach Starts with experiment and prototypes Allows an organization to move forward at considerably less expense Allows to evaluate the benefits of the technology before making significant commitments 3. Combined Approach Exploits the planned and strategic nature of the top-down approach Retains the rapid implementation and opportunistic application of the bottom-up approach Four views of Data warehouse Top-down, bottom-up approaches or a combination of both Top-down: Starts with overall design and planning (mature) Bottom-up: Starts with experiments and prototypes (rapid) From software engineering point of view

7 Waterfall: structured and systematic analysis at each step before proceeding to the next Spiral: rapid generation of increasingly functional systems, short turn around time, quick turn around Typical data warehouse design process Choose a business process to model, e.g., orders, invoices, etc. Choose the grain (atomic level of data) of the business process Choose the dimensions that will apply to each fact table record Choose the measure that will populate each fact table record Three tier architecture Data Mart Contains a subset of corporate-wide data that is of value to a specific group of users. Scope is confined to specific selected subjects Implemented on low-cost departmental servers that are UNIX or Windows NT based. Implementation cycle is measured in weeks rather than months or years They are further classified as: Independent Sourced from data captured from one or more operational systems or external information providers Sourced from data generated locally within a particular department or geographic area Dependent Sourced directly from enterprise data warehouses

8 Virtual Warehouse It is a Set of Views over operational databases For efficient query processing; only some of the possible summary views may be materialized It is easy to build but requires excess capacity on operational database servers Data Warehouse Development: A Recommended Approach Distributed Data Marts Multi-Tier Data Warehouse Data Mart Data Mart Enterprise Data Warehouse Model refinement Model refinement Define a high-level corporate data model August 17, 2010 Data Mining: Concepts and Techniques Data Warehouse Back-End Tools and Utilities Data extraction get data from multiple, heterogeneous, and external sources Data cleaning detect errors in the data and rectify them when possible Data transformation convert data from legacy or host format to warehouse format Load sort, summarize, consolidate, compute views, check integrity, and build indicies and partitions Refresh propagate the updates from the data sources to the warehouse

9 4.2.4 Metadata Repository Meta data is the data defining warehouse objects. It stores: Description of the structure of the data warehouse schema, view, dimensions, hierarchies, derived data defn, data mart locations and contents Operational meta-data data lineage (history of migrated data and transformation path), currency of data (active, archived, or purged), monitoring information (warehouse usage statistics, error reports, audit trails) The algorithms used for summarization The mapping from operational environment to the data warehouse Data related to system performance Warehouse schema, view and derived data definitions Business data Business terms and definitions, ownership of data, charging policies OLAP Server Architectures Relational OLAP (ROLAP) Use relational or extended-relational DBMS to store and manage warehouse data and OLAP middle ware Include optimization of DBMS backend, implementation of aggregation navigation logic, and additional tools and services Greater scalability Multidimensional OLAP (MOLAP) Sparse array-based multidimensional storage engine Fast indexing to pre-computed summarized data Hybrid OLAP (HOLAP) (e.g., Microsoft SQLServer) Flexibility, e.g., low level: relational, high-level: array Specialized SQL servers (e.g., Redbricks) Specialized support for SQL queries over star/snowflake schemas OLAP operations 1. Roll-up Performs aggregation on a data cube, either by climbing up a concept hierarchy for a dimension or by dimension reduction 2. Drill-down Can be realized by either stepping down a concept hierarchy for a dimension or introducing additional dimensions 3.Slice and Dice Slice performs a selection on one dimension of the given cube, resulting in a sub cube Dice defines a subcube by performing a selection on two or more dimensions

10 4.Pivot (rotate) It s a visualization operation that rotates the data axes in view in order to provide an alternative presentation of the data location (cities) Toronto 395 Vancouver time (quarters) Q1 Q2 605 computer home entertainment item (types) dice for (location = Toronto or Vancouver ) and (time = Q1 or Q2 ) and (item = home entertainment or computer ) location (countries) USA 2000 Canada time (quarters) roll-up on location (from cities to countries) Q Q2 Q3 Q4 computer home entertainment phone item (types) security location (cities) item (types) Chicago New York Toronto Vancouver home entertainment computer phone security computer home entertainment item (types) Chicago pivot phone New York security Toronto location (cities) Vancouver slice for time = Q1 Chicago 440 New York Toronto Vancouver Q1 Q2 Q3 Q location (cities) time (quarters) computer home entertainment phone item (types) security time (months) Chicago New York Toronto Vancouver January February March April May June July August September October November December location (cities) drill-down on time (from quarters to months) computer security home phone entertainment item (types)

11 4.2.7 Concept hierarchies It is a sequence of mappings from a set of low-level concepts to higher-level, more general concepts Data warehouse Implementation Cube: A Lattice of Cuboids

12 all 0-D(apex) cuboid time item location supplier 1-D cuboids time,location item,location location,supplier time,supplier item,supplier 2-D cuboids time,location,supplier 3-D cuboids time,item,supplier item,location,supplier 4-D(base) cuboid Mining query Mining result Layer4 User Interface User GUI API OLAM Engine OLAP Engine Layer3 OLAP/OLAM Data Cube API MDDB Layer2 MDDB Meta Data Filtering&Integration Databases Database API Data Data integration Filtering Data Warehouse Layer1 Data Repository Review Key terms OLAP Rollup Concept hierarchies

13 ROLAP Drill down Metadata Repository Data cube pivot Data mart cuboid slice Dice Multiple choice questions 1) Which of the following is not a characteristic of data warehouse? (a)volatile (b)subject oriented (c)integrated (d)time variant 2) Data warehousing uses approach for storing and direct query analysis. (a)query driven (b)update driven (c)bottom up (d)spiral 3) The two operations of data warehousing are (a)initial loading of data (b)access of data (c)decision support (d)a and b 4) The fact table contains (a)measures (b)dimensions (c)values (d)all of the above 5) The lattice of cubboids can be formed by of data. (a)groupby (b)integrated (c)minimum (d)maximum 6) Which schema is the best one? (a)star (b)snowflake (c)fact constelation (d)all of the above 7) sum is related with measures (a)distributive (b)holistic (c)algebraic (d)none of the above 8) avg()is related with measures (a)distributive (b)algebraic (c)holistic (d)all of the above 9) Sequence of mappings from low level concepts to high level concepts is called as (a)distributive (b)concept hierarchy (c)conversion (d)none of the above 10) One or more dimensions can be removed from data cube by (a)roll-up (b)drill down (c)slice (d)dice 11) Selection of one dimension is called as (a)rollup (b)drill down (c)slice (d)dice 12) HOLAP means (a)hash OLAP (b)hybrid OLAP (c)high OLAP (d)all of the above 13) The cuboid contains all the dimensions (a)base (b)apex (c)3d cuboid (d)all of the above

14 14) Which materialization is the best? (a)partial (b)full (c)all of the above (d)none of the above 15) The method is popular in OLAP products for quick searching in data cubes (a)materialization (b)meta data repository (c)bit map indexing (d)data mart 16) commercial tools for discrepancy detection (a)data scrubbing tools (b)data mining tools (c)data auditing tools 17) Attribute construction is also known as (a)feature construction (b) Feature selection (c) attribute selection (d) none of the above 18) In irrelevant attributes can be detected and removed. (a) Attribute selection (b) Data transformation (c) data integration cleaning (d) data Review questions PART A 1. Define data warehouse 2. Define the characteristics of data warehouse 3. Define a Data mart? 4. Define enterprise data warehouse 5. What are the schemas used for multidimensional model? 6. What is the disadvantage of star schema? 7. What is the disadvantage of snowflake schema? 8. What is data warehouse performance issue? 9. What is Data Inconsistency Cleaning? 10. Merits of Data warehouse. 11. What are the characteristics of metadata repository? 12. List some of the data warehouse tools. 13. Define concept hierarchy. 14. Explain OLAP. 15. Explain ROLAP. 16. Explain MOLAP. 17. Explain HOLAP. 18. Define specialized SQL server. PART B 1. Explain about data warehousing architecture.

15 2. Discuss in detail. OLAP operations. 3. Describe about concept hierarchies. 4. What are Schemas used for Multidimensional model? 5. Discuss about data warehouse implementation. REFERENCES 1. Jiawei Han, Micheline Kamber, "Data Mining: Concepts and Techniques", Morgan Kaufmann Publishers, Alex Berson,Stephen J. Smith, Data Warehousing, Data Mining,& OLAP, Tata McGraw- Hill, Usama M.Fayyad, Gregory Piatetsky - Shapiro, Padhrai Smyth And Ramasamy Uthurusamy, "Advances In Knowledge Discovery And Data Mining", The M.I.T Press, Ralph Kimball, "The Data Warehouse Life Cycle Toolkit", John Wiley & Sons Inc., Sean Kelly, "Data Warehousing In Action", John Wiley & Sons Inc., For Further references