By Mahesh R. Sanghavi Associate professor, SNJB s KBJ CoE, Chandwad

Transcription

1 By Mahesh R. Sanghavi Associate professor, SNJB s KBJ CoE, Chandwad

2 All the content of these PPTs were taken from PPTS of renown author via internet. These PPTs are only mean to share the knowledge among the Students. I specially thanks to the authors and owner due to whom it is possible to prepare these PPTs.

3 Introduction: Data Warehouse Modeling Data Warehouse Design Data-ware-house Technology Distributed Data Warehouse Materialized view

4 A data warehouse is the foremost repository for the data available for developing business intelligence architectures and decision support systems. The term data warehousing indicates the whole set of interrelated activities involved in designing, implementing and using a data warehouse A data warehouse is a relational database for query and analysis. It contains historical data It uses OLAP(online analytical processing) engine It s Key features are as follows: subject-oriented, integrated, time-variant, nonvolatile February 3, 2016 Data Mining: Concepts and Techniques 4

5 Three main categories of data feeding into a data warehouse Internal Data Bac-koffice, front office, web-based External Data Collected from market survey, GIS etc. Personal Data BI Analyst will produce this data

6 OLTP (on-line transaction processing) Major task of traditional relational DBMS Day-to-day operations: purchasing, inventory, banking, manufacturing, payroll, registration, accounting, etc. OLAP (on-line analytical processing) Major task of data warehouse system Data analysis and decision making Distinct features (OLTP vs. OLAP): User and system orientation: year vs. market Data contents: current, detailed vs. historical, consolicityd Database design: ER + application vs. star + subject View: current, local vs. evolutionary, integrated Access patterns: upcity vs. read-only but complex queries February 3, 2016 Data Mining: Concepts and Techniques 6

7 OLTP OLAP users clerk, IT professional knowledge worker function day to day operations decision support DB design application-oriented subject-oriented data current, up-to-date detailed, flat relational isolated usage repetitive ad-hoc access read/write lots of scans index/hash on prim. key unit of work short, simple transaction complex query # records accessed tens millions #users thousands hundreds DB size 100MB-GB 100GB-TB historical, summarized, multidimensional integrated, consolidated metric transaction throughput query throughput, response February 3, 2016 Data Mining: Concepts and Techniques 7

8 Integration Quality Efficiency Extendibility Entity Oriented Integrated Time-variant Persistent Consolidated Denormalized

9 Data warehouse Subject oriented Large (hundreds of GB up to several TB) Historic data De-normalized table structure (few tables, many columns per table) Batch updates Usually very complex queries Operational system Transaction oriented Small (MB up to several GB) Current data Normalized table structure (many tables, few columns per table) Continuous updates Simple to complex queries

10 Data marts are systems that gather all the data required by a specific company department, such as marketing or logistics, for the purpose of performing business intelligence analyses and executing decision support applications specific to the function itself Therefore, a data mart can be considered as a functional or departmental data warehouse of a smaller size and a more specific type than the overall company data warehouse.

11 Contd.. Data warehouses and data marts thus share the same technological framework. In order to implement business intelligence applications, some companies prefer to design and develop in an incremental way a series of integrated data marts rather than a central data warehouse, in order to reduce the implementation time and uncertainties connected with the project.

12 Data Warehouse Enterprise-wide data multiple subject areas difficult to build takes more time to build larger memory Data Mart Department-wide data single subject area easy to build less time to build limited memory

13 Accuracy Completeness Consistency Timeliness Non-redundency Relevance Interoperability Accessibility Accessibility

14 The table above is an example of a crosstabulation (cross-tab), also referred to as a pivot-table.

15 Cross-tabs can be represented as relations We use the value all is used to represent aggregates The SQL:1999 standard actually uses null values in place of all despite confusion with regular null values

16 A data cube is a multidimensional generalization of a cross-tab Can have n dimensions; we show 3 below Cross-tabs can be used as views on a data cube

17 Modeling data warehouses: dimensions & measures Star schema: A fact table in the middle connected to a set of dimension tables Snowflake schema: A refinement of star schema where some dimensional hierarchy is normalized into a set of smaller dimension tables, forming a shape similar to snowflake Fact constellations: Multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation February 3, 2016 Data Mining: Concepts and Techniques 17

18 time time_key day day_of_the_week month quarter year branch branch_key branch_name branch_type Measures Sales Fact Table time_key item_key branch_key location_key units_sold dollars_sold avg_sales item item_key item_name brand type supplier_type location location_key street city state_or_province country February 3, 2016 Data Mining: Concepts and Techniques 18

19 time time_key day day_of_the_week month quarter year Sales Fact Table time_key item_key item item_key item_name brand type supplier_key supplier supplier_key supplier_type branch branch_key branch_name branch_type Measures branch_key location_key units_sold dollars_sold avg_sales location location_key street city_key city city_key city state_or_province country February 3, 2016 Data Mining: Concepts and Techniques 19

20 time time_key day day_of_the_week month quarter year Sales Fact Table time_key item_key branch_key item item_key item_name brand type supplier_type Shipping Fact Table time_key item_key shipper_key from_location branch branch_key branch_name branch_type Measures location_key units_sold dollars_sold avg_sales location location_key street city province_or_state country to_location dollars_cost units_shipped shipper shipper_key shipper_name location_key shipper_type February 3, 2016 Data Mining: Concepts and Techniques 20

21 all all region Europe... North_America country Germany... Spain Canada... Mexico city Frankfurt... Vancouver... Toronto office L. Chan... M. Wind February 3, 2016 Data Mining: Concepts and Techniques 21

22

23 Cross-tabs can be easily extended to deal with hierarchies Multi levels of hierarchy can be displayed in the cross-tab Can drill down or roll up on a hierarchy Figure 18.5 Figure 18.1 Cross tab without hierarchy

24 Country TV PC VCR sum city 1Qtr 2Qtr 3Qtr 4Qtr sum Total annual sales of TV in U.S.A. U.S.A Canada Mexico sum February 3, 2016 Data Mining: Concepts and Techniques 25

25 Roll up (drill-up): Drill down (roll down): Slice : Dice: Pivot (rotate): drill across: drill through: by climbing up hierarchy or by dimension reduction summarize data, perform aggregate on data cube from higher level summary to lower level summary or detailed data, or introducing new dimensions reverse of roll-up Selection of one dimension from cube, resulting in sub cube Defines sub cube by selecting two or more dimension reorient the cube, visualization, 3D to series of 2D planes It is a visualization operation involving (across) more than one fact table through the bottom level of the cube to its back-end relational tables (using SQL) February 3, 2016 Data Mining: Concepts and Techniques 26

26 February 3, 2016 Data Mining: Concepts and Techniques 27

27 Four views regarding the design of a data warehouse Top-down view Data source view Data warehouse view Business query view allows selection of the relevant information necessary for the data warehouse for future business needs exposes the information being captured, stored, and managed by operational systems consists of fact tables and dimension tables sees the perspectives of data in the warehouse from the view of end-user February 3, 2016 Data Mining: Concepts and Techniques 28

28 Choose a business process to model, e.g., orders, invoices, etc. Choose the grain of the business process. Grain is fundamental level of data to be represented in the fact table e.g. individual transaction, individual daily snap shot Choose the dimensions that will apply to each fact table record e.g. Item, year, supplier Choose the measure that will populate each fact table record like dollars_sold, unit_sold February 3, 2016 Data Mining: Concepts and Techniques 29

29 Top-down, bottom-up approaches or a combination of both Top-down: Starts with overall design and planning (mature) Bottom-up: Starts with experiments and prototypes (rapid) From software engineering point of view Waterfall: structured and systematic analysis at each step before proceeding to the next Spiral: rapid generation of increasingly functional systems, short turn around time, quick turn around February 3, 2016 Data Mining: Concepts and Techniques 30

30 Two approaches for Centralized Data Warehouse Inmon s (Top-Down) Kimball s (Bottom-Up) In a nutshell Bill Inmon s enterprise data warehouse approach (the top-down design): A normalized data model is designed first. Then the dimensional data marts, which contain data required for specific business processes or specific departments are created from the data warehouse. Ralph Kimball s dimensional design approach (the bottom-up design): The data marts facilitating reports and analysis are created first; these are then combined together to create a broad data warehouse.

31

32 Inmon defines the data warehouse in the following terms: 1. Subject-oriented: The data in the data warehouse is organized so that all the data elements relating to the same real-world event or object are linked together 2. Time-variant: The changes to the data in the database are tracked and recorded so that reports can be produced showing changes over time 3. Non-volatile: Data in the data warehouse is never over-written or deleted -- once committed, the data is static, read-only, and retained for future reporting 4. Integrated: The database contains data from most or all of an organization's operational applications, and that this data is made consistent

33

34 Data marts are created first. These provide a thin view into the organizational data, and as and when required these can be combined into a larger data warehouse. Kimball defines data warehouse as A copy of transaction data specifically structured for query and analysis Kimball s data warehousing architecture is also known as Data Warehouse Bus (BUS). Dimensional modeling focuses on ease of end user accessibility and provides a high level of performance to the data warehouse.

35

36 Other sources Operational DBs Metadata Extract Clean Transform Load Refresh Monitor & Integrator Data Warehouse OLAP Server Output Analysis Query Reports Data mining Data Marts Data Sources Bottom Tier Middle tier Top tier February 3, 2016 Data Mining: Concepts and Techniques 37

37 Data extraction get data from multiple, heterogeneous, and external sources Data cleaning Data transformation Load detect errors in the data and rectify them when possible convert data from legacy or host format to warehouse format sort, summarize, consolicity, compute views, check integrity, and build indicies and partitions Refresh propagate the upcitys from the data sources to the warehouse February 3, 2016 Data Mining: Concepts and Techniques 38

38 Meta data is the data defining warehouse objects. It stores: Description of the structure of the data warehouse schema, view, dimensions, hierarchies, derived data defn, data mart locations and contents Operational meta-data data lineage (history of migrated data and transformation path), currency of data (active, archived, or purged), monitoring information (warehouse usage statistics, error reports, audit trails) The algorithms used for summarization The mapping from operational environment to the data warehouse Data related to system performance Includes indices and profiles that improve data access Rules for timing and scheduling of refresh, upcity and replication Business metadata business terms and definitions, ownership of data, charging policies February 3, 2016 Data Mining: Concepts and Techniques 39

39 Bottom Tier Ware house database server Middle Tier OLAP Server (ROLAP or MOLAP) Top Tier (Front-end-client Layer) contains reporting tool, analysis tool or data mining tool February 3, 2016 Data Mining: Concepts and Techniques 40

40 Enterprise Data Warehouse (EDW) collects all of the information about subjects spanning the entire organization. It contains detailed data as well s summarized data and can range in size from gigabytes to terabytes Contains detailed data and require years to design and build Data Mart (DM) a subset of corporate-wide data that is of value to a specific groups of users. Its scope is confined to specific, selected groups, such as marketing data mart may confine its subjects to year, item and sale. It is in the summarized form and requires few weeks for implementation Data mart can be independent or dependent February 3, 2016 Data Mining: Concepts and Techniques 41

41 Virtual Data Warehouse (VDW) A set of views over operational databases For efficient query processing, only some of the possible summary views may be materialized Easy to build Requires excess capacity on operational database servers February 3, 2016 Data Mining: Concepts and Techniques 42

42 Relational OLAP (ROLAP) Use relational or extended-relational DBMS to store and manage warehouse data and OLAP middle ware Include optimization of DBMS backend, implementation of aggregation navigation logic, and additional tools and services Greater scalability Multidimensional OLAP (MOLAP) Sparse array-based multidimensional storage engine Fast indexing to pre-computed summarized data Hybrid OLAP (HOLAP) (e.g., Microsoft SQLServer) Flexibility, e.g., low level: relational, high-level: array Specialized SQL servers (e.g., Redbricks) Specialized support for SQL queries over star/snowflake schemas February 3, 2016 Data Mining: Concepts and Techniques 43

43 Distributed Data Marts Multi-Tier Data Warehouse Data Mart Data Mart Enterprise Data Warehouse Model refinement Model refinement Define a high-level corporate data model

44 Information processing supports querying, basic statistical analysis, and reporting using crosstabs, tables, charts and graphs Analytical processing multidimensional analysis of data warehouse data supports basic OLAP operations, slice-dice, drilling, pivoting Data mining knowledge discovery from hidden patterns supports associations, constructing analytical models, performing classification and prediction, and presenting the mining results using visualization tools

45 ETL Extraction Transformation Loading

46 February 3, 2016 Data Mining: Concepts and Techniques 47

47 February 3, 2016 Data Mining: Concepts and Techniques 48

48 To get data out of the source and load it into the data warehouse simply a process of copying data from one database to other Data is extracted from an OLTP database, transformed to match the data warehouse schema and loaded into the data warehouse database Many data warehouses also incorporate data from non-oltp systems such as text files, legacy systems, and spreadsheets; such data also requires extraction, transformation, and loading

49 Data is extracted from heterogeneous data sources Each data source has its distinct set of characteristics that need to be managed and integrated into the ETL system in order to effectively extract data.

50 ETL process needs to effectively integrate systems that have different: DBMS Operating Systems Hardware Communication protocols Need to have a logical data map before the physical data can be transformed The logical data map describes the relationship between the extreme starting points and the extreme ending points of your ETL system usually presented in a table or spreadsheet

51 Target Source Transformation Table Name Column Name Data Type Table Name Column Name Data Type The table type gives us our queue for the ordinal position of our data load processes first dimensions, then facts. The primary purpose of this document is to provide the ETL developer with a clear-cut blueprint of exactly what is expected from the ETL process. This table must depict, without question, the course of action involved in the transformation process The transformation can contain anything from the absolute solution to nothing at all. Most often, the transformation can be expressed in SQL. The SQL may or may not be the complete statement

52 Actually changes data and provides guidance whether data can be used for its intended purposes Performed in staging area

53 Correct Unambiguous Consistent Complete Data quality checks are run at 2 places - after extraction and after cleaning and confirming additional check are run at this point

54 Column Property Enforcement Null Values in required columns Numeric values that fall outside of expected high and lows Cols whose lengths are exceptionally short/long Cols with certain values outside of discrete valid value sets Adherence to a required pattern/ member of a set of pattern Spell check Outlier check

55 Structure Enforcement Tables have proper primary and foreign keys Obey referential integrity Data and Rule value enforcement Simple business rules Logical data checks

56 February 3, 2016 Data Mining: Concepts and Techniques 57

57 Stop Yes Staged Data Cleaning And Confirming Fatal Errors No Loading

58 Physically built to have the minimal sets of components The primary key is a single field containing meaningless unique integer Surrogate Keys The DW owns these keys and never allows any other entity to assign them De-normalized flat tables all attributes in a dimension must take on a single value in the presence of a dimension primary key. Should possess one or more other fields that compose the natural key of the dimension

59

60 The data loading module consists of all the steps required to administer slowly changing dimensions (SCD) and write the dimension to disk as a physical table in the proper dimensional format with correct primary keys, correct natural keys, and final descriptive attributes. Creating and assigning the surrogate keys occur in this module. Surrogate keys are keys that have no business meaning and are solely used to identify a record in the table. Such keys are either database generated (example: Identity in SQL Server, Sequence in Oracle, Sequence/Identity in DB2 UDB etc.) The table is definitely staged, since it is the object to be loaded into the presentation system of the data warehouse.

61 Efficient Data Cube Computation Cube operation Iceberg Indexing

62 Data cube can be viewed as a lattice of cuboids The bottom-most cuboid is the base cuboid The top-most cuboid (apex) contains only one cell How many cuboids in an n-dimensional cube with L levels? T n ( L i 1) i 1 Materialization of data cube Materialize every (cuboid) (full materialization), none (no materialization), or some (partial materialization) Selection of which cuboids to materialize Based on size, sharing, access frequency, etc. February 3, 2016 Data Mining: Concepts and Techniques 63

63 Cube definition and computation in DMQL define cube sales[item, city, year]: sum(sales_in_dollars) compute cube sales Transform it into a SQL-like language (with a new operator cube by, introduced by Gray et al. 96) SELECT item, city, year, SUM (amount) FROM SALES CUBE BY item, city, year Need compute the following Group-Bys (city, item, year), (city, Item),(city, year), (item, year), (city), (item), (year) () (city) (item) (year) (city, item) (city, year) (item, year) (city, item, year) February 3, 2016 Data Mining: Concepts and Techniques 64 ()

64 Computing only the cuboid cells whose count or other aggregates satisfying the condition like HAVING COUNT(*) >= minsup Motivation Only a small portion of cube cells may be above the water in a sparse cube Only calculate interesting cells data above certain threshold Avoid explosive growth of the cube Suppose 100 dimensions, only 1 base cell. How many aggregate cells if count >= 1? What about count >= 2? February 3, 2016 Data Mining: Concepts and Techniques 65

65 Index on a particular column Each value in the column has a bit vector: bit-op is fast The length of the bit vector: # of records in the base table The i-th bit is set if the i-th row of the base table has the value for the indexed column not suitable for high cardinality domains Base table Index on Region Index on Type Cust Region Type C1 Asia Retail C2 Europe Dealer C3 Asia Dealer C4 America Retail C5 Europe Dealer RecIDAsia Europe America RecID Retail Dealer February 3, 2016 Data Mining: Concepts and Techniques 66

66 Ab Initio DMExpress Talend Infosphere Datastage Informatica PowerCenter Oracle OWB and DI February 3, 2016 Data Mining: Concepts and Techniques 67

67 A centralized Data Warehouse is a single physical repository which contains integrated data extracted from external information system. Advantages : Security Ease of Management Experience Performance Expandability Reliability Vendor Dependency Disadvantages Server fails entire system get collapse May not serve for the Large Scale Application Costly

68 Site A Head Quarters Operational Processing Site B

69 Many organizations have physically distributed databases with extremely large amounts of data. Traditionally the data warehouse would be seen as a centralized repository, whereby data from all sources would be imported into that large centralized repository for analysis. Nowadays the speed and bandwidth of wide-area computer networks enables a distributed approach, whereby parts of the data may reside in different places, parts being cached and/or replicated for performance reasons, and the system functions to the outside world as a single global access-transparent repository. As the amount of data and number of sites grow, this distributed approach becomes crucial, as a single centralized data warehouse importing data from all the sources has obvious scalability limitations.

70 3 Types Local / Global Distributed DW Serving global businesses where there are local operations and a central opeations Technologically Distributed DW Where the volume of data is such that it is spread over multiple physical volumes Independently Evolving Distributed DW It grows-up in uncoordinated manner

71 Technical Requirement Managing large amount of data Managing multiple media Indexing & Monitoring Data Efficient loading of Data Efficient Index Utilization Data Compaction Lock Management Fast Restore