STRATEGIC INFORMATION SYSTEMS IV STV401T / B BTIP05 / BTIX05 - BTECH DEPARTMENT OF INFORMATICS. By: Dr. Tendani J. Lavhengwa

Transcription

1 STRATEGIC INFORMATION SYSTEMS IV STV401T / B BTIP05 / BTIX05 - BTECH DEPARTMENT OF INFORMATICS LECTURE: 05 (A) DATA WAREHOUSING (DW) By: Dr. Tendani J. Lavhengwa lavhengwatj@tut.ac.za 1

2 My personal quote: Inspirational Quotes Always be a thought ahead. Do not fear the blank page, everything started somewhere Quotes to consider as inspiration: "Errors using inadequate data are much less than those using no data at all" ~ Charles Babbage One is too small a number to achieve greatness ~ John C. Maxwell Your quotes????

3 #. Start-up Items to discuss 1. Literature in context and evolution 2. Data Warehousing (DW) - multiple definitions 3. Data Warehousing concepts 4. Data Warehousing fundamental characteristics 5. Key points from business on Data warehouses (IBM, 2018) 6. Traditional integration, Data Warehouse vs. Operational DBMS 7. OLTP systems vs. Data Warehouse 8. Application-Orientation vs. Subject-Orientation 9. Data Warehouse Models 10. Modelling of Data Warehouse dimensions and measures 11. Organising data for Data Warehouses 12. Data Mart Centric 13. Data Warehouse Architecture - Base 14. Extract, Transform and Load (ETL) process

4 data warehouse architecture 1. Literature in context and evolution -was born in the 1980s as an architectural model designed to support the flow of data from operational systems to decision support systems Data Warehousing really saw its genesis in the late 1980s. An IBM Systems Journal article published in 1988, An architecture for a business information system, coined the term business data warehouse, although a future progenitor of the practice, Bill Inmon, used a similar term in the 1970s. Later in the 1990s Inmon developed the concept of the Corporate Information Factory, an enterprise level view of an organization s data of which Data Warehousing plays one part. Russom (2015) Data warehouse architecture is being influenced by business practices and goals that continue to evolve. The reason: a well-aligned data warehouse reflects the business it serves.

5 2. Data Warehousing (DW) - multiple definitions Turban et al. (2011) -a pool of data produced to support decision making -a repository of current and historical data of potential interest to managers througout the organisation Bocij et al. (2015) Large database systems containing current and historical data that can be analysed to produce information to support organisational decision making IBM.com (2018) databases provide a decision support system (DSS) environment in which you can evaluate the performance of an entire enterprise over time William H. Inmon... -a subject-oriented, integrated, time-variant and nonvolatile collection of data that supports management's decisionmaking process. Others... -a collection of corporate information and data derived from operational systems and external data sources. -designed to support business decisions by allowing data consolidation, analysis and reporting at different aggregate levels. -a federated repository for all the data that an enterprise's various business systems collect. -the Data Warehouse repository may be physical or logical.

6 3. Data Warehousing concepts Data Warehouses are aimed at decision making Data is populated into the DW through the processes of extraction, transformation and loading. Data warehouse databases are optimized for data retrieval. extraction, transformation and loading (ETL) - add figure

7 4. Data Warehousing fundamental characteristics (1 of 2) -subject orientated - -- data organised by detailed subject, only relevant for decision support --eg. sales, products, customer -integrated - --is closed related to subject orientation -- DW must place data from different sources into a consistent format --presumed to be totally integrated -time variant - --maintains historical data -the data does not necessarily provide current status (except for real-time systems) -they detect trends, deviations and long term relationships for forecasting and comparisons leading to decision making -non-volatile - --once data is on the DW, users cannot change or update the data --Obsolete data are discarded and changes are recorded as new data

8 4. Data Warehousing fundamental characteristics (2 of 2) Additional characteristics -Web based -Relational / Multidimensional -client / server -Real-time -Include metadata

9 5. Key points from business on Data warehouses (IBM, 2018) -A database that is optimized for data retrieval to facilitate reporting and analysis. -A data warehouse incorporates information about many subject areas, often the entire enterprise. -Typically you use a dimensional data model to design a data warehouse. -The data is organized into dimension tables and fact tables using star and snowflake schemas. -The data is denormalized to improve query performance.

10 6. Traditional integration, Data Warehouse vs. Operational DBMS Traditional heterogeneous DB integration A query driven approach Data Warehouse vs. Operational DBMS Data Warehouse: update-driven, high performance

11 7. OLTP systems vs. Data Warehouse

12 8. Application-Orientation vs. Subject-Orientation

13 9. Data Warehouse Models Enterprise Warehouse Data Mart Virtual Warehouse Collects all the information about subjects in the entire organisation/enterprise A subset of corporate-wide data that is of value to a specific group of users Example: Marketing, Sales, Finance A set of views over operational databases Only some of the possible summary views may be materialised

14 10. Modelling of Data Warehouse dimensions and measures Start schema --a fact table in the middle connected to a set of dimension tables Snowflake schema --a refinement of star schema where some dimensional hierarchy is normalised into a set of smaller dimension tables, forming a shape similar to snowflake -Fact constellations --multiple fact tables share dimension tables, viewed as a collection of stars, therefore called galaxy schema or fact constellation

15 11. Organising data for Data Warehouses The data is organized into: -dimension tables -fact tables (using star and snowflake schemas) sample snowflake schema with DAILY_SALES table as the fact table data mart with the DAILY_SALES fact table

16 12. Data Mart Centric

17 13. Data Warehouse Architecture - Base Two-Tier Data Warehouse Architecture Three-Tier Data Warehouse Architecture Web-based Data Warehouse Architecture

18 14. Extract, Transform and Load (ETL) process a process in database usage and especially in data warehousing Some activities carried out at "Transforming" stage: Cleaning (e.g. Male to M and Female to F etc.) Filtering (e.g. selecting only certain columns to load) Enriching (e.g. Full name to First Name, Middle Name, Last Name) Splitting a column into multiple columns and vice versa Joining together data from multiple sources Extracts data from homogeneous or heterogeneous data sources Extracting the data from different sources the data sources can be files (like CSV, JSON, XML) or RDBMS etc Transforms the data for storing it in proper format or structure for querying and analysis purpose Transforming the data this may involve cleaning, filtering, validating and applying business rules Loads it into the final target (database, more specifically, operational data store, data mart, or data warehouse) Loading data is loaded into a data warehouse or any other database or application that houses data

19 LECTURE: 05 (A) - DATA WAREHOUSING (DW) --- QUESTIONS & ENQUIRIES LAVHENGWATJ@TUT.AC.ZA 19