Data-Driven Driven Business Intelligence Systems: Parts I. Lecture Outline. Learning Objectives

Transcription

1 Data-Driven Driven Business Intelligence Systems: Parts I Week 5 Dr. Jocelyn San Pedro School of Information Management & Systems Monash University IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, 2004 Lecture Outline Data-driven BIS Data warehouse Data warehouse architectures Entity-Relationship Modelling Multi-dimensional Modelling Star Schema An Example: Retail Trading IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Learning Objectives At the end of this lecture, the students will Have better understanding of concepts, tools and technology underlying data-driven business intelligence systems Have knowledge of multidimensional modelling and star schema for data modelling for data warehouses IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

2 Data-Driven Business Intelligence Systems IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data-Driven Driven BIS Data-driven BIS information systems that provide BI through access and manipulation of large databases of structured data includes tools for drill down for more detailed information drill up for broader, more summarised view slice and dice for a change in data dimensions IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data-Driven Driven BIS IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

3 Data-Driven Driven BIS IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data-Driven Driven BIS Slicing the cube $225,764 Peacock Leverling Fuller Davolio Callahan Suyama King Dodsworth Buchanan $201,196 $162,504 $182,500 $123,033 $72,528 $116,963 $75,048 $68,792 Salesperson Time IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, P r o d u c t Data-Driven Driven BIS Dicing the cube $225,764 $201,196 $162,504 $182,500 $123,033 $72,528 $116,963 $75,048 $68,792 Peacock Leverling Fuller Davolio Callahan Suyaman King Dodsworth Buchanan $22,719 $6,858 $16,035 Q1 Q2 Q3 Q4 IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, $23,181

4 Data Warehouse IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data Warehouse A data warehouse is a subject-oriented, integrated, time-variant, nonvolatile collection of data in support of management s decision making process Bill Inmon (1995) Subject-oriented: focus is on subjects related to business or organisational activity like customers, employees, suppliers (instead of applications-oriented (finance, marketing, production) Integrated: data from various databases is stored in a consistent format through use of naming conventions, domain constraints, physical attributes and measurements Time-variant: associating data with specific points in time Nonvolatile: data does not change once it is in the data warehouse and stored in data warehouse IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data Warehouse Data warehouse is a copy of transaction data specifically structured for query and analysis Ralph Kimball (1996) Data warehouse is a specific database designed and populated to provide decision support in an organisation - Gray and Watson (1998) IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

5 Data Warehouse Data warehousing emerged as result of improvements in database technology relational data model and relational database management systems (DBMS) advances in computer hardware - emergence of affordable mass storage and parallel computer architectures emergence of end-user computing, facilitated by powerful, intuitive computer interfaces and tools advances in middleware products that enable enterprise database connectivity across heterogeneous platforms IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data Warehouse triggered by recognition of fundamental differences between operational (or production) systems and informational (or decision support) systems Operational system system that is used to run a business in real time, based on current data e.g. sales order processing, reservation systems, patient registration Informational systems designed to support decision making based on stable point-in-time or historical data; for complex read-only queries or data mining applications e.g. sales trend analysis, customer segmentation, human resources planning IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data Warehouse Comparison of Operational and informational Systems McFadden, Hoffer and Prescott 1999 Characteristic Operational Systems Informational Systems Primary purpose Type of data Primary users Scope of usage Design goal Run the business on a current basis Current representation of state of the business Clerks, salespersons, administrators Narrow vs. simple updates and queries performance Support managerial decision making Historical or point-in-time (snapshots) Managers, business analysts, customers Broad vs. complex queries and analysis Ease of access and use IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

6 Generic two-level architecture Source (file) Source database) Source (database) Source (database) Transformation and Integration Data warehouse IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Three-level architecture Source (file) Transformation and Integration Source (database) Source database) Source (database) Enterprise Data warehouse Selection and aggregation Data mart Data mart IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data mart a data warehouse that is limited in scope contains selected and summarised data to support specific decision support applications of specific end-user group e.g., marketing data mart, finance data mart IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

7 Three-layer data architecture Derived data Data mart Data mart metada Reconciled data Enterprise data warehouse EDW metadata Enterprise data model Operational data Operational systems Operational metadata IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Enterprise data model Presents a total picture explaining the data required by an organisation Must be developed prior to designing a data warehouse Entity-Relationship Models traditional approach in relational database design Multidimensional Models are commonly used in data warehouses and data marts for faster retrieval for querying and analysis IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Operational Data current or transient, not historical restricted in scope to a particular application poor quality not normalised (there are multi-valued attributes or repeating groups, partial dependencies, transitive dependencies in data relations) IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

8 Sample Operational Data from Northwind database IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Reconciled Data Detailed - rather than summarised Historical snapshots, periodic Comprehensive should reflect enterprise-wide perspective; conform to enterprise data model Quality controlled Normalised 3NF or higher 3NF no multi-valued attributes, no partial dependencies, no transitive dependencies IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Steps in Normalisation Table with multi-valued attributes 1 st Normal Form 2 nd Normal Form 3 rd Normal Form Remove multivalued attributes Remove partial dependencies Remove transitive dependencies IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

9 Sales relation with sample data Relation in 3NF Salesperson Region Cust_ID Name Salesperson Region Cust_ID Name Salesperson IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Derived Data selected, formatted, aggregated provides ease of use for decision support applications provides fast response for user queries supports ad-hoc queries and data mining applications data model commonly used is star schema IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Metadata data that describe the properties or characteristics of other data Operational metadata describe the data in various operational systems (as well as external data) that feed the EDW EDW metadata describe the reconciled data layer as well as the rules for transforming operational data to reconciled data Data mart metadata describe the data in derived data layer and rules of transforming reconciled data to derived data IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

10 Sample data description IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Sample data description IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Data Reconciliation Process Stage 1: Initial load, when EDW is first created Stage 2: Subsequent updates Steps in Data Reconciliation Process Capture extract relevant data from source/s IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

11 Scrub clean or upgrade the quality of raw data before transformation and loading (using pattern recognition, artificial intelligence techniques) Track and correct errors: misspelled names, erroneous birthdates, missing data; inconsistent data formats IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Transform - includes converting data format or representation from source to target system partitioning data according to predefined criteria aggregating data from detailed to summary level IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Load and Index Refresh mode filling the EDW by bulk rewriting of target data Update mode only changes in source data are written to the data warehouse; at periodic intervals, data warehouse is rewritten, replacing previous contents without overwriting or deleting previous contents Create necessary indexes IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

12 Entity-Relationship Modelling IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Entity-Relationship Modelling Customer Type groups Customer within Region makes contains Product Type groups Product in Sale located at Store within Period (based on Kimball (1996), p29, and Simsion-Bowles (1996), p2) IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Entity-Relationship Modelling Entities, attributes and relationships Rules of normalisation 3NF is typical Protection of integrity of database by avoiding anomalies Every logical thing is represented only once Separate consideration of logical and physical aspects IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

13 Entity-Relationship Modelling ER Model for the Northwind sample database IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Entity-Relationship Modelling Large numbers of tables Oracle Financials - 1,800; SAP 7 up to 8,000 Commonly used Feels natural once you get used to it Research shows that they are not easily understood by IT people Especially concepts like abstraction, generalisation, sub-types, etc. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

14 Multi-dimensional Modelling It is possible to conceptualise data as multi-dimensional Difficult to design Easy to use resulting reports Advocated by Ralph Kimball (see his manifesto, and a rebuttal, available on the web site). A logical design technique that seeks to present data in a standard framework that is intuitive and allows for highperformance access. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling An approach to database design that provides an easy to understand and navigate database The aim is to encourage understanding, exploration and learning Each number in a database has a set of associated attributes What it measures, what point of time it was created, what location its from, what product it s associated with, what promotion, etc. This makes the number meaningful. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling Each attribute associated with each number represents a dimension Measure, time, location, product, location, etc. Resulting views are easy to navigate and move around Slice and dice Report template IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

15 Multi-dimensional Modelling Example Widget Sales ($Million) One Dimension (State): Vic NSW QLD WA SA TA S State Two Dimensions (location x time): Vic NSW QLD WA SA TA State S Year IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling Three Dimensions (location x time x product): Year Vic NSW QLD WA SA TA S State Sprockets Widgets Flanges Gaskets Product IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling Usually talk about information spaces as cubes, or hypercubes, or n-cubes Resulting views of databases are easy to navigate and move around Slicing and dicing Report Template IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

16 Multi-dimensional Modelling Slicing and Dicing Select certain dimension values to examine a set of data: IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling Report Templates One template is produced for a set of slices Data changes, layout doesn t Location Drop Down Box Product Sales: Victoria, 2001 Year Drop Down Box Widgets Sprockets Flanges Gaskets IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Multi-dimensional Modelling From Traditional Relational to Multi-dimensional Typical relational data-base From Pilot Software OLAP White Paper Same data displayed in twodimensions Easy! (The key is to identify the continuous and discrete variables in the flat file.) IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

17 Star Schema IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Star Schema Used to implement dimensional analysis using standard relational database technology Very common in data warehousing Many variations Two components: Fact Table contains measurements of business, eg. sales, purchase order, shipment Dimension Tables stores the textual descriptions of the dimensions of the business, eg. product, customer, vendor, store. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Star Schema Fact tables store the hard data Dimension tables store all the information about our dimensions. The fact table has a many-to-one relationship with each dimension table Each dimension table has a primary key that appears as a foreign key in the fact table, whose primary key is a concatenation of all of the foreign keys. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

18 Star Schema Dimension tables in star schemas are denormalised resulting in: Fewer tables Simpler for users to navigate Reduced number of complex multi-join tables. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Star schema Customer Customer key Name Customer type Product Product key Product type weight Sale Time key Store key Customer key Product key Dollar sales Unit sales Store Store key Address Region Time Time key Day Month Legend: Primary Key Foreign key IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Snowflake schema Do not snowflake your dimensions, even if very large. If you do snowflake your dimensions, prepare to live with poor performance Kimball (1996) Customer Type Customer Product Type Product Sale Store Region Time IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

19 Star Schema Dimensions can be shared amongst fact tables. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Star Schema ER schemas are useful for data mapping to legacy systems and for integration of the data warehouse Star schemas are useful for the design of warehouse databases as they are efficient and easy to understand and use Allow relational databases to support multi-dimensional data cubes IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Star Schema Steps in the design process 1. Choose a business process 2. Choose the grain of the fact table Too fine > Oversized database Too large > Loss of meaningful information 3. Choose the dimensions 4. Choose the measured facts (usually numeric, additive quantities) 5. Complete the dimension tables Kimball (1996) IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

20 Extra steps in the design process 6. Determine strategy for slowly changing dimensions 7. Create aggregations and other physical storage components 8. Determine the historical duration of the database 9. Determine the urgency with which the data is to be extracted and loaded into the data warehouse. Kimball (1996) IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, An Example: Retail Trading A large grocery store with approx. 500 stores Each store has approx. 60,000 products on shelves Need to maximise profit and keep shelves stocked Important decisions concern pricing and promotion Promotion types are: Temporary price reductions Newspaper advertisements Shelf and end-aisle displays Coupons IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, An Example: Retail Trading 1. Choose a Business Process Daily Item Movement 2. Choose the grain of the fact table Stock Keeping Unit (SKU) by store by promotion by day 3. Choose the Dimensions Time, product, store and promotion IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

21 An Example: Retail Trading Retail Trading Dimensions Promotion Promotion key Other Promotion attributes Product Product key Other product attributes Sale Time key Product key Store Key Promotion Key Facts to be detailed next Store Store key Other Store Attributes Time Time key Other Time Attributes IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, An Example: Retail Trading 4. Choose the measured facts Promotion Promotion key Other Promotion attributes Product Product key Other product attributes Sale Time key Product key Store Key Promotion Key Dollar Sales Unit Sales Dollar Costs Customer Count Store Store key Other Store Attributes Time Time key Other Time Attributes IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, An Example: Retail Trading 5. Complete the dimension tables Promotion Promotion key Other Promotion attributes Product Product key SKU Description SKU Number Package Size Brand Sub Category Department Package Type Diet Type Weight Weight unit of measure Units per retail case Units per ship case Cases per pallet Sale Time key Product key Store Key Promotion Key Dollar Sales Unit Sales Dollar Costs Customer Count Time Time key Other Time Attributes Store Store key Other Store Attributes IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,

22 References Inmon, W. H. (1996) Building the Data Warehouse (2 nd ed), Wiley, NY. Kimball, R. (1996) The Data Warehouse Toolkit, Wiley, NY. McFadden, F., Hoffer, J. and Prescott, M. (1999) Modern Database Management, Addison-Wesley. IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1, Questions? School of Information Management and Systems, Monash University T1.28, T Block, Caulfield Campus IMS3001 BUSINESS INTELLIGENCE SYSTEMS SEM 1,