Knowledge Modelling and Management. Part B (9)

Transcription

1 Knowledge Modelling and Management Part B (9) Yun-Heh Chen-Burger 1

2 A Brief Introduction to Business Intelligence 2

3 What is Business Intelligence? Business Intelligence is an umbrella term that combines architectures, tools, databases, analytical tools, applications and methodologies. It means different things to different people. Its main objective is to enable interactive access to data, to enable manipulation of data, and to give business managers and analysts the ability to conduct appropriate analysis. By analysing historical and current data, situations and performances, decision makers get valuable insights that enable them to make more informed and better decisions. The process of BI is based on the transformation of data to information, then to decisions and finally to actions. 3

4 History of BI Term firstly coined by the Gartner Group in mid-1990s, but it was rooted from the MIS (management information system) where information used to be static, 2 dimensional and no analytical abilities. From 1980s, Executive Information Systems (EIS) start to introduce dynamic multi-dimensional reporting, forecasting, prediction, trend analysis, drill down to details, status access and critical success factors. By 2005, AI capabilities are included in BI systems. 4

5 BI Technologies Financial Reporting OLAP Querying and Reporting EIS/ESS Metadata Data warehouse Data marts ETL Spreadsheets (MS Excel) DSS Digital Cockpits And dashboards Scorecards and dashboards workflow Business Intelligence portals Broadcasting tools Process mining Business performance mgmt Alerts and Notifications Data and text mining Predictive analytics 5

6 Glossary ETL - Extract, transform, and load (ETL) is a process in database usage and especially in data warehousing that involves: Extracting data from outside sources; Transforming it to fit operational needs (which can include quality levels); Loading it into the end target (database or data warehouse). A data mart (DM) is the access layer of the data warehouse (DW) environment that is used to get data out to the users. The DM is a subset of the DW, usually oriented to a specific business line or team. EIS is sometimes also called ESS (Executive Support System) a specialised decision support system for top executives while making strategic decisions. Example users are members of board of directors and executives, e.g. CEO, COO (Chief Operation Officer), CFO, etc. The goal of information broadcasting is to distribute the right information, in the appropriate format, to the right people, through different channels, at the right time. OLAP on-line analytical processing makes use of background knowledge regarding the domain of the data being studied in order to allow the presentation of data at different levels of abstraction to allow interactive analysis of multi-dimensional data. Such operations accommodate different user view points. Example operations are drill-down (details) and roll-up (summary), e.g. sales data based on country, city, quarter, month to day to day. 6

7 Business Performance Management An outgrowth of BI; BI + planning Uses BI technologies, applications and techniques BI includes technologies used to access, analyse and report on data relevant to an enterprise; it can be departmental or application area focused BPM use BI to enable enterprise-wise planning, monitoring and analysing; to prevent local optimisation at the expenses of overall enterprise optimisation. Makes use of existing planning, monitoring and analysing business processes; but integrate them and their data to achieve performance targets, execute enterprise strategies and deliver value to stakeholders. 7

8 Data Warehouse - 1 A corner stone of any medium-to-large BI system. Originally includes only historical data; today some systems also include current data to provide real-time decision support. A repository of multi-heterogeneous data sources, organised under a unified schema at a single site to facilitate decision making. A database that is maintained separately from an organisation s operational databases. They support information processing by providing a solid platform of consolidated historical data for analysis. [2] A subject-oriented, integrated, time-variant and non-volatile collection of data in support of management s decision making process. [2] Facilities include data cleansing, data integration and OLAP (on-line analytical processing) analysis techniques such as summarisation, consolidation, aggregation, view information from different angles. Often seem as an important pre-processing step for data mining. 8

9 Multi-Dimensional Data Model The multidimensional data model is an integral part of On- Line Analytical Processing, or OLAP. Because OLAP is on-line, it must provide answers quickly; analysts pose iterative queries during interactive sessions, not in batch jobs that run overnight. And because OLAP is also analytic, the queries are complex. The multidimensional data model is designed to solve complex queries in real time. The multidimensional data model is composed of logical cubes, measures, dimensions, hierarchies, levels, and attributes. The simplicity of the model is inherent because it defines objects that represent real-world business entities. Analysts know which business measures they are interested in examining, which dimensions and attributes make the data meaningful, and how the dimensions of their business are organized into levels and hierarchies. [source: Standard University] 9

10 Example Multi-Dimensional Data Model 10

11 Data Warehouse - 2 Subject-oriented - A data warehouse is organised around major subjects, such as customers, supplier, product and sales. Rather than concentrating on the day-to-day operations and transaction processing of an organisation, a data warehouse focuses on the modelling and analysis of data for decision makers. Hence, data warehouses typically provide a simple and concise view around particular subject issues by excluding data that are not useful in the decision support process. Integrated A data ware house is usually constructed by integrating multi-heterogeneous sources, such as relational database, flat files, and on-line transaction records. Data cleaning and data integration techniques are applied to ensure consistency in, e.g. naming conventions, encoding structures, attribute measures, etc. Time-variant Data are often stored to provide information from a historical perspective, e.g. the past 5-10 years. Every key structure in the data warehouse contains, either implicitly or explicitly, an element of time. Non-volatile a data warehouse is always a physically separate store of data transformed from the application data found in the operational environment. Due to this separation, a data warehouse does not require transaction processing, recovery and concurrency control mechanism. It usually requires only two operations in data accessing: initial loading of data and access of data. 11

12 What is Data Mining? Extracting or mining knowledge from large amounts of data knowledge mining Knowledge Discovery in Databases (KDD) The process that finds a small set of precious nuggets from a great deal of raw material. The process of discovering interesting knowledge from large amounts of data stored either in databases, data warehouses or other information repositories. Steps including: data cleaning, integration, selection, transformation, mining, pattern evaluation and knowledge presentation. [2] 12

13 Data Minding Process Step 1 Business Understanding to know what the study is for require explicit specifications of business objectives: e.g. What are the common characteristics of the customers we have lost to our competitors recently? What are the typical profiles of our customers and how much value does each of them provide to us? A project plan should be formed, together with people responsible for data collection, analysing and reporting, as well as a budget allocated for the project. 13

14 Data Minding Process Step 2 Data Understanding DM is to address a welldefined business task that require specific sets of data. Acquire a well-defined specification for DM tasks Identify relevant data from many available databases What is the data collection process: who collects them? How often is it updated? What are the most relevant variables are they independent of each other? Is there any synonymous or homonymous variables? Are these variables complete information? Are they overlapping or conflicting information? 14

15 Data Minding Process Step 3 Data Preparation (or pre-processing) Takes data collected in step 2 and prepare them for DM tools Time-consuming costs about 80% of DM project Real-world data is often incomplete (lack of attribute values, attributes of interests, contains only aggregated data); Noises; Inconsistencies (in codes or names) Steps: data consolidation collect/select/integrate data data cleaning add missing value, reduce noises, eliminate inconsistency data transformation normalise, discretise/aggregate data, construct new attributes data reduction reduce number of variables (dimensional reduction) and cases; choosing a subset for analysis (representational sampling); balance skewed data (stratified sampling) 15

16 Data Minding Process Step 4 Model Building select several modelling techniques; assess and comparative analysis of models built to select a best method; experiment with parameters to get optimal results May need to get back to data preparation step DM task types: Prediction: classification or regression Association Clustering 16

17 Data Minding Process Step 5 Testing and Evaluation Models are assessed for accuracy and generality Meet business objectives? If so, to what extent? Test against real-world scenarios Business values can be obtained/established from knowledge discovered by DM tasks Determine business value may difficult good communication with business analysts and decision makers are essential Tabulation and visualisation techniques are often used for communication, e.g. pivot tables, cross-tabulation of findings, pie charts, histograms, box plots and scatter plots. 17

18 Data Minding Process Step 6 Deployment Knowledge needs to be organised and presented so that the end user can understand and benefit from it May be report generating or another DM tasks Deployment is carried by the customer, not be DM modellers May include maintaining activities on the DM models Establish a carefully designed monitoring and maintenance strategy to help DM results up-todate. 18

19 Example Data Mining Applications Customer relationship management Banking Retailing and logistics Manufacturing and production Brokerage and securities trading Insurance Computer hardware and software Government and defence Travel industry Health care Medicine Entertainment industry Homeland security and law enforcement Sports 19

20 Reference [1] Business Intelligence - a managerial approach, by Efraim Turban, Ramesh Sharda, Dursun Delen and David King. Published by Pearson Education, Inc. Publishing as Prentice Hall. ISBN 13: Copyright (c) [2] Data Mining - Concepts and Techniques by Jiawei Han and Micheline Kamber, published by Morgan Kaufmann. 20

21 Additional Information 21

22 Data Mining Function - 1 Concept/class Description Data summary and characterisation a summary of the general characteristics of a target class of data, e.g. a study of the characteristics of software products whose sales increased by 10% last year. Can use SQL queries; OLAP roll-up operation; or attributeoriented induction. Data discrimination compare features of a target class of data to other (contracting) classes, as specified by the user, e.g. to compare the products whose sales went up 10% with those whose sales went down 30% last year. 22

23 Data Mining Function - 2 Association Analysis The discovery of association rules showing attribute value conditions that occur frequently together in a given set of data. Association rules are often logical rules:» X Y» A1 A2 A3. B1 B2 B3, where Ai and Bi are attribute values.» age(x, ) income(x, 20k-29k ) buys(x, CD player ) [support = 2%; confidence = 60%]» Support: 2% of all transactions are customers of age between 20-29; have income between 20-29k and have purchased a CD player.» Confidence: 60% probability that a customer who is of the above age group and income will purchase a CD player. 23

24 Data Mining Function - 3 Classification - the process of finding a set of models (or functions) that describe and distinguish data classes or concepts for the purpose of being able to use the model to predict the class of objects whose class label is unknown. Example techniques: IF-THEN rules, decision trees, mathematical formulae, neural network. Prediction often used to predict missing data values; may also include the prediction of distribution trends based on available data. Both of the above techniques need to be preceded by relevance analysis so can remove irrelevant attributes in advance. Example: a company wants to classify all products according to their sales campaign into three categories: good response, mild response and no response. 24

25 Data Mining Function - 4 Cluster Analysis The process of grouping data objects based on their features to discover class labels. Training data often does not have any class label to start with. Objects are often grouped based on the principle of maximising the intraclass similarity and minimising the interclass similarity. 25

26 Data Mining Function - 5 Outlier Analysis the discovery and analysis of irregular data (the outlier) the outlier mining. Techniques: Use statistical test that assume a distribution or probability model; or use distance measures where objects that are a substantial distance from other clusters are considered outlier. Example: Credit card fraud detection 26

27 Data Mining Function - 6 Evolution Analysis Describes and models regularities or trends for objects whose behaviour changes over time. Techniques: may use a combination of any of the previously mentioned techniques Also includes time-series data analysis, sequence or periodicity pattern matching and similarity based data analysis. Example: study of stock performance of a sector or a company 27

28 Example Pivot Table Table: Sales figures in shirts sold 28

29 Example Cross-table report 29