QUALITY MONITORING AND

Transcription

1 BUSINESS INTELLIGENCE FOR CMS DATA QUALITY MONITORING AND DATA CERTIFICATION. Author: Daina Dirmaite Supervisor: Broen van Besien CERN&Vilnius University 2016/08/16

2 WHAT IS BI? Business intelligence is a set of integrated tools, technologies and programmed products used to collect, integrate, analyze, and transform data into information. This information is then used to enable effective business decision making. BI systems combine operational data with analytical tools to present complex and competitive information to planners and decision makers.

3 WHY DO WE NEED IT? BI seeks to provide the capability to access and analyze information, so that massive data from many different sources of a large enterprise can be integrated into a coherent group to provide a 360 view of its business case. What do we have now: INADEQUATE ANALYSIS AND REPORTING: Collecting information and cobbling it together via spreadsheets is cumbersome. The detailed information that decision-makers need can be hard to get or not even available. TIME CONSUMING AND LABOR INTENSIVE TO SET UP AND MAINTAIN: Establishing a company-wide model, creating organizational plans, distributing and collecting information from different sources, consolidating multiple spreadsheets, and debugging broken macros and formulas becomes unwieldy.

4 AND SO..? The BI system extracts data from various data sources, transforms it into specified formats, and then loads the newly formatted data into specially designated data warehouses that are available to all three levels of decision making within the organization: operational, strategic, and tactical.

5 MAIN CONCEPTS The role of business intelligence is to extract the information deemed central to the business and to present or manipulate that data into information. Understanding business intelligence systems enables any organization to implement an analytical approach that transforms data into information, information into knowledge and then knowledge into decisions All business intelligence systems require, at a minimum, four specific components to produce business intelligence: data warehouses, ETL tools, OLAP techniques, data mining.

6 Let s look deeper into it Data Warehouse ETL OLAP techniques Data mining DW is a collection of relevant business data that is organized and validated. Data warehouses are populated with data that has been extracted from distributed databases, often heterogeneous. The data warehouse is considered the core component of a business intelligence system. ETL tools and processes are responsible for the extraction of data from one or many source systems, as they transform data from many different formats into a common format and then load that data into a data warehouse. On-Line Analytical Processing is an improvement to earlier single dimensional analysis tools that allowed managers to analyze data from only one perspective at a time. By providing managers with a multi-dimensional tool, OLAP enables managers to analyze data from multiple perspectives and explore it in order to discover hidden information. Data mining techniques are designed to identify relationships and rules within a data warehouse, then create a report of these relationships and rules. The data mining process involves discovering various patterns, generalizations, regularities and rules in data resources.

7 ETL TOOLS ETL solutions are divided into three distinct stages that find and convert data from various sources and inserts the resulting product into a data warehouse. The three stages of ETL: The extraction stage This stage involves obtaining access to data originating from different, often heterogeneous sources. These sources are often distributed across multiple platforms and can be part of a customer's information system. The transformation stage This stage transforms the extracted data and is considered the most complex stage of the ETL process. The transformation stage converts the data into the same schema of the data warehouse to which it is to be loaded. The load stage The load stage pushes the transformed data and loads the data warehouses with data that are aggregated and filtered

8 OLAP TECHNIQUES OLAP allows user access, analysis and modeling of business problems and sharing of information that is stored in data warehouses. OLAP offers techniques for data analysis and drilling data and the tools are mainly used for interactive report generations. Drilling down data means delving deep into the data, so that you can get exact answers filtered by time, location, source and other factors. This is a very important feature with the increasing pressure of real-time decision making. It allows you to get down to the nitty-gritty in just a click, saving hours poring over spreadsheets.

9 OLTP vs OLAP ONLINE TRANSACTION PROCESSING (OLTP) Relational databases are best when the primary goal is to record many transactions which is the case for most companies. OLTP systems are great for recording transactions, but not so great at extracting and analyzing strategic data quickly. A different type of database is used for analysis. ONLINE ANALYTICAL PROCESSING (OLAP) OLAP database contains only historical data, with little or no updates to the data itself. Another distinguishing characteristic is that there are far fewer tables in an OLAP database than is usually found in an OLTP database, and the idea of repeating data in the tables is no longer a consideration. Instead, the focus in on how quickly the database can access the information needed to perform the analyses required. OLAP database is primarily concerned with numerical data, also called measures or facts.

10 OLAP databases: Slice and Dice, Drill Down/Up, Roll Up Slice and Dice A slice of a cube is selecting data from the cube by fixing one dimension to a single value. For instance, the analyst might look at sales for all products for all months in the year for a single store, or sales for all stores and all products for a specific month. Dicing refers to selecting a range of possible values along one or more dimensions, for example, sales for the first quarter for all products in only the southeastern stores. Drill Down/Up Allows the analyst to look at summary data ( drill up ) or more detailed data ( drill down ). Roll Up Summarizes or aggregates the data along one or more dimensions.

11 DATA WAREHOUSE A data warehouse is a relational database that is designed for query and analysis rather than for transaction processing. It usually contains historical data derived from transaction data, but it can include data from other sources. It separates analysis workload from transaction workload and enables an organization to consolidate data from several sources.

12 GENERAL BI ARCHITECTURE FROM THE DECISION SUPPORT SYSTEM VIEW

13 DIMENSIONAL MODELING: STAR SCHEMA Much like a database, a data warehouse also requires to maintain a schema. A database uses relational model, while a data warehouse uses Star, Snowflake, and Fact Constellation schema. We will look deeper into Star schema. Dimension The core structure of the star schema comprises basic fact and dimension tables. The center of the star consists of fact table and the points of the star are the dimension tables. Usually the fact tables in a star schema are in third normal form(3nf) whereas dimensional tables are de-normalized. Dimension Fact Dimension Dimension

14 FACT TABLE A fact table typically has two types of columns: foreign keys to dimension tables and measures those that contain numeric facts. These are numeric and additive across some or all of the dimensions. The granularity, of the data is determined by the lowest level of granularity of each dimension table(e.g. luminosity per lumi section).the lower the granularity, the more records that will exist in the fact table. The granularity also determines how far users can drill down without returning to the base, transaction-level data. Facts records the measures of business events (such as sales). Fact_Luminosity ID Time_FK Date_FK Lumi_Section_FK Delivered_Online_Luminosity Delivered_Offline_Luminosity Recorded_Online_Luminosity Recorded_Offline_Luminosity Avgpu Inst_Luminosity Beam1_Intensity Beam2_Intensity Foreign Keys Measures

15 DIMENSION TABLE Dimension tables are used to describe the data we want to store. For example: a retailer might want to store the date, store, and employee involved in a specific purchase. Each dimension table is its own category (date, employee, store) and can have one or more attributes. For each store, we can save its location at the city, region, state and country level. For each date, we can store the year, month, day of the month, day of the week, etc. This is related to the hierarchy of attributes in the dimension table. Dimensions represents the who, what, where, and when of business events. Dim_Lumi_Section Lumi_Section_Pk Era Fill_Number Run_Number Run_Class Lumi_Section_Number Hierarchy: Form Era to Lumi section

16 MAIN ADVANTAGES OF STAR SCHEMA REAT QUERY EFFECTIVENESS Small number of tables to join RAPID AGGREGATIONAL ACTIONS Tasks like sum, average, count, and others are performed quickly on this systems. SIMPLE STRUCTURE Easily understandable schema Simplified queries and decreased query execution time PERFORMANCE ENHANCEMENTS The performance has substantial gains due to the de-normalized form of the data.

17 MY STAR SCHEMA Input and Events analyzed DQM^2 stream files WBM database Run registry database BRILCALC files DQM rate Luminosity, downtime events Detector high voltage on/off, Sub-detector status(good/bad) Luminosity

18

19 Tool for BI: Tableau Desktop (Free for students and instructors at accredited academic institutions) About One of the leader in BI tools market. Whether it s in a spreadsheet, a SQL database, Hadoop, or the cloud, with Tableau it is possible connect to any data. Tech specs OPERATING SYSTEMS Windows Vista or later OSX 10.9 or later (Alternative for Linux: WEB based solution-tableau Online) Big data, live or in-memory: Tableau s Data Engine lets extract data for ad-hoc analysis of massive data in seconds. It combines advances in database and computer graphics technology so you can analyze huge datasets on a laptop. Compatible with Tableau Online and Tableau Server.

20 Luminosity

21

22

23 DQM Rate

24

25

26 Literature Business intelligence guidebook : from data integration to analytics, Rick Sherman, 2015 Business Intelligence Roadmap: The Complete Project Lifecycle for Decision- Support Applications' by Larissa T. Moss and Shaku Atre, An Introduction to Business Intelligence Concept What is Business Intelligence, and Why Should You Care? Laurie McCabe, 2010/07/30 1/What-is-Business-Intelligence-and-Why-Should-You-Care.htm

27 THANK YOU Daina Dirmaite Vilnius University, Lithuania