Business Intelligence Architecture Kim Setälä 37E00550 Business Intelligence

Transcription

1 Business Intelligence Architecture Kim Setälä 37E00550 Business Intelligence

2 Digital Aalto Data-driven university

3 Setting the problem

4 I am Kim Setälä Development Manager, Master Data Management Aalto University, IT Services Master of Science Aalto University, School of Electrical Engineering, 2014, mobile technology Elisa Oyj, analyst , customer experience in mobile network and churn analytics Learning by doing and never under estimate capability to learn and survive through it 4

5 Agenda Data Architecture Data Governance Analytics in Practice Considerations when designing data architecture in company Why data should be managed? How to get what you want from data scientist (leadership and content) 5

6 Enterprise Architecture Data Governance Data Architecture For the successful analytics, we need to have Transparent and deployed Enterprise Architecture, which controls how new applications are implemented and what principals should be followed Scalable, structured and well documented Data Architecture to provide basis for analytics and reporting and new data sources Clear and accurate Data Governance, which respond to real world and helps to develop data utilization and offers framework to use the data Analytics & Business Intelligence 6

7 Enterprise Architecture Enterprise architecture describes the principals for the system development. The principles help to achieve common understanding how to develop technology, based on strategy and actions The principles are guidelines to choose technology, applications and solutions 7

8 Enterprise Architecture in Aalto Common Use Applications Basic functionality any solution must be able to be used with Webbrowser Modular design Use of open standards Information is Accessible Common Vocabulary and Data Definitions Information Security Solution technology is modular, reusable and supports environmentally sustainability. Information is an Asset Ease-of-Use Interoperability Solution continuity assurance Requirements-Based Change Aalto IT services are provided location independently The use of Aalto IT services does not require special IT knowledge Aalto IT service users are informed about Aalto IT services and associated conditions Service usage is completely separated from service management and service provision Buy before in-house development Information and processes are considered first in the solution development Information security must be considered at beginning of the IT development process Location independent IT Systems interoperability Plan for Integration Reusable, Shared Services 8

9 Data Architecture Describes how the analytics and data are provided Based on EA principles Cloudera s six principles for data architecture 9

10 Data Governance Data Governance helps to manage and control and guide data use It describes common understanding how the can be used and stored De-facto framework is DAMA s Wheel, which includes very detailed guidelines to develop data governance DAMA: The Global Data Management Community 10

11 Analytics & Business Intelligence Analytics in the broadest sense applies to all technology-enabled problemsolving activities. Experts generally divide analytics into four categories descriptive analytics (What happened?) diagnostic analytics (Why did it happen?) predictive analytics (What is going to happen?) prescriptive analytics (What should be done?) Business intelligence leverages software and services to transform data into actionable intelligence that informs an organization s strategic and tactical business decisions. It is what enables an organization to collect, analyze and present analysis of data. 11

12 What is data warehouse? Consolidates data from multiple sources Provides a consolidated version of the truth Saves time in building reports Enables reporting and analyses of data in ways you could not do using applications Ensures high-performance data access for reports and BI Data Mart: Subset of the data warehouse that is usually oriented to specific subject 12

13 Extract, Transform, Load (ETL) taken, converted, stored Three step process used to blend data from multiple sources in many cases to data warehouse. Currently, the ELT has become more popular as its faster data processing Extract Data is loaded, i.e. from REST API with JSON from the sources i.e. to staging area for transforming and validating Good overview and examples from SAS Transform Most complex step, which can include quality validation, mapping to destination format Load During the step, data can be choose to be virtualized to another view, scheduled and master data will be stored to it s source 13

14 Maturity of Data Warehouse 14

15 Designing the data warehouse What subject areas need to be analyzed using DW? Where the ETL process will be done What source systems will be integrated? If similar data is stored in multiple sources, which one is the master? Is the target to store all data from an integrated source, or only the data needed for reporting? What is the time-variance of DW? How long in history is the data kept? What is the legislation in the subject areas? Volatility of the DW If the data is updated or deleted in the source system, is it updated or deleted also in the DW? Many Data Warehouses are used so that no data is deleted or updated How often there is need to refresh the reports? Once or twice a day or near real-time? Building a near real-time reporting system using DW requires a lot more effort compared to batch-oriented DW What data model will be used? Which database product will be used to store the data? Which development tools will be used? 15

16 Data models for Data Warehouse 16

17 Data models for Data Warehouse: Inmon Inmon s Entity-Relationship (ER) model Bill Inmon (born 1945) is an American computer scientist An ER model is composed of entity types (which classify the things of interest) and specifies relationships that can exist between instances of those entity types Each entity has a primary key that uniquely identifies each instance of the entity Foreign keys are used to refer to primary keys of entities Centralized atomic normalized tabled Integration via enterprise data model 3-tier; data warehouse, data mart, cube + duplication of data James Serra, Big Data/Data Warehouse Evangelist at Microsoft 17

18 Data models for Data Warehouse: Kimball Introduced by Ralph Kimball (born 1944) Consists of dimension tables and fact tables Decentralized data marts (not required to be separate physical data store) Independent dimensional data marts optimized for reporting/analytics Dimension tables Records corresponds to nouns Examples: location, product, time, organisation etc. Tables are typically short 10s to 1000s of records Data changes slowly Rich set of attributes Tables can contain many columns Lots of redundancy James Serra, Big Data/Data Warehouse Evangelist at Microsoft 18

19 What about Data Vault? (Lindstedt) 19

20 Basic of Data Vault Good summary about Data Vault vs others 1) Instead of each master table (Inmon) in we add a hub and a satellite 1) Hub has unique keys for i.e. customer or orders or products 2) Satellite has the attributes 2) Instead of the transactional table (Kimball), we add Link table and Satellite 1) Link unites i.e. customers to orders (many-to-many relationship) 3) Instead of the joins between master tables, we add Link tables. 20

21 Data warehouse in Aalto Data sources (Data Marts) are mainly applications like Oodi, HR or finance applications The master data is at the applications own databases Most of the data like personal information or transactional data (payments) are transferred from application to application with integration solution/interface) Data warehouse has important role as source of reporting/analytics and sharing the data to applications and interfaces 21

22 Student data Oodi ObjectContent hetu oppijanumero sukunimi etunimet kutsumanimi lahiosoite postinumero postitoimipaikka maa kansalaisuus kaksoiskansalaisuus syntymaaika matkapuhelin puhelin sahkoposti kotikunta Students personal data is created by Opintopolku/MoveOn to Oodi and shared to application with integration and from Aalto data warehouse 22

23 tables solutions Data Transfers in Aalto Transfer protocols: file transfer, http, https, SQL Data formats: CSV, XML, fixed format text Data is transferred incrementally, if the application supports incremental transfer Data transfer is automated Data from all sources is transferred each nigh Bookkeeping data is transferred twice a day, at night and before noon D E S T I N A T I O N A P P L I C A T I O N S A N D T A B L E S ( D W & I N T E G R A T I O N S ) Target_applicattion Integ_OUT Target_taulut

24 Data Model in Aalto DW Data model follows Inmon s ER model The implementation is based on XDW model developed by CSC, (national) IT Center of Science The model includes schema figures and detailed descriptions s-kaavio.html Each table includes also technical columns Id (internal unique id for each row) Data source Concept name Timestamps of creation, latest update and recognition 24

25 Size of Aalto DW Physical size: 155 GB Columns: > Tables: Functions: 42 Views: 889 Procedures:

26 Operational work in Aalto DW The goal is to automate operational work as far as possible There are more and more citizen analytics in Aalto, who are using selfservice analytical tools; Qlikview since 2015 student and finance reporting restricted with personal licenses Microsoft Power BI since 2017 license for the whole university uses the Microsoft s DAX-language flexiable with other programming languages Automation includes Data transfer from source applications Data transfer to target applications Alarms when a DW job returned an error Alarms when server is overloaded (e.g. disk space almost full, high CPU load) Daily backups Refresh of database statistics to provide faster access 26

27 Best practices with reporting Use clear naming Avoid own abbreviations in table and column names Use naming conventions E.g. loading procedures are named as load_<table_name> Design clear code structure with comments Manage the schema so that enhancements can be done without breaking the existing interfaces/reports If possible, test always with real data Test also with large amount of data to check the performance Do not deploy to production without testing Keep track of code changes Implement automated data transfer Maintain documentation 27

28 Data Governance Data governance best practices Why data governance increasing data volumes from more and more sources, causing data inconsistencies that need to be identified and addressed, before decisions are made using incorrect information more self-service reporting and analytics (data democratization), creating the need for a common understanding of data across the organization the continuing impact of regulatory requirements such as GDPR, making it even more important to have a strong handle on what data is where, and how it s being used an increasing need for a common business language to enable cross-departmental analysis and decisions 28

29 Dimensions for the Data Governance 29

30 Remarks about Data Governance Strategy Data strategy should be direct from organizations other goals and actions so it will serve the purpose through the organization Strategy should declare the data as asset (instead of applications) How the data is modelled and described How the data is available on platforms and interfaces How the data quality is measured How the data is secured based on regulations Architecture Lifecycle of data; where it s created, stored and how it will be available for use Description of the data architecture includes reporting, analyses, data extracts and transfers and use of AI / ML The understanding of difference between substance i.e. business and IT and communications Organization and Roles Clear understanding of roles and responsibilities helps communication Competence and it s development on different levels Outsourcing decisions and areas 30

31 Analytics in practice The truth: Analytics and business intelligence are everything between Excel sheet data to learned routes in iphones 31

32 Call Call Call Analyses and findings in Elisa NPS vs. customers technical experience Regularly measured Net Promoter Score correlates with KPI, which is measured from customers devices and describes their perception of mobile network (throughput, breaking) Churn predictions and actions Long-term churn rates correlates with several different attributes i.e. campaigns, devices, contract age and actions where focused on critical customer groups Results of actions and behavior of target customers were followed frequently and cost of actions were evaluated Churn retension actions NoAction 95 5 Offer NotAnswered Decline Offer % 20 % 40 % 60 % 80 % 100 % Churn NotChurn 32

33 You are a Citizen Data Scientist Gartner defines a "citizen data scientist" as a person who creates or generates models that leverage predictive or prescriptive analytics, but whose primary job function is outside of the field of statistics and analytics. By 2019, citizen data scientists will surpass data scientists in the amount of advanced analysis produced By 2020, more than 40% of data science tasks will be automated, resulting in increased productivity and broader usage by citizen data scientists 4 steps to citizen data scientist 33

34 Roles in business intelligence Business people with analytical skills Business Owner Analysts / Data Scientist IT (hardware) IT (BI) Management Understands the need Coordinates the actions Describes the problem Part of separate organization Works for different business owners Who prioritize the duties and tasks? Maintains the data warehouse and operation system Understand the restrictions of storing the data and interfaces Serves the whole organization Maintains the BI tools roadmap Consolidates the businesses needs Works with vendors and partners Helps to develop competence Feed the business owners and analysts with new ideas Demands the return of investment from the data 34

35 Data science needs people Question to ask from data scientists: What question should we ask?. What data do we need? How do we obtain the data? Is the data clean and easy to analyze? Is the model too complicated?

36 36