Data Governance: Are Governance Models Keeping Up?

Transcription

1 Data Governance: Are Governance Models Keeping Up? Jim Crompton and Paul Haines Noah Consulting Calgary Data Management Symposium Oct 2016 Copyright 2012 Noah Consulting LLC. All Rights Reserved. Page 1

2 Are Governance Models keeping up with the demand for Analytics? 2 Bottoms Up: Self Service BI Tools + Domain Experience Top Down: Enterprise Dashboards, Holistic Asset Performance, Full Asset Lifecycle 2016 Noah Confidential

3 Are Governance Models Keeping Up?

4 The Audience is Changing 4 Financial & Business Analysts Power users Data Science teams Self-service BI tools for everyone Embedded Analytics 2016 Noah Confidential

5 Self-Service BI: What's Not So Great (Steve Swoyer, TWDI) Self-Service Currently Difficult to Scale Data Integration By Any Other Name Metadata? We Don't Need No Stinkin' Metadata! The Tragedy of the Commons (point-to-point data flows) A Spreadmart in Disguise Working Out the Kinks (auditability, accountability. Regulations) Copyright 2012 Noah Consulting LLC. All Rights Reserved. Page 5

6 Are Algorithms Taking Over the World?

7 What is Governance? Governance refers to "all of processes of governing, whether undertaken by a government, market or network, whether over a family, tribe, formal or informal organization or territory and whether through the laws, norms, power or language." It relates to "the processes of interaction and decisionmaking among the actors involved in a collective problem that lead to the creation, reinforcement, or reproduction of social norms and institutions. (Wikipedia) Copyright 2012 Noah Consulting LLC. All Rights Reserved. Page 7

8 What is Data Governance? Data Governance Framework of people, process & technology that enables, delivers, & sustains to positively affect Data Management Data Governance lets us take ownership of our information as a team where everyone clearly understands their role & the data they are responsible for Data Governance defines expectations, sets authority, & monitors & verifies performance Data Governance identifies what business must do to achieve & sustain known quality Data Governance includes principles & processes, standards & rules, roles & responsibilities Data Stewardship Data Governance Data Quality You can improve Data Quality for a short period of time, but without Data Governance & Data Stewardship, it won t last! Copyright 2015 Noah Consulting LLC. All Rights Reserved Page 9

9 What happens when the business can t find or trust the data it needs to make good business decisions? Is this YOUR company? Can you AFFORD to keep operating this way? Copyright 2015 Noah Consulting LLC. All Rights Reserved Page 4

10 Data Governance a New Type of Agreement Data Governance is not a one-time action carried out by IT but an ongoing, iterative effort in which functional areas agree which data types are most important, then take ownership of their data. Through a coordinated effort, designated roles, & sustainable processes, data quality is improved & maintained. Data Governance is the structure that encourages consistent, desired behavior and establishes how the business makes decisions to define, manage, & maintain high quality data. Copyright 2015 Noah Consulting LLC. All Rights Reserved Page 10

11 But Not Everyone Thinks that Governance is a Good Thing Overly restrictive governance prevents people from getting access to data they need or make needed corrections and updates Less responsive forms of governance fails to react to business challenges Too many standards and rigid structures results in shadow IT solutions as data escapes the control of governance processes Too few standards and loose structures create confusion in data integration and loss of productivity What do I do about legacy data or data recently acquired from mergers and acquisitions??

12 Real Time Real Time Integration / Adapters / APIs Micro Batch Transform Analyze Integration / Adapters / APIs Micro Batch Batch Batch Dimensional Views/ Aggregates Reference Architecture for Modern Data Management Data Access 14 GIS Interpretation Viewer Valuation Model Inventory Completeness Data Completeness Production Dashboard Data Discovery Analytics Reservoir Dashboard Producer Structured Information Store Data Mart #1 Data Mart #2 Data Mart #3 Data Mart #4 Data Mart #5 Wells E&P Structured Data Consumer Logs Surveys Services Wells Logs Facilities Seismic Leases Formation Tops Master Data Service Data Quality Service Exception & Error Handling Backup & DR Security Surveys Facilities Seismic Leases Reservoir Data Production Data Well Tests Bottom Hole Data Schematics Data Reservoir RAW CLEANSED TRANSFORMED HARMONIZED Analytics Sandbox Data Lake SQL Engine Data Factory Real Time Event Engine (In Memory) Batch Processing Ops Machine Learning Workflow/Scheduling Formation Tops Reservoir Data Well Tests Production Data Bottom Hole Data Consulting, An Infosys Company

13 OMG: Isn t There Something Simpler? Consulting, An Infosys Company

14 BECOMING A DATA-DRIVEN ORGANIZATION

15 @Noah Consulting, An Infosys Company 17

16 Do I Really Need Standards & Governance? Not if I am only thinking about me (benefits realized somewhere else) Not if I am in a hurry (standards take too long) Not if I delegate to my preferred supplier (cartel) Not if emerging technologies will take care of my problems (data lakes and data scientists) I can start with a standard and customize it Page 18

17 Data Governance Maturity an Evolution Where is your company? Understand your company s data governance maturity level We think our data is fine We notice some problems We know our problems & are engaged in solution options Standards defined, Processes evolving, Critical data governed Competitive Advantage Unaware Aware Engaged Defined Standards Fully Adopted Copyright 2015 Noah Consulting LLC. All Rights Reserved. Page 19

18 2016 Noah Confidential Data - Core to Business Data Image from the Data Management Book of Knowledge (DMBOK) published by Data Management International (DAMA) Data as an Asset Data as a Decision-Driver Data as a Strategic Differentiator Data Quality is a key factor Data Quality Enablers Data Stewardship Data Governance The weapon of choice here is information, because the more information we have the better we are able to make decisions * * Lt. Col. David Haworth - USAF

19 Data Governance Components Framework Process Workflow Architecture Organizational Model Roadmap Copyright 2015 Noah Consulting LLC. All Rights Reserved. Page 21

20 A Proven Approach A high percentage of Data Governance efforts fail to deliver on the vision or anticipated results Successful & repeated implemented industry relevant Data Governance with significant measurable improvements in data availability & trustworthiness Leverage specific accelerators to boost the efficiency, effectiveness, & business value Client-specific, pragmatic, iterative, adaptive, & phased approach based on proven successes in Data Governance, Data Quality, & Data Management segments Copyright 2015 Noah Consulting LLC. All Rights Reserved Page 22

21 Data Governance Design & Implementation Guiding Principles: Plan for an iterative, evolutionary governance model Set realistic, achievable goals for each iteration Build Monitoring Tools early Deliver Business Value in every iteration Goal of data quality metrics is to drive effective & efficient business processes to support business decisions, not data quality for data quality sake Copyright 2015 Noah Consulting LLC. All Rights Reserved Page 23

22 Data Governance Design & Implementation Real-life Learnings: Big Bang approach to implementing Data Governance rarely works If people can t SEE IT easily, you can t realistically govern it Successes come from a phase approach with a tight focus for each iteration & delivering recognized business value There is no one size fits all engagement model or organizational model Company culture & maturity must be considered before deciding on organization structure & titles Avoid command/control/compliance-focused organization structures when data governance maturity is in early stages Leverage business people who care about data quality & work data corrections Data Quality issues are often traceable to a broken or incomplete Business Processes or a system process involved in authoring/loading the data Information Stewards must not solely be accountable for data quality. Copyright 2015 Noah Consulting LLC. All Rights Reserved Page 22

23 Will the Real Data Scientist Please Stand Up