Good analytics needs good data and that needs good metadata

Transcription

1 Good analytics needs good data and that needs good metadata 28 th February 2018 Mandy Chessell CBE FREng CEng FBCS Distinguished Engineer, Master Inventor Analytics Chief Data Office

2 Introduction and background Mandy Chessell CBE FREng Distinguished Engineer, Master Inventor Focused on Analytics, Open Metadata and Governance 30 years at IBM working on exchange and integration of data between business systems. 2

3 Agenda Why is metadata so important today? What is the challenge? Building an open ecosystem Apache Atlas and the specifics ODPI Data Governance PMC Progress report and call to action 3

4 Data and analytics affects every aspect of modern life Healthcare Government Manufacturing Telecommunication Insurance Energy & Utilities Financial Services Retail High Tech Hospital Oil & Gas Travel & Hotel Transportation Multi-channel integration Stock Market 4

5 Today s technology is inherently ethics-agnostic but it pushes the art of the possible to new limits The availability of a wide range of data from many sources. The ability to cheaply process and link this data together to understand a bigger picture. The accuracy with which an individual can be identified and targeted. The ability to pinpoint location for contextual insight and surveillance. The application of this new insight to a wide range of activities and actions. The operation of the insight in real-time or near real-time. Increasing both the ability and the obligation to act 5

6 Open Metadata and Governance MANAGING DATA FOR DATA SCIENCE 6

7 The perils of reusing data Callie Quartile Data Scientist Callie Quartile uses (1) open data from the local government registrar and (2) data from the employee directory to (3) create a birthday card service for the company. 1 Open Data Site Data Lake 3 Employee Directory 2 7

8 The perils of reusing data Callie Quartile Data Scientist But its not my birthday Happy Birthday 1 Open Data Site Data Lake 3 Employee Directory 2 Unfortunately the obvious date in the registrar record was the registration of birth date not the date of birth. Date of birth was not published in the open data. Callie needed better information about the open data to realise she had the wrong data. 8

9 Metadata should bring as much information about the data sets to Callie s data science as is known collectively by the organization. Data Set Name: Employee Directory Description: Core attributes describing all employees of OCO pharmaceuticals created from a daily extract from Kenexa. Owner: Penny Payer Classification Ranges: Confidentiality: Public, Confidential, Sensitive Confidence: Authoritative Retention: Indefinitely Status: Last accessed: 6 th May 2016 Records: 3488 Last Update: 1 st May 2016 Contents: Structure Contents Lineage X Column: Band Description Characteristi Lineage cs Position reference number for nonexempt employees. The value ranges from 01 to 06 where 01 is the most senior and 06 is the most junior. Type: String Classification: Public Employee Directory NameBand Job Title X X 9

10 Curation I wonder what this means Lemmie Stage New York 4 27 DataStage Expert Code St Harlem NY Callie Quartile New York 4 27 Data Scientist Code St Harlem NY Tanya Tidie New York 4 27 Data Steward Code St Harlem NY 1 3 I know 10

11 Scared to share Faith Broker Business Team Faith Broker has been doing some simple analysis on the HR data of the company. She wants to share this data with Callie Quartile to do some detailed work. However, she does not want Callie to see the sensitive personal information in the record Lemmie Stage New York 4 27 DataStage Expert Code St Harlem NY Callie Quartile New York 4 27 Data Scientist Code St Harlem NY Tanya Tidie New York 4 27 Data Steward Code St Harlem NY Lemmie Stage New York 4 27 DataStage Expert 1 XXXXX XXX 27 Code St Harlem NY Callie Quartile New York 4 27 Data Scientist 1 XXXXX XXX 27 Code St Harlem NY Tanya Tidie New York 4 27 Data Steward 1 XXXXX XXX 27 Code St Harlem NY 1 3 Callie Quartile Data Scientist 11

12 Why do we need metadata? Metadata enables data to be used outside of the application that created it. Analytics and decision making New business applications Reporting and compliance Metadata describes the format and content of data allowing people to judge which data set to use for a new project Structure Meaning Origin Valid values and quality Usage and ownership Regulations and classifications that apply <more> Metadata describes the business context and classification of data allowing automated governance processes to operate. 12

13 Open Metadata and Governance WHAT METADATA DO WE NEED? 13

14 Scope of metadata for a data driven organization Governance Glossary Collaboration Models and Reference Data Lineage Base Types, Systems and Infrastructure Data Assets Metadata Discovery 14

15 Open Metadata and Governance TODAY S REALITY 15

16 It is a complex world Many systems Different design Different technology Different quality Different aims 16

17 Data-centric Organisations Systems of Record Mobile Apps Systems of Engagement Databases Files Applications Data Lake 17 External Data Systems of Insight

18 Today s reality 18

19 What needs to change? Open and Unified Metadata 19

20 Automatic metadata capture 20 20

21 Manual metadata capture 21

22 A new manifesto for metadata and governance Metadata management must be automated Metadata management must become ubiquitous Metadata must become open and remotely accessible Metadata should be used to drive the governance of data The discovery, maintenance and user of metadata has to be an integral part of all tools that access, change and move information

23 Open metadata management ecosystem Analytics Platform Metadata Collaboration Space Metadata Application Metadata Hadoop Platform Metadata Cloud SaaS platform Metadata Peer-to-peer network of repositories Metadata stored and managed close to its source Each repository/tool brings unique value. Open, extensible metadata structures for metadata exchange and federation extending coverage of the types of resources that need to be described. Open source infrastructure sharing cost of development and maintenance between vendors Support for open standards where available 23

24 Open Metadata and Governance BUILDING THE OPEN ECOSYSTEM 24

25 Apache Atlas Apache Atlas has just graduated to become a top-level project. It began as an incubator open source project on 5 th May 2015 to deliver an open source governance capability focused primarily on the Hadoop platform. Apache Atlas is designed to localize operational governance to the operating data platform such as Hadoop. At its heart is a type-agnostic metadata store that can be access through restful interfaces. We see Apache Atlas as the reference implementation for open metadata and governance, for vendors to pick up and use; or test their integration against. Being open source allows all vendors to enrich/enhance standard. 25

26 Apache Atlas today 26

27 Updates to Apache Atlas Automation Capture of metadata from data platforms, data movement engines and data protection engines. Exception management and stewardship Business Value Specialized services for key data roles such as CDO, Data Scientist, Developer, DevOps Operator, Asset Owner, Applications Connectivity Metadata Highway offering open metadata exchange, linking and federation between heterogeneous metadata repositories. 27

28 Metadata type system 28

29 Open metadata meta-types, types and instances Meta-Type Model (enumerations, structs, entities, classifications, relationships, ) «entity» DataStore * datastore «relationship» PhysicalStoreForDataSet dataset * «entity» DataSet Type Model «entity» Database «entity» DeployedDatabaseSchema Instance Model «entity» Server «classification» DatabaseServer 29

30 Open metadata type model summary 0 Policy Metadata (Principles, Regulations, Standards, Approaches, Rule Specifications, Roles and Metrics) Governance Actions and Processes Strategy Campaigns and Projects Implementation Instrument Rollout Rights Management Information Process Instrumentation (design lineage) Basic Types, Infrastructure and Systems Classification Schemes 4 Classification 7 Association Access Subject Area Definition Business Objects and Relationships, Taxonomies and Ontologies 3 Organization Business Attributes Mapping Asset Collections (Sets, Typed Sets, Type Organized Sets) Information Views Physical Asset Descriptions (Data stores, APIs, models and components) Connectors 2 1 Augmentation 5 Augmentation 6 Teaming Metadata (people profiles, communities, projects, notebooks, ) Feedback Metadata (tags, comments, ratings, ) Models and Schemas Reference Data Discovery Metadata (profile data, technical classification, data classification, data quality assessment, ) 30

31 More detail here 31

32 Metadata and governance digital platform Reporting Platform Governance Platform Analytics Platform Open Metadata and Governance Data Platform ETL Platform 32 Virtualization Platform

33 Types of tools that may integrate with an open metadata repository BI and visualization tools locating data assets and related information about them; defining reports and publishing their metadata; viewing lineage Data Science tool wanting to find out about data assets available and manage user lineage of transformations and analytics models may also manage metadata for analytics models API developer tool wanting to understand proper data structures and data meaning to use for APIs plus additional governance requirements that need to be implemented by API because of the data it exchanges. Counter-fraud tools ad hoc analysis of logs and error reports, setting up rules Curator/owner tool for managing the curation of assets, providing access, verifying use of assets, reviewing discovery results and exceptions, approving change requests. Glossary tool for subject matter experts and information architects to share expertise about a particular subject area may also define structures and related reference data Enterprise architect tools defining the data landscape and related systems. DevOps tools conformance to polices and standards in development metadata capture at deployment validation of deployment platform requirements Data integration engine locating appropriate data and component assets, log design lineage, log operational lineage Information Virtualisation tools locate appropriate data assets, build views and publish them, add design lineage, log operational lineage Governance tools setting up and monitoring governance program, data quality, Stewardship tools reviewing assigned exceptions, making data changes and requesting approval Information security tools setting up data access policies and enforcement Auditor tools view compliance reports and validate policies and policy implementations 33

34 Open Metadata Access Services Asset Owner Subject Area Information View Asset Catalog Community Profile Stewardship Project Management Governance Program Data Science Information Process Developer Information Protection Information Landscape Governance Engine DevOps Information Infrastructure Data Platform Connected Asset Asset Consumer Discovery 34

35 Inside the server OMAS REST APIs and Topics Server Configuration Open Metadata Access Services (OMAS) OMAG Administration REST APIs Open Metadata Repository Services (OMRS) OMRS Repository REST APIs OMRS AuditLog Connector OMRS Archive Connector OMRS Cohort Registry Store Connector OMRS Topic Connector OMRS Repository Connector OMRS Event Mapper Connector 35 Open Metadata and Governance (OMAG) Server

36 Inside the server OMAS REST APIs and Topics Server Configuration Open Metadata Access Services (OMAS) OMAG Administration REST APIs Administration Cohort Services Enterprise Repository Services Local Repository Services OMRS Repository REST APIs OMRS AuditLog Connector OMRS Archive Connector OMRS Cohort Registry Store Connector OMRS Topic Connector OMRS Repository Connector OMRS Event Mapper Connector 36 Open Metadata and Governance (OMAG) Server

37 Simple cohort Data Lake Cohort A Chief Data Office Systems of Record 37

38 Multiple cohorts Marketing Data Lake Data Lake Cohort A Cohort B Chief Data Office 38 Systems of Record Mobile Apps Systems of Record

39 Integration patterns 39

40 Caller Pattern A metadata tool or engine can access the consumer-specific APIs to work with metadata. The Access Layer handles the calls to metadata repositories connected to the metadata highway 40

41 Adapter Pattern Simple components plug into a repository proxy to connect in an existing metadata repository. 41

42 Native Pattern Native implementation of the open metadata governance APIs Apache Atlas is a native implementation of the open metadata and governance APIs. 42

43 First server 43

44 Establishing contact 44

45 Federated queries 45

46 Caching metadata for availability and performance 46

47 @ODPiOrg ODPI - CO-CREATION WITH PRACTITIONERS Compliance assistance and certification for vendors Subject matter experts sharing best practices and co-creating content packs

48 Call to action how can you help? Direct code contribution to the Apache Atlas project. There are many features that still need to be coded. Collaboration through the ODPI Data Governance PMC Encouraging your customers/partners and projects internal to your organization to embrace Apache Atlas and its standards to grow the ecosystem of data and processing that is assured by metadata and governance capability

49 49

50 z zzz z z z Questions? 50

51 51