Data Clairvoyance. A business approach to data. Real data practitioners, delivering real improvements to your enterprise data assets.

Transcription

1

2 Data Clairvoyance A business approach to data. A professional services firm that provides a very unique and holistic approach that enables your organization to be successful in traversing the data challenges of today. Quantifying Trust & Confidence of Business Critical Data Designing, Enabling and Growing a Data Centric Culture Real transparency into shared data responsibilities Real data practitioners, delivering real improvements to your enterprise data assets.

3 Agenda

4 Agenda Data Quality Solution Overview Human Resource Requirements Technology Requirements Example Deliverables Advanced Solution 4

5 Data Quality Solution Overview

6 Data Quality Solution Overview Data represents the data sources to be analyzed Profile Data represents the profiling tool used to infer metadata directly from the data content Data Quality Assessment represents the process utilized to analyze the data content Monitoring Rules represent the code created to validate the data content over time Data Quality Report represents the reports created to validate the data content over time Remediation Requirements represents the specifications for correcting the identified data quality problems in the data content 6

7 Detailed DQ Solution Overview Data is identified by the flat file and relational data source and the connections and metadata required Profile Data represents the profiling tool used to infer metadata directly from the data content and the inferred metadata produced by the tool Data Quality Assessment represents the process utilized to analyze the data content and the review of the assessment results by the lead data quality analyst Monitoring Rules represent the code created to validate the data content over time that are reviewed with the subject matter experts and/or the business Data Quality Report represents the reports created to validate the data content over time by the subject matter experts and/or the business Remediation Requirements represents the specifications for correcting the identified data quality problems 7

8 Data Sources Flat file data sources that are normalized with a single record type The layout for the flat file is required by the profiling tool in order to analyze the data properly Relational data sources are databases that contain the data to be analyzed Connections are required to the relational database in order to analyze the data content 8

9 Data Profiles The solution is software agnostic but does require a profiling tool The profiling tool analyzes the data content to infer metadata The inferred metadata is copied into the assessment template for each attribute in the table/file The inferred metadata includes the number of distinct values, null rule, inferred data type, range, value frequencies, and pattern frequencies 9

10 Data Quality Assessment A step by step process for analyzing the inferred metadata by column to identify anomalies and/or inconsistencies in the data content The inferred metadata is copied from the profiling tool into the assessment template The anomalies and/or inconsistencies are categorized as being invalid, inconsistent, duplicate content, mixed content, and/or questionable content The assessment results are reviewed and validated by the lead data quality analyst, who also suggests monitoring rules specific to each attribute 10

11 DQ Assessment SME Review The categorized anomalies are reviewed with the subject matter experts and the business The subject matter experts and the business determine if the identified problems are valid or not Notes are captured that identify business or data rules specific to the content The suggested monitoring rules are also reviewed and validated for coding purposes Validated data quality problems are identified for remediation 11

12 Data Quality Monitoring Rules Monitoring rules are suggested by the lead data quality analyst Additional rules are identified during the assessment review session The rules are coded and tested The results of the rules are reviewed with the subject matter expert and the business Additional rules and changes to the initial rules are identified and implemented The review process is repeated until the subject matter experts and the business are satisfied with the rules The subject matter experts and the business sign off on the rules The DQ analysts finalize the monitoring rules by creating the appropriate documentation The finalized rules are then deployed to begin monitoring the quality of the data over time The results of the monitoring rules are also utilized to identify data content for remediation 12

13 Data Quality Reports Reports are created from the results of the monitoring rules Thresholds are set for the rules based on the importance and frequency of change for the data with the help from the subject matter experts and the business The reports are scheduled to run based on input from the subject matter experts and the business The reports are then deployed to automatically run on a regular basis The generated reports are validated by the data quality analysts prior to forwarding to the subject matter experts and the business for review 13

14 Data Quality Closed Loop Solution The last step in the process ensures that the quality of both the data content and monitoring rules are maintained over time This is accomplished by reviewing the data quality reports and determining if the data problems identified are valid or if the monitoring rules need to be updated to reflect changes in the business and the data content Remediation requirements are created for the anomalies identified as true data quality problems Changes to the monitoring rules are identified and implemented when the anomalies turn out to be valid data content This creates a closed loop solution 14

15 Human Resource Requirements

16 Data Quality Team The Data Quality Project Manager tracks the completion of the data quality process across data sources and maintains the project plan The Lead Data Quality Analyst assigns and manages the work for the administrator and the senior analysts, and also schedules and leads the review meetings with the subject matter experts and the business for the assessments and rule creation The Senior Data Quality Analyst distributes the work across the analyst and junior analysts, and performs data quality assessment and creates monitoring rules The Data Quality Analyst is responsible for performing assessments and creating monitoring rules The Junior Data Quality Analyst is responsible for performing assessments and creating monitoring rules The Data Quality Administrator is responsible for gathering metadata for the flat file data sources and creating the connections for the relational data sources 16

17 Subject Matter Experts The Data Stewards are usually business resources that have expert knowledge about specific subject areas and related data content The Data Custodians are usually information technology resources that have expert knowledge about the metadata for specific data content The Business or true subject matter experts that utilize the data on a regular basis, ensuring intimate knowledge about the data and how it is used Data Modelers that have extensive experience and knowledge about the data from creating data structures to capture and store the data content 17

18 Data Governance Team The Data Governance Team participates in the review sessions with the subject matter experts The Data Governance Enterprise Architect is usually an experienced modeler, analyst, database administrator, and technician The Data Governance Business Analyst with a background working with the data from a business perspective 18

19 Typical SME Review Session The review session is similar to a JAD session because you are gathering participants from across the organization to walk through the assessments and monitoring rule results to determine if identified problems are true data quality problems The subject matter experts provide intimate knowledge about the data from specific applications The data quality team drives the meeting and provides insight as to why specific problems have been identified The Data Governance team provides guidance on standards and view the data from an enterprise perspective as opposed to an application perspective 19

20 Technology Requirements

21 Technology Requirements Enterprise Level Data Profiling tool that profiles at the column, table, and crosstable levels (column, dependency, and redundancy profiling) Enterprise Level Data Quality tool for creating and maintaining the data quality monitoring rules MS Office MS Excel specifically SQL tool for accessing relational data sources File editing tool that allows for viewing data in hex format Data modeling software capable of reverse engineering relational data sources 21

22 Example Deliverables

23 DQ Assessment Deliverable 23

24 Data Quality Monitoring Rules Deliverable 24

25 Advanced Solution

26 Advanced Solution Business Glossary Metadata Repository Data Quality Repository Executive Summary Balance Sheet of Data 26

27 Basic Analytics & Data Intelligence Knowing what assets to invest into, remediate and enrich.

28 Example Set of Enterprise DQ Metrics Critical to Quality Metric Description Level Measurement Method Accuracy Does the data accurately represent reality or a verifiable source? Column then aggregated to cross table TI, TII, TIII Rules (TIII can also include a Manual Audit) Integrity Do broken links exist across physical data assets in data that should be related? Cross Table TI, TII, TIII Rules (TIII can also include a Manual Audit) Does the structure or pattern fit to a standard or defined TI, TII, TIII Rules (TIII can also include Consistency Conformance structure? Is there a single representative of data? Column and Cross Table a Manual Audit) Completeness Is there data missing, blank, null, etc when it isnt supposed to? Column and Cross Table TI, TII, TIII Rules (TIII can also include a Manual Audit) Validity Is the data stored in an acceptable fromat and contain valid values? Where possible, does the data adhere to a known architectural standard? Column TI, TII, TIII Rules (TIII can also include a Manual Audit) Accessibility Is the data understandable, easily accessed and match its contextual use cases? Column and Cross Table Survey Timeliness Is information recorded and available to consumers when it is needed? Column and Cross Table Survey Detail Data Quality Analysis Overall Data Quality Score This is an advanced method that is tailored to the specific Data Quality observations identified Column and Cross Table Custom Algorithm What is the overall Data Quality considering all quality metrics above? Aggregation of all data quality scores against a control limit Custom Algorithm Overall Data Quality Score What is the overall Data Quality considering all quality metrics above? Aggregation of all data quality scores against a control limit Custom Algorithm 28

29 Synthesis of the Expected Finished Good (Output of the Machine) Business Data Element n (i.e. City Name from Address) Overall MetaQuality Score TM Conceptual Assets (i.e. Business Data Elements) Required metatags with degradation Logical Asset n Required metatags with degradation Physical Asset n Required metatags with degradation Overall Data Quality Score Physical Asset n Column Level Completeness Pass/Fail Percentage Business Defined Rule n Conformance Pass/Fail Percentage Business Defined Rule n Integrity Pass/Fail Percentage Business Defined Rule n Validity Pass/Fail Percentage Business Defined Rule n Accuracy Pass/Fail Percentage Business Defined Rule n The Ultimate Finished Good Asset #1 Asset #2 Asset #3 Asset #4 Overall Data Certification Score Overall Data Quality Score BDE: SSN BDC: Items in force Asset #5 29

30 Data Quality Scorecard Social Security Number Confidence Interval 58.33% data Data Quality Metric Score Target % to Target Last Updated Measurement Method Accuracy % 7/9/2014 Automated control chart lineage Integrity % 7/9/2014 Automated control chart lineage Consistency % 7/9/2014 Automated control chart lineage Completeness % 7/9/2014 Automated control chart lineage Validity % 7/9/2014 Automated control chart lineage Accessibility % 5/25/2014 Manual Timeliness % 6/6/2014 Manual Overall Data Quality % MetaQuality Metric Target Score Last Updated Measurement Method Count of Physical Instances % 6/24/2014 Automated control chart lineage MetaQuality Score % 6/27/2014 Automated control chart lineage Data Certification Score % 30

31 Enterprise Data Quality Trend Charts Conformance 31

32 Advanced Analytics in Data as an Intangible Asset Knowing what assets to invest into, remediate and enrich.

33 Physical instances of data Integrity Score Business Metadata Accuracy Score Completeness Score Data Quality Metadata People Metadata Business Data Element Core Metadata Technical Metadata Consistency Score Conformance Score Copyright Data Clairvoyance Group 2014 All rights reserved by Data Clairvoyance Group 2015.

34 Systems of Record Systems of Authority (Access points) Physical instance of data Integrity Score Accuracy Score Business Metadata Where is creation & consumption happening? Technical Data Lineage Completeness Score Data Quality Metadata People Metadata Business Data Element (I.e. written premium) Core Metadata Technical Metadata Consistency Score Where does it live and come from? Conformance Score All rights reserved by Data Clairvoyance Group 2015.

35 Enriched data elements lead to quantifiable confidence surrounding a given data element Overall Meta Quality + = Data Overall Data Quality Certification All rights reserved by Data Clairvoyance Group 2015.

36 Leveraging intelligence to manage data similar to how tangible assets are managed i n s Overall Meta Quality + Overall Data Quality Balance Sheet of Data X n s Y Algorithm All rights reserved by Data Clairvoyance Group 2015.

37 Balance Sheet of Data The Management Mechanism for all Chief Data Officers Data Steward Data Certification Score Julius Stocking 3.00 Juliana Lum 4.25 Kayleigh Berryhill 4.30 Erica Demma 4.67 Allen Hayton 4.67 Kimberely Truesdell 4.93 Business Subject Area Business Data Element Data Certification Data Steward Score Customer Active Customer 3.4 Kimberely Truesdell Customer Social Security Number 2.2 Vivien Rebelo Customer Address Line Text 1.4 Carina Sypher Customer Birth Date 4.1 Dorothy Oglesby Customer Business Process Channel 3.3 Darline Shearer Customer Business Process Channel Subtype 4.2 Dorothy Oglesby Customer City Name 2.0 Vivien Rebelo Customer Party 2.0 Phillip Kroenke All rights reserved by Data Clairvoyance Group 2015.

38 Capturing & structuring names, roles & functions can be extremely powerful metadata John, Marketing (Data Steward) Anna, IT (Technologist) Customer Name Sally, Enterprise Architecture (Architect) Bill, Marketing (Consumer) 38

39 Data Certification Score 3.6 Overall Data Quality Score 4.2 Customer Name Overall Data Completeness 94% Data Accuracy 65% 39

40 Name, Function (Role) Name, Function (Role) Data Element Name, Function (Role) Name, Function (Role) 40

41 Analysis on performance & Human Behavior related to data. 41

42 6 0 Data Steward Technologists Consumers Architects Data Custodians 42

43 Role or Function All rights reserved by Data Clairvoyance Group /15/

44 Community Cards The Management Mechanism to drive REAL accountability Data Steward Data Certification Score Data Steward Julius Stocking 3.00 Juliana Lum 4.25 Kayleigh Berryhill 4.30 Erica Demma 4.67 Allen Hayton 4.67 Kimberely Truesdell

45 Community Cards The Management Mechanism to drive REAL accountability Data Custodian Data Certification Score Carol Winters 4 Julia Townsand 3 Bob Sanders 2 Bill Flibert 1 45

46 Data Clairvoyance A business approach to data. Business Driven Technology Enablement Business Data Strategy for Leverage & Management Data Improvement Portfolio Planning & Management Organizational Design for Data Centric Companies Metadata Design, Management and Implementation Data Quality Analysis & Measurement Data Improvement SDLC Integration Real data practitioners, delivering real improvements to your enterprise data assets.