Building an Effective Data Quality Management Program

Transcription

1 Building an Effective Data Quality Management Program May 14, 2007 TDWI Boston, MA David Loshin 6/4/2007 1

2 The Data Quality Value Proposition Growing awareness of value of high quality data: 2006 TDWI Report on Enterprise Data Quality reports that nearly half of survey respondents claim that the quality of their organization s data is worse than everyone thinks. 2005, Ted Friedman from Gartner: Fortune 1000 companies will spend or lose more on operational efficiencies in the back office than on data warehousing or CRM. These seem like shocking statistics, but the points of business failure include denial about data quality issues." 2003 Cutter survey reports 60% to 90% of all data warehouse projects either fail to meet expectations or are abandoned PWC 2001 Data Management Survey reports that 75% of the senior executives reported significant problems as a result of defective data TDWI Report on Data Quality calculates that the cost of poor data quality to US businesses exceeds $600 billion each year 2

3 Reality Organizational information is increasingly coming under scrutiny, due to both internal and external factors The more data is shared, the more opportunities there are for exposing problems There is a growing acceptance and appreciation for the value of enterprise data quality 3

4 Agenda Objectives of a Data Quality Program Building the Business Case Expectations and Dimensions of Data Quality Governance and Oversight Policies and Protocols Data Standards Data Quality Technology Data Quality Management 4

5 Objectives of a Data Quality Program Data quality defined Information compliance Data quality management goals 5

6 What is Data Quality? Fitness for purposes Instead of defect-freedom, measured compliance with user expectations above agreed-to thresholds When is data quality at an acceptable level? Fuzzy notions of good vs. bad data Different criteria for different users Data sets are used in ways they were never intended Data Quality is Contextual 6

7 Data Quality Challenges Consumer data validation of supplied data provides little value unless supplier has an incentive to improve its product Data errors introduced within the enterprise drain resources for scrap and rework, yet the remediation process seldom results in long-term improvements Reacting to data integrity issues by cleansing the data does not improve productivity or operational efficiency Ambiguous data definitions and lack of data standards prevents most effective use of centralized source of truth and limits automation of workflow Proper data and application techniques must be employed to ensure ability to respond to business opportunities Centralization of integrated reference data opens up possibilities for reuse, both of the data and the process 7

8 Information Compliance The coordinated, measurable conformance of a collection of data instances with a set of explicitly defined data expectations expressed using a formal rule language. Can be used: To characterize fitness for purpose To motivate the definition of data expectations To determine validity of information within a set of defined constraints As the basis for business performance metrics and measurements of those metrics To enable rapid root cause analysis of information scrap and rework 8

9 Information Compliance and Data Quality Effective measurement of compliance with expectations is a core component of a data quality strategy Defining information consumer expectations provides clarity as to data quality requirements Expectations must be tied to specific business impacts Well-defined requirements provide insight into objective metrics and key data quality indicators 9

10 Addressing the Problem To effectively ultimately address data quality, we must be able to manage the Identification of customer data quality expectations Definition of contextual metrics Assessment of levels of data quality Track issues for process management Determination of best opportunities for improvement Elimination of the sources of problems Continuous measurement of improvement against baseline 10

11 A Repeatable Data Quality Process Identify actual problems with the data as they relate to business client expectations Identify specific business impacts attributable to those problems Quantify the size of those impacts for prioritization Evaluate the costs to reconcile the data quality problems Once these details have been identified, the value of improved data quality can be quantified Prioritize and select projects for improvement 11

12 DQ Management Goals Evaluate business impact of poor data quality and develop performance models for Data Quality activities Document the information architecture showing data models, metadata, information usage, and information flow throughout enterprise Identify, document, and validate Data Quality expectations Educate your staff in ways to integrate Data Quality as an integral component of system development lifecycle Management framework for Data Quality event tracking and ongoing Data Quality measurement, monitoring, and reporting of compliance with customer expectations Consolidate current and planned Data Quality guidelines, policies, and activities 12

13 Business Data Quality Management Initiatives Establishing data quality monitoring and improvement as a business imperative Acquiring, then deploying the proper tools, methods, and expertise to improve the exploitation of reference information Transitioning from a reactive to a proactive organization with respect to data quality Prepare the organization to be a high-quality information environment 13

14 Turning Data Quality into Process Quality Institute a data governance framework Use business-driven data validity assessment to baseline current state and to measure ongoing improvement Establish data quality issues tracking to improve internal remediation within an accountability chain Develop a services-based approach to your centralized reference master(s) Establish best practices for data management for other enterprise data sets 14

15 Building the Business Case Identify key business performance criteria related to information quality assurance Review how data problems contribute to each business impact Determine the frequency that each impact occurs Sum the measurable costs associated with each impact incurred by a data quality issue Assign an average cost to each occurrence of the problem Validate the evaluation within a data governance forum 15

16 Business Impacts Regulatory or Legislative risk System Development risk Information Integration Risk Investment risk Health risk Privacy risk Competitive risk Fraud Detection Delayed/lost collections Customer attrition Lost opportunities Increased cost/volume Decreased Revenues Increased Risk Low Confidence Increased Costs Detection and correction Prevention Spin control Scrap and rework Penalties Overpayments Increased resource costs System delays Increased workloads Increased process times Organizational trust issues Impaired decision-making Lowered predictability Impaired forecasting Inconsistent management reporting 16

17 Examples Increased Costs Large Office Supply Company: Redundant data and unused data accounted to large percentage of their storage usage Elimination of unused or redundant data would result in significant (20%) reduction in DASD (and corresponding data management) costs DoD Guidelines on Data Quality: the inability to match payroll records to the official employment record can cost millions in payroll overpayments to deserters, prisoners, and ghost soldiers. the inability to correlate purchase orders to invoices is a major problem in unmatched disbursements. Manufacturing Company: Inability to determine that similar components had already been designed and built incurred duplicated design and development costs exceeding $70,000 per item 17

18 Examples Decreased Revenues Telecommunications company: Applied revenue assurance to detect underbilling indicated revenue leakage of just over 3 percent of total revenue due to poor data quality Identified 49 misconfigured (but assumed to be unusable) high-bandwidth circuits that could be returned to productive use Federal Agency: 55 percent of the records in a building database were wrong, resulting in the underbilling of $12 million in rent. Another Federal Agency: Stale contact addresses slows process of collecting debt obligations 18

19 Examples Decreased Confidence Pharmaceutical company: Large investment made in creating front-end sales application fed by back-end database Application clients refused to use new application due to mistrust of back-end database Agriculture company: Multiple sales databases conflicted with accounting databases Sales staff did not trust that their commissions were being properly calculated 19

20 Examples Increased Risk Pharmaceutical/Medical Device company Party database used to manage grantees Grantees may also be providers Inability to properly track grantees exposed company to risk of violating Federal Anti-Kickback statute Banking industry, credit risk: PWC estimates that 90% of the top 100 world banks are deficient in credit risk data management in maintenance of clean counterparty static data repositories, common counterparty identifiers,, staff dedicated to data quality, consistent data standards. 20

21 Cause and Effect(s) The Information Flow Graph A data flaw introduced here may be irrelevant A data flaw introduced at this processing stage Determining the value of fixing the process where the flaw is introduced must be correlated to the cost of the eventual business impacts. But you also have to find out where the flaw is introduced! propagates through this processing stage and ultimately impacts business results at these stages 21

22 Data Flaws Incur Business Impacts An Example Missing product id, inaccurate product description at data entry point Inventory Fulfillment 1. Slower turnover of stock 2. Stock write downs 4. Out of stock at customers 5. Inability to deliver orders 6. Inefficiencies in sales promotions Product data is not standardized, multiple systems have inconsistent data Logistics 7. Distribution errors and rework 8. Unnecessary deliveries 9. Extra shipping costs 22

23 Correlating Data Flaws to Business Impacts Problem Missing product id, inaccurate product description at data entry point Issue Inability to clearly identify known products leads to inaccurate forecasts Business Impact Slower turnover of stock Stock write downs Out of stock at customers Inability to deliver orders Inefficiencies in sales promotions Distribution errors and rework Shipping costs Unnecessary deliveries 23

24 Researching Costs and Impacts Historical data associated with work/process flows during critical events can provide cost/impact details Consult: Issues tracking system event logs Management reports on staff allocation for problem resolution Interview key personnel Review external impacts (e.g., stock price, management spin) Identify key quantifiers for business impact 24

25 Correlating Data Flaws to Business Impacts Problem Issue Business Impact Quantifier Missing product id, inaccurate product description at data entry point Inability to clearly identify known products leads to inaccurate forecasts Slower turnover of stock Increased costs Stock write downs Increased costs Out of stock at customers Lost revenue Inability to deliver orders Lost revenue Inefficiencies in sales promotions Speed to market Distribution errors and rework Staff time Shipping costs Shipping costs Unnecessary deliveries Staff time 25

26 Assigning Costs Evaluate actual costs based on quantifying variables Examples: Count the number of extra hours of staff time are incurred when flaws require increased manual intervention Sum up increased shipping, logistics, inventory costs Secondary risk associated with potential restatement of revenue Assign probabilities to risk impacts Collect incurred costs as related to underlying problems (this will come in handy when prioritizing solutions) 26

27 Quantification Variables For Data Failures Problem Issue Business Impact Quantifier Yearly Incurred Impact Missing product id, inaccurate product description at data entry point Inability to clearly identify known products leads to inaccurate forecasts Slower turnover of stock Increased cost $30, Stock write downs Increased cost $20, Out of stocks at customers Lost revenue Inability to deliver orders Lost revenue $250, Inefficiencies in sales promotions Speed to market (and lost revenue) $20, Distribution errors and rework Staff time $24, Shipping costs Increased shipping costs $78, Unnecessary deliveries Staff time $23,

28 Costs to Remediate Assess costs to develop and maintain a solution: Tools System design, development, implementation Hardware Maintenance Staffing 28

29 Assessing Solution Investment - Example Problem Issue Solution Software Costs Staffing Missing product id, inaccurate product description at data entry point Inability to clearly identify known products leads to inaccurate forecasts Parsing and Standardization, Record, monitoring linkage tools for cleansing $150, for license 15% annual maintenance.75 FTE for 1 year.15 FTE for annual maintenance 29

30 Examples and Their Results Pharmaceutical company: Our data quality assessment and impact analysis is being used to justify the staffing of a data quality program Manufacturing company: Assembling a business case to justify a Master Data Management approach to consolidating a global product master repository Health Insurer: Very small investment in proactive data quality monitoring yields over 10X return on investment Federal Agency: Now investing in proactive address standardization and cleansing to reduce or eliminate delays in debt collection Telecommunications: A marketing campaign based on cleansed data results in a 23 percent decrease in the cost of customer acquisition relative to prior efforts 30

31 Business Expectations & Dimensions of Data Quality Review of impacts of poor data quality Introduction to Dimensions of Data Quality Review of Dimensions of Data Quality Data Quality Performance Monitoring 31

32 Business Expectations and Data Quality Data Quality Rules Business Expectations Duplicates Inconsistencies Missing values Unusable data Throughput Scrap/rework Failed transactions Response to opportunities 32

33 Data Flaws Incur Business Impacts An Example Missing product id, inaccurate product description at data entry point Inventory Fulfillment 1. Slower turnover of stock 2. Stock write downs 4. Out of stock at customers 5. Inability to deliver orders 6. Inefficiencies in sales promotions Product data is not standardized, multiple systems have inconsistent data Logistics 7. Distribution errors and rework 8. Unnecessary deliveries 9. Extra shipping costs 33

34 What are Dimensions of Data Quality? The concept of a dimension evokes thoughts of measurement Different dimensions are intended to represent different measurable aspects of data quality Used in characterizing relevance across a set of application domains and to ensure an enterprise standard of data quality Measurements are taken to review data quality performance at different levels of the operational hierarchy Monitoring overall both line-of-business and enterprise performance Each group within the organization has the freedom to introduce its own dimensions with customized characteristics. 34

35 Categorization of Dimensions Conformance Governance Application Operations Policies Standards 35

36 Objective Data Quality Dimensions Characteristics Criteria Metrics Thresholds 36

37 Dimensions of Data Quality Intrinsic Contextual Conformance Accuracy Timeliness Fitness Provenance Currency Compliance Semantic Completeness Structure Consistency Rating Aggregation Transformation 37

38 Caveat This list is intended as guidance and as a starting point for defining the dimensions that are relevant within the organization The methods for measurement should be identified before agreeing to the selection of a metric The metrics should be correlated with the impacts identified during the impact analysis 38

39 Accuracy Integral to ensuring that the data values that are managed are accurate with respect to systems of record System of Record A registry of data sets for accuracy corroboration exists Precision Data elements are defined with the proper level of precision Value Acceptability Acceptable values for each data element are defined Domain Definitions Commonly used data value sets have conceptual domains defined Value domains for conceptual domains are enumerated once Value Accuracy Data values are accurate 39

40 Provenance Critical to managing compliance with information policies (e.g., limitation of use), root cause analysis, and quality ratings Documentation of originating data source Do data elements incorporate attribution of its original source and date Are provenance audit trails archived? 40

41 Semantic Semantic consistency refers to: consistency of definitions among attributes within a data model similarly named attributes in different enterprise data sets the degree to which similar data objects share consistent names and meanings 41

42 Semantic Data definitions A metadata repository with all data elements named and defined is available for all participants Conformance to naming conventions An enterprise naming convention has been documented and all data element names conform to the convention Name ambiguity No two elements share the same name Semantic consistency Similarly named data attributes are assured to refer to the same business concept 42

43 Structure Syntactic consistency Formats of shared data elements that share the same value set have the same size and data type Documentation of common types Data element length and type are specified in the metadata repository 43

44 Timeliness Accessibility Newly posted records should be available to enterprise applications within a specified time period Policies specifying acceptable time delays must be provided. Response time Ensure that requested data is provided within the acceptable time period Expectations for response time must be specified 44

45 Currency Age/Freshness The acceptable time period lifetime between updates for each data element is defined - Expiry date Time of release The date/time upon which the data becomes available is defined If data is expected to be delivered to specified participants, the release date/time should be specified Synchronization of replicas Data synchronizations and replication policies between systems must be specified Correction/update promulgation Polices for promulgation of corrections and updates, must be specified. Temporal Temporal Consistency rules are valid 45

46 Completeness Population Density Specify the minimum degree of population for each data element Optionality Mandatory attributes are expected to have assigned values in a data set Optionality must be specified for all data elements Null validation Null value rules for all data elements are defined Null value rules are conformed to 46

47 Consistency Presentation Common presentation formats for each data element are defined Presentation completeness Each data presentation format can convey all information within the attributes Null presentation Standards for the presentation of missing information for each data type are defined Capture and collection Data entry edits and data importation rules should be defined for each data element 47

48 Aggregation Entity uniqueness No entity exists more than once within the system Search and match A probability of a successful or partial match for the identifying information associated with a specific record will be defined Coverage The central repository is expected to identify the universe of unique entities across the enterprise The potential total universe of entities by classification must be defined Linkage Links between data records in different data sets is properly maintained 48

49 Transformation Transformation metadata Data transforms are documented within the enterprise metadata repository Transformation validation Data values are correctly transformed from source to target 49

50 Compliance Standards and policies Enterprise-wide information standards are specified Enterprise-wide information policies are specified Participant applications conform to standards and policies 50

51 Fitness for Use Authoritative sources Trust Anonymity 51

52 What Makes a Good Metric? Clarity of Definition Measurability Business Relevance Controllability Representation Reportability Trackability Drilldown Capability 52

53 Data Quality Dashboard Identify key performance indicators Select them from the list of characteristics of data quality dimensions Provide a reporting scheme Use existing reporting and dashboard applications, if possible Integrate with the data governance framework 53

54 Practical Steps Review data quality goals to associate preferred results in terms of units of performance Specify desired results for data quality by reprioritizing or changing the performance criteria Ensure the desired results directly contribute to the enterprise results Identify more first-level data quality dimensions measures to evaluate if and how well desired results are being achieved Document a performance plan -- including desired results and measures Conduct ongoing observations and measurements to track performance Exchange ongoing feedback about performance Conduct a performance appraisal (sometimes called performance review) If performance meets the desired performance standard, then provide incentives for performance 54

55 Governance and Oversight Awareness of the problem Establishing a value proposition Engaging the appropriate sponsors Data Governance frameworks 55

56 Does Data Quality Imply Business Value? Data Quality Scorecard Prioritizing Impacts Root Cause Analysis Productivity Decision-making 56

57 The Challenge of Sponsorship Business partners will expect business justification and a reasonable business case ( ROI ) before funding an ongoing program But The right people in the organization who get it can transcend the need for ROI Objective: Socialize the value of improved data quality to support ongoing business productivity improvement as well as operational process improvement Engage stakeholders with implicit or explicit ownership Foster a collaborative approach to oversight and management 57

58 Roles and Responsibilities Executive Sponsorship Data Governance Oversight Provide senior management support at the C-level, warrants the enterprise adoption of measurably high quality data, and negotiates quality SLAs with external data suppliers. Strategic committee composed of business clients to oversee the governance program, ensure that governance priorities are set and abided by, delineates data accountability. Data Coordination Council LOB Data Governance LOB Data Governance LOB Data Governance LOB Data Governance Tactical team tasked with ensuring that data activities have defined metrics and acceptance thresholds for quality meeting business client expectations, manages governance across lines of business, sets priorities for LOBs and communicates opportunities to the Governance Oversight committee. Data governance structure at the line of business level, defines data quality criteria for LOB applications, delineates stewardship roles, reports activities and issues to Data Coordination Council 58

59 Data Governance Architecture Policies and Procedures Roles & Responsibilities Ongoing Monitoring Audit & Compliance Standards Oversight Performance Metrics Data Definitions Master Reference Data Taxonomies Enterprise Architecture Exchange Standards Data Quality Data Profiling Data Cleansing Parsing & Standardization Record Linkage Data Integration Data Access Transformation Discovery & Assessment Metadata Management Auditing & Monitoring Delivery 59

60 Data Governance Oversight Board Guides activities Approves governance policies Oversees proper compliance with governance Reviews and Endorses/Approves policies and protocols 60

61 The Coordination Council Provides direction to those tasked with data quality and metadata management Authorize workgroup activities Provide direction for development of semantics, taxonomies, and ontologies Recommend standards to the Data Governance Oversight Board Ensure that data quality controls are in place Ensure that key data quality indicators are communicated to stakeholders and data owners 61

62 Technical Advisors Tasked with: Providing technical input to workgroup definitions and standards development Identifying technical and infrastructure issues with standard definitions and expected uses Assess business needs for tools and technology Updating & maintaining technical specs Providing guidance on implementation Identifying and documenting existence of source of truth data sets 62

63 Metadata Developers Encapsulate data element definitions, format specification, and semantics in a formal representation Facilitate development of: Enterprise data definitions Exchange/sharing schemas (e.g., fixed-format, XML) Exchange application support (e.g., class definitions, code development, application objects) Functional support for shared application capabilities for information life cycle 63

64 Metadata Registrar Provides support and configuration management for standards within the Metadata Registry Manages access to the Metadata Registry Facilitates and manages data standards activity workflows Helps develop procedures Promote reuse across applications 64

65 Data Steward Tasked with: Determining the relevant data sets that are relevant to the business Identifying correlation between data quality and achievement of business objectives Managing data quality techniques, tools, dimensions, tracking, reporting Documenting, communicating, and tracking issues and concerns to relevant stakeholders Verifying the metadata Assume accountability for managing the quality of data 65

66 Coordinating the Data Governance Processes Manages the various data quality activities of data owners and workgroups Compiles, maintains, and monitors data quality performance indicators in process Supports the metadata and data quality rules definition, registration, and development processes Develops policies and procedures Provides training and knowledge transfer 66

67 Policies and Protocols Root Cause Analysis The Data Quality Life Cycle Validation and Certification Data Stewardship Remediation and Manual Intervention 67

68 Root-Cause Analysis Impacts are typically associated with the discovery location of a data quality problem In fact, one impact may be related to a combination of problems Alternately, a single problem may have multiple impacts A key to improving information quality is to identify the root cause of problems and eliminate them at their sources A key to managing information quality include: Setting policies for data quality issue remediation Establishing best practices for data management Describing protocols and service level agreements for documenting, tracking, and eliminating data quality issues 68

69 Root Cause Analysis Map information flow to understand how data moves across the organization Use the dimensions of data quality to define probing measurements to assess the quality of data at points along the information data When data failures occur, employ the probes to determine at which stage the flaw was introduced 69

70 Developing Metrics Develop metrics based on relationship of information to relevant business activities Master reference information Human capital productivity Business productivity Sales channel Service level compliance Vision compliance Behavior Risk 70

71 Key Information Quality Indicators A key indicator reflects a rolled-up summary of some important aspect of the current state of the organization s information quality Sample indicators: Number of unique reference data objects (e.g., customers, vendors, products) vs. duplicate entries Number of transaction back outs Financial inconsistencies Null or missing data values Exposures to risk 71

72 A Process-Driven Approach to Data Quality 5. Monitor Data Quality against Targets 1. Identify & Measure how poor Data Quality impedes Business Objectives Data Analysis and Assessment 2. Define businessrelated Data Quality Rules & Performance Targets 4. Implement Quality Improvement Methods and Processes Data Quality Improvement and Monitoring 3. Design Quality Improvement Processes that remediate process flaws Source: Informatica 72

73 Data Quality Life Cycle Errors Time 73

74 Data Quality Life Cycle Initial investment in: Assessment Consulting Tools Time investment in: Impact analysis Building a business justification Architecting governance model Understanding data ownership paradigms Education and socialization Early life cycle resource investment in: Root cause analysis Reaction to, and evaluation to underlying issues Resolution techniques Identify dimensions of data quality Assess improvement opportunities and target measures of data quality Identify key pilot projects Evaluate business needs for tools acquisition Tools acquisition Maturation phase: Continuous improvement Identify and solidify best practices Transition from reactive to proactive Pareto Principle: 80% of benefit achieved through 20% of work Diminishing returns Later life cycle: Maturity models Proactive management Transition of governance to all staff Reduction in staff dedicated to reaction Data stewardship managed by lines of business Initial Stage Early life cycle Continuous Process Improvement Mature 74

75 Business Rules and Validation Business Rules Business Rules Management System Generated Program Information Compliance Report 75

76 Certification Ensuring that data meets defined expectation levels For each area of data quality dimension: Assert expectations based on defined business needs Identify an acceptance threshold Ensure that the metrics can be measured Assemble a set of processes to measure against business rules Assemble a process to collect and report on levels of data quality 76

77 Rule-Based Validation 77

78 Data Stewardship Established role for accountability and responsibility for the quality of a data set Formalized processes for managing data sets and processes that impact those data sets Effectiveness of data stewardship increases as the accountability is diffused across the organization 78

79 Remediation and Manual Intervention Issues with addressing data quality events: Immediate remediation of flawed data does this imply data correction? Not all data flaws can be captured via automated processes this implies manual reviews Accuracy may only be measured by comparing values directly Carefully integrate manual intervention when necessary in a controlled manner 79

80 Data Standards Standards for information sharing Critical Data Elements Data Harmonization Master data 80

81 Data Standards Partners In any enterprise, we can be confident that information exchanges are understood the same way through the definition and use of data standards Standard definitions, processes, exchanges Customers LoBs 81

82 What Makes it a Data Standard? Participants desire a common framework for communicating Participants select key stakeholders to define and publish a draft standard Participants have opportunity to read and comment on draft Most Importantly. All participants agree to use the standard 82

83 Benefits of Defining Data Standards Enable effective communication between multiple parties expressing an interest in sharing information Reduce manual intervention in data movement processes Develop opportunities for increased automation Establish a shared catalog for enterprise business terms and related exchanged data elements Streamline on-demand client access Support ongoing system upgrade requirements Improve data quality 83

84 Data Standards Challenges Absence of clarity makes it difficult to determine semantics Ambiguity in definition introduces conflict into the process Lack of Precision leads to inconsistency in representation and reporting Variant source systems and frameworks encourage turf-oriented biases Flexibility of data motion mechanisms leads to multitude of approaches for data movement 84

85 Common Business Terms and Metadata Business Definitions Extract Metadata Metadata Repository Data Standards for Information Sharing A 123 X12 A 123 T63 A 123 Y38 A 123 Data Data Model Model 123 X Y T63 85

86 Standards Review and the Metadata Registry Draft proposal is presented to broad constituency Limited time for submitting comments Comments are reviewed and addressed by workgroup members Multiple rounds of review may be allowed until all major issues are resolved Workflow activity is tracked within the metadata registry 86

87 Infrastructure Services ISO XML.GOV ebxml GJXDM Dublin Core External Standards Good Design Practices ODS DW OLAP Metadata Registry Reusable Metadata and Data Objects Facilitation of Data Standards Process Data Access Dashboards Classic Sharing Enterprise Standards XML Schemas Future Sharing 87

88 What are Critical Data Elements (CDEs)? Business facts that are deemed critical to the organization Meets one or more of these criteria: Is used by one or more external applications Contains Identifying Information Critical employee information Critical customer information Critical vendor/supplier information Is designated as critical for operational decision-making or scorecard performance Data Ownership/Responsibility/Accountability Data governance policies provide guidance for how critical information is managed and monitored Governance must account for: Ownership = ultimate accountability Stewardship = sets of responsibilities 88

89 ISO & Data Standards Object Class A set of ideas, abstractions, or things in the real world that can be identified with explicit boundaries and meaning and whose properties and behavior follow the same rules Data Type A set of distinct values, characterized by properties of those values and by operations on those values. Property A characteristic common to all members of an Object Class. Unit of Measure The unit in which any associated Data Element values are specified. Data Element Concept A concept that can be represented in the form of a Data Element, described independently of any particular representation. Non-Enumerated Conceptual Domain A Conceptual Domain that cannot be expressed as a finite set of Value Meanings. It can be expressed via a description or specification. Data Element A basic unit of data of interest to an organization (formed when a Data Element Concept is assigned a representation). Enumerated Conceptual Domain A Conceptual Domain containing a finite allowed inventory of notions (expressed as a set of Value Meanings). Representation Class A Classification Scheme for representation (a mechanism for conveying to an individual functional or presentational categories). Non-Enumerated Value Domain An expression of a Value Domain as description or specification such as a rule, a procedure, or a range. Conceptual Domain A set of Value Meanings, which may either be enumerated or expressed via a description. Enumerated Value Domain An expression of a Value Domain as an explicit set of two or more Permissible Values. Value Domain Provides representation, but has no implication as to what data element concept the values are associated nor what the values mean. Permissible Value An expression of a Value Meaning within an Enumerated Value Domain. Glossary Not in ISO 11179; a set of controlled terms with their definitions as used within a data standards environment. Value Meaning An expression that provides distinction between one member and the other members in an Enumerated Conceptual Domain. 89

90 Conceptual and Value Domains A conceptual domain is a set of valid value meanings Example: US States The set of primary governmental divisions of the United States A value domain is a set of permissible values Example: Alabama, Alaska, Arizona,.., Wisconsin, Wyoming More than one Value Domain may be associated with a single Conceptual Domain One Value Domain may be associated with more than one Conceptual Domain 90

91 Conceptual & Value Domains More than one Value Domain may be associated with a single Conceptual Domain US States AL, AK, AZ, AR, CA, CO, CT, DE, DC, FL, GA, OK, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY Alabama, Alaska, Arizona, Arkansas, California, Colorado, Connecticut, Delaware, District of Columbia, Florida, Georgia, Oklahoma, Hawaii, Idaho, Illinois, Indiana, Iowa, Kansas, Kentucky, Louisiana, Maine, Maryland, Massachusetts, Michigan, Minnesota, Mississippi, Missouri, Montana, Nebraska, Nevada, New Hampshire, New Jersey, New Mexico, New York, North Carolina, North Dakota, Ohio, Oregon, Pennsylvania, Rhode Island, South Carolina, South Dakota, Tennessee, Texas, Utah, Vermont, Virginia, Washington, West Virginia, Wisconsin, Wyoming 01, 02, 04, 05, 06, 08, 09, 10, 11, 12, 13, 40, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 41, 42, 44, 45, 46, 47, 48, 49, 50, 51, 53, 54, 55, 56 91

92 Value & Conceptual Domains Colors One Value Domain may be associated with more than one Conceptual Domain Weekdays 0,1,2,3,4,5,6 Employment Level Status Codes 92

93 Data Element A data element concept is a concept that can be represented in the form of a Data Element, described independently of any particular representation. Issuing Jurisdiction: The governing body in which a license was issued A data element is a unit of data for which the definition, identification, representation, and permissible values are specified by means of a set of attributes. Name Data Element Concept Used By Definition Data Domain Data Type Presentation Business Rules Unit of Measure Data Steward Meta Tag Name IssuingState Issuing State E-DMV The state in which the license was issued FIPS 2-Character State Abbreviation CHAR(2) CHAR(2) Not null N/A John Doe <IssuingState> 93

94 Value Domain Reuse IssuingState One Value Domain may be used by many Data Elements DriverState FIPS 2-Character State Abbreviation AL, AK, AZ, AR, CA, CO, CT, DE, DC, FL, GA, OK, HI, ID, IL, IN, IA, KS, KY, LA, ME, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NJ, NM, NY, NC, ND, OH, OR, PA, RI, SC, SD, TN, TX, UT, VT, VA, WA, WV, WI, WY CaseState DMVWorkerState 94

95 Metadata Registries and Data Standards A system for organizing information about data that is exchanged A system and tool for administering the data standards process A browsable gateway to access organizational metadata Data element definitions, business terms, object structure, exchange schemas, business rules, exchange workflows 95

96 Metadata Registry What Data Element Definitions Data Element Formats Object Definitions Object Relationships Schemas Reference Data Exchange Rules Who Recipient Sender When, Where, How Access Methods Exchange Rules Frequency Classification Administration Access Security Management Dissemination Sender Data Exchange Rules Reference Data Data Exchange Partners Recipient Data Element Group of Data Elements Related Data Elements Exchanged Message Management Access Control Security The Registry is a single resource for information necessary to define, specify, and implement information exchange Complies with ISO/IEC Standard on Metadata Registries Consistent with Enterprise Architectures Compatible with existing industry data standards 96

97 The 4-Step Data Harmonization Process Extract & Collect Integrate & Collate Identify Anomalies Resolve and Standardize Regulations Policies Documents Forms Extract data definitions from current guidance documents, etc. and categorize definitions using standard terminology Integrate data elements into a single reference source, then combine common data elements Identify inconsistent definitions, conflicts in data domains, format variations Identify authoritative source for definitions Document in metadata registry 97

98 Step 1: Extract & Collect Wage Amount Source: Exchange Output Record CHART 5: OUTPUT DETAIL RECORD Field Name Locati on Leng th Alpha/Nu meric Comments QW Employee Wage Amount A/N This field will contain the information as provided from the QW record submitted to the Employee Directory. Source: Employment Registry Name: WAGE AMOUNT Type: Output Field Condition: Conditional for the following output record: Employee Match Response Record Length: 11 Format Signed Numeric Values: through Description: The amount of a person s wages during a Reporting Quarter. The last two positions are implied to be to the right of the decimal point. Source: Employment Registry Guidance Document Data Dictionary 98

99 Step 2: Integrate and Collate Integrate data elements into a single reference source Annotate with relevant metadata Combine common data elements when feasible 99

100 Step 3: Identify Anomalies Identify potential issues or inconsistencies that become evident through the collation process Review the issues with business subject matter experts to determine the source of any anomalies and resolutions 100

101 Step 4: Resolve & Standardize Identify authoritative sources Statute Regulation Organizational Policy Contracts Subject Matter Expert Convention Assign standardized name using the Enterprise Naming Convention Document in metadata registry Capture name, definition, data type, size, etc. for general reuse 101

102 Master Data and Master Data Management Master data: Core business objects used in the different applications across the organization, along with their associated metadata, attributes, definitions, roles, connections, and taxonomies, e.g.: Customers Suppliers Parts Products Locations Contact mechanisms 102

103 Transactions Use Master Data David Loshin purchased seat 15B on US Airways flight 238 from Baltimore (BWI) to San Francisco (SFO) on July 20, Master Data Object Value Customer Product Flight Location Location David Loshin Seat 15B 238 BWI SFO 103

104 What is Master Data Management? Master Data Management (MDM) incorporates the business applications, information management methods, and data management tools to implement the policies, procedures, infrastructure that support the capture, integration, and subsequent shared use of accurate, timely, consistent and complete master data. 104

105 MDM Objectives An MDM program is intended to: Assess the use of core information objects, data value domains and business rules Identify core information objects relevant to business success Instantiate a shared standardized object model Manage collected and discovered metadata as an accessible, browsable resource Collect and harmonize replicated data instances Integrate the harmonized view via a service-oriented approach Institute the proper data governance policies and procedures at the corporate or organizational level 105

106 Data Quality Technology Sources of data flaws and errors The process-driven approach Data quality activities and technologies 106

107 What Can Go Wrong? Data entry errors Format errors or invalid data values Valid, but not correct Lack of clear agreed-to definitions Mismatched syntax, formats, and structures Multiple data suppliers to same target system Data conversion errors Changes in use and perception of data Data failure paradigms fall into two classes: Syntactic, e.g., data entry errors Semantic, e.g., nonconformance to enterprise data standards 107

108 Data Entry Errors Attribute Granularity Data granularity is not at the proper level creates confusion when more than one data element can be represented in the same attribute Example: name vs. last name, first name Finger Flubs This happens whem the incorrect letter is typed on the keybpard Also, sometimes mnore than one letter is hit by mistake Also, a leter might be missing Format Conformance When the format is too restrictive, the user may not be able to properly enter the data Example: Requirement for First name, middle initial, last name, but some people go by their middle name 108

109 More Data Entry Errors Semi-structured form There may be multiple valid formats that appear in freeform, such as corporate structure laid out at web sites, or names: (first name) (middle initial) (last name) or (last name), (first name) Transcription Errors Data is collected through fuzzy media and is not properly transcribed because of mispronounced data and/or incorrect spellings Transformation Flubs Automated processing may introduce errors, such as a database of names was found to have an inordinately large number of high-frequency word fragments, such as INCORP, ATIONAL, COMPA. 109

110 Even More Data Entry Errors Misfielded Data Data that is placed in the wrong field Example:street addresses Fields may not be big enough Text spills over to next field Floating Data Information that belongs in one field is contained in different fields in different records in the database See examples in housing authority database Overloaded Attributes More than one entity shows up in data Example: John and Mary Smith, TTES, Smith Foundation 110

111 A Process-Driven Approach to Data Quality 5. Monitor Data Quality against Targets 1. Identify & Measure how poor Data Quality impedes Business Objectives Data Analysis and Assessment 2. Define businessrelated Data Quality Rules & Performance Targets 4. Implement Quality Improvement Methods and Processes Data Quality Improvement and Monitoring 3. Design Quality Improvement Processes that remediate process flaws Source: Informatica 111

112 Engineering Data Quality into the System Flat File RDBMS Analyze/profile data Assess data quality dimensions Data quality, Validity, & Transformation rules Create monitoring system Recommend data transformations IMS VSAM Improved enterprise data quality Application Generate data quality reports Send data quality reports to data owners 112

113 Data Quality Activities and Technologies Analysis Cleansing Enhancement Monitoring Data profiling Parsing and standardization Matching Transformation Matching Enhancement Data profiling Reporting 113

114 Data Analysis Using Data Profiling Empirical analysis of ground truth Statistical analysis Functional dependency analysis Association rule analysis Rule validation can be used for monitoring Three activities: Column Cross-column Cross-table 114

115 Column Profiling Techniques Range Analysis Sparseness Format Evaluation Cardinality and Uniqueness Frequency Distribution Value Absence Abstract Type Recognition Overloading 115

116 Range Analysis Determination of restriction of values within particular value ranges Applies to: Integers Numeric Dates Character strings 116

117 Sparseness Evaluates the percentage of the elements populated May indicate an aging or unused attribute Depending on percentage populated, may relate in a consistency relationship to other attributes 117

118 Format Evaluation A process to identify patterns within sets of values Determination of the use of a known standard or format Examples: dates, SSNs, telephone numbers, some names Also may be extant in organizational definitions (e.g., product codes, SKUs, policy numbers) 118

119 Cardinality and Uniqueness Cardinality defines the number of distinct values that the attribute takes A cardinality equal to the number of records implies uniqueness Low cardinality implies a restricted value set Compare with range analysis and sparseness Unique value assignment to attribute indicates absence of functional dependency 119

120 Frequency Distribution Looks at the number of times each value appears Low frequency values ( outliers ) are potential data quality violations Small number of very high frequency values may indicate a flag or code attribute 120

121 Value Absence Two different issues: Absent values when they are expected Presence of values when absence is expected Different meanings for absent values 121

122 Abstract Type Recognition An abstract type is a predefined pattern-based sub-class of a character string type. This process can exploit format evaluation to try to deduce attribute types 122

123 Overloading An overloaded attribute is one that carries more than one piece of information Two ways this is manifested: Multiple bits of information in single value Use of field for multiple uses depending on context 123

124 Cross-Column Analysis Key discovery Normalization & structure analysis Derived-value columns Business rule discovery 124

125 Ongoing Monitoring Using Data Profiling Rule validation can be used to assert data quality expectations throughout the processing flow Use profiling jobs as probes across the information flow graph to identify where flaws are introduced Correlate occurrences of errors to documented business impact for prioritization 125

126 Parsing and Standardization (301) (999) (999) (999) (999) Area Code Exchange Line

127 Triggering Actions Whether the string matches a pattern or not, actions can be triggered, e.g.: If the string can be parsed: The tokens can be extracted and forwarded into component data element attributes Tokens can be transformed into a standard form If the string cannot be parsed into a success pattern: There may be common error patterns than can trigger corrections Uncorrectable errors can be forwarded back to the data owner 127

128 Data Correction If we can automatically recognize data as not conforming to a standard, can we automate its correction? If we have translation rules or mappings from incorrect values to correct values This is how many data cleansing applications work example: Internatinal International 128

129 Data Standardization Use standard form as a pivot for linkage and consolidation Example Elizabeth R. Johnson, 123 Main St Beth R. Johnson, 123 Main St It s a good hunch that these records represent the same person We can standardize components based on nicknames, abbreviations, etc. 129

130 Transformation Rules Standardization is a process of transforming nonconforming forms to conforming forms Use mappings/transformation rules Create a rule engine instance and integrate the rules Engine becomes a filter This capability is embedded in many data cleansing tools, as well as in many ETL/integration tools 130

131 Transformation Engine Application of context-sensitive consistency and derivation rules transforms a data instance into appropriate form In this case, derivation rules act on non-standard values Also referred to as edits Rule base grows as violations are noted 131

132 Parsing and Standardization - Approach Simplest approach: standard regular expression or contextfree parsing with actions attached to success states More complex: Standard parsing with variant lookahead Context-sensitive parsing Table-driven string/pattern matching 132

133 Data Cleansing Lookup Table International Bus Mach IBM Int l Bus Mach Intl Business Machines IBM Inc. International Business Machines 133

134 Matching/Record Linkage Identity recognition and harmonization Approaches used to evaluate similarity of records Use in: Duplicate analysis and elimination Merge/Purge Householding Data Enhancement Data Cleansing Customer Data Integration 134

135 Example Who is Who? Howard David Loshin Mr. David Loshin Loshin, Howard Loshin David D Loshin Jill and David Loshin Mr. Loshin HD Loshin The Loshin Family Howard Loshin David Loshin David Howard Loshin H David Loshin David H Loshin David Loshing David Losrin David Lotion David Loskin David Lashin David Lasrin David Laskin David Loshing 135

136 No, Really Who is Who? Enterprise Knowledge Management : The Data Quality Approach (The Morgan Kaufmann Series in Data Management Systems) (Paperback) by David Loshin Me Business Intelligence: The Savvy Manager's Guide (The Savvy Manager's Guides) (Paperback - June 2003) by David Loshin Me The Geometrical Optics Workbook(Paperback - June 1991) by David S Loshin Not Me 136

137 Matching/Record Linkage - Example Knowledge Integrity, Inc David Loshin David Lotion 1163 Kersey Rd Howard David Loshin Knowledge Integrity Knowledge Integrity Incorporated

138 Matching/Record Linkage Pairwise comparisons reliant on similarity scoring (inefficient) More efficient algorithms block records into candidate sets, then do pairwise comparisons Probabilistic/Statistical (Fellegi & Sunter, Jaro, Winkler) Deterministic (rule-based) Approximate string matching Phonetic compression (Soundex, NYSIIS) N-grams Winkler (probability-based) 138

139 Data Correction Correction by consolidation Makes use of record linkage Find a pivot attribute across which to link The pivot should be unique (such as social security number) Link records together and consolidate correct name based on other factors, such as data source, timestamp, etc. 139

140 Data Enhancement Data improvement process that relies on record linkage Value-added improvement from third-party data sets: Address correction Geo-Demographic/Psychographic imports List append Typically partnered with data providers 140

141 Subject Number Percent Subject Number Percent SCHOOL ENROLLMENT OCCUPANCY STATUS Total housing units Occupied housing units 1,704 1, Population 3 years and over enrolled in school 1, Nursery school, preschool Kindergarten Elementary school (grades 1-8) High school (grades 9-12) College or graduate school Vacant housing units Tenure Occupied housing units Owner-occupied housing units Renter-occupied housing units 23 1,681 1, EDUCATIONAL ATTAINMENT Population 25 years and over 3, Less than 9th grade th to 12th grade, no diploma High school graduate (includes equivalency) Some college, no degree Associate degree Bachelor's degree Graduate or professional degree Percent high school graduate or higher 89.2 (X) Percent bachelor's degree or higher 35.4 (X) 141

142 Enhancement: USPS Address Standardization Multiple address lines Recipient line Delivery Address line Last line Person or entity to whom mail is to be delivered Knowledge Integrity, Inc Kersey Rd Suite 100 Silver Spring, MD Contains location information City, state, and ZIP 142

143 Address Standardization First: Is the address already in standard form? Address special cases (East West Hwy) Identify all addressing elements Make sure placement is correct; if not, correct it Are street and city names valid? Is the address number valid within the street address ranges? Next: Correct if necessary Identify all address elements Look up proper city name Look up correct ZIP+4 Move elements to proper location in address block Transform elements into standard abbreviated form Generate bar code (if needed) 143

144 Auditing and Monitoring Data Quality Performance Data Quality Scorecard Data quality, Validity, & Transformation rules Prioritizing Impacts Generate data quality reports Root Cause Analysis Send data quality reports to data owners Productivity Decision-making 144