Data Quality Blueprint for Pentaho: Better Data Leads to Better Results. Charles Gaddy Director Global Sales & Alliances, Melissa Data

Size: px

Start display at page:

Download "Data Quality Blueprint for Pentaho: Better Data Leads to Better Results. Charles Gaddy Director Global Sales & Alliances, Melissa Data"

Eric Black
5 years ago
Views:

1 Data Quality Blueprint for Pentaho: Better Data Leads to Better Results Charles Gaddy Director Global Sales & Alliances, Melissa Data

2 Agenda What is Data Quality, and What Role Does it Play? 6 Concepts of Data Quality Full Data Quality Lifecycle

3 What is Data Quality? Data quality is an assessment of data s fitness to serve its stated purpose. Think Fitness for use. Data Quality can be any kind of data; scientific, transactional, customers, products, assets, locations, financial Data cleansing is the verb used to achieve Data Quality.

Information Industry The data governance discipline,

Management (MDM) discipline are closely related.

4 Information Industry The data governance discipline, the data quality discipline and the Master Data Management (MDM) discipline are closely related. Data quality improvement is important within data governance and MDM. Furthermore you seldom see an MDM implementation without a (master) data governance work stream.

5 Information Industry What is data used for? Revenues 63% Service 54% Marketing 38% Risk Reduction 37% Channel Pipeline 36% New Projects 34% Regulatory 32% Survey Conducted by Melissa Global Intelligence

6 Atomic Domains of Data Quality Basic data domains represent data such as: age, date of birth, and sales amount, that are common to many businesses. Advanced data domains span the range of data classifications to provide more specific cases for your use. In addition, the rule conditions for these advanced data rule definitions can be more complex.

7 Atomic Domains of Data Quality Atomic or Entity Domains that need special handling and available domain based knowledge Personal Identity Age Date of Birth US SSN CA SIN Passport Number Asset Identity IP address Information Phone Number Address VIN Number Financial Policy Portfolio Bank Account Number Orders and Sales Order Amount Sales Amount Order ID Location Address Name Zip Code Latitude Longitude State City Country

8 Advanced Atomic Domains of Data Quality Atomic or Entity Domains that need special handling and requires custom domain based knowledge Product Items Assemblies Parts SKUs

9 6 Concepts of Data Quality Duplication Integrity Accuracy Consistency Conformity Completeness

In some cases, missing data is irrelevant, but when the

10 1. Completeness Is all the requisite information available? Are data values missing, or in an unusable state? In some cases, missing data is irrelevant, but when the information that is missing is critical to a specific business process, completeness becomes an issue.

11 2. Conformity Are there expectations that data values conform to specified formats? If so, do all the values conform to those formats? Maintaining conformance to specific formats is important in data representation, presentation, aggregate reporting, search, and establishing key relationships.

3. Consistency Do distinct data instances provide conflicting information about the same underlying data object? Are values consistent across data sets?

12 3. Consistency Do distinct data instances provide conflicting information about the same underlying data object? Are values consistent across data sets? Do interdependent attributes always appropriately reflect their expected consistency? Inconsistency between data values plagues companies attempting to reconcile between systems and applications.

13 4. Accuracy Do data objects accurately represent the real-world values they are expected to model? Incorrect spellings of product or person names, addresses, and even untimely or not current data can impact operational and analytical applications.

14 5. Duplication Are there multiple, unnecessary representations of the same data objects within your data set? The inability to maintain a single representation for each entity across your systems poses numerous vulnerabilities and risks.

15 6. Integrity What data is missing important relationship linkages? The inability to link related records together may actually introduce duplication across your systems. Not only that, as more value is derived from analyzing connectivity and relationships, the inability to link related data instance together impedes this valuable analysis.

U.S. addresses declines 17% per year Phone

16 Why Data is Always in Flux 40 million Americans (1 in 6) move annually More than 100,000 changes (adds, deletes, or modifications) every month Quality of stored U.S. addresses declines 17% per year Phone Area Code Splits Domain Changes Disconnected Phone Numbers

17 The Full Life Cycle of Data Quality

18 Profiling Gathering Metadata for Analysis Data about your data Identify the Problems NULLs/Blanks, Unnecessary Spaces, Incorrect Patterns, Unstandardized Data, etc. Overall status of the Quality of Data Statistical Analysis

19 Hygiene Data Standardization/Normalization Proper Casing Proper Formatting Removal of Unnecessary Characters Data Cleansing Misspellings Parsing Abbreviations

20 Data Verification Verifying the actual content of data Do the Addresses actually exist? Are the Phone Numbers callable? Are Addresses deliverable? Are the names actually people s names? Do the Address, Name, Phone and correspond to each other?

21 Enrich and Update Missing Information Appending Fill in missing data pieces such as a missing phone number or address Enrichment of Data Property Information, Geographic Information, Firmographics, Demographics Retrieve the latest information (eg. Move Address and Latest Phone Number) Data becomes outdated over time

22 Matching De-Duplication Duplicate data is bad data Fuzzy Matching Application of fuzzy logic algorithms for inexact matches Deep Domain Knowledge Handles matching problems in international data and in multiple domains

23 Merging Golden Record Selection Selection of the best record Consolidation and Survivorship Merging the best pieces of data according to intelligent rules

24 Monitoring Profiling Over Time Continuously gather metadata Allows for maintenance of Data Quality Data profiling with a good tool can also be employed as an active monitoring solution. Active monitoring is something that can be employed to safeguard collected data By using the same profiling techniques, it is possible to reassess the current state of the quality of data

25 Summary What is Data Quality and what role it plays? 6 concepts of Data Quality Full Data Quality Lifecycle

The Importance of Data Profiling

The Importance of Data Profiling By Bud Walker and Joseph Vertido A Melissa Data Whitepaper Introduction Data profiling is a commonly used term in the discipline of data management, yet the perception