Data Quality in the MDM Ecosystem

Transcription

1 Solution Guide Data Quality in the MDM Ecosystem What is MDM? The premise of Master Data Management (MDM) is to create, maintain, and deliver the most complete and comprehensive view possible from disparate enterprise data. These composite views consolidate data related to customer, product, supplier, employee, asset, material, and location information. Final, integrated master views are then made available to applications, increasing the value they can provide as part of operations and analysis. Gartner recommends that organizations change their thinking about data integration and data quality as they are no longer just tactical issues they are in fact highly strategic. Gartner Research, 2007 Mitigating Risk of MDM Failure Past CRM project failures are most frequently attributed to poor source data quality and insufficient data insights. Such factors impact project timelines, resource allocation, the assessment and scoping of critical data, and present significant data integration and migration challenges. These same issues can be anticipated as obstacles to MDM success. Data profiling and integration challenges will weigh heavily on the success of any MDM endeavor. Without data quality there is no master data. And without master data, there is no MDM. To unlock the value of your corporate data and accelerate the insights that drive your business, you need to focus on understanding your data and improving its quality as you embark on MDM readiness initiatives.

2 Most MDM systems conceptually encompass four key elements as shown in the following MDM Ecosystem diagram. Interface & Workflow Services - Data entry & editing - Exception processing - Validation & approval processes - Hierarchy management <<< New Values Exceptions >>> <<< Queries Repository Services - Persistent storage (key x-ref & attributes) - Security & access policies - History tracking/ time-variance - Metadata Cleansed Data >>> Data Quality Services - Data standardization & validation - Profiling & monitoring - Automated enrichment - Matching & linking - Survivorship & merging records <<< Incoming Data Source & Subscriber Applications <<< Publish Integration Backbone - Subscribe & publish mechanism - Synchronous, asynch & batch - Real-time messaging - Service brokering & integration MDM Architecture As data flows throughout a best-of-breed MDM ecosystem, it will transition through four key areas. The Repository Services stores a persistent view of information that represents the most consistent and complete view of customer, supplier, product, or other master data. Which fields and how much history should be stored will vary for each organization depending on business requirements. The Integration Backbone lets source systems provide data to, and receive data from the MDM repository. There are a number of different technologies that organizations use to move data including: ETL tools, middleware, and custom code. Data Quality Services ensure that data moved into and around the MDM ecosystem adheres to defined standards. These services should be invoked prior to any new data entering the MDM repository. Robust Data Quality Services automate cleansing and enrichment functions so that all new data is consistent and maintains a pristine master data view. Interface and Workflow Services provide the capability to define and implement workflows for new master data creation as well as perform any manual data remediation necessary, based upon errors identified by the automated data quality services. At present, no single vendor can provide all these infrastructure elements. Analysts recommend that customers look for best-of-breed components to craft an MDM solution that truly meets their needs. 2

3 Some MDM vendors offer primitive data quality capabilities in their products. While this functionality can be helpful for organizations with simple data and low volumes, it is not robust enough to deal effectively with cleansing, standardizing, matching, and de-duplicating global data that possesses varying patterns and data structures from multiple sources. The Power of Profiling When you think about it, implementing an MDM solution is essentially a massive data migration and integration project. The final 360 business data view will be a composite of whatever data exists in your source systems regardless of its condition. Data accuracy, consistency, and completeness will all impact the MDM vision. Data profiling lets you gain a thorough understanding of your data quickly, comprehensively, and efficiently. It automates the process of identifying data characteristics such as the formats and structure of data attributes and whether or not they conform to required business specifications, the content and ranges of values and whether these fall within acceptable limits and domain values. It will verify the uniqueness of primary keys and dependencies between attributes. Automated data profiling can also validate rules across entities and disparate data sources relational, non-relational, flat-file, delimited, non-delimited, and different technology platforms. Profiling helps you assess risk and create accurate and realistic project plans to secure your longer term vision. The Trillium Software System provides powerful profiling that: Scopes and prioritizes data at project inception Identifies specific data validation and transformation rules during the design phase Automates and formalizes data testing and result analysis during QA Supports data governance efforts and monitors ongoing master data health Why You Need Data Quality Arguably, the data quality engine and associated rule-sets are the most important part of the whole MDM ecosystem. Without a strong capability to produce valid, complete and clean data, the end result is an expensive, new repository filled with the same unreliable information that already exists in source systems. A powerful data quality engine lets project teams repurpose and extend out-of-the-box and customized business rules and make them available as services to the enterprise. These rules can be put in place as Data Quality Services and used repeatedly to cleanse both operational source systems as well as the MDM environment itself. 3

4 Gartner Research proposes that for MDM success, MDM and all data integration activities must be infused with data quality controls, metrics and governance from a process and technology angle and that data integration without data quality will fail. (Source: Gartner Leader s Toolkit, August 2007). What Can We Do for You? The Trillium Software System applies rich and powerful capabilities to the following activities: Legacy Data Cleansing, Consolidation and Migration Building an MDM repository requires cleansing, integrating, and loading existing legacy data. Support periodic batch/ bulk loads with cleansing, validation, deduplication, and enrichment activities when adding new source systems or monthly feeds to the MDM system. New Master Record Data Entry Support and enhance the capturing of new master records directly in the MDM interface or in upstream business applications, such as the call center. Use real-time calls to validate, verify, enrich, and de-duplicate data. Inbound Data Validation, Cleansing, and Standardization Leverage a range of data validation capabilities and automate the process of cleansing and standardizing data to reduce manual correction of data issues. Our automated approach ensures standardization and auditability as it corrects erroneous or non-standard values entering the MDM system (in real-time). Enrich Data Enhance master data with appended attributes for geographical/ location data through out-of-the-box tables and with other enrichment capabilities. Duplicate Record Prevention Prevent the creation of duplicate records in MDM systems and in business applications, such as ERP and CRM, through a real-time call that checks whether a new record will duplicate a record already in the MDM repository. Identify and Maintain Duplicate Record Key Mappings Maintain a key cross reference table showing how different representations of customers, products, etc. relate to each other for MDM implementations not intended to prevent or remove duplicate records in subscribing applications. Survivor Records Identify which record or pieces of a record, among a set of identified duplicates, should be incorporated into the master view even when the requirements call for a complex approach that calls for field-specific update logic. 4

5 Creating Golden Records Create a best representation of a given customer or product by stitching together a golden record based on individual attribute values from several source systems. Master Record Search Many MDM systems need to access a particular customer, product, or other master entity based on some loose search criteria. Automate searches using powerful matching capabilities. MDM Repository Data Monitoring Monitor and report proactively on the data residing in the MDM repository and the data flows that feed into it. Track data quality and communicate via web-based, graphical views to report these results to a wide range of users and stakeholders. Ongoing Data Governance During the course of implementing an MDM strategy, organizations will endure the challenges of defining corporate data standards and a common understanding of critical fields. Use data quality services to enforce these standards and measure compliance with standards as part of ongoing operations. Without a solid data quality strategy, MDM projects are at risk of failure. The Trillium Software System offers a single user experience for complete data quality life cycle management that spans the distributed enterprise. When vendors innovate to provide rudimentary technical capabilities in areas where other vendors specialize, they come up short. Although some MDM vendors offer cumbersome data quality capabilities in their products, MDM applications are not purpose-built to improve data quality, just as applications lack the rich capabilities found in word processing applications. Using technology not expressly designed for required functions severely limits organizational capabilities, business process alignment, and final results. Achieving a high level of data quality entails using a platform built specifically to deliver high data quality. Reap the benefits of impeccable and total data quality with the Trillium Software System and see results that save you money. 5

6 Functionality Trillium Software System Typical MDM Platform Profiling Automated Identification and Routing Context- Sensitive Data Cleansing Verification & Enrichment Matching Merging and Survivorship Ongoing Governance Highly interactive data analysis workbench Rich views of patterns, relationships and anomalies in the data Deductive approach based on all available data Out-of-the-box rules Context-sensitive Out-of-the-box rules Address verification and geo-coding out-of-the-box Easy to append other enrichment data Robust matching based on tunable and auditable rules Highly granular, multi-matching options Robust selection of surviving values on field-by-field basis Measure conformance to established standards Flag records for manual remediation efforts Not included Deterministic logic based on a limited set of fields Rules must be created One-to-one substitution based on literal values Rules must be created Not included Capabilities vary from very basic (exact) matching to fairly robust fuzzy logic Capabilities vary from very basic logic to select surviving record to fairly robust fuzzy logic For some, remediation interfaces exist No automation of non-compliant record identification Corporate Headquarters 300 Concord Road, Suite 200 Billerica, MA USA trlinfo@trilliumsoftware.com Europe, Middle East, Africa Theale Court High Street Theale RG7 5AH UK +44 (0) trillium.uk@trilliumsoftware.com Germany, Switzerland Trillium Software GmbH Otto-Lilienthal-Straße Böblingen Germany Australia Office C Toorak Road TOORAK VIC 3142 Australia