Informatica Data Quality (Version 9.5.1) User Guide

Transcription

1 Informatica Data Quality (Version 9.5.1) User Guide

2 Informatica Data Quality User Guide Version December 2012 Copyright (c) Informatica. All rights reserved. This software and documentation contain proprietary information of Informatica Corporation and are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. This Software may be protected by U.S. and/or international Patents and other Patents Pending. Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided in DFARS (a) and (a) (1995), DFARS (1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. The information in this product or documentation is subject to change without notice. If you find any problems in this product or documentation, please report them to us in writing. Informatica, Informatica Platform, Informatica Data Services, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange, PowerMart, Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica B2B Data Transformation, Informatica B2B Data Exchange Informatica On Demand, Informatica Identity Resolution, Informatica Application Information Lifecycle Management, Informatica Complex Event Processing, Ultra Messaging and Informatica Master Data Management are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies. All rights reserved. Copyright Sun Microsystems. All rights reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All rights reserved.copyright Aandacht c.v. All rights reserved. Copyright Genivia, Inc. All rights reserved. Copyright Isomorphic Software. All rights reserved. Copyright Meta Integration Technology, Inc. All rights reserved. Copyright Intalio. All rights reserved. Copyright Oracle. All rights reserved. Copyright Adobe Systems Incorporated. All rights reserved. Copyright DataArt, Inc. All rights reserved. Copyright ComponentSource. All rights reserved. Copyright Microsoft Corporation. All rights reserved. Copyright Rogue Wave Software, Inc. All rights reserved. Copyright Teradata Corporation. All rights reserved. Copyright Yahoo! Inc. All rights reserved. Copyright Glyph & Cog, LLC. All rights reserved. Copyright Thinkmap, Inc. All rights reserved. Copyright Clearpace Software Limited. All rights reserved. Copyright Information Builders, Inc. All rights reserved. Copyright OSS Nokalva, Inc. All rights reserved. Copyright Edifecs, Inc. All rights reserved. Copyright Cleo Communications, Inc. All rights reserved. Copyright International Organization for Standardization All rights reserved. Copyright ej-technologies GmbH. All rights reserved. Copyright Jaspersoft Corporation. All rights reserved. Copyright is International Business Machines Corporation. All rights reserved. Copyright yworks GmbH. All rights reserved. Copyright Lucent Technologies. All rights reserved. Copyright (c) University of Toronto. All rights reserved. Copyright Daniel Veillard. All rights reserved. Copyright Unicode, Inc. Copyright IBM Corp. All rights reserved. Copyright MicroQuill Software Publishing, Inc. All rights reserved. Copyright PassMark Software Pty Ltd. All rights reserved. Copyright LogiXML, Inc. All rights reserved. Copyright Lorenzi Davide, All rights reserved. Copyright Red Hat, Inc. All rights reserved. Copyright The Board of Trustees of the Leland Stanford Junior University. All rights reserved. Copyright EMC Corporation. All rights reserved. Copyright Flexera Software. All rights reserved. This product includes software developed by the Apache Software Foundation ( and other software which is licensed under the Apache License, Version 2.0 (the "License"). You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This product includes software which was developed by Mozilla ( software copyright The JBoss Group, LLC, all rights reserved; software copyright by Bruno Lowagie and Paulo Soares and other software which is licensed under the GNU Lesser General Public License Agreement, which may be found at The materials are provided free of charge by Informatica, "as-is", without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. The product includes ACE(TM) and TAO(TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University, University of California, Irvine, and Vanderbilt University, Copyright ( ) , all rights reserved. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (copyright The OpenSSL Project. All Rights Reserved) and redistribution of this software is subject to terms available at and This product includes Curl software which is Copyright , Daniel Stenberg, <daniel@haxx.se>. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. The product includes software copyright ( ) MetaStuff, Ltd. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at license.html. The product includes software copyright , The Dojo Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes ICU software which is copyright International Business Machines Corporation and others. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright Per Bothner. All rights reserved. Your right to use such materials is set forth in the license which may be found at kawa/software-license.html. This product includes OSSP UUID software which is Copyright 2002 Ralf S. Engelschall, Copyright 2002 The OSSP Project Copyright 2002 Cable & Wireless Deutschland. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by Boost ( or under the Boost software license. Permissions and limitations regarding this software are subject to terms available at / This product includes software copyright University of Cambridge. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright 2007 The Eclipse Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software licensed under the terms at doc/ license.html, license.html, licenseagreement; license.html;

3 software/tcltk/license.html, iodbc/license; and This product includes software licensed under the Academic Free License ( the Common Development and Distribution License ( the Common Public License ( the Sun Binary Code License Agreement Supplemental License Terms, the BSD License ( the MIT License ( and the Artistic License ( This product includes software copyright Joe WaInes, XStream Committers. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by the Indiana University Extreme! Lab. For further information please visit This product includes software developed by Andrew Kachites McCallum. "MALLET: A Machine Learning for Language Toolkit." (2002). This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,501; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775; 6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,162,643; 7,243,110, 7,254,590; 7,281,001; 7,421,458; 7,496,588; 7,523,121; 7,584,422; ; 7,720,842; 7,721,270; and 7,774,791, international Patents and other Patents Pending. DISCLAIMER: Informatica Corporation provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of noninfringement, merchantability, or use for a particular purpose. Informatica Corporation does not warrant that this software or documentation is error free. The information provided in this software or documentation may include technical inaccuracies or typographical errors. The information in this software and documentation is subject to change at any time without notice. NOTICES This Informatica product (the "Software") includes certain drivers (the "DataDirect Drivers") from DataDirect Technologies, an operating company of Progress Software Corporation ("DataDirect") which are subject to the following terms and conditions: 1. THE DATADIRECT DRIVERS ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. 2. IN NO EVENT WILL DATADIRECT OR ITS THIRD PARTY SUPPLIERS BE LIABLE TO THE END-USER CUSTOMER FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR OTHER DAMAGES ARISING OUT OF THE USE OF THE ODBC DRIVERS, WHETHER OR NOT INFORMED OF THE POSSIBILITIES OF DAMAGES IN ADVANCE. THESE LIMITATIONS APPLY TO ALL CAUSES OF ACTION, INCLUDING, WITHOUT LIMITATION, BREACH OF CONTRACT, BREACH OF WARRANTY, NEGLIGENCE, STRICT LIABILITY, MISREPRESENTATION AND OTHER TORTS. Part Number: DQ-UG

4 Table of Contents Preface.... vi Informatica Resources.... vi Informatica Customer Portal.... vi Informatica Documentation.... vi Informatica Web Site.... vi Informatica How-To Library.... vi Informatica Knowledge Base.... vii Informatica Multimedia Knowledge Base.... vii Informatica Global Customer Support.... vii Part I: Informatica Data Quality Concepts Chapter 1: Introduction to Data Quality Data Quality Overview Chapter 2: Reference Data Reference Data Overview User-Defined Reference Data Informatica Reference Data Reference Data and Transformations Reference Tables Reference Table Structure Managed and Unmanaged Reference Tables Content Sets Character Sets Classifier Models Pattern Sets Probabilistic Models Regular Expressions Token Sets Creating a Content Set Creating a Reusable Content Expression Part II: Data Quality Features in Informatica Developer Chapter 3: Column Profiles in Informatica Developer Column Profile Concepts Overview Column Profile Options Rules Table of Contents i

5 Scorecards Column Profiles in Informatica Developer Filtering Options Sampling Properties Creating a Single Data Object Profile Chapter 4: Column Profile Results in Informatica Developer Column Profile Results in Informatica Developer Column Value Properties Column Pattern Properties Column Statistics Properties Exporting Profile Results from Informatica Developer Chapter 5: Rules in Informatica Developer Rules in Informatica Developer Overview Creating a Rule in Informatica Developer Applying a Rule in Informatica Developer Chapter 6: Scorecards in Informatica Developer Scorecards in Informatica Developer Overview Creating a Scorecard Chapter 7: Mapplet and Mapping Profiling Mapplet and Mapping Profiling Overview Running a Profile on a Mapplet or Mapping Object Comparing Profiles for Mapping or Mapplet Objects Generating a Mapping from a Profile Chapter 8: Reference Data Reference Tables Overview Reference Table Data Properties Creating a Reference Table Object Creating a Reference Table from a Flat File Creating a Reference Table from a Relational Source Copying a Reference Table in the Model Repository Part III: Data Quality Features in Informatica Analyst Chapter 9: Column Profiles in Informatica Analyst Column Profiles in Informatica Analyst Overview Column Profiling Process Profile Options Profile Results Option ii Table of Contents

6 Sampling Options Drilldown Options Creating a Column Profile in the Analyst Tool Editing a Column Profile Running a Profile Creating a Filter Managing Filters Synchronizing a Flat File Data Object Synchronizing a Relational Data Object Chapter 10: Column Profile Results in Informatica Analyst Column Profile Results in Informatica Analyst Overview Profile Summary Column Values Column Patterns Column Statistics Column Profile Drilldown Drilling Down on Row Data Applying Filters to Drilldown Data Column Profile Export Files in Informatica Analyst Profile Export Results in a CSV File Profile Export Results in Microsoft Excel Exporting Profile Results from Informatica Analyst Chapter 11: Rules in Informatica Analyst Rules in Informatica Analyst Overview Predefined Rules Predefined Rules Process Applying a Predefined Rule Expression Rules Expression Rules Process Creating an Expression Rule Chapter 12: Scorecards in Informatica Analyst Scorecards in Informatica Analyst Overview Informatica Analyst Scorecard Process Metrics Metric Weights Adding Columns to a Scorecard Running a Scorecard Viewing a Scorecard Editing a Scorecard Defining Thresholds Table of Contents iii

7 Metric Groups Drilling Down on Columns Viewing Trend Charts Scorecard Notifications Notification Message Template Setting Up Scorecard Notifications Configuring Global Settings for Scorecard Notifications Scorecard Integration with External Applications Viewing a Scorecard in External Applications Chapter 13: Exception Record Management Exception Record Management Overview Exception Management Process Flow Reserved Column Names Exception Management Tasks Viewing and Editing Bad Records Updating Bad Record Status Viewing and Filtering Duplicate Record Clusters Editing Duplicate Record Clusters Consolidating Duplicate Record Clusters Viewing the Audit Trail Chapter 14: Reference Tables Reference Tables Overview Reference Table Properties General Reference Table Properties Reference Table Column Properties Create Reference Tables Creating a Reference Table in the Reference Table Editor Create a Reference Table from Profile Data Creating a Reference Table from Profile Columns Creating a Reference Table from Column Values Creating a Reference Table from Column Patterns Create a Reference Table From a Flat File Analyst Tool Flat File Properties Creating a Reference Table from a Flat File Create a Reference Table from a Database Table Creating a Database Connection Creating a Reference Table from a Database Table Copying a Reference Table in the Model Repository Reference Table Management Managing Columns Managing Rows iv Table of Contents

8 Finding and Replacing Values Exporting a Reference Table Audit Trail Events Viewing Audit Trail Events Rules and Guidelines for Reference Tables Index Table of Contents v

9 Preface The Informatica Data Quality User Guide is written for Informatica users who create and run data quality processes in the Informatica Developer and Informatica Analyst client applications. The Informatica Data Quality User Guide contains information about profiles and other objects that you can use to analyze the content and structure of data and to find and fix data quality issues. Informatica Resources Informatica Customer Portal As an Informatica customer, you can access the Informatica Customer Portal site at The site contains product information, user group information, newsletters, access to the Informatica customer support case management system (ATLAS), the Informatica How-To Library, the Informatica Knowledge Base, the Informatica Multimedia Knowledge Base, Informatica Product Documentation, and access to the Informatica user community. Informatica Documentation The Informatica Documentation team takes every effort to create accurate, usable documentation. If you have questions, comments, or ideas about this documentation, contact the Informatica Documentation team through at We will use your feedback to improve our documentation. Let us know if we can contact you regarding your comments. The Documentation team updates documentation as needed. To get the latest documentation for your product, navigate to Product Documentation from Informatica Web Site You can access the Informatica corporate web site at The site contains information about Informatica, its background, upcoming events, and sales offices. You will also find product and partner information. The services area of the site includes important information about technical support, training and education, and implementation services. Informatica How-To Library As an Informatica customer, you can access the Informatica How-To Library at The How-To Library is a collection of resources to help you learn more about Informatica products and features. It includes articles and interactive demonstrations that provide solutions to common problems, compare features and behaviors, and guide you through performing specific real-world tasks. vi

10 Informatica Knowledge Base As an Informatica customer, you can access the Informatica Knowledge Base at Use the Knowledge Base to search for documented solutions to known technical issues about Informatica products. You can also find answers to frequently asked questions, technical white papers, and technical tips. If you have questions, comments, or ideas about the Knowledge Base, contact the Informatica Knowledge Base team through at Informatica Multimedia Knowledge Base As an Informatica customer, you can access the Informatica Multimedia Knowledge Base at The Multimedia Knowledge Base is a collection of instructional multimedia files that help you learn about common concepts and guide you through performing specific tasks. If you have questions, comments, or ideas about the Multimedia Knowledge Base, contact the Informatica Knowledge Base team through at KB_Feedback@informatica.com. Informatica Global Customer Support You can contact a Customer Support Center by telephone or through the Online Support. Online Support requires a user name and password. You can request a user name and password at Use the following telephone numbers to contact Informatica Global Customer Support: North America / South America Europe / Middle East / Africa Asia / Australia Toll Free Brazil: Mexico: North America: Toll Free France: Germany: Italy: Netherlands: Portugal: Spain: Switzerland: United Kingdom: Toll Free Australia: New Zealand: Standard Rate India: Standard Rate Belgium: France: Germany: Netherlands: United Kingdom: Preface vii

11 viii

12 Part I: Informatica Data Quality Concepts This part contains the following chapters: Introduction to Data Quality, 2 Reference Data, 4 1

13 C H A P T E R 1 Introduction to Data Quality This chapter includes the following topic: Data Quality Overview, 2 Data Quality Overview Use Informatica Data Quality to analyze the content and structure of your data and enhance the data in ways that meet your business needs. You use Informatica applications to design and run processes to complete the following tasks: Profile data. Profiling reveals the content and structure of data. Profiling is a key step in any data project, as it can identify strengths and weaknesses in data and help you define a project plan. Create scorecards to review data quality. A scorecard is a graphical representation of the quality measurements in a profile. Standardize data values. Standardize data to remove errors and inconsistencies that you find when you run a profile. You can standardize variations in punctuation, formatting, and spelling. For example, you can ensure that the city, state, and ZIP code values are consistent. Parse data. Parsing reads a field composed of multiple values and creates a field for each value according to the type of information it contains. Parsing can also add information to records. For example, you can define a parsing operation to add units of measurement to product data. Validate postal addresses. Address validation evaluates and enhances the accuracy and deliverability of postal address data. Address validation corrects errors in addresses and completes partial addresses by comparing address records against address reference data from national postal carriers. Address validation can also add postal information that speeds mail delivery and reduces mail costs. Find duplicate records. Duplicate analysis calculates the degrees of similarity between records by comparing data from one or more fields in each record. You select the fields to be analyzed, and you select the comparison strategies to apply to the data. The Developer tool enables two types of duplicate analysis: field matching, which identifies similar or duplicate records, and identity matching, which identifies similar or duplicate identities in record data. Manage exceptions. An exception is a record that contains data quality issues that you correct by hand. You can run a mapping to capture any exception record that remains in a data set after you run other data quality processes. You review and edit exception records in the Analyst tool or in Informatica Data Director for Data Quality. Create reference data tables. Informatica provides reference data that can enhance several types of data quality process, including standardization and parsing. You can create reference tables using data from profile results. 2

14 Create and run data quality rules. Informatica provides rules that you can run or edit to meet your project objectives. You can create mapplets and validate them as rules in the Developer tool. Collaborate with Informatica users. The Model repository stores reference data and rules, and this repository is available to users of the Developer tool and Analyst tool. Users can collaborate on projects, and different users can take ownership of objects at different stages of a project. Export mappings to PowerCenter. You can export and run mappings in PowerCenter. You can export mappings to PowerCenter to reuse the metadata for physical data integration or to create web services. Data Quality Overview 3

15 C H A P T E R 2 Reference Data This chapter includes the following topics: Reference Data Overview, 4 User-Defined Reference Data, 5 Informatica Reference Data, 6 Reference Data and Transformations, 6 Reference Tables, 7 Content Sets, 8 Reference Data Overview A reference data object contains a set of data values that you perform search operations in source data. You can create reference data objects in the Developer tool and Analyst tool, and you can import reference data objects to the Model repository. The Data Quality Content installer includes reference data objects that you can import. You can create and edit the following types of reference data: Reference tables A reference table contains standard and alternative versions of a set of data values. You add a reference table to a transformation in the Developer tool to verify that source data values are accurate and correctly formatted. A database table contains at least two columns. One column contains the standard or preferred version of a string, and other columns contain alternative versions. When you add a reference table to a transformation, the transformation searches the input port data for values that also appear in the table. You can create tables with any data that is useful to the data project you work on. Content Sets Content sets are repository and file objects that contain reference data values. Content sets are similar in structure to reference tables but they are more commonly used for lower-level There are different types of content sets. When you add a content set to a transformation, the transformation searches the input port data for values that appear in the content or for strings that match the data patterns defined in the content set. The Data Quality Content installer includes reference data objects that you can import. You download the Data Quality Content Installer from Informatica. The Data Quality Content installer includes the following types of reference data: 4

16 Informatica reference tables Database tables created by Informatica. You import Informatica reference tables when you import accelerator objects from the Content Installer. The reference tables contain standard and alternative versions of common business terms from several countries. The types of reference information include telephone area codes, postcode formats, first names, Social Security number formats, occupations, and acronyms. You can edit Informatica reference tables. Informatica content sets Content sets created by Informatica. You import content sets when you import accelerator objects from the Content Installer. A content set contains different types of reference data that you can use to perform search operations in data quality transformations. Address reference data files Reference data files that identify all valid addresses in a country. The Address Validator transformation reads this data. You cannot create or edit address reference data files. The Content Installer installs files for the countries that you have purchased. Address reference data is current for a defined period and you must refresh your data regularly, for example every quarter. You cannot view or edit address reference data. Identity population files Contain information on types of personal, household, and corporate identities. The Match transformation and the Comparison transformation use this data to parse potential identities from input fields. You cannot create or edit address identity population files. The Content Installer writes population files to the file system. User-Defined Reference Data You can use the values in a data object to create a reference data object. For example, you can select a data object or profile column that contains values that are specific to a project or organization. The column values let you create custom reference data objects for a project. You can build a reference data object from a data column in the following cases: The data rows in the column contain the same type of information. The column contains a set of data values that are either correct or incorrect for the project. Note: Create a reference object with incorrect values when you want to search a data set for incorrect values. The following table lists common examples of project data columns that can contain reference data: Information Stock Keeping Unit (SKU) codes Employee codes Reference Data Example Use an SKU column to create a reference table of valid SKU code for an organization. Use the reference table to find correct or incorrect SKU codes in a data set. Use an employee code or employee ID column to create a reference table of valid employee codes. Use the reference table to find errors in employee data. User-Defined Reference Data 5

17 Information Customer account numbers Customer names Reference Data Example Run a profile on a customer account column to identify account number patterns. Use the profile to create a token set of incorrect data patterns. Use the token set to find account numbers that do not conform to the correct account number structure. When a customer name column contains first, middle, and last names, you can create a probabilistic model that defines the expected structure of the strings in the column. Use the probabilistic model to find data strings that do not belong in the column. Informatica Reference Data You purchase and download address reference data and identity population data from Informatica. You purchase an annual subscription to address data for a country, and you can download the latest address data from Informatica at any time during the subscription period. The Content Installer user downloads and installs reference data separately from the applications. Contact an Administrator tool user for information about the reference data installed on your system Reference Data and Transformations Several transformations read reference data to perform data quality tasks. The following transformations can read reference data: Address Validator. Reads address reference data to verify the accuracy of addresses. Case Converter. Reads reference data tables to identify strings that must change case. Classifier. Reads content set data to identify the type of information in a string. Comparison. Reads identity population data during duplicate analysis. Labeler. Reads content set data to identify and label strings. Match. Reads identity population data during duplicate analysis. Parser. Reads content set data to parse strings based on the information the contain. Standardizer. Reads reference data tables to standardize strings to a common format. You can create reference data objects in the Developer tool and Analyst tool. For example, you can create a reference table from column profile data. You can export reference tables to the file system. The Data Quality Content Installer file set includes Informatica reference data objects that you can import. 6 Chapter 2: Reference Data

18 Reference Tables A reference table contains the standard versions of a set of data values and any alternative version of the values that you may want to find. You add reference tables to transformations in the Developer tool. You create reference tables in the following ways: Create a reference table object and enter data values. Create a reference table from column profile results. Create a reference table from data in a flat file. Create a reference table from data in another database table. When you create a reference table, the Model repository stores the table metadata. The staging database or another database stores the column data values. After you create a reference table, you can add and edit columns, rows, and data values. You can also search and replace values in reference table rows. Reference Table Structure Most reference tables contain at least two columns. One column contains the correct or required versions of the data values. Other columns contain different versions of the values, including alternative versions that may appear in the source data. The column that contains the correct or required values is called the valid column. When a transformation reads a reference table in a mapping, the transformation looks for values in the non-valid columns. When the transformation finds a non-valid value, it returns the corresponding value from the valid column. You can also configure a transformation to return a single common value instead of the valid values. The valid column can contain data that is formally correct, such as ZIP codes. It can contain data that is relevant to a project, such as stock keeping unit (SKU) numbers that are unique to an organization. You can also create a valid column from bad data, such as values that contain known data errors that you want to search for. For example, a Developer tool user creates a reference table that contains a list of valid SKU numbers in a retail organization. The user adds the reference table to a Labeler transformation and creates a mapping with the transformation. The user runs the mapping on a product database table. When the mapping runs, the Labeler creates a column that identifies the product records that do not contain valid SKU numbers. Reference Tables and the Parser Transformation You create a reference table with a single column when you want to use the table data in a pattern-based parsing operation. You configure the Parser transformation to perform pattern-based parsing, and you import the data to the transformation configuration. Managed and Unmanaged Reference Tables Reference tables store metadata in the Model repository. Reference tables can store column data in the reference data database or in another database. The Content Management Service stores the database connection for the reference data database. A managed reference table stores column data in the reference data database. You can edit the values of a managed table in the Analyst tool and Developer tool. An unmanaged reference table stores column data in a database other than the reference data database. You cannot edit the values of an unmanaged table in the Analyst tool or Developer tool. Reference Tables 7

19 Content Sets A content set is a Model repository object that you use to store reusable content expressions. A content expression is an expression that you can use in Labeler and Parser transformations to identify data. You can create content sets to organize content expressions into logical groups. For example, if you create a number of content expressions that identify Portuguese strings, you can create a content set that groups these content expressions. Create content sets in the Developer tool. Content expressions include character sets, pattern sets, regular expressions, and token sets. Content expressions can be system-defined or user-defined. System-defined content expressions cannot be added to content sets. User-defined content expressions can be reusable or non-reusable. Character Sets A character set contains expressions that identify specific characters and character ranges. You can use character sets in Labeler transformations that use character labeling mode. Character ranges specify a sequential range of character codes. For example, the character range "[A-C]" matches the uppercase characters "A," "B," and "C." This character range does not match the lowercase characters "a," "b," or "c." Use character sets to identify a specific character or range of characters as part of labeling operations. For example, you can label all numerals in a column that contains telephone numbers. After labeling the numbers, you can identify patterns with a Parser transformation and write problematic patterns to separate output ports. Character Set Properties Configure properties that determine character labeling operations for a character set. The following table describes the properties for a user-defined character set: Property Label Standard Mode Start Range End Range Advanced Mode Range Character Delimiter Character Description Defines the label that a Labeler transformation applies to data that matches the character set. Enables a simple editing view that includes fields for the start range and end range. Specifies the first character in a character range. Specifies the last character in a character range. For a range with a single character, leave this field blank. Enables an advanced editing view where you can manually enter character ranges using range characters and delimiter characters. Temporarily changes the symbol that signifies a character range. The range character reverts to the default character when you close the character set. Temporarily changes the symbol that separates character ranges. The delimiter character reverts to the default character when you close the character set. 8 Chapter 2: Reference Data

20 Classifier Models A classifier model analyzes input strings and determines the types of information they contain. You use a classifier model in a Classifier transformation. You can use a classifier model when input strings contain significant amounts of data. For example, you can use a classifier model and Classifier transformation to identify the types of information in a set of documents. You export the text from each document, and you store the text of each document as a separate field in a single data column. The Classifier transformation reads the data and classifies the information in each field according to the labels defined in the model. The classifier model contains the following columns: A column that contains the words and phrases that may exist in the input data. The transformation compares the input data with the data in this column. A column that contains descriptive labels that may define the information in the data. The transformation returns a label from this column as output. The classifier model also contains logic that the Classifier transformation uses to calculate the correct information type for the input data. The Model repository stores the metadata for the classifier model object. The column data and logic is stored in a file in the Informatica installation directory structure. Note: You cannot create or edit a classifier model in the Developer tool. Classifier Models and the Core Accelerator Informatica includes a classifier model in the set of prebuilt mappings and reference data objects called the Core Accelerator. The Core Accelerator is part of the Informatica Data Quality product. You download the Core Accelerator from Informatica with the Data Quality Content Installer. When you download the Data Quality Content Installer, find the Core Accelerator xml file in the Content Installer file set. Use the Developer tool to import the accelerator objects. The import operation writes the model object to the Model repository and the model data file to the Informatica file system. Pattern Sets A pattern set contains expressions that identify data patterns in the output of a token labeling operation. You can use pattern sets to analyze the Tokenized Data output port and write matching strings to one or more output ports. Use pattern sets in Parser transformations that use pattern parsing mode. For example, you can configure a Parser transformation to use pattern sets that identify names and initials. This transformation uses the pattern sets to analyze the output of a Labler transformation in token labeling mode. You can configure the Parser transformation to write names and initials in the output to separate ports. Pattern Set Properties Configure properties that determine the patterns in a pattern set. The following table describes the property for a user-defined pattern set: Property Pattern Description Defines the patterns that the pattern parser searches for. You can enter multiple patterns for one pattern set. You can enter Content Sets 9

21 Property Description patterns constructed from a combination of wildcards, characters, and strings. Probabilistic Models A probabilistic model identifies tokens by the types of information they contain and by their positions in an input string. You use probabilistic models with the Labeler and Parser transformations. Select a probabilistic model when you want to label or parse values on an input port into separate output ports. A probabilistic model uses a structured set of tokens as a reference data set. A labeling or parsing operation can use a probabilistic model to answer the following questions about the data that it reads on a port: Does the port data contain a token that matches the reference data in the model? What type of information does the token contain? A probabilistic model contains the following columns: An input column that represents the data on the input port. You populate the column with sample data from the input port. The model uses the sample data as reference data in parsing and labeling operations. One or more label columns that identify the types of information in each input string. You add the columns to the model, and you assign labels to the tokens in each string. Use the label columns to indicate the correct position of the tokens in the string. 10 Chapter 2: Reference Data

22 The following figure shows a probabilistic model in the Developer tool: When you configure a token labeling operation with a probabilistic model, the Labeler transformation writes the column name from the probabilistic model to an output port on the transformation. For example, the Labeler can use a probabilistic model to label the string "Franklin Delano Roosevelt" as "FIRSTNAME MIDDLENAME LASTNAME." When you configure a token parsing operation with a probabilistic model, each column you add to the model becomes an output port on the Parser transformation. The transformation writes each token to an output port based on its position in the model. Probabilistic Logic Probabilistic models behave differently to other types of content set. Data Quality can infer a match between the input port data values and the model data values even if the port data is not listed in the model. This means that a probabilistic model does not need to list every token in a data set to correctly label or parse the tokens in the data set. Data Quality uses probabilistic or fuzzy logic to identify tokens on the transformation input port that match tokens in the probabilistic model. The engine updates the fuzzy logic rules when you compile the probabilistic model. Content Sets 11

23 Probabilistic Model Advanced Properties The Advanced Properties dialog box exposes the computational properties that are built into a probabilistic model when you compile the model. The basic element in the compilation of probabilistic models is the n-gram. An n-gram is a series of letters that can be followed or preceded by one or more letters to complete a word. Probabilistic analysis creates n-grams for each value in the Input column of the probabilistic model. The analysis adds one or more letters to each n-gram to create different words. If the probabilistic analysis can create a word that matches a value on a Labeler or Parser transformation input port, then the analysis determines that the Input value in the probabilistic model matches the input value on the transformation port. The advanced properties on a probabilistic model determine how the probabilistic model handles n-grams and other model features. Note: The default property values represent the preferred settings for probabilistic analysis and probabilistic model compilation in Informatica. If you edit an advanced property, you may adversely affect the accuracy of the probabilistic analysis. Do not edit the advanced properties unless you understand the effects of the changes you make. Steps to Create a Probabilistic Model You create a probabilistic model in multiple stages. Complete the tasks associated with each stage to create and configure a model that you can use in a transformation. Complete the following tasks: Create the probabilistic model object in the repository You can use a data object to create the model, or you can create an empty model. Assign labels to the input data If the probabilistic model does not contain labels for the input data values, you must assign the labels. Compile the probabilistic model When you have entered the input data and configured the labels, you compile the model. You compile every time you edit the model. Creating an Empty Probabilistic Model You can use a data object as the source for the data in a probabilistic model, or you can create an empty model. Create an empty probabilistic model when you want to enter the reference data at a later time. Complete the following steps to create an empty probabilistic model: 1. In Object Explorer, open or create a content set. 2. Select the Content view. 3. Select Probabilistic Models, and click Add. The Probabilistic Model wizard opens. 4. Select the Probabilistic Model option. Click Next. 5. Enter a name for the model. Click Finish and save the model. The probabilistic model opens in the Developer tool. After you create the empty model, you must add input data. 12 Chapter 2: Reference Data

24 Creating a Probabilistic Model from a Data Object You can use a data object as the source for the data in a probabilistic model. For example, use the source data object from the mapping that will read the probabilistic model. You can also profile an object in the mapping and create a data object from the profile results. Probabilistic model logic works best when you use data from the input port on the transformation to populate the input and label columns in the model. Complete the following steps to create a probabilistic model from a data object: 1. In Object Explorer, open or create a content set. 2. Select the Content view. 3. Select Probabilistic Models, and click Add. The Probabilistic Model wizard opens. 4. Select the Probabilistic Model from Data Objects option. Click Next. 5. Enter a name for the model, and browse to the data object you want to use. Click Next. 6. Review the available data columns on the data object, and select a column to add as input data or label data to the model. To add a data source column to the Input column in the model, select the column name and click Data >. To use a data source column as a label source for the model, select the column name and click Label >. Click Next. 7. Select the number of rows to copy from the data source. Select all rows, or enter the number of rows to copy. If you enter a number, the model counts the rows from the start of the data set. 8. Set the delimiters to use for the Input column and Data columns. The delimiters apply when the columns contain multiple tokens. The default delimiter is \s, which represents a character space. 9. Enter a name for a column to contain any token that the labeling or parsing operation cannot recognize. The default name is O, which stands for Overflow. 10. Click Finish and save the model. The probabilistic model opens in the Developer tool. 11. Click Compile to build the probabilistic logic rules for the model. Assigning Labels to Probabilistic Model Data If the data object you use to create the probabilistic model does not contain columns for label data, you must add the data. A label is a column name in the probabilistic model. The model uses the column name to identify different types of information in the input data. You create the label columns, and you assign a label to each token in each input row. When you assign a label to a token, the model adds the token to the label column. Follow these guidelines when you assign labels to input data: A label identifies the type of information that the token represents. A token may represent multiple types of information if it appears in multiple locations in the input string. For example, you can assign the labels FIRSTNAME LASTNAME to the names "John Blake" and "Blake Smith." You must assign a label to every token in every row, even if the tokens repeat in multiple rows. Content Sets 13