Informatica (Version 9.1.0) Data Explorer User Guide

Transcription

1 Informatica (Version 9.1.0) Data Explorer User Guide

2 Informatica Data Explorer User Guide Version March 2011 Copyright (c) Informatica. All rights reserved. This software and documentation contain proprietary information of Informatica Corporation and are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. This Software may be protected by U.S. and/or international Patents and other Patents Pending. Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided in DFARS (a) and (a) (1995), DFARS (1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. The information in this product or documentation is subject to change without notice. If you find any problems in this product or documentation, please report them to us in writing. Informatica, Informatica Platform, Informatica Data Services, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange, PowerMart, Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica B2B Data Transformation, Informatica B2B Data Exchange, Informatica On Demand, Informatica Identity Resolution, Informatica Application Information Lifecycle Management, Informatica Complex Event Processing, Ultra Messaging and Informatica Master Data Management are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies. All rights reserved. Copyright Sun Microsystems. All rights reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All rights reserved.copyright Aandacht c.v. All rights reserved. Copyright Genivia, Inc. All rights reserved. Copyright 2007 Isomorphic Software. All rights reserved. Copyright Meta Integration Technology, Inc. All rights reserved. Copyright Oracle. All rights reserved. Copyright Adobe Systems Incorporated. All rights reserved. Copyright DataArt, Inc. All rights reserved. Copyright ComponentSource. All rights reserved. Copyright Microsoft Corporation. All rights reserved. Copyright Rogue Wave Software, Inc. All rights reserved. Copyright Teradata Corporation. All rights reserved. Copyright Yahoo! Inc. All rights reserved. Copyright Glyph & Cog, LLC. All rights reserved. Copyright Thinkmap, Inc. All rights reserved. Copyright Clearpace Software Limited. All rights reserved. Copyright Information Builders, Inc. All rights reserved. Copyright OSS Nokalva, Inc. All rights reserved. Copyright Edifecs, Inc. All rights reserved. This product includes software developed by the Apache Software Foundation ( and other software which is licensed under the Apache License, Version 2.0 (the "License"). You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This product includes software which was developed by Mozilla ( software copyright The JBoss Group, LLC, all rights reserved; software copyright by Bruno Lowagie and Paulo Soares and other software which is licensed under the GNU Lesser General Public License Agreement, which may be found at The materials are provided free of charge by Informatica, "as-is", without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. The product includes ACE(TM) and TAO(TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University, University of California, Irvine, and Vanderbilt University, Copyright ( ) , all rights reserved. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (copyright The OpenSSL Project. All Rights Reserved) and redistribution of this software is subject to terms available at This product includes Curl software which is Copyright , Daniel Stenberg, <daniel@haxx.se>. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. The product includes software copyright ( ) MetaStuff, Ltd. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at license.html. The product includes software copyright , The Dojo Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at svn.dojotoolkit.org/dojo/trunk/license. This product includes ICU software which is copyright International Business Machines Corporation and others. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright Per Bothner. All rights reserved. Your right to use such materials is set forth in the license which may be found at kawa/software-license.html. This product includes OSSP UUID software which is Copyright 2002 Ralf S. Engelschall, Copyright 2002 The OSSP Project Copyright 2002 Cable & Wireless Deutschland. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by Boost ( or under the Boost software license. Permissions and limitations regarding this software are subject to terms available at / This product includes software copyright University of Cambridge. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright 2007 The Eclipse Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software licensed under the terms at license.html, license.html, and This product includes software licensed under the Academic Free License ( the Common Development and Distribution License ( the Common Public License ( and the BSD License ( This product includes software copyright Joe WaInes, XStream Committers. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by the Indiana University Extreme! Lab. For further information please visit

3 This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,501; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775; 6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,162,643; 7,254,590; 7,281,001; 7,421,458; 7,496,588; 7,523,121; 7,584,422; 7,720,842; 7,721,270; and 7,774,791, international Patents and other Patents Pending. DISCLAIMER: Informatica Corporation provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of non-infringement, merchantability, or use for a particular purpose. Informatica Corporation does not warrant that this software or documentation is error free. The information provided in this software or documentation may include technical inaccuracies or typographical errors. The information in this software and documentation is subject to change at any time without notice. NOTICES This Informatica product (the Software ) includes certain drivers (the DataDirect Drivers ) from DataDirect Technologies, an operating company of Progress Software Corporation ( DataDirect ) which are subject to the following terms and conditions: 1. THE DATADIRECT DRIVERS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. 2. IN NO EVENT WILL DATADIRECT OR ITS THIRD PARTY SUPPLIERS BE LIABLE TO THE END-USER CUSTOMER FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR OTHER DAMAGES ARISING OUT OF THE USE OF THE ODBC DRIVERS, WHETHER OR NOT INFORMED OF THE POSSIBILITIES OF DAMAGES IN ADVANCE. THESE LIMITATIONS APPLY TO ALL CAUSES OF ACTION, INCLUDING, WITHOUT LIMITATION, BREACH OF CONTRACT, BREACH OF WARRANTY, NEGLIGENCE, STRICT LIABILITY, MISREPRESENTATION AND OTHER TORTS. Part Number: IN-PRG

4 Table of Contents Preface.... iv Informatica Resources.... iv Informatica Customer Portal.... iv Informatica Documentation.... iv Informatica Web Site.... iv Informatica How-To Library.... iv Informatica Knowledge Base.... v Informatica Multimedia Knowledge Base.... v Informatica Global Customer Support.... v Chapter 1: Introduction to Data Profiling Data Profiling Overview Types of Profile Chapter 2: Data Profiling in the Analyst Tool Analyst Tool Data Profiling Overview Profile Options Creating a Column Profile in the Analyst Tool Creating a Quick Profile Running a Profile Previewing a Table or File... 6 Profile Results Profile Summary Column Values Column Patterns Column Statistics Viewing Profile Results Exporting Profile Results Viewing and Adding Profile Comments Wide Tables and Files Rules and Guidelines for Profiles Chapter 3: Profiles and Rules Rules Overview Process Flow for Profiles and Rules Rule Types Predefined Rules Expression Rules Table of Contents i

5 Mapplets and Rules Adding a Rule to a Column Profile Chapter 4: Export and Import Profiles Object Import and Export Overview Object Export Exporting Objects Importing Objects Chapter 5: Scorecards Scorecards Overview Creating a Scorecard Adding Columns to a Scorecard Running a Scorecard Viewing a Scorecard Drilling Down on Columns Viewing Column Data in a Scorecard Editing a Scorecard Defining Thresholds Creating a Group Moving Scores to a Group Editing a Group Deleting a Group Viewing Trend Charts Rules and Guidelines for Scorecards Chapter 6: Data Profiling in the Developer Tool Developer Tool Data Profiling Overview Profile Process Flows Data Profile Options Inferred and Validated Relationships Column Analysis Creating a Column Profile in the Developer Tool Column Profile Options Column Profile Results Multiple Data Source Profiles Creating Column Profiles for Multiple Data Objects Primary Key Analysis Primary Key Analysis Properties Functional Dependency Analysis Functional Dependency Analysis Properties Foreign Key Analysis Creating a Profile Model ii Table of Contents

6 Defining Parent and Child Object Relationships Finding Foreign Key Relationships Between Data Objects Foreign Key Analysis Results Join Analysis Creating a Join Profile Join Analysis Results Exporting Join Profile Results to File Index Table of Contents iii

7 Preface The Informatica Data Explorer Guide is written for Informatica Analyst and Informatica Developer users. It contains information about how you can use profiles to analyze the content and structure of data. Use profiles to discover data quality issues in a data set and to understand the relationships between columns in one or more data sets. Informatica Resources Informatica Customer Portal As an Informatica customer, you can access the Informatica Customer Portal site at The site contains product information, user group information, newsletters, access to the Informatica customer support case management system (ATLAS), the Informatica How-To Library, the Informatica Knowledge Base, the Informatica Multimedia Knowledge Base, Informatica Product Documentation, and access to the Informatica user community. Informatica Documentation The Informatica Documentation team takes every effort to create accurate, usable documentation. If you have questions, comments, or ideas about this documentation, contact the Informatica Documentation team through at infa_documentation@informatica.com. We will use your feedback to improve our documentation. Let us know if we can contact you regarding your comments. The Documentation team updates documentation as needed. To get the latest documentation for your product, navigate to Product Documentation from Informatica Web Site You can access the Informatica corporate web site at The site contains information about Informatica, its background, upcoming events, and sales offices. You will also find product and partner information. The services area of the site includes important information about technical support, training and education, and implementation services. Informatica How-To Library As an Informatica customer, you can access the Informatica How-To Library at The How-To Library is a collection of resources to help you learn more about Informatica products and features. It iv

8 includes articles and interactive demonstrations that provide solutions to common problems, compare features and behaviors, and guide you through performing specific real-world tasks. Informatica Knowledge Base As an Informatica customer, you can access the Informatica Knowledge Base at Use the Knowledge Base to search for documented solutions to known technical issues about Informatica products. You can also find answers to frequently asked questions, technical white papers, and technical tips. If you have questions, comments, or ideas about the Knowledge Base, contact the Informatica Knowledge Base team through at Informatica Multimedia Knowledge Base As an Informatica customer, you can access the Informatica Multimedia Knowledge Base at The Multimedia Knowledge Base is a collection of instructional multimedia files that help you learn about common concepts and guide you through performing specific tasks. If you have questions, comments, or ideas about the Multimedia Knowledge Base, contact the Informatica Knowledge Base team through at KB_Feedback@informatica.com. Informatica Global Customer Support You can contact a Customer Support Center by telephone or through the Online Support. Online Support requires a user name and password. You can request a user name and password at Use the following telephone numbers to contact Informatica Global Customer Support: North America / South America Europe / Middle East / Africa Asia / Australia Toll Free Brazil: Mexico: North America: Standard Rate North America: Toll Free France: Germany: Israel: Italy: Netherlands: Portugal: Spain: Switzerland: or United Kingdom: or Toll Free Australia: New Zealand: Singapore: Standard Rate India: Standard Rate France: Germany: Netherlands: Preface v

9 vi

10 C H A P T E R 1 Introduction to Data Profiling This chapter includes the following topics: Data Profiling Overview, 1 Types of Profile, 1 Data Profiling Overview A profile is an analysis of the content and structure of data. Create and run profiles to identify data quality issues in a data set and to understand the relationships between columns in a data set. You can use profiles to create scorecards and to create and update reference data tables. You can drill down on data rows in profiled data. You can define a profile to analyze data in a single data object or across multiple data objects. Data discovery is the first step in a data project. You run profiles to evaluate the data structure and to verify that data columns contain the types of information you expect. If a profile reveals problems in the data, you can define mappings and rules to fix those problems. You can add comments to profiles so that you can track the profiling process effectively. Types of Profile You can create several types of profile in the Analyst tool and Developer tool. You create profiles to perform the following types of data analysis: Column Analysis Analyze data quality in selected columns in a table or file. You can define profiles for column analysis in the Analyst tool and Developer tool. Primary Key Analysis Discover primary key relationships between columns in a table or file. You can define profiles for primary key analysis in the Developer tool. Functional Dependency Analysis Discover functional dependencies between columns in a table or file. 1

11 You can define profiles for functional dependency analysis in the Developer tool. Foreign Key Analysis Discover foreign key relationships between columns across multiple tables or multiple files. You can define profiles for foreign key analysis in the Developer tool. Join Analysis Determine the degree of overlap between columns in a data source or across multiple data sources. You can define profiles for join analysis in the Developer tool. Note: You can create and run column profiles in the Developer tool and the Analyst tool. Changes that you make to profiles in the Analyst tool do not appear in the Developer tool until you refresh the Developer tool connection to the repository. Disconnect from the repository in the Developer tool and then reconnect to refresh the connection. 2 Chapter 1: Introduction to Data Profiling

12 C H A P T E R 2 Data Profiling in the Analyst Tool This chapter includes the following topics: Analyst Tool Data Profiling Overview, 3 Profile Results, 6 Viewing and Adding Profile Comments, 10 Wide Tables and Files, 10 Rules and Guidelines for Profiles, 11 Analyst Tool Data Profiling Overview You perform column profiling on data objects in the Analyst tool. Column profiling discovers the following facts about data: The number of unique and null values in each column, expressed as a number and a percentage. The patterns of data in each column, and the frequencies with which these values occur. Statistics about the column values, such as the maximum and minimum lengths of values and the first and last values in each column. You can also use a profile to drill down on data rows in a data source. Profile Options You can create quick and custom profiles for a data object. You configure the profile columns, sampling, and drilldown options when you create a profile. Use a quick profile to include all columns for a data object and use the default profile options. Use a custom profile to select the columns for a data object and to configure the profile results, sampling, and drilldown options. Profile Results Option The following table describes the profile results option for a profile: Option Discard profiling results for columns and rules not selected for re-profiling Description The Analyst tool discards the profile results for previously profiled columns from the profiling warehouse and displays results for the columns and rules selected for the latest profile 3

13 Option Description run. Do not select this option if you want the Analyst tool to display profile results for previously profiled columns. Sampling Options The following table describes the sampling options for a profile: Option All Rows First <number> Rows Random Sample <number> Rows Random sample Description The Analyst tool profiles all rows in the data object. This is the default option for a quick profile. Number of rows that the Analyst tool profiles. The Analyst tool chooses the rows from the first rows in the table or file. Number of rows to randomly sample in the table or file. Random sampling forces the drill down to be performed on staged data stored in the profiling warehouse that can slow drill down performance. The Analyst tool chooses the sample size, based on the number of rows in the data object. Random sampling forces the drill down to be performed on staged data stored in the profiling warehouse that can slow drill down performance. Drilldown Options The following table describes the drilldown options for a profile: Options Enable Row Drilldown Select Columns Drilldown on live or staged data Description Drill down to row data in the profile results. Click Select Columns to identify columns for drilldown that you did not select for profiling. Note: In the Developer tool, you enable drilldown for the profiled columns or you select a custom set of columns. Select one of the following options: - Drilldown on live data. Reads the current data in the data source. - Drilldown on staged data. Reads profile data that is staged in the profiling warehouse. Creating a Column Profile in the Analyst Tool Select a data object and create a custom profile to configure the columns and sampling and drilldown options. The Analyst tool creates the profile in the same project and folder as the data object. 1. In the Navigator, select the project that contains the data object that you want to create a custom profile for. 2. In the Contents panel, select the data object. Do not click the link for the object. 4 Chapter 2: Data Profiling in the Analyst Tool

14 3. Click Actions > New > New Profile. The New Profile wizard appears. 4. In the Sources panel, select a data object. 5. Click Next. 6. Enter a name and an optional description for the profile. 7. In the Folders panel, select the project or folder where you want to create the profile. The Analyst tool displays the project that you selected and shared projects that contain folders where you can create the profile. The profile objects in the folder appear in the Profiles panel. 8. Click Next. 9. In the Columns panel, select the columns you want to profile. The columns include any rules you applied to the profile. The Analyst tool lists the name, datatype, precision, and scale for each column. Optionally, select Name to select all columns. 10. Accept the default option in the Profile Results Options panel. The first time you run the profile, the Analyst tool displays profile results for all columns selected for profiling. 11. In the Sampling Options panel, configure the sampling options. 12. In the Drilldown Options panel, configure the drilldown options. Optionally, click Select Columns to select columns to drill down on. In the Drilldown Columns window, select the columns for drill down and click OK. 13. Click Next. 14. Optionally, define a filter for the profile. 15. Click Save to create the profile, or click Save & Run to create the profile and then run the profile. Creating a Quick Profile Select a data object and create a quick profile to use the default settings for all columns and sampling and drilldown options. The Analyst tool creates the profile in the same project and folder as the data object. 1. In the Navigator, select the project that contains the data object that you want to create a quick profile for. 2. In the Contents panel, select the data object. Do not click the link for the object. 3. Click Actions > New > New Profile. The New Profile wizard appears. 4. Click Save to create the profile. Running a Profile Run a profile to analyze a data source for content and structure and select columns and rules for drill down. You can drill down on live or staged data for columns and rules. You can profile a column or rule without profiling all the source columns again after running the profile. 1. In the Navigator, select the project or folder that contains the profile you want to run. 2. Click the profile to open it. The profile appears in a tab. 3. Click Actions > Run Profile. Analyst Tool Data Profiling Overview 5

15 The Column Profile window appears. 4. In the Columns panel, select the columns you want to profile. The columns include rules you applied to the profile. Optionally, select Name to select all columns. The Analyst tool lists the name, datatype, precision, and scale for each column. 5. Optionally, in the Profile Results Options panel, select the option to discard profile results for previously profiled columns and rules. The first time you run a profile, the Analyst tool displays profile results for all columns selected for profiling. 6. In the Sampling Options panel, configure the sampling options. 7. In the Drilldown Options panel, configure the drilldown options. 8. Optionally, click Select Columns to select columns to drill down on. In the Drilldown Columns window, select the columns for drill down. 9. Click OK. 10. Click Run. The Analyst tool displays the profile results. Previewing a Table or File You can preview the rows for the table or file for a profile. You can preview the rows before or after you run the profile. You can preview the first 100 rows. 1. In the Navigator, select the project or folder that contains the profile that you preview rows for or create a profile. 2. Click the profile to open it. The profile appears in a tab. 3. Click the Data Preview view. The rows for the data source appear in the panel. 4. Click Refresh to refresh the data. Profile Results View profile results to understand the structure of the data and analyze data quality. You can view the profile results after you run a profile. You can view a summary of the columns and rules in the profile and the values, patterns, and statistics for columns and rules. You can view properties for the profile and properties for the columns and rules in the profile. You can preview profile data. After you run a profile, the Analyst tool purges the last profile run results from the profiling warehouse. If you drill down on profile results while another analyst runs the profile again, the Analyst tool displays a warning message to refresh the profile view. After you run a profile, you can view the profile results in the Column Profiling, Properties, and Data Preview views. In the Column Profiling view, you can view the summary information for columns for a profile run. You can view values, patterns, and statistics for each column in the Values, Patterns, and Statistics views. 6 Chapter 2: Data Profiling in the Analyst Tool

16 The Analyst tool displays rules as columns in profile results. The profile results for a rule appear as a profiled column. The profile results that appear depend on the profile configuration and sampling options. The following profiling results appear in the Column Profiling view: Summary. Summary information for the profile run, including the number of unique and null values, inferred datatype, and last run date and time. Values. Values for columns and the frequency in which the value appears for the column. The frequency appears as a number, a percentage, and a chart. Patterns. Value patterns for the profiled columns and the frequency in which the pattern appears. The frequency appears as a number and a percentage. Statistics. Statistics about the column values, such as average, length, and top and bottom values. Note: You can select a value or pattern and view profiled rows that match the value or pattern on the Details panel. In the Properties view, you can view profile properties on the Properties panel. You can view properties for columns and rules on the Columns and Rules panel. In the Data Preview view, you can preview the profile data. The Analyst tool includes all columns in the profile displays the first 100 rows of data. Profile Summary The summary for a profile run includes the number of unique and null values expressed as a number and a percentage, inferred datatypes, and last run date and time. You can click each profile summary property to sort on values of the property. The following table describes the profile summary properties: Property Name Unique Values Unique % Null Null % Datatype Inferred % Documented Datatype Max Value Min Value Description Name of the column in the profile. In the Developer tool, this is the Column property. Number of unique values for the column. Percentage of unique values for the column. Number of null values for the column. Percentage of null values for the column. Data type derived from the values for the column. The Analyst tool can derive the following datatypes from the datatypes of values in columns: - String - Varchar - Decimal - Integer - "-" for Nulls Percentage of values that match the data type inferred by the Analyst tool. Not displayed in the Developer tool. Data type declared for the column in the profiled object. Maximum value in the column. Minimum value in the column. Profile Results 7

17 Property Last Profiled Drilldown Description Date and time you last ran the profile. If selected, enables drilldown on live data for the column. Column Values The column values include values for columns and the frequency in which the value appears for the column. The following table describes the properties for the column values: Property Value Frequency Percent Chart Description List of all values for the column in the profile. Number of times a value appears for a column, expressed as a number, a percentage, and a chart. Percentage that a value appears for a column. Chart for the percentage. Note: You can sort the Value and Frequency columns by selecting the columns. When you sort the results of the Frequency column, the Analyst tool sorts the results based on the datatype of the column. Column Patterns The column patterns include the value patterns for the columns and the frequency in which the pattern appears. The profiling warehouse stores 16,000 unique highest frequency values including NULL values for profile results by default. If there is at least one NULL value in the profile results, the Analyst tool can display NULL values as patterns. The following table describes the properties for the column patterns: Property Pattern Frequency Percent Chart Description Pattern for the column in the profile. Number of times a pattern appears for a column, expressed as a number. Percentage that a pattern appears for a column. Chart for the percentage. Column Statistics The column statistics include statistics about the column values, such as average, length, and top and bottom values. For a string column, the Analyst tool displays the Maximum Length, Minimum Length, Bottom, and Top statisctics. 8 Chapter 2: Data Profiling in the Analyst Tool

18 For an integer column, the Analyst tool displays the Average, Standard Deviation, Maximum Length, Minimum Length, Bottom, and Top statistics. The following table describes the types of column statistics: Statistic Average Standard Deviation Maximum Length Minimum Length Bottom Top Description Average of the values for the column. The standard deviation, or variability between column values, for all values of the column. Length of the longest value for the column. Length of the shortest value for the column. Lowest values for the column. Highest values for the column. Viewing Profile Results View profile results to see a summary of the columns and rules in the profile and the values, patterns, and statistics for columns and rules. You can view the rows from the data source that represent a specific value or pattern in the profile results. 1. In the Navigator, select the project or folder that contains the profile. 2. Click the profile to open it. The profile appears in a tab. 3. In the Summary view, select a column name to view the profile results for the column. The values, patterns, and statistics for the column appear in the Results panel. Optionally, you can sort on the Values and Patterns for the column. 4. Select a column value on the Values tab or select a column pattern on the Patterns tab. Use CONTROL and SHIFT keys to select multiple values. Optionally, on the Values tab, sort on the Value and Frequency. Optionally, on the Patterns tab, sort on the Pattern and Frequency. 5. Click Actions > Drilldown to view the rows of data. The Drilldown panel displays the rows that contain the values or patterns. The column value or pattern appears at the top of the panel. Note: You can choose to drill down on live data or staged data. Exporting Profile Results You can export the results of a profile to a CSV file to view the data in a file. For each column in the profile, you can select the type of data to export. You can select value frequencies, pattern frequencies, or drilldown data for selected values and patterns. 1. In the Navigator, select the project or folder that contains the profile. 2. Click the profile to open it. The profile opens in a tab. 3. In the Column Profiling view, select the column that you want to export. Profile Results 9

19 4. Click Actions > Export Data. The Export Data to a file window appears. 5. Enter the file name. Optionally, use the default file name. 6. Select the type of data to export. You can select one of the following data options to export: Value frequencies for the selected column. Pattern frequencies for the selected column. Drilldown data for the selected values or pattern. 7. Enter a file format. Default format is CSV. 8. Select the code page of the file. 9. Click OK. Viewing and Adding Profile Comments You can view and add comments to any profile object. Use comments to share information about profiles with other Analyst tool users. 1. In the Navigator, select the project and folder that contains the profile. 2. Click the profile to open it. The profile appears in a tab. 3. Click Actions > Show Comments. The Comments panel appears. All comments appear in descending order by date. 4. To add a comment, enter the text for the comment in the comment box and click Add. The comment appears in the Comments panel. 5. Optionally, click Actions > Hide Comments to hide all comments. Wide Tables and Files You can profile wide tables and flat files that have a large number of columns. You can profile tables with more than 30 columns and flat files with more than 100 columns. When you create or run a profile, you can choose to select all the columns or select each column you want to include for profiling. The Analyst tool displays the first 30 columns in the data preview. You can select all columns for drilldown and view value frequencies for these columns. You can use rules that have more than 50 output fields and include the rule columns for profiling when you run the profile again. 10 Chapter 2: Data Profiling in the Analyst Tool

20 Rules and Guidelines for Profiles The rules and guidelines for profiles can enable you to profile data more effectively. Consider the following rules and guidelines when you work with profiles: Exporting profile results. The Analyst tool displays only the first 200 values for value frequencies, pattern frequencies, and drilldown data for the selected value or column. When you export the profile results, the Analyst tool exports all values to the CSV file. Exporting column patterns. When you export inferred column patterns in the profile results, the Analyst tool exports a different format of the column pattern to the CSV file. For example, when you export the inferred column pattern X(5), the Analyst tool displays the following format of the column pattern in the CSV file: XXXXX Inferred datatypes. The Analyst tool cannot derive the datatype from the values of a numeric column that has a precision greater than 38. The Analyst tool cannot derive the datatype from the values of a string column that has a precision greater than 255. If you have a date column on which you are creating a column profile with a year value earlier than 1800, the inferred datatype may show up as fixed length string. Change the default value for the year-minimum parameter in the InferDateTimeConfig.xml, as necessary. Column patterns. The Analyst tool cannot derive the pattern for a numeric column that has a precision greater than 38. The Analyst tool cannot derive the pattern for a string column that has a precision greater than 255. Datatypes not included in profiling. The Analyst tool excludes the CLOB, BLOB, Raw, and Binary datatypes in column values in a profile. Column names. You cannot add a column with the same name to an existing profile. You cannot add the same column twice to a profile even if you change the column name. Profiling multilingual data. You can profile multilingual data from different sources and view profile results based on the locale settings in the browser. The Analyst tool changes the Datetime, Numeric, and Decimal datatypes based on the browser locale. Profiling Unicode data on IBM DB2 databases. To profile Unicode data in a DB2 database, set the DB2CODEPAGE database environment variable in the database and restart the Data Integration Service. Sorting on multilingual data. You can sort on multilingual data. The Analyst tool displays the sort order based on the browser locale. Rules and Guidelines for Profiles 11

21 C H A P T E R 3 Profiles and Rules This chapter includes the following topics: Rules Overview, 12 Process Flow for Profiles and Rules, 12 Rule Types, 13 Adding a Rule to a Column Profile, 16 Rules Overview You can add a rule to the profile to cleanse, modify, or validate the data. When you create a profile, you select the columns in the data object for which you want to profile data. You can set or configure sampling and drilldown options for faster profiling. After you run the profile, you can examine the profiling statistics to understand the data. Run the profile again to view the data. Process Flow for Profiles and Rules When you create a profile, you select the columns in the data object for which you want to profile data. You can set or configure sampling and drilldown options for faster profiling. After you run the profile, you can examine the profiling statistics to understand the data. You can add a rule to the profile to cleanse, modify, or validate the data. Run the profile again to view the data. The following steps describe how to configure a profile and add rules to perform data quality analysis on the data: 1. Create a profile. 2. Save and run the profile. 3. Examine the profile results. 4. Determine if you need to add a rule to the profile. For example: You can add a rule because you want to cleanse one or more data columns. You can add a lookup rule that provides information that the source data does not provide. You can add a rule to validate the correctness of a cleansing rule that you intend to use in a data quality or data integration project. 12

22 5. Add the rule to the profile. You can select a predefined rule or create an expression rule. 6. Save and run the profile. 7. View the profile results. Rule Types You can add rules after you create a profile. You can add predefined rules to a profile or create expression rules to add to a profile. After you create an expression rule, you can make it reusable. Add rules to a profile by selecting a predefined rule or create an expression rule. You can create or apply the following rule types for a profile: Expression rules. Use expression functions and columns to define rule logic. Create expression rules in the Analyst tool. Predefined rules. Includes system-defined, user-defined, and reusable rules. System-defined rules are packaged with the content installer for the Developer tool and can appear as reusable rules in the Analyst tool. Rules that are created in the Developer tool as mapplets can appear in the Analyst tool as user-defined rules. An analyst can create an expression rule and promote it to a reusable rule that other analysts can use in multiple profiles. After you add a rule to a profile, you can choose to run the profile again for the rule column. The Analyst tool displays profile results for the rule column. You can modify the rule and run the profile again to view changes to the profile results. The output of a rule can be one or more virtual columns. The virtual columns exist only in the profile results. The Analyst tool profiles the virtual columns. For example, you use a predefined rule that splits a column that contains first and last names into FIRST_NAME and LAST_NAME virtual columns. The Analyst tool profiles the FIRST_NAME and LAST_NAME columns. Note: If you delete a rule object that other object types reference, the Analyst tool displays a message that lists the object types being referenced. Determine the impact of deleting the rule before you choose to delete it. Predefined Rules Predefined rules are rules created in the Developer tool or provided with the Developer tool and the Analyst tool. Apply predefined rules to the Analyst tool profiles to modify or validate source data. Predefined rules use transformations to define rule logic. You can use predefined rules with multiple profiles. In the Model repository, a predefined rule is a mapplet with an input group, an output group, and transformations that define the rule logic. You can apply the following types of predefined rules: System-defined. Packaged with the content installer for the Developer tool and can appear as reusable rules in the Analyst tool. User-defined. Created in the Developer tool as a mapplet and validated as a rule. Can appear as reusable rules in the Analyst tool. Use the New Rule Wizard to apply a predefined rule to a profile. You must open a profile before you apply a predefined rule. When you apply a predefined rule, you configure the name and description for the rule and configure the columns in the profile or reference tables as parameters for the rule. You can then include the rule in profile results and configure sampling and drilldown options. Rule Types 13

23 Applying a Predefined Rule Use the New Rule Wizard to apply a predefined rule to a profile. When you apply a predefined rule, you select the rule and configure the input and output columns for the rule. Apply a predefined rule to use a rule promoted as a reusable rule or use a rule created by a developer. 1. In the Navigator, select the project or folder that contains the profile that you want to add the rule to. 2. Click the profile to open it. The profile appears in a tab. 3. Click Actions > Add Rule. The New Rule window appears. 4. Select the option to Apply a Rule. 5. Click Next. 6. In the Rules panel, select the rule that you want to apply. The name, datatype, descripton, and precision columns appear for the Inputs and Outputs columns in the Rules Parameters panel. 7. Click Next. 8. In the Inputs section, select an input column. The input column is a column name in the profile. 9. Optionally, in the Outputs section, configure the label of the output columns. 10. Click Next. 11. In the Columns panel, select the columns you want to profile. The columns include any rules you applied to the profile. Optionally, select Name to include all columns. The Analyst tool lists the name, datatype, precision, and scale for each column. 12. In the Sampling Options panel, configure the sampling options. 13. In the Drilldown Options panel, configure the drilldown options. 14. Click Save to apply the rule or click Save & Run to apply the rule and then run the profile. Expression Rules Expression rules use expression functions and columns to define rule logic. Create expression rules and add them to a profile in the Analyst tool. Use expression rules to modify or validate values for columns in a profile. You can create one or more expression rules to use in a profile. Expression functions are SQL-like functions used to transform source data. The Analyst tool provides the following types of functions that you can use to create expression rule logic: Character Conversion Data Cleansing Date Encoding Financial Numerical Scientific 14 Chapter 3: Profiles and Rules

24 Special Test Use the New Rule Wizard to create an expression rule and add it to a profile. You must open a profile before you create an expression rule. When you create an expression rule, you configure the name and description for the rule and configure the rule logic using expression functions and columns as parameters. You can then include the rule in profile results and configure sampling options. The New Rule Wizard includes an expression editor. Use the expression editor to add expression functions, configure columns as input to the functions, validate the expression, and configure the return type, precision, and scale. The output of an expression rule is a virtual column that uses the name of the rule as the column name. The Analyst tool profiles the virtual column. For example, you use an expression rule to validate a ZIP code. The rule returns 1 if the ZIP Code is valid and 0 if the ZIP code is not valid. Informatica Analyst profiles the 1 and 0 output values of the rule. Creating an Expression Rule Use the New Rule Wizard to create an expression rule and add it to a profile. Create an expression rule to modify or validate values for columns in a profile. 1. In the Navigator, select the project or folder that contains the profile that you want to add the rule to. 2. In the Contents panel, click the profile to open it. The profile appears in a tab. 3. Click Actions > Add Rule. The New Rule window appears. 4. Select Create a rule. 5. Click Next. 6. Enter a name and optional description for the rule. 7. Optionally, choose to promote the rule as a reusable rule and configure the project and folder location. If you promote a rule to a reusable rule, you or other users can use the rule in another profile as a predefined rule. 8. In the Functions tab, select a function and click the right arrow to enter the parameters for the function. 9. In the Columns tab, select an input column and click the right arrow to add the expression in the Expression editor. You can also add logical operators to the expression. 10. Click Validate. You can proceed to the next step if the expression is valid. 11. Optionally, click Edit to configure the return type, precision, and scale. 12. Click Next. 13. In the Columns panel, select the columns you want to profile. The columns include any rules you applied to the profile. Optionally, select Name to select all columns. The Analyst tool lists the name, datatype, precision, and scale for each column. 14. In the Sampling Options panel, configure the sampling options. 15. In the Drilldown Options panel, configure the drilldown options. 16. Click Save to create the rule or click Save & Run to create the rule and then run the profile. Rule Types 15

25 Mapplets and Rules A rule is business logic that defines conditions applied to source data when you run a profile. It is a midstream mapplet that you use in a profile. A rule must meet the following requirements: It must contain an Input and Output transformation. You cannot use data sources in a rule. It can contain Expression transformations, Lookup transformations, and passive data quality transformations. It cannot contain any other type of transformation. For example, a rule cannot contain a Match transformation, as it is an active transformation. It does not specify cardinality between input groups. Note: Rule functionality is not limited to profiling. You can add any mapplet that you validate as a rule to a profile in the Analyst tool. For example, you can evaluate postal address data quality by selecting a rule configured to validate postal addresses and adding it to a profile. Validating a Mapplet Validate a mapplet before you add it to a mapping. You can also validate a mapplet as a rule to include it in a profile. 1. Right-click the mapplet canvas. 2. Select Validate As > Mapplet or Validate As > Rule. The Validation Log displays mapplet error messages. Adding a Rule to a Column Profile You can add a rule to a saved column profile. You cannot add a rule to a profile configured for join analysis. 1. Browse the Object Explorer and find the profile you need. 2. Right-click the profile and select Open. The profile opens in the editor. 3. Click the Definition tab, and select Rules. 4. Click Add. The Apply Rule dialog box opens. 5. Click Browse to find the rule you want to apply. Select a rule from a repository project, and click OK. 6. Click the Value column under Input Values to select an input port for the rule. 7. Optionally, click the Value column under Output Values to edit the name of the rule output port. The rule appears in the Definition tab. 16 Chapter 3: Profiles and Rules