Informatica PowerCenter Data Validation Option (Version 9.5.1) Installation and User Guide

Transcription

1 Informatica PowerCenter Data Validation Option (Version 9.5.1) Installation and User Guide

2 Informatica PowerCenter Data Validation Option Version February 2013 Copyright (c) Informatica Corporation. All rights reserved. This software and documentation contain proprietary information of Informatica Corporation and are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. This Software may be protected by U.S. and/or international Patents and other Patents Pending. Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided in DFARS (a) and (a) (1995), DFARS (1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. The information in this product or documentation is subject to change without notice. If you find any problems in this product or documentation, please report them to us in writing. Informatica, Informatica Platform, Informatica Data Services, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerExchange, PowerMart, Metadata Manager, Informatica Data Quality, Informatica Data Explorer, Informatica B2B Data Transformation, Informatica B2B Data Exchange Informatica On Demand, Informatica Identity Resolution, Informatica Application Information Lifecycle Management, Informatica Complex Event Processing, Ultra Messaging and Informatica Master Data Management are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies. All rights reserved. Copyright Sun Microsystems. All rights reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All rights reserved.copyright Aandacht c.v. All rights reserved. Copyright Genivia, Inc. All rights reserved. Copyright Isomorphic Software. All rights reserved. Copyright Meta Integration Technology, Inc. All rights reserved. Copyright Intalio. All rights reserved. Copyright Oracle. All rights reserved. Copyright Adobe Systems Incorporated. All rights reserved. Copyright DataArt, Inc. All rights reserved. Copyright ComponentSource. All rights reserved. Copyright Microsoft Corporation. All rights reserved. Copyright Rogue Wave Software, Inc. All rights reserved. Copyright Teradata Corporation. All rights reserved. Copyright Yahoo! Inc. All rights reserved. Copyright Glyph & Cog, LLC. All rights reserved. Copyright Thinkmap, Inc. All rights reserved. Copyright Clearpace Software Limited. All rights reserved. Copyright Information Builders, Inc. All rights reserved. Copyright OSS Nokalva, Inc. All rights reserved. Copyright Edifecs, Inc. All rights reserved. Copyright Cleo Communications, Inc. All rights reserved. Copyright International Organization for Standardization All rights reserved. Copyright ej-technologies GmbH. All rights reserved. Copyright Jaspersoft Corporation. All rights reserved. Copyright is International Business Machines Corporation. All rights reserved. Copyright yworks GmbH. All rights reserved. Copyright Lucent Technologies. All rights reserved. Copyright (c) University of Toronto. All rights reserved. Copyright Daniel Veillard. All rights reserved. Copyright Unicode, Inc. Copyright IBM Corp. All rights reserved. Copyright MicroQuill Software Publishing, Inc. All rights reserved. Copyright PassMark Software Pty Ltd. All rights reserved. Copyright LogiXML, Inc. All rights reserved. Copyright Lorenzi Davide, All rights reserved. Copyright Red Hat, Inc. All rights reserved. Copyright The Board of Trustees of the Leland Stanford Junior University. All rights reserved. Copyright EMC Corporation. All rights reserved. Copyright Flexera Software. All rights reserved. Copyright Jinfonet Software. All rights reserved. Copyright Apple Inc. All rights reserved. This product includes software developed by the Apache Software Foundation ( and other software which is licensed under the Apache License, Version 2.0 (the "License"). You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This product includes software which was developed by Mozilla ( software copyright The JBoss Group, LLC, all rights reserved; software copyright by Bruno Lowagie and Paulo Soares and other software which is licensed under the GNU Lesser General Public License Agreement, which may be found at The materials are provided free of charge by Informatica, "as-is", without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. The product includes ACE(TM) and TAO(TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University, University of California, Irvine, and Vanderbilt University, Copyright ( ) , all rights reserved. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit (copyright The OpenSSL Project. All Rights Reserved) and redistribution of this software is subject to terms available at and This product includes Curl software which is Copyright , Daniel Stenberg, <daniel@haxx.se>. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies. The product includes software copyright ( ) MetaStuff, Ltd. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at license.html. The product includes software copyright , The Dojo Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes ICU software which is copyright International Business Machines Corporation and others. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright Per Bothner. All rights reserved. Your right to use such materials is set forth in the license which may be found at kawa/software-license.html. This product includes OSSP UUID software which is Copyright 2002 Ralf S. Engelschall, Copyright 2002 The OSSP Project Copyright 2002 Cable & Wireless Deutschland. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by Boost ( or under the Boost software license. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright University of Cambridge. Permissions and limitations regarding this software are subject to terms available at This product includes software copyright 2007 The Eclipse Foundation. All Rights Reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software licensed under the terms at license.html, asm.ow2.org/license.html, license.html, jsch/license.txt; nanoxml.sourceforge.net/orig/copyright.html;

3 license.txt; and CreateJS/EaselJS/blob/master/src/easeljs/display/Bitmap.js; This product includes software licensed under the Academic Free License ( the Common Development and Distribution License ( the Common Public License ( the Sun Binary Code License Agreement Supplemental License Terms, the BSD License ( the MIT License ( and the Artistic License ( This product includes software copyright Joe WaInes, XStream Committers. All rights reserved. Permissions and limitations regarding this software are subject to terms available at This product includes software developed by the Indiana University Extreme! Lab. For further information please visit This Software is protected by U.S. Patent Numbers 5,794,246; 6,014,670; 6,016,501; 6,029,178; 6,032,158; 6,035,307; 6,044,374; 6,092,086; 6,208,990; 6,339,775; 6,640,226; 6,789,096; 6,820,077; 6,823,373; 6,850,947; 6,895,471; 7,117,215; 7,162,643; 7,243,110, 7,254,590; 7,281,001; 7,421,458; 7,496,588; 7,523,121; 7,584,422; ; 7,720, 842; 7,721,270; and 7,774,791, international Patents and other Patents Pending. DISCLAIMER: Informatica Corporation provides this documentation "as is" without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of noninfringement, merchantability, or use for a particular purpose. Informatica Corporation does not warrant that this software or documentation is error free. The information provided in this software or documentation may include technical inaccuracies or typographical errors. The information in this software and documentation is subject to change at any time without notice. NOTICES This Informatica product (the "Software") includes certain drivers (the "DataDirect Drivers") from DataDirect Technologies, an operating company of Progress Software Corporation ("DataDirect") which are subject to the following terms and conditions: 1. THE DATADIRECT DRIVERS ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. 2. IN NO EVENT WILL DATADIRECT OR ITS THIRD PARTY SUPPLIERS BE LIABLE TO THE END-USER CUSTOMER FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL OR OTHER DAMAGES ARISING OUT OF THE USE OF THE ODBC DRIVERS, WHETHER OR NOT INFORMED OF THE POSSIBILITIES OF DAMAGES IN ADVANCE. THESE LIMITATIONS APPLY TO ALL CAUSES OF ACTION, INCLUDING, WITHOUT LIMITATION, BREACH OF CONTRACT, BREACH OF WARRANTY, NEGLIGENCE, STRICT LIABILITY, MISREPRESENTATION AND OTHER TORTS. Part Number: PC-DVO

4 Table of Contents Preface.... ix Informatica Resources.... ix Informatica MySupport Portal.... ix Informatica Documentation.... ix Informatica Web Site.... ix Informatica How-To Library.... ix Informatica Knowledge Base.... x Informatica Multimedia Knowledge Base.... x Informatica Global Customer Support.... x Chapter 1: Data Validation Option Overview Data Validation Option Overview Data Validation Option Users Data Validation in an Enterprise Data Validation Workflow Architecture Data Validation Methodology Basic Counts, Sums, and Aggregate Tests... 3 Check Referential Integrity of Target Tables... 4 Enforce Constraints on Target Tables Compare Individual Records between Sources and Targets... 5 Data Validation with Views Data Validation Option Reporting Data Validation Option Licenses Chapter 2: New Features and Behavior Changes New Features and Enhancements New Features and Enhancements in New Features and Enhancements in New Features and Enhancements in New Features and Enhancements in New Features and Enhancements Behavior Changes Behavior Changes in Behavior Changes in Behavior Changes in Behavior Changes Table of Contents i

5 Chapter 3: Data Validation Option Installation and Configuration Before You Begin Verify the Minimum System Requirements Verify the PowerCenter Requirements License Key Pre-Installation Tasks Create the Database for the Data Validation Option Repository Verify Code Page Compatibility Install the PowerCenter Client Gather PowerCenter Information Required for Installation Configure the Associated PowerCenter User Prepare to Install DVOCmd on UNIX Install Data Validation Option in Graphical Mode Install Data Validation Option in Silent Mode PowerCenter Configuration for Data Validation Option Verify the PowerCenter Repository Service Is Running Create a Connection to the Data Validation Option Repository Create the Target Folder in the PowerCenter Repository Verify the domains.infa File Location Set the INFA_HOME Environment Variable Copy dvoct.jar to the PowerCenter Integration Service Machine Data Validation Option Configuration Data Validation Option Repository Data Validation Option Mappings PowerCenter Repository Test the Data Validation Option Installation and Configuration JasperReports Server Configuration Install JasperReports Server Install the SSL Certificate Copy the DataDirect Drivers JasperReports Properties Configure JasperReports Properties Install DVOCmd on UNIX Data Validation Option Upgrade Upgrade from Data Validation Option Upgrade from Data Validation Option 9.1.x Upgrade from Data Validation Option Upgrade from Data Validation Option Chapter 4: Data Validation Option Client Layout Data Validation Option Client Overview Tabs Available in Data Validation Option Client Layout ii Table of Contents

6 Tests Tab SQL Views Tab Lookup Views Tab Join Views Tab Folders Copying Folders Copying Objects Menus Settings Folder Chapter 5: Data Validation Option Management Data Validation Option Management Overview User Configuration Directory Preference Files Multiple Data Validation Option Repositories Creating an Additional Data Validation Option User Specifying the User Configuration Directory Creating an Additional Data Validation Option Repository Multiple PowerCenter Versions Batch Files Sample Batch Files for Multiple Users Sample Batch Files for Multiple PowerCenter Instances Chapter 6: Informatica Authentication Informatica Authentication Overview Data Validation Option Users and Informatica Users Security Informatica Authentication Properties Enabling Informatica User Authentication Chapter 7: Repositories Repositories Overview Adding a Repository Editing Repositories Deleting Repositories Refreshing Repositories Exporting Repository Metadata Metadata Export and Import Exporting Metadata Importing Metadata Metadata Manager Integration Metadata Manager Properties Configuring Metadata Manager Integration Table of Contents iii

7 Chapter 8: Table Pairs Table Pairs Overview Basic Properties Connection Properties Relational Connection Properties SAS and Salesforce Connection Properties SAP Connection Properties Flat File Connection Properties Advanced Properties Database Processing Pushing Test Logic to the Database WHERE Clauses Table Joins Parameterization Bad Records Configuration Bad Records in Flat File Bad Records in Database Schema Mode Data Sampling Rules and Guidelines for Data Sampling PowerCenter Cache Adding Table Pairs Editing Table Pairs Deleting Table Pairs Viewing Overall Test Results Chapter 9: Tests for Table Pairs Tests for Table Pairs Overview Test Properties Tests Fields A and B Conditions A and B Operator Threshold Max Bad Records Case Insensitive Trim Trailing Spaces Null = Null Comments Expression Definitions Expression Tips Adding Tests Editing Tests iv Table of Contents

8 Deleting Tests Running Tests Automatic Test Generation Generating Table Pairs and Tests Generating Tests for Table Pairs Compare Columns by Position Bad Records Troubleshooting Table Pair Tests Chapter 10: Single-Table Constraints Single-Table Constraints Overview Basic Properties Connection Properties Relational Connection Properties SAS and Salesforce Connection Properties SAP Connection Properties Flat File Connection Properties Advanced Properties Parameterization Bad Records Configuration Bad Records in Database Schema Mode Bad Records in Flat File Data Sampling Rules and Guidelines for Data Sampling PowerCenter Cache Adding Single Tables Editing Single Tables Deleting Single Tables Viewing Overall Test Results Chapter 11: Tests for Single-Table Constraints Tests for Single-Table Constraints Overview Test Properties Tests Field Condition Operator Constraint Value Remaining Controls on Test Editor Adding Tests Editing Tests Deleting Tests Running Tests Table of Contents v

9 Bad Records Chapter 12: SQL Views SQL Views Overview SQL View Properties Description Table Definitions and Connection Column Definition SQL Statement Comment Adding SQL Views Editing SQL Views Deleting SQL Views Chapter 13: Lookup Views Lookup Views Overview Lookup View Properties Selecting Source and Lookup Tables Selecting Connections Overriding Owner Name Source Directory and File Description Source to Lookup Relationship Adding Lookup Views Editing Lookup Views Deleting Lookup Views Lookup Views Example Joining Flat Files or Heterogeneous Tables using a Lookup View Chapter 14: Join Views Join Views Overview Join View Data Sources Join View Properties Connection Properties Database Optimization in a Join View Join Types Alias in Join View Join Conditions Adding a Join View Configuring a Table Definition Configuring a Join Condition Managing Join Views Join View Example vi Table of Contents

10 Chapter 15: Reports Reports Overview Business Intelligence and Reporting Tools (BIRT) Reports BIRT Report Generation SQL and Lookup View Definitions Custom Reports Viewing Reports Jasper Reports Status in Jasper Reports Generating a Report Jasper Report Types Dashboards Metadata Manager Integration Chapter 16: Command Line Integration Command Line Integration Overview Rules and Guidelines for Running DVOCmd Commands CopyFolder CreateUserConfig DisableInformaticaAuthentication ExportMetadata ImportMetadata InstallTests Cache Settings LinkDVOUsersToInformatica PurgeRuns RefreshRepository RunTests Cache Settings UpdateInformaticaAuthenticationConfiguration UpgradeRepository Chapter 17: Troubleshooting Troubleshooting Overview Troubleshooting Initial Errors Troubleshooting Ongoing Errors Troubleshooting Command Line Errors Appendix A: Datatype Reference Test, Operator, and Datatypes Matrix for Table Pair Tests Test, Operator, and Datatypes Matrix for Single-Table Constraints Table of Contents vii

11 Appendix B: BIRT Report Examples Summary of Testing Activities Table Pair Summary Detailed Test Results Test Page Detailed Test Results Bad Records Page Appendix C: Jasper Report Examples Home Dashboard Repository Dashboard Folder Dashboard Tests Run Vs Tests Passed Total Rows Vs Percentage of Bad Records Most Recent Failed Runs Last Run Summary Appendix D: Reporting Views Reporting Views Overview results_summary_view rs_bad_records_view results_id_view meta_sv_view meta_lv_view meta_jv_view meta_ds_view meta_tp_view rs_sv_id_view, rs_lv_id_view, and rs_jv_id_view Appendix E: Metadata Import Syntax Metadata Import Syntax Overview Table Pair with One Test Table Pair with an SQL View as a Source Table Pair with Two Flat Files Single-Table Constraint SQL View Lookup View Appendix F: Glossary Index viii Table of Contents

12 Preface The PowerCenter Data Validation Option Installation and User Guide describes how you can test and validate data across multiple data sources. It is written for database administrators and testers who are responsible for validating enterprise data. This guide assumes you have knowledge of the data sources and PowerCenter. Informatica Resources Informatica MySupport Portal As an Informatica customer, you can access the Informatica MySupport Portal at The site contains product information, user group information, newsletters, access to the Informatica customer support case management system (ATLAS), the Informatica How-To Library, the Informatica Knowledge Base, the Informatica Multimedia Knowledge Base, Informatica Product Documentation, and access to the Informatica user community. Informatica Documentation The Informatica Documentation team takes every effort to create accurate, usable documentation. If you have questions, comments, or ideas about this documentation, contact the Informatica Documentation team through at We will use your feedback to improve our documentation. Let us know if we can contact you regarding your comments. The Documentation team updates documentation as needed. To get the latest documentation for your product, navigate to Product Documentation from Informatica Web Site You can access the Informatica corporate web site at The site contains information about Informatica, its background, upcoming events, and sales offices. You will also find product and partner information. The services area of the site includes important information about technical support, training and education, and implementation services. Informatica How-To Library As an Informatica customer, you can access the Informatica How-To Library at The How-To Library is a collection of resources to help you learn more about Informatica products and features. It includes articles and interactive demonstrations that provide solutions to common problems, compare features and behaviors, and guide you through performing specific real-world tasks. ix

13 Informatica Knowledge Base As an Informatica customer, you can access the Informatica Knowledge Base at Use the Knowledge Base to search for documented solutions to known technical issues about Informatica products. You can also find answers to frequently asked questions, technical white papers, and technical tips. If you have questions, comments, or ideas about the Knowledge Base, contact the Informatica Knowledge Base team through at Informatica Multimedia Knowledge Base As an Informatica customer, you can access the Informatica Multimedia Knowledge Base at The Multimedia Knowledge Base is a collection of instructional multimedia files that help you learn about common concepts and guide you through performing specific tasks. If you have questions, comments, or ideas about the Multimedia Knowledge Base, contact the Informatica Knowledge Base team through at KB_Feedback@informatica.com. Informatica Global Customer Support You can contact a Customer Support Center by telephone or through the Online Support. Online Support requires a user name and password. You can request a user name and password at Use the following telephone numbers to contact Informatica Global Customer Support: North America / South America Europe / Middle East / Africa Asia / Australia Toll Free Brazil: Mexico: North America: Toll Free France: Germany: Italy: Netherlands: Portugal: Spain: Switzerland: United Kingdom: Toll Free Australia: New Zealand: Standard Rate India: Standard Rate Belgium: France: Germany: Netherlands: United Kingdom: x Preface

14 C H A P T E R 1 Introduction to Data Validation Option This chapter includes the following topics: Data Validation Option Overview, 1 Architecture, 2 Data Validation Methodology, 3 Data Validation Option Reporting, 5 Data Validation Option Licenses, 6 Data Validation Option Overview Data Validation Option is a solution that you use with PowerCenter to validate data. You can validate target data to verify that it is accurate and the transformation process did not introduce errors or inconsistencies. Data validation is the process to verify whether the moved or transformed data is complete and accurate. You must also verify that there are no errors in the movement or transformation process that changes the data. Use PowerCenter Data Validation Option to verify that your data is complete and accurate. You might have the standard license or the enterprise license for Data Validation Option. If you have the enterprise license, you can use Data Validation Option in a production environment. If you have the standard license, you can use Data Validation Option only in a non-production environment. Data Validation Option Users There are many possible users of Data Validation Option: Business or Data Analysts Data Warehouse Testers ETL Developers Database Administrators 1

15 Data Validation in an Enterprise There are two types of data validation generally performed in a data integration setting, source to target comparisons and production to development comparisons. You can do the source to target validation at the end of development of a data integration project on the initial load of a data warehouse, or as reconciliation of the ongoing daily or incremental loads. You can perform data validation to compare production and development environments, when you upgrade data integration software or RDBMS database software. Finally, you can perform data validation as part of the testing process or as part of the production process, called the reconciliation or Audit/Balance/Control process. Data Validation Option supports all of the use cases described above. Data Validation Option reads table definitions from PowerCenter metadata repositories, and checks the data at either end of the process. It does not check the correctness of transformations or mappings. Data Validation Option identifies problems or inconsistencies but does not attempt to identify the source of the problem in the ETL process. Data Validation Workflow A typical workflow for data validation consists of multiple tasks. 1. Data Validation Option reads one or more PowerCenter metadata repositories. 2. Define the validation rules in Data Validation Option. 3. Run the rules to ensure the data conforms to the validation rules. When you do this, Data Validation Option performs the following tasks: Creates and executes all tests through PowerCenter. Loads results into the Data Validation Option results database and displays them in the Data Validation Option Client. 4. Examine the results and identify sources of inconsistencies in the ETL process or the source systems. 5. Repeat this process for new records. Architecture Data Validation Option requires installation and setup of PowerCenter. Source and target data table and file definitions are imported from PowerCenter repositories. You set up table pairs and test rules in Data Validation Option. This test metadata is stored in the Data Validation Option repository. When the tests are run, Data Validation Option communicates with PowerCenter through an API to create appropriate mappings, sessions, and workflows, and to execute them. PowerCenter connects to the data being tested instead of Data Validation Option. After the tests are executed, results are stored in the Data Validation Option repository and displayed in the Data Validation Option Client. You can configure Data Validation Option to authenticate users based on Informatica domain login credentials. After you enable Informatica authentication, the users must use their Informatica domain login credentials to use the Data Validation Option client. 2 Chapter 1: Introduction to Data Validation Option

16 Data Validation Methodology A sample methodology to help you design a rigorous data validation process is presented in the following section. Most users have some kind of a testing process already in place. Usually, it is a combination of SQL code and Excel spreadsheets. A common temptation is to replicate the current SQL-based process. The first question often asked is, "How do I do this with Data Validation Option?" Use the following guidelines to set up a data validation approach: 1. Test data, not mappings or workflows. Your test framework should not contain parallel mappings, sessions, and workflows. Testing mappings is unit testing, which is different from data validation. 2. Do not try to mimic SQL. Step back and think of what you are trying to accomplish. Data Validation Option can make things a lot easier. 3. Assume the worst. If data needs to be moved from last_name to last_name, it may have been moved to city by mistake. If an IF statement was used, assume it was coded wrong. It is always prudent to assume a mistake has been made and be pleasantly surprised when tests return no errors. 4. Do the easy things first. Complicated problems often manifest themselves in simple ways. Simple counts and constraints can point out some obvious errors. 5. Design the initial test framework without taking performance into account. After you are satisfied with your approach, begin to optimize. 6. Try to split complex SQL into more than one table pair. For example, if you see something like the following statements: Select CASE (code='x', TableA.Fld1, TableB.Fld1) Select CASE (code='x', TableA.Fld2, TableB.Fld2) You can create two table pairs: Table A vs. Target WHERE clause A: code='x' Table B vs. Target WHERE clause A: code <> 'X' 7. Do not copy formulas from the ETL mapping into Data Validation Option. Sometimes when you need to test a complex transformation such as complex IF statements with SUBSTR, you might be tempted to just copy it from the mapping. This approach produces an obvious problem. If there is an error in the ETL mapping formula, you will replicate it in Data Validation Option, and Data Validation Option will not catch it. Therefore, you must always maintain a proper separation between ETL and testing. 8. Do not try to do everything in Data Validation Option. If you think that a particular step can be accomplished more easily with SQL, use SQL. If you run 95% of your validation in Data Validation Option, and can document it with the audit trail, this is more than enough. Basic Counts, Sums, and Aggregate Tests The goal of basic counts, sums, and aggregate tests are to make sure that all records were moved. This approach detects the following problems: Lack of referential integrity in the source. (Child with no parent will not be moved.) Row rejection by the target system. Incorrect ETL logic in the WHERE clauses. Other problems that do not move all the required records. Approach data validation in the following order: COUNT and COUNT_ROWS to count the number of records Data Validation Methodology 3

17 SUM for numeric fields COUNT DISTINCT to compare detail vs. aggregate tables Check Referential Integrity of Target Tables Check referential integrity of table tables to find lack of referential integrity in the target, either child without a parent or a fact record without corresponding dimension table. In a scenario where Table A is child table; Table B is parent table and Field A is child foreign key; Field B is parent primary key, approach data validation in the following order : Test is SETA_in_B. (Every child FK is in parent PK.) In case of a star schema, fact table is child, dimension is parent. (Every fact FK needs to be in parent PK.) In case of composite keys, create an expression that concatenates the keys, and run these tests on that expression. Enforce Constraints on Target Tables Often the errors in very complicated transformations manifest themselves in rather simple ways such as NULLs in the target, missing rows, or incorrect formats. You can test for such scenarios by enforcing constraints on target tables. This is one of the most overlooked yet most effective data testing strategies. The following examples explain target table constraints: Unique Primary Keys UNIQUE(PK) If a composite PK, use UNIQUE(expression). Valid Individual Values For example: VALUE(FldA) Between 10, 50 VALUE(FldB) In ('A','B','C') VALUE(FldC) > 0 NOT_NULL(FldD) FORMAT(Phone) = 'Reg expression' (optional) You can have more than one test on a specific field and you can create tests on an expression. Aggregate Constraints These are often used for sanity checks in terms of rows moved, totals, etc. For example, is this correct? Source (staging file) Target 1 laptop laptop desktop desktop 1500 This looks correct, but if this is XYZ company s daily sales, it is not correct, even though it was moved correctly. Somehow you know that XYZ sells more than $2500/day. Therefore, you can say that anything less than 1000 records and anything less than $2m and more than $15m is suspect. 4 Chapter 1: Introduction to Data Validation Option

18 Therefore: COUNT_ROWS(any fld) > 1000 SUM(Amount) Between , Compare Individual Records between Sources and Targets If a field was moved without transformation or if it was transformed, comparing individual records between sources and targets ensures whether the value is correct. This is another critical step in testing. Read the section that explains the difference between VALUE and OUTER_VALUE tests and the expected results when using each test. Approach data validation in the following order : Simple comparison. Create a table pair, join on a common keys, and then either set up tests automatically (rightclick/generate) or manually if field names are different. Any row based expression (concatenation, calculation) can be tested similarly, for example: VALUE(first_name '_' last_name '@dvosoft.com' = ) Data Validation with Views You can use lookup views, sql views, and join views to create complex data validation scenarios. SQL Views After you complete the other data validation methods, you can use Data Validation Option SQL views to construct complicated SQL-based scenarios that involve multiple tables and complicated transformations. Lookup Views Testing lookups is an important step in data testing. Data Validation Option lookup views allow you to test the validity of the lookup logic in your transformation layer. Join Views You can create complex join relationships between heterogeneous data sources in a join view. You can test the validity of data across related tables with join views. Data Validation Option Reporting You can run reports to display test definitions and results that are stored in the Data Validation Option repository. You can use the BIRT reporting engine to generate the reports on one or more table pairs or single tables. For example, you can run the Summary of Testing Activities report to determine the number of table pairs or single tables, the number of tests for each table pair or single table, and the overall test results. If you have the enterprise license, you can use JasperReports Server to generate the reports and dashboards. A dashboard shows displays the test results from one or more reports. Data Validation Option Reporting 5

19 Data Validation Option Licenses Data Validation Option is available with a standard license and an enterprise license. With the standard license, you can use Data Validation Option in a non-production environment. With the enterprise license, you can use Data Validation Option in a production environment. You can perform the following additional tasks if you have the enterprise license: Use parameters in the WHERE clause of a table pair or single table to perform incremental data validation. Store up to 16 million bad records in the Data Validation repository or as a flat file for each test. If you have the standard license, you can store up to 1,000 bad records for reporting and analysis. Use JasperReports Server to generate different reports. Use JasperReports Server to view and generate dashboards. You must enter the license key after you install Data Validation Option. 6 Chapter 1: Introduction to Data Validation Option

20 C H A P T E R 2 New Features and Behavior Changes This chapter includes the following topics: New Features and Enhancements, 7 Behavior Changes, 12 New Features and Enhancements This section contains information about new enhancements in different versions of Data Validation Option. New Features and Enhancements in This section describes new features and enhancements in version Caching You can configure caching for table pairs and single-table constraint tests in the Data Validation Option Client. Connections You can add a file list to the connection for a single-table constraint or table pair. When you specify a file list, Data Validation Option processes all files in the list. Data Validation Option Client The user interface contains the following enhancements: The Table Pair Test Editor and Single Table Test Editor dialog boxes have a new layout. You can view longer column names in the Column Definition area of the SQL View Editor dialog box. DVOCmd Commands on UNIX You can run the following DVOCmd commands on UNIX: InstallTests RefreshRepository RunTests UpgradeRepository 7

21 Join Views You can select an SQL view as a data source for a join view. Metadata Export and Import You can skip objects that are not valid when you export metadata. You can use the Data Validation Option Client to edit the metadata before you import it. Operating System Support Data Validation Option supports Informatica Services versions and higher on AIX 7.1. PowerCenter Session Logs You can right-click a single table or table pair to view test errors in the PowerCenter session log. Sampling You can configure sampling to reduce the number of rows processed when you test data sets in table pairs and single-table constraints. You can use a seed value to repeat the same sample in a test. If you are sampling data from IBM DB2, Microsoft SQL Server, Oracle, or Teradata, you can perform native sampling in the database. Push sampling to the database to increase performance. New Features and Enhancements in This section describes new features and enhancements in version Authentication LDAP Authentication Administrators can configure LDAP authentication on the Informatica domain for a Data Validation Option schema with PowerCenter or later. Licensing Data Validation Option is available with a standard license and an enterprise license. You must enter the license key after you install Data Validation Option. Command Line Utility InstallTests When you run the InstallTests command you can use the forceinstall option to recreate mappings for the tests that are already installed. Reports JasperReports Server Reports If you have the enterprise license, you can generate different reports and dashboards with the JasperReports Server. Repository Metadata Manager Integration You can view the details of the PowerCenter repository objects in Metadata Manager. 8 Chapter 2: New Features and Behavior Changes

22 SQL Views, Lookup Views, and Join Views Expressions in Join Conditions You can enter and validate PowerCenter expressions as fields in a join condition when you configure a join view. Expression Validation You can validate the PowerCenter expressions that you enter in the Data Validation Option Client when you configure a source to lookup relationship in a lookup view. Table Pairs and Single Tables Enhanced Bad Records Storage If you have the enterprise license, you can store up to 16 million bad records in the Data Validation repository or as a flat file for each test. Expression Validation You can validate the PowerCenter expressions that you enter in the Data Validation Option Client when you configure table pairs and single tables. Parameterization If you have the enterprise license, you can configure a parameter file and use the parameters in the WHERE clause of a table pair or single table. Copy Objects Tests You can copy table pairs and single tables in a folder to another folder. Expression Validation You can validate the PowerCenter expressions that you enter in the Data Validation Option Client when you configure test conditions and expression definitions. Automatic Test Generation You can generate tests based the position of columns of the tables in a table pair. New Features and Enhancements in This section describes new features and enhancements in version Authentication Informatica Authentication Administrators can configure Informatica authentication for a Data Validation Option schema with PowerCenter or later. After you enable Informatica authentication, users must use Informatica domain credentials to log in to the Data Validation Option Client. Repositories Data Sources You can use the following data sources in Data Validation Option: PowerExchange for DB2 z/os Netezza New Features and Enhancements 9

23 SAP Salesforce.com SAS Single Tables and Table Pairs Threshold You can enter the threshold for aggregate and value tests in table pairs and single tables as a percentage value in addition to the absolute value. Max Bad Records You can enter a percentage value for the maximum number of bad records for tests in table pairs and single tables in addition to the absolute value. Number of Processed Records After you run a test, the test report displays the details of the processed records. You can view the number of bad records, total number of records processed through the join, and the number of records read from the database. Join Views You can create an object with complex join conditions in multiple heterogeneous data sources. You can use a join view as a table in table pairs and single tables. New Features and Enhancements in This section describes new features and enhancements in version Command Line Utility RunTests Command When you run tests with the RunTests command, you can use the following options: Send an with the test results once Data Validation Option completes the test. Provide cache memory setting for mapping transformations. InstallTests Command Tests When you install tests with the InstallTests command, you can provide the cache memory setting for mapping transformations Automatic Test Generation You can generate count tests along with value tests in automatic test generation. Repositories Data Sources Data Validation Option supports the following data sources through PowerExchange for ODBC: DB2 z/os DB2 AS/400 IMS Adabas 10 Chapter 2: New Features and Behavior Changes

24 VSAM Mainframe flat files Metadata Import Significant performance improvement when you import or refresh metadata from a PowerCenter repository. New Features and Enhancements in This section describes new features and enhancements in version Client Layout Folders. You can organize single tables and table pairs by placing them in folders. When you upgrade from version 3.1, the installation program creates a Default folder for each user and places the table pairs and single tables in the folder. When you create a new user, Data Validation Option creates a Default folder. Error reporting. If a test fails to run, Data Validation Option displays the test run error on the Tests tab. Previously, you had to examine the PowerCenter session log file to view test run errors. Single Tables tab. The details area contains separate tabs for table pairs and single tables. Command Line Utility New Commands. The Data Validation Option command line utility, DVOCmd.exe, contains new commands that allow you to create users and refresh repositories. PowerCenter Version PowerCenter Data Validation Option works with PowerCenter versions 8.5 and later, except for PowerCenter version 9.0. Reports Reports for tests in folders. You can run reports for all tests in a folder. Report information. Reports display the folder name, error messages, join expressions, and conditions, if applicable. Repositories Refreshing repositories. When you refresh a repository, you can refresh the entire repository, the connection objects, the folders, or the sources and targets. You can also refresh repository folders individually. Single Tables and Table Pairs Expressions in join conditions. When you join two tables in a table pair, you can enter a PowerCenter expression as a field in the join condition. Enter an expression to join tables with key fields that are not identical. Large table processing. When you include a large table in a table pair, you can optimize the way Data Validation Option joins table data. You can specify which table Data Validation Option uses for the master or detail table. You can also use sorted output for the join. Pushing sorting logic to the database. To increase the performance of table pair and single table tests, you can push the sorting logic for joins to the source database. Pushing sorting logic to the database causes the database to sort records before it loads them to PowerCenter which minimizes disk input and output. Tests Filter conditions. You can apply a filter condition to table pair and single table tests. If you apply a filter condition to a table pair test, Data Validation Option applies the filter condition after it joins the tables in the table pair. New Features and Enhancements 11

25 Behavior Changes This section contains information on behavior changes in different versions of Data Validation Option. Behavior Changes in This section describes behavior changes in version Caching Effective in version 9.5.1, you specify the total cache size in the --cachesize option in the InstallTests and RunTests DVOCmd commands. The PowerCenter Integration Service divides the cache size among all Aggregator, Joiner, Lookup, and Sorter transformations in the underlying PowerCenter session. The PowerCenter Integration Service allocates cache memory based on the transformation requirements, such as the number of rows processed, number of tests, and precision and scale of the table columns. Previously, the cache size that you specified in the command option was the cache size for each transformation, except the Sorter transformation. Data Validation Option assigned 8 MB for each Sorter transformation. For example, if you set the cache size to 200 MB and the session contained two Aggregator transformations, two Joiner transformations, and one Sorter transformation, then each Aggregator and Joiner transformation received 200 MB of cache memory and the Sorter transformation received 8 MB of cache memory. Connections Effective in version 9.5.1, connections appear in alphabetic order. Previously, connections were not sorted alphabetically. JasperReports Configuration Effective in version 9.5.1, you do not have to re-enter JasperReports configuration after you disable JasperReports. Previously, when you disabled JasperReports, the Data Validation Option Client deleted the previous configuration. Lookup Views Effective in version 9.5.1, you cannot enter an expression for the Lookup field in a lookup view if the lookup table is from any of the following data source types: flat file, mainframe flat file, ADABAS, IMS, or VSAM. Previously, you could enter an expression, but an error occurred when you ran a test on the lookup view. If the lookup table is from a flat file, mainframe flat file, ADABAS, IMS, or VSAM data source and you upgrade to 9.5.1, you must delete the expression from the Lookup field in the lookup view. Metadata Refresh Effective in version 9.5.1, Informatica significantly improved repository refresh performance for networks with high latency between the Data Validation Option Client and the Data Validation Option repository. Repository Selection Effective in version 9.5.1, you can select a repository faster when you have a large number of repositories, such as 10,000 repositories, defined in the Data Validation Option Client. Previously, it took a few seconds to select a repository when a large number of repositories were defined in the Data Validation Option Client. Behavior Changes in This section describes behavior changes in version Chapter 2: New Features and Behavior Changes

26 Client Layout SQL Views Tab and Lookup Views Tab You can no longer add SQL views and lookup views to a table pair or single table from the right-click menu in SQL Views Tab and Lookup Views Tab. Repositories PowerCenter Repository Support Tests You can use repositories from PowerCenter HotFix 10 and later. Previously, you could use repositories from PowerCenter 8.5 and later. Automatic Test Generation You select whether to enable trim trailing spaces for the tests that you generate automatically. Previously, you had to manually update the automatically generated tests to enable trim trailing spaces. You can also apply separate data source sorting for the tables in the table pair. Previously, you could not provide separate sorting for the tables. Behavior Changes in This section describes behavior changes in version Single Tables and Table Pairs Database connection Tests Name of the database connection that you provide in Data Validation Option table pairs and single tables are no longer case sensitive. Previously, if you edit the database connection name to a different case in PowerCenter, the existing table pairs and single tables would be invalidated. Autogeneration of Tests Use Compare Table menu item to autogenerate of table pairs and tests between tables and flat files in folders and contains several options. Previously, you had to select the two folders and right-click to autogenerate table pairs and tests. Behavior Changes in This section describes behavior changes in version Client Layout Data Sources tab Folders The Data Sources tab is removed. Table pairs and single tables appear in the Default folder in the Navigator. Previously, single tables and table pairs appeared in the Single Tables and Table Pairs nodes in the Navigator. Properties area The Properties area is moved to the bottom right side of the Data Validation Option Client to make more room for the Navigator. Previously, the Properties area appeared in the bottom left side of the screen. Behavior Changes 13

27 Results tab The tab that lists bad records for tests is renamed to Results. The Results tab displays test summary information for table pairs, single tables, and tests. It also displays bad records for certain types of tests. Previously, Data Validation Option displayed the Details tab only for tests. It displayed bad records only. Single Tables tab In the details area, single tables are listed on the Single Tables tab. Previously, single tables were listed on the Table Pairs tab. Installation Executable name The Data Validation Option executable file name is DVOClient.exe. Previously, the executable file name was DataValidator.exe. Installation directory The Data Validation Option default installation directory is C:\Program Files\Informatica<version>\DVO on 32-bit operating systems and C:\Program Files (x86)\informatica<version>\dvo on 64-bit operating systems. Previously, the default installation directory was C:\Program Files\DVOSoft. Reports Reports for single-table constraints Reports for single-table constraints display information about the single table only. Previously, reports for singletable constraints were the same as reports for table pairs except they displayed Table B values as null values. Repositories Importing metadata When you add a PowerCenter repository, Data Validation Option imports folder names. If the repository is a target repository, Data Validation Option also imports connection metadata. To import source and target metadata, you must refresh the repository. Previously, when you saved a repository for the first time, Data Validation Option imported folder names and all source and target metadata. If the repository was a target repository, Data Validation Option also imported connection metadata. Refreshing repositories When you refresh a repository, you do not have to close and restart Data Validation Option. Previously, you had to restart Data Validation Option after you refreshed a repository. Single Tables and Table Pairs Single Tables Editor You create single tables using the Single Table Editor dialog box. Previously, you created single tables using the Table Pairs Editor dialog box. SQL and Lookup Views SQL view definition When you create an SQL view, you can select the tables and columns to use in the view. Data Validation Option detects the datatype, precision, and scale of the columns. Previously, when you created an SQL view, you had to define the SQL relationships across tables, define the columns, and define the datatype, precision, and scale of each column manually. 14 Chapter 2: New Features and Behavior Changes

28 C H A P T E R 3 Data Validation Option Installation and Configuration This chapter includes the following topics: Before You Begin, 15 Pre-Installation Tasks, 16 Install Data Validation Option in Graphical Mode, 20 Install Data Validation Option in Silent Mode, 20 PowerCenter Configuration for Data Validation Option, 21 Data Validation Option Configuration, 25 Test the Data Validation Option Installation and Configuration, 31 JasperReports Server Configuration, 32 Install DVOCmd on UNIX, 34 Data Validation Option Upgrade, 35 Before You Begin You must verify the requirements and license key before you perform Data Validation Option pre-installation and installation tasks. The machine on which you install Data Validation Option must meet requirements from Data Validation Option and the PowerCenter Client. You install the PowerCenter Client and Data Validation Option on the same machine. Data Validation Option requires a client installation only. Verify the Minimum System Requirements Verify that the machine where you plan to install Data Validation Option meets the minimum system requirements. Machine requirements The machine where you install Data Validation Option must have at least 1 GB of RAM and 256 MB of disk space. Note: The machine must also meet the minimum system requirements of the PowerCenter Client. 15

29 Permissions You must have permissions on the Data Validation Option Client machine to complete the following tasks: Create and modify environment variables on the Data Validation Option Client machine. Read from and write to the Data Validation Option installation directory and subdirectories. Verify the PowerCenter Requirements Verify that the environment where you plan to install Data Validation Option meets the PowerCenter requirements. PowerCenter must be installed in the same network as the Data Validation Option Client machine. The PowerCenter version must be at least HotFix 10. Install a later version of PowerCenter if you implement the following Data Validation Option features: - Informatica authentication. Requires PowerCenter or later. - SAP R/3 data source. Requires PowerCenter or later. - SAS data source. Requires PowerCenter or later. The Informatica domain must contain at least one PowerCenter Repository Service and one PowerCenter Integration Service. The Informatica domain must contain a PowerCenter repository where you can store Data Validation Option objects, such as workflows. The Data Validation Option Client machine must meet the minimum system requirements of the PowerCenter Client. Later, you will install the PowerCenter Client on the machine where you install Data Validation Option. License Key Before you install the software, verify that you have the license key available. You can get the license key in the following ways: Installation DVD. If you receive the Informatica installation files in a DVD, the license key file is included in the Informatica License Key CD. FTP download. If you download the Informatica installation files from the Informatica Electronic Software Download (ESD) site, the license key is in an message from Informatica. Copy the license key file to a directory accessible to the user account that installs the product. Contact Informatica Global Customer Support if you do not have a license key. Pre-Installation Tasks Complete the pre-installation tasks before you can install Data Validation Option. Complete the following pre-installation tasks: Create the Data Validation Option repository database. Verify the code page compatibility. Install the PowerCenter Client. Gather the PowerCenter information required for Data Validation Option installation. Configure the associated PowerCenter user. Optionally, prepare to run DVOCmd on a UNIX machine. 16 Chapter 3: Data Validation Option Installation and Configuration

30 Create the Database for the Data Validation Option Repository Create the database for the Data Validation Option repository. After you install Data Validation Option, you can create the Data Validation Option repository in this database. The Data Validation Option repository database must meet the following requirements: The database type must be IBM DB2, Microsoft SQL Server, or Oracle. The Data Validation Option repository must not reside on an IBM DB2 database cluster. The database server must have at least 100 MB of disk space to create the Data Validation Option repository. The database user must have privileges to create, alter, and delete tables, indexes, sequences, and views during installation. If the Data Validation Option repository database is IBM DB2, ensure that the page size is at least 16 KB and use the default schema. Also, ensure that the user name and schema name are the same. Verify Code Page Compatibility Verify that the code page for the PowerCenter Integration Service process is compatible with the NLS setting of the Data Validation Option repository database. If the settings are not compatible, test results might be inaccurate. If you run the PowerCenter Integration Service on a grid, verify that the code page of every PowerCenter Integration Service process is compatible with the NLS setting of the Data Validation Option repository database. Install the PowerCenter Client Install the PowerCenter Client on the machine where you plan to install Data Validation Option. For more information about installing the PowerCenter Client, see the Informatica PowerCenter Installation and Configuration Guide. Gather PowerCenter Information Required for Installation Gather the PowerCenter information that is required to install Data Validation Option. Optionally, you can integration Metadata Manager with Data Validation Option. To integrate Metadata Manager, gather the Metadata Manager host name and port number, and the resource name of the PowerCenter repository that stores Data Validation mappings, sessions, and workflows. The following table provides PowerCenter information that you must gather before you install Data Validation Option: Name Informatica domain PowerCenter Integration Service PowerCenter Repository Service PowerCenter repository user name Description Name of the Informatica domain that contains the PowerCenter repository for Data Validation Option. Name of the PowerCenter Integration Service that runs Data Validation Option workflows. Name of the PowerCenter Repository Service that manages the PowerCenter repository for Data Validation Option. User name of the PowerCenter repository. The user must have permission to create, edit, and delete mappings, sessions, and workflows in the Data Validation Option folder. Pre-Installation Tasks 17

31 Name PowerCenter repository password Location of the domains.infa file Metadata Manager Service host Description Password for the PowerCenter repository. Location of the domains.infa file on machine where you installed Informatica services. The domains.infa file contains details about the Informatica domain and PowerCenter repository. Optional. Host name of the machine on which the Metadata Manager Service runs. Metadata Manager Service port Optional. Port on which the Metadata Manager Service runs. Default is Resource name Optional. Name of the Metadata Manager resource for the PowerCenter repository for Data Validation Option. Configure the Associated PowerCenter User Each Data Validation Option user is associated with a PowerCenter user. When you run a Data Validation Option test, PowerCenter runs the underlying workflow based on the credentials of the associated PowerCenter user. You must configure the user permissions and user privileges on the corresponding PowerCenter Repository Service. 1. Log in to the Administrator tool. 2. Click the Security tab. 3. In the Users area, click the Actions button, and then select Create User. 4. Enter the user information and then click OK. You specify this PowerCenter user when you add the PowerCenter repository for Data Validation Option. 5. Select the user in the Users area. 6. Click the Privileges tab. The following figure shows that no privileges assigned to the user: 7. Select the PowerCenter Repository Service that runs the Data Validation Option sessions and workflows. The Edit Roles and Privileges dialog box appears. 8. On the Roles tab, expand the folder for the PowerCenter Repository Service. 9. Enable the PowerCenter Connection Administrator custom role. 10. Click OK. 18 Chapter 3: Data Validation Option Installation and Configuration

32 11. Click the Permissions tab. The Permissions tab shows that no permissions are assigned to the user. 12. Select the PowerCenter Repository Service that runs the Data Validation Option sessions and workflows. 13. Click the Edit Direct Permissions button. The Edit Direct Permissions dialog box appears. 14. In the Access column, select Allow. 15. Click OK. The Permissions tab shows that the user has permission on the PowerCenter Repository Service. Prepare to Install DVOCmd on UNIX Optionally, you can run a subset of the DVOCmd commands on UNIX. Before you install DVOCmd on UNIX, you must perform prerequiste tasks on the UNIX machine, such as configuring PowerCenter for Data Validation Option. DVOCmd uses pmrep, pmcmd, and other PowerCenter libraries. Complete the following prerequisite tasks to install DVOCmd on UNIX: Install Java Runtime Environment (JRE) 1.6 or higher on the UNIX machine. Use 32-bit JRE for 32-bit Informatica Services and 64-bit JRE for 64-bit Informatica Services. If you installed a previous version of DVOCmd, delete the DVO folder before installing DVOCmd. Copy the domains.infa file from the PowerCenter Client installation directory to the following directory on the UNIX machine: /home/<user>/informatica/<powercenter version> You can copy the domains.infa file from the following directory where you installed the PowerCenter Client: <InformaticaInstallationDir>\clients\PowerCenterClient Set the following environment variables on the UNIX machine: INFA_HOME Set to the root directory of the machine where the PowerCenter Repository Service runs. For example, specify the following command to set the directory: setenv INFA_HOME /home/<user>/informatica/<powercenter version> JRE Set to the directory where JRE is installed with Informatica Services. For example, specify $INFA_HOME/server/ java/jre/bin. INFA_DOMAINS_FILE Set to the following location of the domains.infa file on the Data Validation Option Client machine: $INFA_HOME/domains.infa PC_CLIENT_INSTALL_PATH PATH Set to $INFA_HOME/server/bin. Add the following directories to the PATH environment variable: $INFA_HOME/server/bin and $JRE/bin. LD_LIBRARY_PATH If you run DVOCmd on Linux or Solaris, set the shared library environment variable on the Linux or Solaris machine to $INFA_HOME/server/bin. Pre-Installation Tasks 19

33 LIBPATH If you run DVOCmd on AIX, set the shared library environment variable on the AIX machine to $INFA_HOME/ server/bin. SHLIB_PATH If you run DVOCmd on HP-UX, set the shared library environment variable on the HP-UX machine to $INFA_HOME/server/bin. Install Data Validation Option in Graphical Mode You can install Data Validation Option in graphical mode. 1. Close all other applications. 2. Run Install_DataValidator_<version>_ReleaseBuild.exe on the machine where you want to install Data Validation Option. 3. Click Next. 4. Enter the absolute path for the installation directory. The installation directory must be on the current machine. The path must not contain spaces or the dollar sign ($). The maximum length of the path must be less than 260 characters. 5. Click Next. 6. Optionally, select the Start Menu folder or enter the folder name. If you do not want to create a Start Menu folder, select Don't Create a Start Menu Folder. 7. To create a shortcut for all users, select Create Shortcuts for All Users. 8. Click Next. The installer copies the Data Validation Option Client files to the installation directory. 9. After the installer copies the files, click Finish. A message appears indicating that the Data Validation Option needs to be configured. 10. Click OK to begin configuring the Data Validation Option repository. The Preferences dialog box appears. 11. Configure the Data Validation Option repository. You can view the installation log file to get more information about the tasks performed by the installer. You can find the installation.log file in the following directory: <DataValidationOptionInstallationDir>\.install4j Install Data Validation Option in Silent Mode You can install Data Validation Option in silent mode. Use silent mode to install Data Validation Option on multiple machines. You can create a batch file to run the installation on each machine. 1. Copy the Data Validation Option installer to the client machine. You can use software, such as Tivoli, to copy the installer. 20 Chapter 3: Data Validation Option Installation and Configuration

34 2. Run the following command on the client machine: Install_DataValidator_<version>_Release_Build.exe -q -dir <DataValidationOptionInstallationDir> Enclose the directory path with double quotes if the directory path has spaces. For example, you can use the following syntax: Install_DataValidator_ _Release_Build.exe -q -dir "C:\Program Files (x86)\informatica<version>\dvo" After you complete the silent installation, you must configure Data Validation Option on each machine. PowerCenter Configuration for Data Validation Option You must configure the PowerCenter environment to run Data Validation Option tests. Complete the following configuration tasks: Verify that the PowerCenter Repository Service is running. Create a connection to the Data Validation Option repository. Create the target folder in the PowerCenter repository. Verify the domains.infa file location. Set the INFA_HOME environment variable. Copy the dvoct.jar file to the PowerCenter Integration Service machine. Verify the PowerCenter Repository Service Is Running Before you can access the PowerCenter repository, you must verify that the PowerCenter Repository Service is running. 1. In the Administrator tool, on the Domain tab, select the Services and Nodes view. 2. Select the PowerCenter Repository Service in the Navigator. The following figure shows the status of the PowerCenter Repository Service on the Properties tab: 3. Verify that the PowerCenter Repository Service is available. Create a Connection to the Data Validation Option Repository Create a connection to the Data Validation Option repository to enable the PowerCenter Integration Service to write test results to the repository. 1. In the PowerCenter Workflow Manager, click Connections > Relational. 2. Click New. PowerCenter Configuration for Data Validation Option 21

35 3. In the Select Subtype dialog box, select the database type of the Data Validation Option repository. 4. Click OK. 5. Enter the connection name and database properties. Enter a meaningful connection name that implies that the connection is for Data Validation Option repository. For example, enter the name DataValidationOption_Repository. 6. Click OK. 7. Click Permissions. The Permissions dialog box appears: 8. Select the PowerCenter user associated with Data Validation Option. 9. Grant Read, Write, and Execute permissions on this folder to the user. 10. Record the connection name. You must specify this connection name when you add a PowerCenter repository for Data Validation Option. 11. Click OK. Create the Target Folder in the PowerCenter Repository The target folder in the PowerCenter repository stores the Data Validation Option objects, such as mappings, sessions, and workflows. 1. In the PowerCenter Repository Manager, right-click the PowerCenter repository and then select Connect. 2. Click Folders > Create. The Create Folder dialog box appears. 3. Enter the folder properties. Enter a meaningful folder name that implies that the folder is for Data Validation Option objects. For example, enter the name DataValidationOption_Objects. 22 Chapter 3: Data Validation Option Installation and Configuration

36 4. Click the Permissions tab. The Permissions tab shows the permissions for each user on the folder: 5. Select the PowerCenter user associated with the Data Validation Option. 6. Grant Read, Write, and Execute permissions on this folder to the user. 7. Record the PowerCenter repository name. You must specify this repository name when you add a PowerCenter repository for Data Validation Option. 8. Record the folder name. You must specify this folder name as the target folder when you add a PowerCenter repository for Data Validation Option. Verify the domains.infa File Location The domains.infa file contains connection information to the domain, such as host name and port number of the master gateway node. The PowerCenter Client uses this file to connect to the domain, PowerCenter repository for Data Validation Option, and Data Validation Option repository. 1. Verify that the domains.infa file is in the following PowerCenter Client directory on the Data Validation Option Client machine: <InformaticaInstallationDir>\clients\PowerCenterClient\ 2. If the domains.infa file is not in this location, copy the domains.infa file from the Informatica services installation directory on the machine where you installed PowerCenter. PowerCenter Configuration for Data Validation Option 23

37 Set the INFA_HOME Environment Variable The INFA_HOME environment variable specifies the location where Informatica Services is installed and where the domains.infa file is stored. 1. Create the INFA_HOME environment variable on the machine where you installed Data Validation Option. 2. Set the value of INFA_HOME to the path of the domains.infa file. Exclude the domains.infa file name from the path. For example, set the value to C:\Informatica\<version>\. 3. Verify that you set the environment variable. a. Open the DOS command window and type set. b. Verify that the INFA_HOME environment variable appears in the list of environment variables. c. If you did not set the environment variable, configure the environment variable again. Copy dvoct.jar to the PowerCenter Integration Service Machine The dvoct.jar file contains Data Validation Option libraries that the PowerCenter Integration Service uses to run Data Validation Option tests. You must copy the dvoct.jar file to the PowerCenter Integration Service machine and then configure the location of the file in the Administrator tool. If you run the PowerCenter Integration Service on a grid, you must copy the file to all PowerCenter Integration Service processes and configure the location for each service process. 1. Copy the dvoct.jar file from the Data Validation Option Client to any location on the PowerCenter Integration Service machine. You can find the dvoct.jar file in the following directory on the Data Validation Option Client machine: <DataValidationOptionInstallationDir>\DVO\powercenterlibs 2. In the Administrator tool, on the Domain tab, select the Services and Nodes view. 3. Select the PowerCenter Integration Service in the Navigator. 4. Select the Processes tab. The Processes tab shows the properties of the PowerCenter Integration Service process: 5. Specify the full path and file name of dvoct.jar in the Java SDK Classpath property. By default, dvoct.jar is installed in the following Windows directory: <DataValidationOptionInstallationDir>/DVO/ powercenterlibs/dvoct.jar. To add multiple values in Java SDK Classpath property, separate the values with a semicolon for Windows or a colon for UNIX or Linux. 24 Chapter 3: Data Validation Option Installation and Configuration

38 Data Validation Option Configuration After you install Data Validation Option, you must configure Data Validation Option. Complete the following configuration tasks: Configure the Data Validation Option repository. Configure the mapping properties for Data Validation Option tests. Add the PowerCenter repository for Data Validation Option. Data Validation Option Repository The Data Validation Option repository stores test metadata and results. The Data Validation Option Client uses this information when connecting to the Data Validation Option repository. Data Validation Option Repository Properties The Data Validation Option repository properties determine how the Data Validation Option Client connects to the Data Validation Option repository. The following table describes the Data Validation Option repository properties: Property User Database Type Database Driver Database URL Database User Database Password Description Unique name for the Data Validation Option user. Use alphanumeric characters only. If you enabled Informatica authentication and the Data Validation Option user name does not match the associated PowerCenter user name, run the DVOCmd LinkDVOUsersToInformatica command to link the Data Validation Option user to the associated PowerCenter user. Type of database for the Data Validation Option repository. This driver of the database type that you selected. The URL of the database type that you selected. Specify the values for the URL parameters, such as database host name, database name, and port number. Examples: IBM DB2: jdbc:informatica:db2:// :50000;DatabaseName=sampleDatabase Oracle: jdbc:informatica:oracle:// :1521;ServiceName=sampleService Microsoft SQL Server: jdbc:informatica:sqlserver:// :1433;DatabaseName=sampleDatabase Database user name. Database user password. Data Validation Option Configuration 25

39 Create the Data Validation Option Repository To create the Data Validation Option repository, you must specify the Data Validation Option user and connection to the repository. The Data Validation Option Client uses this information to connect to the Data Validation Option repository. 1. Open the Data Validation Option Client. 2. Select File > Settings > Preferences. 3. In the Preferences dialog box, select Data Validation Option. The Data Validation Option section appears. 4. Specify the Data Validation Option user. 5. Configure the database connection properties to the Data Validation Option repository. 6. Click Test to test the database connection. If you are creating the Data Validation Option repository, a message appears indicating that the repository needs to be created. 7. Click Yes. 8. Click OK. 9. Click Save. A message appears indicating that you need to shut down the application. 10. Click OK. Data Validation Option shuts down. 11. If you created the Data Validation Option repository, open the Data Validation Option Client. The Informatica Data Validation dialog box prompts you to enter the license key. 12. Browse for the license key file, select the file, and then click OK. The Data Validation Option Client opens. 26 Chapter 3: Data Validation Option Installation and Configuration

40 Data Validation Option Mappings When you run a Data Validation Option test, Data Validation Option creates a PowerCenter mapping, and then runs it. The mapping contains the logic required to complete the test. You configure the mapping properties to determine how the PowerCenter session that contains the mapping processes the test data. Data Validation Option Mapping Properties The mapping properties determine the number of bad records for reporting and analysis, memory allocation, and maximum number of concurrent sessions. The following table describes the Data Validation Option mapping properties: Properties Max Bad Records for Reporting DTM Buffer Size Max Concurrent Runs Max Bad Records for Detailed Analysis File Delimiter Description Maximum number of bad records written to the Data Validation Option repository for each test. Default is 100. Maximum value is 1,000. Amount of memory in megabytes that the DTM process allocates for the PowerCenter session. Specify a numeric value. Or, select Automatic to enable Data Validation Option to use the buffer size specified for the PowerCenter session. Increase the default DTM buffer size if the Data Validation Option tests contain a large number of table pairs. Contact the PowerCenter administrator before changing this value. Default is Automatic. Maximum number of PowerCenter sessions that run at the same time. The PowerCenter Integration Service runs each table pair as one session regardless of the number of tests it contains. You must have an enterprise license to configure this property. Default is 10. Maximum value is 50. Maximum number of bad records stored for detailed error record analysis. You must have an enterprise license to configure this property. Default is Maximum value is 16,000,000. Delimiter character to separate the error records if you choose to store the bad records in a file. Configure the Mapping Properties for Data Validation Option Configure the mapping properties to specify the maximum number of bad records, buffer size, and maximum number of concurrent sessions. If you configure Data Validation Option to write bad records to a file, you can also configure the file delimiter to separate bad records. 1. Open the Data Validation Option Client. 2. Select File > Settings > Preferences. 3. In the Preferences dialog box, select Mapping Properties. The Mapping Properties section appears. Data Validation Option Configuration 27

41 4. Configure the mapping properties. 5. Click Save. PowerCenter Repository You must designate a target folder and one PowerCenter repository to store the mappings, sessions, and workflows created for Data Validation Option tests. You can also add other PowerCenter repositories as data sources for Data Validation Option. Optionally, you can integrate Metadata Manager with Data Validation Option to view additional metadata about data sources. PowerCenter Repository Properties The PowerCenter repository properties determine how to connect to the PowerCenter repository and target folder for Data Validation Option. The properties also determine how to connect to the Metadata Manager Service and the PowerCenter resource. PowerCenter Connection The following table describes the PowerCenter connection properties: Property Name Client Location Description Logical name of the PowerCenter repository for Data Validation Option. Append target to the name to identify the repository that stores Data Validation Option objects. This name appears in the Navigator pane of the Data Validation Option Client. Location of the pmrep.exe file on the PowerCenter Client machine. For example: C:\Informatica\<version> \clients\powercenterclient\client\bin. 28 Chapter 3: Data Validation Option Installation and Configuration

42 Property PowerCenter Version PowerCenter Domain Repository User name Password Security Domain Description The PowerCenter version that runs Data Validation Option. Name of the Informatica domain. Name of the PowerCenter repository for Data Validation Option that is defined in the Informatica domain. User name for the PowerCenter repository. User password for the PowerCenter repository. Security domain for the PowerCenter user. Specify the name of the LDAP security domain. Leave this property blank if you use native authentication. Target Repository The following table describes the properties for the PowerCenter repository that stores Data Validation Option mappings, sessions, and workflows: Property Contains Target Folder Target Folder Integration Service Data Validation Repository Connection Description Indicator that determines whether this repository contains the target folder to store Data Validation Option mappings, sessions, and workflows. If you add multiple PowerCenter repositories, select this option for one PowerCenter repository. Although the Data Validation Option can read from multiple PowerCenter repositories, it can create mappings, sessions, and workflows in one PowerCenter repository. Name of the target folder that stores Data Validation Option mappings, sessions, and workflows. Name of the PowerCenter Integration Service that runs the Data Validation Option workflows. Name of the PowerCenter connection to the Data Validation Option repository. Metadata Manager The following table describes the Metadata Manager properties: Property Enable Metadata Manager Is secure connection Description Optional. Select to integrate Data Validation Option with the Metadata Manager Service. Optional. Indicator that determines whether the Metadata Manager Service runs over a secure connection. Data Validation Option Configuration 29

43 Property Server Host Name Server Port Resource Name Description Optional. Host name of the machine on which the Metadata Manager Service runs. Optional. Port of the Metadata Manager Service. Optional. Name of the PowerCenter resource created in Metadata Manager. Add the PowerCenter Repository You must add one PowerCenter repository to store Data Validation Option mappings, sessions, and workflows. 1. Open the Data Validation Option Client. 2. Right-click INFA Repositories in the Navigator, and select Add Repository. The Repository Editor dialog box appears. 3. Enter the PowerCenter repository properties. 30 Chapter 3: Data Validation Option Installation and Configuration

44 4. Click Test to test the connection to the PowerCenter repository. If the Data Validation Option Client is unable to connect to the PowerCenter repository, verify that you set the INFA_HOME environment variable on the Data Validation Option Client machine. 5. Click Save. 6. Click OK to import the list of folders and connections from the PowerCenter repository into the Data Validation Option Client. Test the Data Validation Option Installation and Configuration To verify that you properly installed and configured Data Validation Option, run a sample test. Create and run a COUNT_ROWS test on a single table that has the least amount of data. 1. Open the Data Validation Option Client. 2. Right-click the folder where you want to store the single table, and then select Add Single Table. The Single Table Editor appears. 3. Enter the description of the single table. 4. Click Browse to select a table. The Select Data Source dialog box appears. 5. Select a table with the least amount of data, and then click Select. The Single Table Editor appears again. 6. Click Edit to select a connection. The Select Connection for Table A dialog box appears. 7. Select the appropriate connection, and then click OK. The Single Table Editor appears again. 8. Select the primary key for the table. 9. Save the single table. 10. Right-click the table in the navigator, and then select Add Constraint Test. The Single Table Test Editor dialog box appears. 11. In the Function field, select COUNT_ROWS to add a COUNT_ROWS test. 12. In the Constraint Value field, enter a constraint value. 13. Click Save. 14. Right-click the test in the details area, and then select Run Test. 15. Verify that the test runs successfully. Test the Data Validation Option Installation and Configuration 31

45 JasperReports Server Configuration If you have the enterprise license, you can use JasperReports Server to view, create, and run Data Validation Option reports. Before you can use the Data Validation Option reports, you must complete the following tasks: Install JasperReports Server. Install the SSL certificate. Copy the DataDirect drivers. Configure the JasperReports settings. Install JasperReports Server Install JasperReports Server to run Data Validation Option reports. 1. Download the JasperReports Server installer from the following location: /. 2. Run the installation program. When you run the installer, install the bundled Tomcat and bundled PostgreSQL database. ireport Designer is not required for Data Validation Option. Install the SSL Certificate You must install the SSL certificate if you use a JasperReports Server that runs on an HTTPS connection. Obtain the SSL certificate that you use with the JasperReports Server. 1. Open JasperReports Server in your web browser. Depending on your web browser, a warning about the certificate appears. 2. Install the SSL certificate and ignore any warning messages. For example, in Internet Explorer, click View Certificate in the warning dialog box to install the SSL certificate. 3. In the command line, browse to the following location: <DataValidationOptionInstallationDir>\DVO\jre\bin 4. Run the keytool -list command to list the current certificates in the keystore. For example, run the following command: keytool -list -keystore..\lib\security\cacerts 5. Enter keystore password. The initial password is changeit. After you enter the password, the keystore type, provider, and entries appear. 6. In the browser, view the installed certificates. For example, in Internet Explorer, go to Tools > Internet Options > Content > Certificates to view the installed certificates. 7. Export the SSL certificate that you just installed. For example, in Internet Explorer, select the certificate under Trusted Root Certification Authorities, and click Export. 8. Save the certificate as a DER-encoded binary file with file name root.cer under the c:\ directory. 9. Run the keytool -import command to import the certificate file into the cacerts keystore. 32 Chapter 3: Data Validation Option Installation and Configuration

46 For example, run the following command: keytool -import -alias myprivateroot -keystore..\lib\security\cacerts -file c:\root.cer 10. Enter keystore password. 11. When asked whether to trust this certificate, enter yes. The certificate is added to keystore. 12. Run keytool -list command again to verify that the SSL certificate is added. For example, run the following command: keytool -list -keystore..\lib\security\cacerts Copy the DataDirect Drivers You must copy the DataDirect drivers available with Data Validation Option to the JasperReports Server. 1. Shut down the JasperReports Server. 2. Copy dwdb2.jar, dworacle.jar, and dwsqlserver.jar from the following location: <DataValidationOptionInstallationDir>\lib. 3. Paste the copied files in the following directory: <JasperReportsServerInstallationDir>\apache-tomcat\lib. 4. Start the JasperReports Server. JasperReports Properties To add the Data Validation Option reports and dashboards to the JasperReports Server, configure the JasperReports properties. The following table describes the JasperReports properties: Properties JasperReports Server Host Name JasperReports Server Port JasperReports Server Web Application Name JasperReports Server Root Folder Is Secure Connection Description Host name of the JasperReports Server. Port number of the JasperReports Server. Name of the JasperReports web application. JasperReports Server folder that stores the Data Validation Option reports. Data Validation Option creates the folder on the Jaspersoft Server after you configure the JasperReports properties. Select if the JasperReports Server runs on a secure connection. You can determine the host name, port, and web application name when you launch the JasperReports Server. The JasperReports Server URL has the following format: name>:<port>/<webapp-name>. For example, JasperReports Server Configuration 33

47 Configure JasperReports Properties Configure the JasperReports properties to enable Data Validation Option reporting and import the Data Validation Option reports and dashboards into JasperReports Server. 1. Click File > Settings > Preferences. The Preferences dialog box appears. 2. Click JasperReports. The JasperReports area appears. 3. Select Enable JasperReports. 4. Configure the JasperReports properties. 5. Click Test to validate the connection to JasperReports Server. 6. Click Install Reports. 7. When prompted, enter the user name and password of a JasperReports Server user with administrator privileges. 8. Click OK. Data Validation Option creates the root folder on the Jaspersoft Server. 9. Click Save to save the properties and import the Data Validation Option reports and dashboards into JasperReports Server. Install DVOCmd on UNIX DVOCmd installs with Data Validation Option. You can also install DVOCmd separately on a UNIX machine. You might run DVOCmd on a UNIX machine to upgrade the Data Validation Option repository, refresh a data source repository, or install and run tests from a UNIX machine. 1. Verify that you have read and write permission on the location of the UNIX machine where you want to untar the DVOCmd files. 34 Chapter 3: Data Validation Option Installation and Configuration

48 2. In the command line of the UNIX machine, run the following command: gunzip install-dvocmd-<version>.tar.gz Note: To unzip the tar file on Solaris, use GNU Tar. You can download GNU Tar from the following web site: 3. Run the following command: cd $HOME tar -xvf install-dvocmd-<version>.tar 4. Create the following directory on the UNIX machine: $HOME/.config/DataValidator 5. Copy the DVConfig.properties file from the Data Validation Option Client machine to the following directory on the UNIX machine: $HOME/.config/DataValidator You can find the DVConfig.properties file in one of the following directories on the Data Validation Option Client machine: Operating System Directory Windows 32-bit C:\Documents and Settings\<user name> \DataValidator\ Windows 64-bit C:\Users\<user name>\datavalidator\ 6. Set the DV_CONFIG_DIR environment variable on the UNIX machine to the directory where you copied the DVConfig.properties file. Specify the full path of the file. 7. Copy the jlmapi.jar file from the following directory on the machine where you installed the Informatica services: $INFA_HOME/server/bin/javalib/jlmapi.jar 8. Copy the jlmapi.jar file to the following directory on the UNIX machine: <DVOCmdInstallationDir>/lib 9. If you run DVCmd on HP-UX or AIX, copy the locale files from $INFA_HOME/server/bin on the Data Validation Option Client machine to <DVOCmdInstallationDir>/locale on the UNIX machine. Copy the following locale files: pm_s_en.res pmlocale.bin 10. Run the following command from DVOCmd installation directory: $HOME/DVO/DVOCmd dos2unix DVOCmd Data Validation Option Upgrade You can upgrade Data Validation Option from multiple versions. If the product version that is currently installed cannot be upgraded to the current version of Data Validation Option, you must first upgrade to a supported version, and then upgrade to the current version. To determine the product version that is currently installed, click Help > About in the Data Validation Option Client. Data Validation Option Upgrade 35

49 The following table shows the upgrade paths from different versions of Data Validation Option: Data Validation Option Version Upgrade Description Upgrade directly to x Upgrade directly to Upgrade to 9.1.0, and then upgrade to Upgrade to 9.1.0, and then upgrade to Upgrade from Data Validation Option You can upgrade from Data Validation Option Back up the database of the Data Validation Option repository. 2. Uninstall the previous version of Data Validation Option. 3. Install Data Validation Option. 4. From the command line, go to the Data Validation Option installation folder. For example, go to the following directory: C:\Program Files (x86)\informatica9.5.0\dvo 5. Set the DV_CONFIG_DIR environment variable to the full path of the new user configuration directory. 6. Enter the DVOCmd UpgradeRepository command to upgrade the Data Validation Option repository. If you upgrade multiple Data Validation Option Clients that connect to the same Data Validation Option repository, upgrade the Data Validation Option repository one time. 7. To change the PowerCenter installation associated with Data Validation Option, reset the INFA_HOME environment variable on the Data Validation Option Client to the location of the domains.infa file. The domains.infa file is available in the following location: <InformaticaInstallationDir>\clients \PowerCenterClient\. 8. Restart Data Validation Option. 9. Run the Data Validation Option Client. Upgrade from Data Validation Option 9.1.x You can upgrade from Data Validation Option 9.1.x. 1. Back up the database of the Data Validation Option repository. 2. Uninstall Data Validation Option 9.1.x. 3. Install Data Validation Option. 4. From the command line, go to the Data Validation Option installation folder. To find the Data validation Option installation folder, right-click the Data Validation Option shortcut and click Open File Location. 5. Set the DV_CONFIG_DIR environment variable to the full path of the new user configuration directory. 6. Enter the DVOCmd UpgradeRepository command to upgrade the Data Validation Option repository. If you upgrade multiple Data Validation Option Clients that connect to the same Data Validation Option repository, upgrade the Data Validation Option repository one time. 36 Chapter 3: Data Validation Option Installation and Configuration

50 7. To change the PowerCenter installation associated with Data Validation Option, reset the INFA_HOME environment variable on the Data Validation Option Client to the location of the domains.infa file. The domains.infa file is available in the following location: <InformaticaInstallationDir>\clients \PowerCenterClient\. 8. Start Data Validation Option. 9. Run the Data Validation Option Client. The Data Validation Option Client prompts you to enter the license key. 10. Click Browse and select the license key. The Data Validation Option Client prompts you to restart the application. Data Validation Option stores the license information in the Data Validation Option database schema. Upgrade from Data Validation Option 3.1 You can upgrade from Data Validation Option 3.1. Upgrade version 3.1 to 9.1.0, and then upgrade to Back up the database of the Data Validation Option repository. 2. Uninstall Data Validation Option Install Data Validation Option From the command line, go to the Data Validation Option installation folder. To find the Data validation Option installation folder, right-click the Data Validation Option shortcut and click Open File Location. 5. Set the DV_CONFIG_DIR environment variable to the full path of the new user configuration directory. 6. Enter the DVOCmd UpgradeRepository command to upgrade the Data Validation Option repository. If you upgrade multiple Data Validation Option Clients that connect to the same Data Validation Option repository, upgrade the Data Validation Option repository one time. 7. To change the PowerCenter installation associated with Data Validation Option, reset the INFA_HOME environment variable on the Data Validation Option Client to the location of the domains.infa file. The domains.infa file is available in the following location: <InformaticaInstallationDir>\clients \PowerCenterClient\ 8. Optionally, to increase performance when you run tests simultaneously, change the number of pmrep processes from 2 to 8 in the following file:c:\program Files< (x86)>\informatica<version>\dvo\config \JMFProperties.properties 9. If Data Validation Option fails to upgrade the database URL, select File > Settings > Preferences > Data Validation Option, and update the database URL. The database URL format changed between versions 3.1 and Upgrade Data Validation to Upgrade from Data Validation Option 3.0 You can upgrade from Data Validation Option 3.0. Upgrade version 3.0 to 9.1.0, and then upgrade to Back up the database of the Data Validation Option repository. 2. In Data Validation Option 3.0, go to Tools > Properties, and record the name of the connection to the Data Validation Option repository. 3. Uninstall Data Validation Option 3.0. Data Validation Option Upgrade 37

51 4. Install Data Validation Option. 5. From the command line, go to the Data Validation Option installation folder. To find the Data validation Option installation folder, right-click the Data Validation Option shortcut and click Open File Location. 6. Set the DV_CONFIG_DIR environment variable to the full path of the new user configuration directory. 7. Enter the DVOCmd UpgradeRepository command to upgrade the Data Validation Option repository. If you upgrade multiple Data Validation Option Clients that connect to the same Data Validation Option repository, upgrade the Data Validation Option repository one time. 8. To change the PowerCenter installation associated with Data Validation Option, reset the INFA_HOME environment variable on the Data Validation Option Client to the location of the domains.infa file. The domains.infa file is available in the following location: <InformaticaInstallationDir>\clients \PowerCenterClient\ 9. Optionally, to increase performance when you run tests simultaneously, change the number of pmrep processes from 2 to 8 in the following file:c:\program Files< (x86)>\informatica<version>\dvo\config \JMFProperties.properties 10. Start Data Validation Option. 11. Edit the following repository information: 12. Click Save. PowerCenter version Name of the connection to the Data Validation Option repository A prompt appears and asks you to verify the test settings. 13. Right-click the repository name in the Navigator and click Refresh. 14. If Data Validation Option fails to upgrade the database URL, select File > Settings > Preferences > Data Validation Option, and update the database URL. The database URL format changed between versions 3.0 and Upgrade Data Validation to Chapter 3: Data Validation Option Installation and Configuration

52 C H A P T E R 4 Data Validation Option Client Layout This chapter includes the following topics: Data Validation Option Client Overview, 39 Tabs, 40 Folders, 42 Menus, 44 Data Validation Option Client Overview The Data Validation Option Client contains multiple areas and menus that allow you to perform different tasks. Statistics Area The statistics area appears when you click the Data Validation Option user. This area displays information about the number of repositories, table pairs, single tables, tests, views, and data sources that exist in the current instance of Data Validation Option. This area also displays information about running tests and the user name. Navigator The Navigator is on the left side of the Data Validation Option Client and contains the following objects: Object INFA Repositories SQL views Lookup views Join views Folders Description Lists all PowerCenter repositories that you add to Data Validation Option. Expand a repository to see the repository folders and the sources and targets in each folder. Lists the SQL views that you create. Lists the lookup views that you create. Lists the join views that you create. Lists the single tables and table pairs that you create. 39

53 Object Table Pairs Single Tables Description Lists all the table pairs that you create. Lists all the single tables that you create. Details Area The details area is in the upper right section of the Data Validation Option Client. This area contains tabs that display details about the objects you create in Data Validation Option such as tests, table pairs, single tables, and views. When you click a folder, table pair, or single table, the details area displays the following information about the tests associated with the object: Number of tests. Number of tests passed. Number of tests failed. Number of tests in progress. Number of tests not run because of errors. Number of tests not run by the user. Properties Area The properties area is in the lower right section of the Data Validation Option Client. This area displays the properties for the object that you select in the details area. The properties area shows the Properties view, Results view, and Bad Records view. Results View The Results view appears in the lower right section of the Data Validation Option Client when you select a test, table pair, or single table in the details area. The Results view displays test summary information, such as the test status, total number of records, number of bad records, record counts, and the test run ID. Bad Records View The Bad Records view appears in the lower right section of the Data Validation Option Client when you select a failed test. The Bad Records view displays the bad records written to the Data Validation Option repository. Status Bar The status bar appears below the Results Area and displays the number of tests in progress and the number of tests in queue to run. Tabs The Data Validation Option Client contains tabs that display different information. The Data Validation Option Client contains the following tabs: Tests Table Pairs Single Tables 40 Chapter 4: Data Validation Option Client Layout

54 SQL Views Lookup Views Join Views Tests Tab The Tests tab in the details area displays all tests set up in this instance of Data Validation Option. By default, tests are sorted in the order they were created. However, you can sort tests by clicking the column header. The following table describes the columns on the Tests tab: Column Test status icon Name Test type Table Pair /Single Table Test Run Date/Time Test Run Error Description Indicates whether tests associated with the table pair have been run and the status of the most recent run. If you hold the pointer over the icon, Data Validation Option displays the meaning of the icon. The test description. Type of test. The name of the table pair or single table. The date and time that the tests were last run. If a test failed, this column lists the error SQL Views Tab The SQL Views tab in the details area displays all SQL views set up in this instance of Data Validation Option. By default, SQL views are sorted in the order they were created. However, you can sort SQL views by clicking the column header. The following table describes the columns on the SQL Views tab: Column Description Table Name SQL Statement Description SQL view description. Tables you use to create the SQL view. SQL statement that you run against the database to retrieve data for the SQL view. The right-click menu on the SQL View tab lists the following options: Add SQL View Edit SQL View Delete SQL View Export Metadata Tabs 41

55 Lookup Views Tab The Lookup Views tab in the details area displays all lookup views set up in this instance of Data Validation Option. By default, lookup views are sorted in the order they were created. However, you can sort lookup views by clicking the column header. The following table describes the columns on the Lookup Views tab: Column Description Source Table Lookup Table Description Lookup view description. Source table name. Lookup table name. The right-click menu on the Lookup Views tab lists the following options: Add Lookup View Edit Lookup View Delete Lookup View Export Metadata Join Views Tab The Join Views tab in the details area displays all join views set up in this instance of Data Validation Option. By default, join views are sorted in the order they were created. However, you can sort Join views by clicking the column header. The following table describes the columns on the Join Views tab: Column Description Joined Tables Description Join view description. List of tables joined in the join view. The right-click menu on the Join Views tab lists the following options: Add Join View Edit Join View Delete Join View Export Metadata Folders Folders store the single tables and table pairs that you create. 42 Chapter 4: Data Validation Option Client Layout

56 By default, Data Validation Option places the single tables and table pairs that you create in the default folder. If you create a folder, you can create single tables or table pairs within the folder. You can move single tables or table pairs between folders. Within a folder, you can expand a single table or table pair to view the tests associated with it. Folder names are case sensitive. You can also copy folders. You can copy the contents of a folder in your workspace to a different folder in your workspace or to a folder in another user workspace. You must copy folder contents to a new folder. You cannot copy folder contents to a folder that exists in the target workspace. When you copy a folder, Data Validation Option copies all table pairs, single tables, and test cases in the source folder to the target folder. Data Validation Option does not copy test runs or the external IDs associated with table pairs or single tables. If a table pair or single table in the source folder uses an SQL view or a lookup view, Data Validation Option copies the view to the target user workspace unless the workspace contains a view with the same name. If the target workspace contains a view with the same name, Data Validation Option gives you the following options: You can use the view in the target workspace. You can copy the view to the target workspace with another name. Data Validation Option names the view in the target workspace "Copy <number> <source view name>." Before Data Validation Option copies a folder, it verifies that the repository and all data sources associated with the objects to copy exist in the target workspace. Object names in Data Validation Option are case sensitive. Therefore, the repository, data sources, and folders that contain the data sources must have identical names in the source and the target workspaces. If the repository or any required data source does not exist in the target workspace, Data Validation Option does not copy the folder. Copying Folders You can copy the contents of a folder in your workspace to a different folder in your workspace or to a folder in another user workspace. 1. Select Edit > Copy Folder. The Copy Folder Contents dialog box opens. 2. Enter the following information: Property Copy from User Copy from Folder Copy to User Copy to Folder Description Name of the source user. Data Validation Option copies the folder in this user workspace. Name of the source folder. Data Validation Option copies this folder. Name of the target user. Data Validation Option copies the folder to this user workspace. The source user and the target user can be the same user. Name of the target folder. The target folder must be unique in the target workspace. 3. Click OK. Folders 43

57 Copying Objects You can copy the objects in a folder in your workspace to a different folder in your workspace or to a folder in another user workspace. 1. Select the object that you want to copy. You can also select multiple objects which are available in different folders that are of the same type. 2. Right-click the object and select Copy. The Copy Object(s) dialog box appears. 3. Enter the following information: Property User Folder Description Name of the target user. Data Validation Option copies the folder to this user workspace. The source user and the target user can be the same user. Name of the target folder. The target folder must be unique in the target workspace. Menus The following table describes the Data Validation Option menu items: Menu Menu Item Definition File New Create a new Data Validation Option object. Metadata Settings Exit Import, export, and reload metadata. Configure preferences and opens the Data Validation Option folder in the Documents and Settings directory in Windows Explorer. Exit the Data Validation Option Client. Edit Edit Edit the selected object. Delete Copy folder Move tables/table pair Delete the selected object. Copy the selected folder. Move tables/table pairs to the selected folder. Action Run Tests Run selected tests. Add Test Generate Value Tests Compare Tables Generate Report Add a new test. Generate value tests for the selected object. Autogenerate tests for all the table pairs. Generate a consolidated test report. 44 Chapter 4: Data Validation Option Client Layout

58 Menu Menu Item Definition Dashboard Launches dashboard. The feature is available if you have the enterprise license. Refresh All Repositories Refresh contents of all the repositories. Everything Folder List Folder (Sources and Targets) Connections Refresh all contents in the selected repository. Refresh the list of folders in the selected repository. Refresh the sources and targets in the selected repository. Refresh all connections in the selected repository. Dashboards Home Launches the Home dashboard. Repository Details Folder Details Table Details Launches the Repository dashboard. Launches the Home dashboard. Launches the Home dashboard. Help Help Opens the help file. About Change License Key Displays information about PowerCenter Data Validation Option. Click the dialog box to close it. Opens the Change License Key dialog box. Note: You can see the Dashboards menu and menu items if you have the enterprise license. Settings Folder When you select File > Settings > Open Settings Folder, Windows Explorer displays the contents of the Data Validation Option folder in the Documents and Settings directory for that installation of the application. The data folder also contains an XML file that contains the information entered in the Preferences dialog box. Menus 45

59 C H A P T E R 5 Data Validation Option Management This chapter includes the following topics: Data Validation Option Management Overview, 46 Multiple Data Validation Option Repositories, 47 Multiple PowerCenter Versions, 48 Batch Files, 48 Data Validation Option Management Overview You can configure multiple users in the Data Validation Option Client to connect to one or more Data Validation Option repositories. You can also configure Data Validation Option to work with one or more PowerCenter installations. Although you can configure multiple users, Data Validation Option repositories, and PowerCenter installations, the Data Validation Option Client uses one Data Validation Option repository and PowerCenter installation for each user. You can configure Data Validation Option to specify the user, repository, and PowerCenter installation. When you configure the Data Validation Option, you specify the user and database connection for the Data Validation Option repository. Data Validation Option stores this information in preferences files in the user configuration directory. You specify the PowerCenter installation directory in the INFA_HOME environment variable. User Configuration Directory The user configuration directory contains preference files, log files, reports, and temporary files. Data Validation Option creates a user configuration directory for each user. By default, you can find the user configuration directory in one of the following locations: Windows 32-bit: C:\Documents and Settings\<user name>\datavalidator\ Windows 64-bit: C:\Users\<user name>\datavalidator\ Preference Files The preference files store user preferences and the database connection for the Data Validation Option repository. Each user has a unique set of preferences files. When you configure the user preferences and the connection to the Data Validation Option repository, Data Validation Option stores this information in the preference files. If you configure multiple Data Validation Option users, each user has a separate set of preference files. When you log in to the Data Validation Option Client, it retrieves the user preferences and database connection from the preference files. 46

60 You can find the following preference files in the user configuration directory: preferences.xml DVConfig.properties Multiple Data Validation Option Repositories You can configure the Data Validation Option Client machine to access multiple Data Validation Option repositories. You might create multiple Data Validation Option repositories if you run multiple versions of Data Validation Option simultaneously or if you have development and production environments. Data Validation Option creates a separate user configuration directory for each user and repository. If you create multiple repositories, you must specify a unique user configuration directory for each repository. To create an additional Data Validation Option repository, complete the following steps: 1. Create an additional Data Validation Option user. 2. Specify the user configuration directory. 3. Create an additional Data Validation Option repository. Creating an Additional Data Validation Option User You can create multiple Data Validation Option users on the same machine. You might create multiple users to access different Data Validation Option repositories. Each user is associated with one Data Validation Option repository. 1. On the Data Validation Option Client, open the command line. 2. Run the DVOCmd command CreateUserConfig to create a Data Validation Option user. The command generates the preference file and stores it in the specified output directory. The name of the preference file is <user name>-preferences.xml. 3. Rename the preferences file to preferences.xml. 4. If you create preference files for multiple users in the same output directory, move the renamed preference file to a unique location for each user. Specifying the User Configuration Directory If you have multiple Data Validation Option repositories, you must specify the user configuration directory. The user configuration directory contains the preferences.xml file, which identifies a particular Data Validation Option repository and user. Use one of the following methods to specify the user configuration directory: Batch file If you installed multiple Data Validation Option Clients and created multiple Data Validation Option repositories, create a batch file. The batch file specifies the location of the Data Validation Option Client executable file and the corresponding user configuration directory. You might use this method to run multiple versions of the Data Validation Option Client. DV_CONFIG_DIR environment variable If you installed one Data Validation Option Client and created multiple Data Validation Option repositories, set the DV_CONFIG_DIR environment variable on the Data Validation Option Client. Set the value of the environment variable to the full path of the user configuration directory. Multiple Data Validation Option Repositories 47

61 --confdir command line option If you installed one Data Validation Option Client and created multiple Data Validation Option repositories, and you want to run a DVOCmd command, add the --confdir option to the command. Set the value of the --confdir option to the full path of the user configuration directory. Creating an Additional Data Validation Option Repository You can create multiple Data Validation Option repositories if you run multiple versions of Data Validation Option simultaneously or if you have development and production environments. 1. Open the Data Validation Option Client. 2. Select File > Settings > Preferences. 3. In the Preferences dialog box, select Data Validation Option. 4. Specify the new database connection for the new Data Validation Option repository. 5. Click Test to test the database connection. 6. Click Save to create the repository. Multiple PowerCenter Versions You can configure multiple PowerCenter versions with Data Validation Option. You might need to use different PowerCenter versions based on different project requirements. For example, you might have two business units that use different versions of PowerCenter. To change the PowerCenter instance associated with Data Validation Option, create a batch file. In each batch file, specify the corresponding PowerCenter Client installation directory in the INFA_HOME environment variable. Batch Files You can create a batch file to specify the Data Validation Option Client, Data Validation Option user, Data Validation Option repository, or PowerCenter instance. Create batch files to switch between multiple Data Validation Option Clients, users, repositories, or PowerCenter instances. Use the following syntax when you create a batch file: SET INFA_HOME=<INFA HOME PATH> "<directory and name of executable file>" <User Configuration Directory> A batch file can contain the following parameters: INFA_HOME environment variable Specify the PowerCenter Client installation directory in the INFA_HOME environment variable to run Data Validation Option with a specific PowerCenter instance. Directory of the Data Validation Option Client executable file Specify the directory of the Data Validation Option Client executable file to run a specific Data Validation Option Client. 48 Chapter 5: Data Validation Option Management

62 Directory of the DVOCmd executable file Specify the directory of the DVOCmd executable file to run a specific instance of DVOCmd. User configuration directory Specify the user configuration directory to launch the Data Validation Option Client under a different user account. Note: If you use a shortcut to launch Data Validation Option, edit the shortcut to call the batch file instead of DVOClient.exe. Sample Batch Files for Multiple Users You can create batch files to run the Data Validation Option Client under different user accounts. For example, you install Data Validation Option in the default directory on Windows 32-bit. You create a separate batch file for the following user configuration directories: Development user configuration directory: C:\DVOConfig_Dev Production user configuration directory: C:\DVOConfig_Prod For the development environment, you create the following batch file: "C:\Program Files\Informatica9.5.0\DVO \DVOClient.exe" C:\DVOConfig_Dev. For the production environment, you create the following batch file: "C:\Program Files\Informatica9.5.0\DVO \DVOClient.exe" C:\DVOConfig_Prod. Sample Batch Files for Multiple PowerCenter Instances You can create batch files to associate different versions of PowerCenter with the Data Validation Option Client. For example, you install Data Validation Option in the default directory on Windows 32-bit. Depending on the test, you run the test on PowerCenter or You create a separate batch file for each PowerCenter version. For PowerCenter version, you create the following batch file: SET INFA_HOME=C:\Informatica\9.1.0 "C:\Program Files\Informatica9.5.0\DVO\DVOClient.exe". For PowerCenter version, you create the following batch file: SET INFA_HOME=C:\Informatica\9.5.0 "C:\Program Files\Informatica9.5.0\DVO\DVOClient.exe". Batch Files 49

63 C H A P T E R 6 Informatica Authentication This chapter includes the following topics: Informatica Authentication Overview, 50 Data Validation Option Users and Informatica Users, 50 Security, 51 Informatica Authentication Properties, 51 Enabling Informatica User Authentication, 52 Informatica Authentication Overview Administrators can enable Informatica authentication so that the users must use valid Informatica domain credentials to use the Data Validation Option Client. By default, the Data Validation Option Client does not validate the users that launch the Data Validation Option Client. You can enable Informatica authentication if you have PowerCenter or later. Informatica authentication validates over a secure connection with TLS if you have enabled TLS in the Informatica domain. To configure Informatica authentication, you must have an Informatica login credential with administrator privileges. The information you require to configure Informatica authentication is available in the nodemeta.xml file in machine that hosts the Informatica services. You can also update the Informatica authentication properties through the DVOCmd command UpdateInformaticaAuthenticationConfiguration. After you enable Informatica authentication, you must enter the Informatica user name and password when you start the Data Validation Option Client. If you use a LDAP domain, you must enter the security domain in addition to the user credentials. Note: Informatica authentication does not authenticate users of the JasperReports Server reports and dashboards. Data Validation Option Users and Informatica Users Data Validation Option users and users with Informatica credentials can launch the Data Validation Option Client. Users connect to a Data Validation Option schema when they use the Data Validation Option Client. You can enable Informatica authentication to enable valid Informatica domain users to log in to Data Validation Option. To configure Informatica authentication, you must have an Informatica login credential with administrator privileges. To enable authentication, map each Data Validation user to an Informatica user. If the two users have the same name, then Data Validation Option maps the users automatically. If the two users have different names, then you can map 50

64 Data Validation Option users to Informatica users with the DVOCmd command LinkDVOUsersToInformatica. The permissions of the Informatica user determines the PowerCenter metadata access for the associated Data Validation Option user, so it is important to ensure that those permissions and privileges are set correctly. Security You can enable the Transport Layer Security (TLS) protocol for secure authentication of Informatica domain users in Data Validation Option. Informatica authentication in Data Validation Option enables connection in a secure network with TLS. Enable TLS in Informatica Administrator to connect to Data Validation Option over a secure connection. You can enable TLS in the Informatica Administrator from the Informatica domain properties. After you enable TLS for the domain, configure Informatica authentication in the Data Validation Option Client to use TLS. The properties for TLS are accessed from the Informatica domain. Informatica Authentication Properties Configure the Informatica services host name and port number to enable Informatica authentication. You must also specify whether the domain uses a secure connection. Use the nodemeta.xml file to determine the values for these properties. The following table describes the Data Validation Option repository properties: Properties Enable Informatica User Authentication Informatica Server Host Name Informatica Server HTTP Port Is Secure Connection Description Enables Informatica authentication in the Data Validation Option Client. By default, Informatica authentication is disabled. Host name of the machine where you installed Informatica services. Port number through which Data Validation Option accesses Informatica domain. Note: Do not use the value available in the httpport element in the nodemeta.xml file. Indicates whether TLS is enabled in the domain. Note: If TLS is not enabled in the Informatica domain, the tlsenabled element is not available in the nodemeta.xml file. You can find the nodemeta.xml file in the following directory on the machine where you installed Informatica services: <InformaticaInstallationDir>\isp\config Security 51

65 Enabling Informatica User Authentication By default, Data Validation Option does not authenticate users. You can configure the Data Validation Option Client to authenticate the Data Validation Option Client users based on their associated Informatica user credentials. You can also use the DVOCmd UpdateInformaticaAuthenticationConfiguration command to enable Informatica user authentication. You must have the Administrator role in the Informatica domain to enable Informatica authentication. The Data Validation Option Client determines the Informatica roles based on the PowerCenter user associated with the user that is logged in to the Data Validation Option Client. 1. In the Data Validation Option Client, click File > Settings > Preferences. The Preferences dialog box appears. 2. Select Informatica Authentication. The Informatica Authentication dialog box appears. 3. Configure the Informatica authentication properties. 4. Click Test to test the domain settings. 5. Click Save to save the domain settings. 6. Close the Data Validation Option Client. 7. Open the Data Validation Option Client. The Login dialog box appears. 8. Enter the PowerCenter username and password. If you use a LDAP domain, you must also enter the security domain. 52 Chapter 6: Informatica Authentication

66 C H A P T E R 7 Repositories This chapter includes the following topics: Repositories Overview, 53 Adding a Repository, 53 Editing Repositories, 54 Deleting Repositories, 54 Refreshing Repositories, 54 Metadata Export and Import, 55 Metadata Manager Integration, 57 Repositories Overview Data Validation Option connects to a PowerCenter repository to import metadata for PowerCenter sources, targets, folders, and connection objects. Data Validation Option also connects to a PowerCenter repository to create mappings, sessions, and workflows in the Data Validation Option target folder. When you add a repository to Data Validation Option, you add either a source or target repository. You can add one target repository and multiple source repositories. The target repository contains metadata for PowerCenter sources, targets, folders, and connection objects. It also contains the Data Validation Option target folder. The target folder stores the mappings, sessions, and workflows that Data Validation Option creates when you run tests. Do not store other PowerCenter mappings, sessions, or workflows in this folder. A source repository contains metadata for PowerCenter sources, targets, and folders. Add source repositories to Data Validation Option if you want to compare tables from different repositories. When you add a source repository, you must verify that all connection objects in the source repository also exist in the target repository. Data Validation Option uses the connection objects in the target repository when you run tests on table pairs. The version number for a source repository can differ from the version number for the target repository. The version numbers for two source repositories can also differ. Adding a Repository You can add one target PowerCenter repository and multiple PowerCenter source repositories. Source repositories contain objects that you want to validate. Data Validation Option stores workflow objects for tests in the target 53

67 repository. The associated PowerCenter user account must have Read permission on source repository connections and Read, Write, and Execute permissions on the target repository connection. 1. Right-click INFA Repositories in the Navigator. 2. Select Add Repository. The Repository Editor dialog box appears. 3. Enter the repository properties. Set Contains Target Folder to false when you add a source repository. Set to true when you add a target repository. 4. Click Test to test the repository connection. Data Validation Option verifies the connection properties. If the repository is a target repository, Data Validation Option also verifies the PowerCenter Integration Service and verifies that the Data Validation Option target folder and option results warehouse connection exist in the repository. Editing Repositories u To edit a repository, right-click the repository, and select Edit Repository, or double-click the repository. The Repository Editor dialog box appears. You can update any property that is enabled. Deleting Repositories u To delete a repository from Data Validation Option, right-click the repository, and select Delete Repository. Data Validation Option deletes the repository and all table pairs, single tables, tests, and views based on the repository data. Refreshing Repositories You refresh a source repository when the contents of the PowerCenter repository have changed. You usually refresh a target repository only when there are additions or changes to connection objects. When you refresh a repository, Data Validation Option imports the source, target, folder, and connection metadata from the PowerCenter repository. Therefore, Data Validation Option objects that use changed or deleted PowerCenter objects might no longer be valid after you refresh a repository. If you created table pairs, single tables, or tests with tables that were deleted from the PowerCenter repository, Data Validation Option deletes them when you refresh the repository. 1. To refresh all repositories at once, right-click INFA Repositories in the Navigator, and select Refresh All Repositories. To refresh one repository, right-click the repository, and select Refresh Repository 54 Chapter 7: Repositories

68 2. When you refresh one repository, you can select the objects to refresh. Select one of the following options: Everything Data Validation Option reimports all source, target, folder, and connection metadata. It updates the folder list, and the Sources and Targets folders in the Navigator. Connections Data Validation Option reimports connection metadata. Select this option when a PowerCenter user adds, removes, or updates connection objects. Folder List Data Validation Option reimports folder metadata. It updates the folder list in the Navigator. Select this option when a PowerCenter user adds or removes folders. Folders (Sources and Targets) Data Validation Option reimports source and target metadata. It refreshes the contents of the Sources and Targets folders in each folder in the repository. Select this option when a PowerCenter user adds, removes, or modifies sources or targets in folders. If the PowerCenter folder contains a shortcut to an object in another folder, you must to refresh the other folder also. You can also refresh repository folders individually. You might refresh a folder after you refresh the folder list and Data Validation Option imports a new folder. To refresh a repository folder, right-click the folder in the Navigator, and select Refresh Folder (Sources and Targets). Data Validation Option refreshes the contents of the Sources and Targets folders within the folder you refresh. Note: Refreshing everything in a repository or refreshing all repositories can take several minutes to several hours, depending on the size of the repositories. If you work with a small number of repository folders, you can shorten refresh time by refreshing the folders individually. Exporting Repository Metadata You can export repository metadata to a file. You might want to export repository metadata when migrating from a development to a production environment or if you are asked to do by Informatica Global Customer Support. To export repository metadata from Data Validation Option, right-click the repository, and select Export Metadata. Data Validation Option prompts you for a file name and file path. Metadata Export and Import Data Validation Option allows you to export and import test metadata from the repositories. Metadata import and export allows users to share tests and allows rapid generation of tests through scripting. Scripting is particularly useful in the following scenarios: You have a very large number of repetitive tests for different table pairs. In this situation, it might be faster to generate the tests programmatically. The source-to-target relationships and rules are defined in a spreadsheet. This often happens during data migration. You can script the actual Data Validation Option tests from the spreadsheets. You can import and export the following metadata: Table Pairs Single Tables Metadata Export and Import 55

69 PowerCenter Sources SQL views Lookup views Join views RELATED TOPICS: Metadata Import Syntax on page 161 Exporting Metadata You can export Data Validation Option metadata objects to an XML file. You can export particular objects or all objects. You can also export dependent objects. When you export objects, you can skip objects that are not valid. 1. Right-click the object and select Export Metadata. 2. To export all objects, select File > Export All Metadata. 3. Enter the name of the export file, and then click Save. 4. If any objects are not valid, specify whether to skip the objects. Importing Metadata You can import Data Validation Option metadata objects. When you import metadata objects, you can edit the metadata, overwrite metadata in the Data Validation Option repository, and generate value tests. Before you import metadata, you can configure the import file to generate value tests. For example, to generate value tests for the CustDetail_CustStage table pair, add the following lines at the end of the table pair definition in the import XML file: <Commands> generate-tests("custdetail_custstage"); When you import metadata from an XML file, you can overwrite repository objects that have the same type and same name as the objects you import. If you do not overwrite objects and an object with the same name and same type exists in the repository, Data Validation Option does not import the metadata. When you import the metadata, you can also edit the metadata. You might change the metadata based on the environment in to which you import the metadata. For example, you can change the connections, flat file locations, Data Validation Option repository, and PowerCenter repository that stores Data Validation Option objects. 1. To generate value tests, add the generate-tests command to the end of the metadata definition for the table pair in the import XML file. 2. To import and overwrite repository objects, select File > Import Metadata (Overwrite). The Import Metadata dialog box appears. 3. To import metadata without overwriting repository objects, select File > Import Metadata. The Import Metadata dialog box appears. 4. Specify whether you want to edit the metadata before importing the metadata. If you do not edit the metadata, the Data Validation Option imports the metadata. If you edit the metadata, the Import Metadata dialog box appears. 5. Change the metadata, and then click Import. 56 Chapter 7: Repositories

70 Metadata Manager Integration You can analyze the impact of test results on data sources if you enable Metadata Manager integration. You can view the metadata of the data source in the PowerCenter repository. To view the metadata of data sources, you must setup a Metadata Manager Service in the Informatica domain. You must create a PowerCenter resource for the PowerCenter repository that contains the data source. Metadata Manager Properties Configure the Metadata Manager properties to view the metadata of the data source in a PowerCenter repository. The following table describes the Metadata Manager properties: Property Enable Metadata Manager Is secure connection Server Host Name Server Port Resource Name Description Optional. Select to integrate Data Validation Option with the Metadata Manager Service. Optional. Indicator that determines whether the Metadata Manager Service runs over a secure connection. Optional. Host name of the machine on which the Metadata Manager Service runs. Optional. Port of the Metadata Manager Service. Optional. Name of the PowerCenter resource created in Metadata Manager. Configuring Metadata Manager Integration Configure the connection to Metadata Manager to view additional metadata about PowerCenter data sources and to view the upstream and downstream impact summaries of data sources. 1. Right-click the repository for which you want to enable Metadata Manager Integration and select Edit Repository. The following figure shows the Metadata Manager properties: 2. Select Enable Metadata Manager Integration. 3. Configure the Metadata Manager properties. Metadata Manager Integration 57

71 4. Click Save. 58 Chapter 7: Repositories

72 C H A P T E R 8 Table Pairs This chapter includes the following topics: Table Pairs Overview, 59 Basic Properties, 60 Connection Properties, 61 Advanced Properties, 62 Database Processing, 63 Pushing Test Logic to the Database, 63 WHERE Clauses, 64 Table Joins, 65 Parameterization, 66 Bad Records Configuration, 66 Data Sampling, 68 PowerCenter Cache, 69 Adding Table Pairs, 69 Editing Table Pairs, 71 Deleting Table Pairs, 71 Overall Test Results, 71 Table Pairs Overview A table pair is the basis for all tests that compare one table to another. You can select a relational table, flat file, lookup view, SQL view, or join view as one or both tables in a table pair. Data Validation Option considers Table A as the master table and Table B as the detailed table. To improve the performance, select the master table as Table A and detailed table as Table B when you select tables in a table pair. If you have the enterprise license, you can store all the error records after you run a test for the table pair. You can also use a parameter file that the PowerCenter Integration Service applies when Data Validation Option runs the sessions associated with the table pair. If a test for a table pair fails to run, right-click the table pair to retrieve the corresponding session log. View the session log to view errors that may have occurred while running the underlying PowerCenter session. 59

73 Basic Properties You can view or configure the basic properties for a table pair. The properties vary based on the types of objects you select for the table pair. The following list describes the basic properties for a table pair: Description Table pair description. By default, Data Validation Option uses "Joined <Table A>-<Table B>" for joined table pairs. It uses "<Table A>-<Table B>" for table pairs that are not joined. Table A/B The first or second table in the table pair. Select Browse to select a table. You can search for a data source by name or path. You can search for a lookup view, SQL view, and join view by name. Conn A/B Connection details for the table. Select Edit to edit the connection properties. Where Clause A/B The WHERE clause filters the records that the PowerCenter Integration Service reads from the data source. If you run the WHERE clause on PowerCenter, enter a valid PowerCenter Boolean expression. If you run the WHERE clause in a relational source or an application source, enter the WHERE clause in the supported data source format. If you choose to run the WHERE clause within the data source, the data source applies the WHERE clause before PowerCenter Integration Service loads data. Execute where clause A/B Select if you want the database to execute the WHERE clause. The database executes the WHERE clause before the PowerCenter Integration Service reads the records. If you select the Where Clause, Sorting and Aggregation in DB optimization level, the database also sorts and aggregates the data. Optimization Level Controls which test logic that Data Validation Option converts to a PowerCenter mapping and which test logic it pushes to the database. You can select one of the following options: Default. Data Validation Option converts all test logic to a PowerCenter mapping and applies sorting to the data source. WHERE clause, Sorting, and Aggregation in DB. Data Validation Option pushes the WHERE clause, sorting logic for joins, and all aggregate tests to the database. Data Validation Option converts all other test logic to a PowerCenter mapping. Already Sorted Input. PowerCenter mapping does not sort the input. If the data source is not sorted, tests might fail. Join Field A/B Join condition for the table pair. Select Expression to insert an expression for a join field. If you entered an expression for a join field, click Validate to validate the expression. Note: If you enter an expression for a join field, you cannot save the table pair until you enter a valid expression. External ID Identifier for the table pair that you can use when you run Data Validation Option tests from the command line. 60 Chapter 8: Table Pairs

74 Connection Properties Choose the connection properties based on the data source type. You must select a connection for all the data sources except for flat files. For flat files, you must provide the source directory and the file name. Connections are PowerCenter connection objects created in the Workflow Manager. Relational Connection Properties Choose the relational connection properties for Microsoft SQL Server, Oracle, IBM DB2, Netezza, and PowerExchange for DB2 data sources. Configure the following properties when you select a relational data source: Property Connection Override Owner Name Description PowerCenter connection object to connect to the relational data source. Override the database name and schema of the source. For example, a Microsoft SQL Server table is identified by <database>.<schema>.<table>. To override the database and the schema, enter <new database name>.<to change the schema, enter <new schema name> in the text box. You cannot change only the database name. SAS and Salesforce Connection Properties You can use a SAS or Salesforce data source in Data Validation Option. Select the PowerCenter connection object when you select an SAS or Salesforce data source. Note: You cannot override the owner name for SAS and Salesforce data sources. SAP Connection Properties You must configure the SAP authentication information to use SAP data sources. Configure the following properties when you select an SAP data source: Property Connection SAP User Name SAP Password SAP Client SAP Language SAP Connect String Description PowerCenter connection object to connect to the SAP data source. SAP source system connection user name. Must be a user for which you have created a source system connection. Password for the user name. SAP client number. Language you want for the mapping. Must be compatible with the PowerCenter Client code page. Data Validation Option does not authenticate the value. Ensure that you enter the correct value so that the tests run successfully. Type A or Type B DEST entry in saprfc.ini. Connection Properties 61

75 SAP Data Sources in Data Validation Option You cannot override the owner name for an SAP data source. Data Validation Option uses the stream mode for installation of the ABAP programs and cannot use the FTP mode. SAP data sources must not contain the backslash (/) character in the field names. Flat File Connection Properties You can use the flat file data sources in the PowerCenter repository. Configure the following properties when you select a flat file data source: Property Source Dir Source File File Type Description Directory that contains the flat file. The path is relative to the machine that hosts Informatica Services. File name with the file name extension. Type of flat file. Select one of the following values: - File Contains Source Data. Select if the flat file contains data. - File Contains a List of Files. Select if the flat file contains a file list. When you select a file list, Data Validation Option processes all files in the list. Advanced Properties You can configure bad records, a parameter file, sampling, and caching for a table pair. You must have the enterprise license to configure bad records and a parameter file. The properties vary based on the types of objects you select for the table pair. The following list describes the advanced properties for a table pair: Save All Bad Records for Test Execution Saves all bad records for value, outer value, and set tests for table pairs. Write Bad Records to: The location where Data Validation Option writes bad records. You can select one of the following options: Flat File. Data Validation Option writes bad records to the specified flat file. DVO Schema. Data Validation Option writes bad records to the Data Validation Option repository. File Name The location and name of the flat file to which Data Validation Option writes bad records. Default location is $INFA_HOME/server/infa_shared/BadFiles/<single table name /table pair name>/<file name>. Parameter File Name of the parameter file. Click Add to add a parameter file. Click Delete to delete the selected parameter file. Enable Data Sampling Indicator that determines if data sampling is enabled. 62 Chapter 8: Table Pairs

76 Apply Sampling on: The table on which you want to apply sampling. Percentage of Rows Percentage of rows that you want to sample. Minimum value is Maximum value is less than 100. Seed Value The starting value used to generate a random number for sampling. You can configure the seed value for the following platforms: PowerCenter. Minimum seed value is 0. Maximum seed value is IBM DB2. Minimum seed value is 1-10³¹. Maximum seed value is 10³¹ - 1. Oracle. Minimum seed value is -2,147,483,648. Maximum seed value is 2,147,483,647. Use Native Sampling The database performs the sampling. If you do not enable this option, PowerCenter performs the sampling. Cache Size Total cache used to run a PowerCenter session associated with the table pair. By default, the units are bytes. Append KB, MB, or GB to the value to specify other units. For example, you can specify , 1024 KB, or 1 MB. Select Automatic to enable PowerCenter to compute the cache size. Database Processing When you include a large table in a table pair, you can optimize the way Data Validation Option joins table data. Data Validation Option uses joins in value and set tests. To run value tests on a table pair, you must join the tables based on the related keys in each table. By default, Data Validation Option joins the tables with an inner equijoin and sorts the rows in the table. The join condition uses the following WHERE clause syntax: Table A.column_name = Table B.column_name If one of the tables in a table pair contains a large volume of data compared to the other table, you can improve test performance by designating the larger table as the detail source and the smaller table as the master source. Select smaller table as Table A and the larger table as Table B. The PowerCenter Integration Service compares each row of the master source against the detail source when it determines whether to join two records. If both tables in a table pair are large, you can improve test performance by using sorted input. When you use sorted data, the PowerCenter Integration Service minimizes disk input and output when it runs a test. You can further increase performance for large relational tables by pushing the sorting logic to the database. Pushing Test Logic to the Database You can push some test logic for a relational table to the source database. By default, Data Validation Option creates a PowerCenter mapping for each table pair. When you run a test, PowerCenter runs the mapping logic in a session. In some cases, you can significantly increase test performance by processing logic in the database instead of PowerCenter. Database Processing 63

77 You can push the WHERE clause, logic for aggregate tests, sorting logic for joins, and sampling to the source database. You can push the WHERE clause, logic for aggregate tests, and sorting logic for joins, to the following databases: IBM DB2, Microsoft SQL Server, Netezza, Oracle, Sybase, and Teradata. You can push sampling to the following databases: IBM DB2, Microsoft SQL Server, Oracle, and Teradata. Pushing the WHERE clause to the database can reduce the number of rows the PowerCenter Integration Service reads when it runs a Data Validation Option test. For example, Table A is a table of U.S. customers, and you want to test data only for customers in California. You enter a WHERE clause such as STATE = 'CA'. Data Validation Option creates a mapping in PowerCenter that reads all U.S. customers. The mapping might contain a Filter transformation that removes all records for customers outside of California. If you push the WHERE clause to the database, the database filters customer records. The PowerCenter Integration Service reads records for California customers only. Pushing the WHERE clause to the database increases the performance of the PowerCenter session because the Integration Service reads a small subset of records instead of all records in the table. Pushing aggregate test logic to the database can also reduce the number of rows the PowerCenter Integration Service reads from the database. For example, you use a COUNT test to compare the number of non-null records in two Customer_ID columns. If you push the test logic to the database, the PowerCenter Integration Service does not have to import all customer records to count them. Pushing the sorting logic for joins to the database causes the database to sort records before it loads them to PowerCenter. This minimizes disk input and output when Data Validation Option runs a test. Push sorting logic to the database when you run tests on tables with large volumes of data. Pushing test logic to the database slightly increases the load on the database. Before you push test logic to the database, decide whether the increased performance of the test outweighs the increased load on the database. Pushing sampling to the database increases the run-time performance because the PowerCenter Integration Service does not have to read all the data from both tables. WHERE Clauses Use a WHERE clause to limit the records returned from a data source and made available for a test. Since PowerCenter pulls data from each data source individually, you can provide separate WHERE clauses or filters for the data pulled from each table. Enter the WHERE clause without the WHERE keyword, for example, CITY <> 'London'. The PowerCenter Integration Service is case-sensitive when it reads WHERE clauses. This functionality corresponds to the use of the Filter transformation in PowerCenter. Data Validation Option does not check the WHERE clause syntax. If the PowerCenter Integration Service executes the WHERE clause, any valid PowerCenter expression, including expressions that use PowerCenter functions, is allowed. If the PowerCenter syntax is not valid, a mapping installation error occurs. Use the following guidelines if the data source executes the WHERE clause: Relational data source. The WHERE clause must be a valid SQL statement. If the SQL statement is not valid, a runtime error occurs. Enter a WHERE clause in the SQL format when you pushdown the WHERE clause into the database. SAP data source. The WHERE clause must be a valid SAP filter condition in the ERP source qualifier. Salesforce data source. The WHERE clause must be a valid SOQL filter condition. 64 Chapter 8: Table Pairs

78 SAS data source. The WHERE clause must be a valid Whereclause Overrule condition in the SAS source qualifier. When you enter a WHERE clause, consider the following issues: Data Validation Option uses two WHERE clauses instead of one. A typical SQL statement has one WHERE clause. Data Validation Option, however, has one WHERE clause for Table A and one for Table B. Therefore, it is possible that more data comes from one table than the other. For example, applying emp_id < 10 to Table A but not Table B results in only nine records coming from Table A and all records from Table B. This affects OUTER_VALUE and aggregate tests, which might or might not be what you intended. However, when you compare production to development where the production environment has three years of data and development only has two weeks, applying a WHERE clause to production equalizes the data sets. Certain validation problems can be solved through a nested SQL WHERE clause. For example, if you want to filter for employees with disciplinary issues, use the following WHERE clause (assuming it is executed in the database): emp_id IN (SELECT DISTINCT emp_id FROM table_discipline) Because the filter condition you enter in the WHERE clause applies to all tests in the table pair, Data Validation Option applies the WHERE clause before it joins the tables. This can improve performance when the WHERE clause filters a large percentage of rows from the source table because the PowerCenter Integration Service processes fewer rows later in the mapping. If you want to enter a condition that filters a small percentage of rows, or you want to apply different filters for different tests, you can enter a filter condition in the Table Pair Test Editor dialog box. Table Joins You can join or unjoin a table pair. You must join the tables if you want to run a VALUE or OUTER_VALUE test. Data Validation Option ignores joins for all set and aggregate tests. To create a joined table pair, define one or more conditions based on equality between the tables. For example, if both tables in a table pair contain employee ID numbers, you can select EMPLOYEE_ID as the join field for one table and EMP_ID as the join field for the other table. Data Validation Option performs an inner equijoin based on the matching ID numbers. You can join tables only on fields of like datatypes. For example, you can join an INT field to a DECIMAL field, but not to a DATETIME or VARCHAR field. Data Validation Option supports numeric, string, datetime and binary/other datatypes. Joins are not allowed on binary/other datatypes. You can create a join with one set of fields or with multiple sets of fields from each table if the table requires this to produce unique records. Note that additional sets of fields increase the time necessary to join two sources. The order of the fields in the join condition can also impact the performance of Data Validation Option tests. If you use multiple sets of fields in the join condition, Data Validation Option compares the ports in the order you specify. When you create a join, you can select a field from each table or enter an expression. Enter an expression to join tables with key fields that are not identical. For example, you have two customer tables that use different cases for the LAST_NAME field. Enter the following expression for one of the join fields: lower(last_name) When you enter an expression for one or both join fields, you must specify the datatype, precision, and scale of the result. The datatype, precision, and scale of both join fields must be compatible. The expression must use valid PowerCenter expression syntax. Data Validation Option does not check the expression syntax. Table Joins 65

79 Parameterization If you have the enterprise license, you can perform incremental data validation through parameterization. You can use parameters in the WHERE clause in Table Pairs and Single Tables. Before you use a parameter in the WHERE clause, you must enter the name of the parameter file and add the parameters on the Advanced tab of a table pair or single table definition. You must specify the data type, scale, and precision of the parameter. After you add the parameters, you can use them in a WHERE clause to perform incremental validation. You can validate the expression with the parameters that you enter in a table pair or single table. The parameter file must be in a location accessible to the Data Validation Option Client. The parameters in the parameter file must be in the format, $$<parameter name>=value. Ensure that the parameter that you add in a table pair or single table is available in the parameter file. When you run a test, Data Validation Option looks up the value of the parameter from the parameter file and runs the WHERE clause based on that value. If the Informatica Services run on a Windows machine, you can place the parameter files in a folder on the server. Ensure that the Data Validation Option Client can access the folder. In the Data Validation Option Client, enter the parameter file location as a network path to the parameter file in the server in the following format: \\<server machine host name>\<shared folder path>\parameter file name If the Informatica Services run on a UNIX machine, you can place the parameter files in a folder on the server. Install DVOCmd on the server. You can configure parameterization on the Data Validation Option Client and provide the absolute path of the parameter file and run the tests from the server with DVOCmd. Alternatively, you can ensure that the Data Validation Option Client can access folder and enter the parameter file location as a network path to the parameter file in the server. Run the tests from the Data Validation Option Client. Bad Records Configuration If you have the enterprise license, you can store up to 16,000,000 bad records per test to perform advanced analysis and up to 1000 bad records for reporting. If you have the standard license, you can store up to 1000 bad records for reporting and analysis. For reporting, you can set the maximum number of bad records in the mapping properties in the Preferences dialog box. The default number of bad records is 100. You can set up to 1000 bad records for reporting. For advanced analysis, you can set the maximum number of bad records for detailed analysis and the file delimiter. You set the properties in the Detailed Error Rows Analysis section in the mapping properties. The default number of bad records is You can set up to 16,000,000 bad records. You can store all the bad records from the tests for a table pair or single table. Select Save all bad records for test execution on the Advanced tab when you create or edit a table pair or single table. Data Validation Option stores the following tests for table pairs: Value Outer Value Set You must select whether you want to store the bad records in a flat file or a table in the Data Validation Option schema. If you store bad records in a flat file, you can optionally enter the name of the file. Data Validation Option appends test information to the name and retains the file extension. 66 Chapter 8: Table Pairs

80 Note: If you modify the file delimiter in the preferences file, run the InstallTests command with the forceinstall option for the existing table pairs or single tables that you already ran. You can also edit and save the table pair or single table from the Data Validation Option Client before you run the test. If you modify the bad records value, you need not reinstall the tests. Bad Records in Flat File If you configure to store bad records in flat file, Data Validation Option creates the flat file in the machine that runs Informatica services. The flat files that Data Validation Option generates after running the tests are stored in the following folder:<powercenter installation directory>\server\infa_shared\tgtfiles You can modify the folder from the Administrator tool. Edit the $PMTargetFileDir property for the PowerCenter Integration Service. Data Validation Option generates a folder for each of the table pair or single table. The name of the table pair or single table is in the following format: TablePairName_TestRunID or SingleTableName_TestRunID Data Validation Option creates flat files for each test inside the folder. The name of the flat file is in the following format: <user defined file name>_ TestCaseType_TestCaseColumn A_TestCaseColumnB_TestCaseIndex.<user defined file extension> You can get the Test Case Index from the Properties tab and the Test Run ID from the results summary of the test in the detail area. You can get the Table Pair ID/Single Table ID from the Table Pair or Single Table properties. For example, you enter the file name as BAD_ROWS.txt when you configure the table pair or single table and you run an outer value test on fields FIRSTNAME and FIRSTNAME. The test-case index is 1 and the test-fields are expressions. The bad records file after you run the test is of the format, BAD_ROWS_OUTER_VALUE_ExprA_ExprB_1.txt. Data Validation Option supports all the file delimiters that PowerCenter supports. When you enter non-printable characters as delimiters, you should enter the corresponding delimiter code in PowerCenter. When you import these files in PowerCenter, you have to manually create the different data fields since the code appears in the place of delimiter in the bad records file. Caution: Data Validation Option uses comma as a delimiter if there a multiple primary keys. You must not use comma as a file delimiter. When you run the tests for the table pair or single table, Data Validation Option stores the details of the bad records along with the following format: Table Pair Name Table A Name Table A Connection Table B Name Table B Connection Test Definition Test Run Time Test Run By User Key A[],Result A[],Key B[],Result B[] If the tests pass, Data Validation Option still creates a flat file without any bad records information. Bad Records Configuration 67

81 Bad Records in Database Schema Mode If you choose to save detail bad records in the Data Validation Option schema, the bad records are written into the detail tables. The following table describes the tables to which Data Validation Option writes the bad records based on the type of test: Test Type Table Columns Value, Outer Value, Value Constraint ALL_VALUE_RESULT_DETAIL TEST_RUN_ID, TEST_CASE_INDEX, KEY_A, VALUE_A, KEY_B, VALUE_B Unique Constraint ALL_UNIQUE_RESULT_DETAIL TEST_RUN_ID, TEST_CASE_INDEX, VALUE Set ALL_SET_RESULT_DETAIL TEST_RUN_ID, TEST_CASE_INDEX, VALUE_A, VALUE_B Note: You must ensure that the database table has enough table space to hold all the bad records. The test details of all the tests that you ran in Data Validation Option is available in the TEST_CASE table. The test installation details of the tests are available in the TEST_INSTALLATION table. You can obtain the TEST_ID of a test from the TEST_INSTALLATION table. You need TEST_ID of a test to query the complete details of a test from the TEST_CASE table. You can get the Test Case Index from the Properties tab and the Test Run ID from the results summary of the test in the detail area. You can get the Table Pair ID/Table ID from the Table Pair or Single Table properties. Foe example, you ran a table pair with a value test and outer value test. The following SQL query is a sample query to retrieve the information of bad records of a test with Test Case Index as 5. select ALL_VALUE_RESULT_DETAIL.*,TEST_CASE.* from ALL_VALUE_RESULT_DETAIL, TEST_RUN, TEST_INSTALLATION, TEST_CASE where ALL_VALUE_RESULT_DETAIL.TEST_RUN_ID=TEST_RUN.TEST_RUN_ID and TEST_RUN.TEST_INSTALLATION_ID=TEST_INSTALLATION.TEST_INSTALLATION_ID and TEST_INSTALLATION.TABLE_PAIR_ID=TEST_CASE.TEST_ID and ALL_VALUE_RESULT_DETAIL.TEST_CASE_INDEX=TEST_CASE.TEST_CASE_INDEX and ALL_VALUE_RESULT_DETAIL.TEST_RUN_ID=220 Data Sampling You can use data sampling to run tests on a subset of a data set. You might use data sampling when the data set is large. You can perform data sampling on table pairs and single tables. When you run a test on a sample data set, Data Validation Option runs the test on a percentage of the data set. The sample percentage that you specify represents the chance that each data is included in the sample. You can use a seed value to repeat the same sample data set in a test. Data Validation uses the seed value as the starting value to generate a random number. If you do not enter a seed value, Data Validation Option generates a random seed value for each test run. You might use a seed value to replicate a Data Validation Option test. By default, PowerCenter performs the sampling. If you sample data from IBM DB2, Microsoft SQL Server, Oracle, or Teradata, you can perform native sampling in the database. Push sampling to the database to increase performance. 68 Chapter 8: Table Pairs

82 If you add the WHERE clause and enable sampling, the order of operations depend on where you perform sampling and execute the WHERE clause. Generally, PowerCenter and the database performs sampling before executing the WHERE clause. However, when you configure the database to execute the WHERE clause and PowerCenter to perform sampling, the database executes the WHERE clause before PowerCenter performs the sampling. Rules and Guidelines for Data Sampling Data sampling is when you run tests on a subset of the dataset. You might use data sampling when the dataset is large. You can perform data sampling on table pairs and single tables. Consider the following rules and guidelines for data sampling: The sample percentage that you specify may differ from the actual sample percentage during a test run. The sample percentage that you specify represents the chance that each data is included in the sample. You cannot use native sampling if the underlying Oracle tables or views from which you are sampling do not have unique primary keys. Instead, use PowerCenter sampling. If you run sampling on a table pair, you select one table to sample. In a master-detail relationship, sample the detail table because it is the bigger table. When you sample the bigger table, the joiner processes a smaller set of data. Table A is the master table and table B is the detail table. PowerCenter Cache You can configure the total cache memory for a table pair. If you enable caching, the PowerCenter Integration Service processes the Lookup, Joiner, Aggregator, and Sorter transformations faster. The PowerCenter Integration Service creates in-memory index and data caches to temporarily store data for each transformation while processing the transformation. If you enable caching and run a test for a table pair, the PowerCenter Integration Service divides the specified cache size among all Aggregator, Joiner, Lookup, and Sorter transformations in the underlying PowerCenter session. The PowerCenter Integration Service allocates cache memory based on the transformation requirements, such as the number of rows processed, number and types of tests, and precision and scale of the table columns. For optimal session performance, configure the cache size so that the PowerCenter Integration Service processes each transformation in memory without paging to disk. Session performance decreases when the Integration Service pages to disk. To determine the required cache size for each transformation, review the cache size specified in the session log after the session completes. Adding Table Pairs You can create a table pair from the file menu or from the shortcut in the menu bar. 1. Select the folder to which you want to add the table pair. 2. Click on the table pair shortcut on the menu bar or click on File > New > Table Pair. The Table Pair Editor window appears. PowerCenter Cache 69

83 3. On the Basic tab, enter the basic properties of the table pair: 4. Click Edit to configure the connection properties for a table. 5. If you have the enterprise license, click the Advanced tab and enter the advanced properties: 6. Click Save to save the table pair. 70 Chapter 8: Table Pairs

84 Editing Table Pairs To edit a table pair, right-click the table pair in the Navigator and select Edit Table Pair. You can also edit a table pair by double-clicking the table pair. When you edit a table pair, the Table Pair Editor dialog box opens. Deleting Table Pairs To delete a table pair, right-click a table pair in the Navigator and select Delete Table Pair. You can also select a table pair and press the Delete key to delete a table pair. When you delete a table pair, Data Validation Option deletes all of the associated tests. Data Validation Option does not delete lookup, join, or SQL views used in the table pair. Overall Test Results When you select a table pair in the navigator, the Tests tab of the details area shows all tests for the corresponding table pair. The Tests tab also shows information about the latest test run for each test. Select a test to view the details about that test in the properties area. The properties area contains multiple views. You can view the test properties in the Properties view, test results in the Results view, and bad records in the Bad Records view. Editing Table Pairs 71

85 C H A P T E R 9 Tests for Table Pairs This chapter includes the following topics: Tests for Table Pairs Overview, 72 Test Properties, 72 Adding Tests, 78 Editing Tests, 78 Deleting Tests, 79 Running Tests, 79 Automatic Test Generation, 79 Bad Records, 82 Troubleshooting Table Pair Tests, 83 Tests for Table Pairs Overview You can run the following types of tests on table pairs: Aggregate Set Value Includes COUNT, COUNT_DISTINCT, COUNT_ROWS, MIN, MAX, AVG, and SUM. Includes AinB, BinA, and AeqB. Includes VALUE and OUTER_VALUE. Note: When you run tests, the target folder must be closed in the Designer and Workflow Manager. If the target folder is open, Data Validation Option cannot write to the folder, and the tests return an error. Test Properties You can apply properties for the table pair test. 72

86 The following table describes the test properties: Property Function Field A/B Operator Threshold Max Bad Records Case Insensitive Trim Trailing Spaces Null=Null Condition A/B Description The test you run such as COUNT, COUNT_DISTINCT, AinB, or VALUE. The field that contains the values you want to compare when you run the test. You must select a field from each table in the table pair. The arithmetic operator that defines how to compare each value in Field A with each value in Field B. The allowable margin of error for an aggregate or value test that uses the approximate operator. You can enter an absolute value or a percentage value. The number of records that can fail comparison for a test to pass. You can enter an absolute value or a percentage value. Ignores case when you run a test that compares string data. Ignores trailing spaces when you run a test that compares string data. Data Validation Option does not remove the leading spaces in the string data. Allows null values in two tables to be considered equal. Allows you to filter records after Data Validation Option joins the tables in the table pair. Enter a valid PowerCenter Boolean expression. Field A/B is Expression Allows you to enter an expression for Field A or Field B. Datatype Precision Scale Comments The datatype for the expression if Field A or Field B is an expression. The precision for the expression if Field A or Field B is an expression. The scale for the expression if Field A or Field B is an expression. Information about a test. Data Validation Option displays the comments when you view test properties in the properties area. Tests The following table describes the table pair tests: Test COUNT COUNT_DISTINCT COUNT_ROWS Description Compares the number of non-null values for each of the selected fields. This test works with any datatype. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. Compares the distinct number of non-null values for each of the selected fields. This test works with any datatype except binary. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. Compares the total number of values for each of the selected fields. This test counts nulls, unlike the COUNT and COUNT_DISTINCT tests. This test works with any datatype. Test Properties 73

87 Test MIN MAX AVG SUM SET_AinB SET_BinA SET_AeqB SET_ANotInB VALUE OUTER_VALUE Description Compares the minimum value for each of the selected fields. This test works with any datatype except binary. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. Compares the maximum value for each of the selected fields. This test works with any datatype except binary. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. Compares the average value for each of the selected fields. This test can only be used with numeric datatypes. Compares the sum of the values for each of the selected fields. This test can only be used with numeric datatypes. Determines whether the entire set of values for Field A exist in the set of values for Field B. This test works with any datatype except binary/other. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. You can use this test to confirm that all values in a field exist in a lookup table. This test examines all values for a column instead of making a row-by-row comparison. Determines whether the entire set of values for Field B exist in the set of values for Field A. Determines whether the entire set of values for the field selected from Table B exist in the set of values for the field selected from Table A. This test works with any datatype except binary/other. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. You can use this test to confirm that all values in a field exist in a lookup table. This test examines all values for a column instead of making a row-by-row comparison. Determines whether the set of values for the selected fields are exactly the same when compared. This test works with any datatype except binary. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. You can use this test to confirm that all values in a field exist in a lookup table. This test examines all values for a column instead of making a row-by-row comparison. Determines whether there are any common values between the selected fields. If there are common values, the test returns an error. If there are no common values, the test succeeds. For joined table pairs, this test compares the values for the fields in each table, row-by-row, and determines whether they are the same. If there are any rows that exist in one table but not the other, the rows are disregarded which implies an inner join between the tables. If the fields are both null and the Null=Null option is disabled, this pair of records fails the test. This test works with any datatype except binary. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-todatetime. For joined table pairs, this test compares the values for the fields in each table, row-by-row, and determines whether they are the same. If there are any rows that exist in one table but not the other, they are listed as not meeting the test rules which implies an outer join between the tables. For the test to pass, the number of rows for the tables, as well as the values for each set of fields must be equal. If the fields are both null and the Null=Null option is disabled, this set of records fails the test. This test works with any datatype except binary. The fields you compare must be of the same general datatype, for example, numeric- to-numeric or datetime-to-datetime. Fields A and B To create a test, you must select the fields that contain the values you want to compare from each table in the table pair. The fields available in each table appear in Field A and Field B. Select a field from each table. 74 Chapter 9: Tests for Table Pairs

88 Conditions A and B You can filter the values for each field in a VALUE or OUTER_VALUE test to exclude rows that do not satisfy the test condition. For example, you want to exclude telephone extension numbers that contain fewer than three characters. Use the following VALUE test: Table A.EXT = Table B.EXT, Condition A = LENGTH(EXT)<3, Condition B = LENGTH(EXT)<3 The filter condition you enter for a test differs from the WHERE clause you enter for a table pair. Data Validation Option applies the WHERE clause to all tests in the table pair, before it joins the tables. It applies the test filter condition after it joins the tables. You might want to use a test filter condition instead of a filter condition in the WHERE clause when the filter condition does not remove a large percentage of rows. This can improve performance if you run one test on the table pair. Data Validation Option does not check the condition syntax. Any valid PowerCenter expression, including expressions that use PowerCenter functions, is allowed. If the PowerCenter syntax is not valid, a mapping installation error occurs when you run the test. To enter a filter condition for either field, enter the filter condition in the Condition A or Condition B field. Because the PowerCenter Integration Service processes the filter condition, it must use valid PowerCenter syntax. Enter the field name in the filter condition, for example, Emp_ID > 0. Do not include the WHERE keyword. Operator The operator defines how to compare the test result for Field A with the test result for Field B. Enter an operator for aggregate and value tests. The following table describes the operators available in the Operator field: Operator Definition Description = Equals Implies that the test result for Field A is the same as the test result for Field B. F or example, SUM(Field A) is the same as SUM(Field B). <> Does not equal Implies that the test result for Field A is not the same as the test result for Field B. < Is less than Implies that the test result for Field A is less than the test result for Field B. <= Is less than or equal to Implies that the test result for Field A is less than or equal to the test result for Field B. > Is greater than Implies that the test result for Field A is greater than the test result for Field B. >= Is greater than or equal to Implies that the test result for Field A is greater than or equal to the test result for Field B. Is approximately the same as Implies that the test result for Field A is approximately the same as the test result for Field B. An approximate test must have a threshold value. You can use this operator with numeric datatypes only. Note: Data Validation Option compares string fields using an ASCII table. Test Properties 75

89 RELATED TOPICS: BIRT Report Examples on page 143 Threshold A threshold is a numeric value that defines an acceptable margin of error for a test. You can enter a threshold for aggregate tests and for value tests with numeric datatypes. An aggregate test fails if the number of non-matching records exceed the threshold value. For example, you run a COUNT test that uses the operator and set the threshold to 10. The test passes when the results are within 10 records of each other. In a value test, the threshold defines the numeric margin of error used when comparing two values. For example, you run a VALUE test that uses the = operator. The test compares a REAL field with a value of to an INTEGER field with a value of 101. The test passes when the threshold value is at least You can enter an absolute value or a percentage value as the threshold. To enter a percentage value as the threshold, suffix the number with a percentage (%) sign. For example, 0.1%. You must configure the threshold if the test uses the approximate operator. Max Bad Records Data Validation Option lists records that do not compare successfully as bad records. You can configure a tolerance value for the acceptable number of bad records. By default, for a set or value test to pass, all records must compare successfully. You can configure an acceptable value for the maximum number of bad records. The test passes if the number of bad records does not exceed the max bad records value. You can enter an absolute value or a percentage value for the max bad records. To enter a percentage value as the max bad records, suffix the number with a percentage (%) sign. For example, 0.1%. Value and set tests display bad records on the Results tab. Case Insensitive String comparison in PowerCenter is case-sensitive. If you want the PowerCenter Integration Service to ignore case when you run a test that compares strings, enable the Case Insensitive option. This option is disabled by default. Trim Trailing Spaces By default, string comparison fails if two strings are identical except one string contains extra spaces. For example, one field value in a test is 'Data Validation' and the other field value is 'Data Validation ' (with three blank spaces after the last character). If you do not trim trailing spaces, the test produces a bad record. If you trim trailing spaces, the comparison passes because the extra spaces are ignored. You might want to trim trailing spaces when the CHAR datatype, which pads a value entered with spaces to the right out to the length of the field, is used. A field of CHAR(20) compared to CHAR(30) fails, even if both fields have the same value, unless you trim the trailing spaces. Enable the Trim Trailing Spaces option if there are spaces after the value entered in a field that should be ignored in the comparison. This option is disabled by default. 76 Chapter 9: Tests for Table Pairs

90 Null = Null If null values in Field A and Field B should be considered equal, enable the Null = Null option. For example, a current employee in a table that contains employee information has a null termination date. If two records with null termination dates were compared by a database, they would not be considered equal because SQL does not consider a null value in one field to be equal to a null value in another field. Because business users often consider null values to be equal, the Null = Null option is enabled by default. Comments You can enter information about a test in the Comments field. Data Validation Option displays comments in the Properties window when you select the test in the Navigator or the Tests tab. Expression Definitions You can use an expression to calculate a value in a join field of a table pair. The expression must use valid PowerCenter expression syntax. The following image shows a sample expression for a join field: The Type value represents the datatype of the expression after the calculation. The datatypes are PowerCenter datatypes. The precision and scale must match the precision and scale used in PowerCenter for the specified datatype. The scale for any string or text datatype is zero. The precision and scale for a datetime datatype is 23 and 3, respectively. If you do not select the correct datatype, precision, or scale, the test might produce an error. Note: Data Validation Option does not support the following PowerCenter functions in expressions: User-defined Custom Lookup Variable Expression Tips Testing often requires the use of different expressions. PowerCenter functions are described at the end of this guide. The following examples demonstrate how to use expressions for data validation. Test Properties 77

91 Concatenation, RTRIM, and SUBSTR Often data transformation involves concatenation or the use of substring functions. The following example tests the result of concatenation transformation: Expression A: UPPER(first_name ' ' last_name) Field B: full_name IF Statements The IF function is arguably the most popular testing function. The syntax for the IF function is as follows: IF(condition, if_true_part, if_false_part) The following example shows the IF function used in testing: Table A sales_usa sales_intl Table B region (either 'USA' or 'INTL') sales The aggregate validation can be accomplished by two tests: Test 1: SUM Field A: sales_usa Expression B: IF(region='USA', sales, 0) Test 2: SUM Field A: sales_intl Expression B: IF(region='INTL', sales, 0) Adding Tests You can add tests to table pairs one at a time or you can generate tests in batches. You can add any test manually. To add a test to a table pair, right-click the name of the table pair in the Navigator or on the Table Pairs tab, or rightclick in the Tests tab, and select Add Test. The Table Pair Test Editor dialog box opens. You can generate value tests in a batch for table pairs that have tables with matching field names and datatype families. You can also generate value tests in a batch for tables or files within two target folders that have matching table or file names, field names, and datatype families. Editing Tests To edit a test, right-click the test in the Navigator or on the Tests tab, and select Edit Test. You can also double-click the test name. The Table Pair Test Editor dialog box opens. 78 Chapter 9: Tests for Table Pairs

92 Deleting Tests To delete a test, right-click the test in the Navigator or on the Tests tab, and select Delete Test. You can also use the Delete key. Running Tests You can run tests from the Data Validation Option client or the command line. Use one of the following methods to run tests: Select one or more table pairs and click Run Tests. Right-click a folder in the Navigator and select Run Folder Tests. Right-click a test on the Tests tab and select Run Selected Tests. Data Validation Option runs all tests for a table pair together. You can run tests individually if only one test is set up for the table pair. If you select an individual test, Data Validation Option runs all tests for the table pair. After you run a test, you can view the results on the Results tab. Data Validation Option uses the following logic to determine whether a test passes or fails: An aggregate test is calculated as A <operator> B. If this relationship is true, the test passes. If the operator chosen is approximate, then the test is calculated as ABS(A-B) <= Threshold. A value or set test must produce fewer or an equal number of records that do not match compared to the threshold value. If there is no threshold value and there are no records that do not match, the test passes. When you run tests, the target repository folders must be closed in the Designer or Workflow Manager. If the target folders are open, Data Validation Option cannot write to the folders, and the tests fail. Automatic Test Generation You can compare two repository folders and generate all table pairs, value tests, and count tests between the tables in the two folders. You can also automatically generate tests for existing table pairs. You can generate value and count tests to compare tables during a PowerCenter upgrade or migration from development to production. You can generate tests based on the column name or column position in the tables. Data Validation Option generates the following tests: OUTER_VALUE test for a set of fields used for the join if the field names and field datatypes match. The OUTER_VALUE test reports any difference when join field values exist. VALUE test for fields in which the field names and datatypes match. The VALUE test reports any difference in actual values for each set of fields. COUNT_ROWS test for a set of fields used for the join if the field names and field datatypes match. A COUNT_ROWS test reports if there are difference in the number of rows between the tables. Data Validation Option does not generate tests for fields when the field names or datatypes do not match. It also does not generate tests for binary fields. Deleting Tests 79

93 Generating Table Pairs and Tests Use the Compare Tables dialog box to generate table pairs between tables in any two folders and generate tests associated with the table pairs. 1. In Data Validation Option, click Action > Compare Tables. The Compare Tables dialog box appears. 2. Select the repositories that contain the tables that you want to compare. 3. Select the folders that contain the tables that you want to compare. 4. Select Sources or Targets from the sub-folders. 5. Select the database connection, for each folder. If there are tables in the folder that require a different database connection, modify the database connection in table pairs and tests after auto-generation is complete. 6. If the database is IBM DB2, enter the database owner name. 7. If the data source is SAP, click Select Source Parameters and configure the SAP source parameters. 8. If the folders contain flat files, enter the path that contains the source files and target files in the Source Dir field. 9. Select whether to compare columns by name or position. 10. If you select compare columns by position, select the column to which you want to skip in the table. 11. Select the folder to store the table pairs and tests. 12. Choose to generate count tests or count and value tests. 13. Select Sort in DB to sort the tables before Data Validation Option generates tests. 14. Choose whether to trim the trailing spaces in the tests. 15. To generate value tests for flat files, Salesforce tables, or tables without primary keys, specify the name and location of the text file that contains information of primary keys for the flat files or tables. The file must contain the name of the flat file or table and the primary key separated by comma. Each entry must be in a new line. If a table has more than one key, each key must be in a new entry. For example: flatfile_dictionary_10rows,field1 flatfile_dictionary_10rows,field2 flatfile_dictionary_10rows_str,field1 16. Select whether you want to skip generation of all the tests or generate count tests if a primary key does not exist for a table. Data Validation Option generates table pairs even if you skip generation of all the tests. 17. If you have the enterprise license, you can select whether to save all the bad records. 18. If you choose to save all the bad records, select whether to save the bad records in a flat file or the Data Validation Option schema. 19. Click Create. Data Validation Option displays the summary of table pairs and tests to create and to skip. 20. Click Ok. Data Validation Option generates all the possible table pairs and tests. 80 Chapter 9: Tests for Table Pairs

94 Generating Tests for Table Pairs Use the Generate Tests option to generate the associated count tests and value tests of an existing table pair. 1. In the Navigator, select the table pair. 2. Right-click on the table pair and select Generate Value Test. 3. Select whether to compare columns by name or position. 4. If you select compare columns by position, select the column to which you want to skip in the table. 5. Click Yes. Data Validation Option generates the tests for the table pair. Compare Columns by Position You can generate tests that compares columns of the tables in a table pair by position instead of name. The following examples describe the various scenarios when you compare columns by position: Table Pair with an SQL View and a Table Suppose you have an SQL view, sample_view and a table, sample1 as Table A and Table B of a table pair. The following table lists the columns in sample_view and sample1: sample_view SampleA.column1 SampleA.column2 sample1 column1 column2 SampleB.column1 SampleB.column2 You need to compare SampleB.column1 and SampleB.column2 with column1 and column2. Select compare columns by position. Select Table A and enter 2 as the offset. Table Pair with tables that have the same number of columns and different names Suppose you have SampleA as Table A and SampleB as Table B of a table pair. The following table lists the columns in sample_view and sample1: SampleA column1 column2 column3 SampleB col1 col2 col3 You need to compare column1, column2, and column3 in SampleA with col1, col2, and col3 in SampleB. Select compare columns by position. Enter 0 as the offset. You can select Table A or Table B. Table Pair with tables that have different number of columns Suppose you have SampleA as Table A and SampleB as Table B of a table pair. Automatic Test Generation 81

95 The following table lists the columns in sample_view and sample1: SampleA column1 column2 column3 SampleB col1 col2 col3 column1 column2 column3 You need to compare column1, column2, and column3 in SampleA with column1, column2, and column3 in SampleB. Select compare columns by position. Select Table B and enter 3 as the offset. Bad Records Bad records are the records that fail a value or a set test. When you select a test on the Tests tab, the bad records appear in the Bad Records view. The columns that appear on the Bad Records view differ based on the test. Aggregate Tests Aggregate tests do not display bad records. The Results view displays the test result value from Field A, the test result value from Field B, and the comparison operator. Value Tests Value tests display the following columns for each bad record: The key for Table A The field or expression from Table A being compared The key for Table B The field or expression from Table B being compared Set Tests Set tests display the following columns for each bad record: The result from Table A The result from Table B Note: If you compare two fields where one field has a value and the other is empty, Data Validation option considers the record as a bad record. If the repository is on Oracle, the database stores the empty field as NULL. A bad record with NULL value in an Oracle repository can be either NULL or an empty field. 82 Chapter 9: Tests for Table Pairs

96 Troubleshooting Table Pair Tests The table pair test fails. If the target folders in PowerCenter Designer or Workflow Manager are open, Data Validation Option cannot write to the folders and the tests fail. Before you run tests, verify that the target repository folders are closed in the the Designer or Workflow Manager. Data Validation Option creates a bad record during a table pair test when one of the values is empty. If you compare two fields where one field has a value and the other is empty, Data Validation option considers the record as a bad record. If the repository is on Oracle, the database stores an empty field as NULL. A bad record with NULL value in an Oracle repository can be either NULL or an empty field. The join failed for a table pair that has a file list connection. If you specify a file list connection and the keys in the join condition are not unique among all files in the file list, the join fails. Choose keys that uniquely identify records among all files in the file list. Troubleshooting Table Pair Tests 83

97 C H A P T E R 1 0 Single-Table Constraints This chapter includes the following topics: Single-Table Constraints Overview, 84 Basic Properties, 85 Connection Properties, 86 Advanced Properties, 87 Parameterization, 88 Bad Records Configuration, 89 Data Sampling, 90 PowerCenter Cache, 91 Adding a Single Table, 92 Editing Single Tables, 92 Deleting Single Tables, 92 Viewing Overall Test Results, 93 Single-Table Constraints Overview Use a single-table constraint to run tests on a single table. Single-table constraints define valid data within a table. You can enforce valid values, aggregates, formats, and uniqueness. For example, you might want to verify that no annual salary in an employee table is less than $10,000. Errors in complex logic often manifest themselves in very simple ways, such as NULL values in the target. Therefore, setting aggregate, value, NOT_NULL, UNIQUE, and FORMAT constraints on a target table is a critical part of any testing plan. To run single-table constraints, you must create a single table. You can select a relational table, flat file, lookup view, SQL view, or join view as a single table. If a single-table constraint fails to run, right-click the single table to retrieve the corresponding session log. View the session log to view errors that may have occurred while running the underlying PowerCenter session. 84

98 Basic Properties You can view or configure the basic properties for a single-table constraint. The properties vary based on the types of objects you select for the single-table constraint. The following list describes the basic properties for a table pair: Description Table Conn A Single-table constraint description. By default, Data Validation Option uses the table name. The table name. Select Browse to select a table. You can search for a data source by name or path. You can search for a lookup view, SQL view, and join view by name. Connection details for the table. Select Edit to edit the connection properties. Where Clause A The WHERE clause filters the records that the PowerCenter Integration Service reads from the data source. If you run the WHERE clause on PowerCenter, enter a valid PowerCenter Boolean expression. If you run the WHERE clause on a relational source or an application source, enter the WHERE clause in the supported data source format. If you run the WHERE clause within the data source, the data source applies the WHERE clause before PowerCenter Integration Service loads data. Execute where clause A/B Select if you want the database to execute the WHERE clause. The database executes the WHERE clause before the PowerCenter Integration Service reads the records. If you select the Where Clause, Sorting and Aggregation in DB optimization level, the database also sorts and aggregates the data. Optimization Level Controls which test logic that Data Validation Option converts to a PowerCenter mapping and which test logic it pushes to the database. You can select one of the following options: Default. Data Validation Option converts all test logic to a PowerCenter mapping and applies sorting to the data source. WHERE clause, Sorting, and Aggregation in DB. Data Validation Option pushes the WHERE clause, sorting logic for joins, and all aggregate tests to the database. Data Validation Option converts all other test logic to a PowerCenter mapping. Already Sorted Input. PowerCenter mapping does not sort the input. If the data source is not sorted, tests might fail. Primary Key Primary key column or columns for the table. External ID Identifier for the single table that you can use when you run Data Validation Option tests at the command line. Basic Properties 85

99 Connection Properties Choose the connection properties based on the data source type. You must select a connection for all the data sources except for flat files. For flat files, you must provide the source directory and the file name. Connections are PowerCenter connection objects created in the Workflow Manager. Relational Connection Properties Choose the relational connection properties for Microsoft SQL Server, Oracle, IBM DB2, Netezza, and PowerExchange for DB2 data sources. Configure the following properties when you select a relational data source: Property Connection Override Owner Name Description PowerCenter connection object to connect to the relational data source. Override the database name and schema of the source. For example, a Microsoft SQL Server table is identified by <database>.<schema>.<table>. To override the database and the schema, enter <new database name>.<to change the schema, enter <new schema name> in the text box. You cannot change only the database name. SAS and Salesforce Connection Properties You can use a SAS or Salesforce data source in Data Validation Option. Select the PowerCenter connection object when you select an SAS or Salesforce data source. Note: You cannot override the owner name for SAS and Salesforce data sources. SAP Connection Properties You must configure the SAP authentication information to use SAP data sources. Configure the following properties when you select an SAP data source: Property Connection SAP User Name SAP Password SAP Client SAP Language SAP Connect String Description PowerCenter connection object to connect to the SAP data source. SAP source system connection user name. Must be a user for which you have created a source system connection. Password for the user name. SAP client number. Language you want for the mapping. Must be compatible with the PowerCenter Client code page. Data Validation Option does not authenticate the value. Ensure that you enter the correct value so that the tests run successfully. Type A or Type B DEST entry in saprfc.ini. 86 Chapter 10: Single-Table Constraints

100 SAP Data Sources in Data Validation Option You cannot override the owner name for an SAP data source. Data Validation Option uses the stream mode for installation of the ABAP programs and cannot use the FTP mode. SAP data sources must not contain the backslash (/) character in the field names. Flat File Connection Properties You can use the flat file data sources in the PowerCenter repository. Configure the following properties when you select a flat file data source: Property Source Dir Source File File Type Description Directory that contains the flat file. The path is relative to the machine that hosts Informatica Services. File name with the file name extension. Type of flat file. Select one of the following values: - File Contains Source Data. Select if the flat file contains data. - File Contains a List of Files. Select if the flat file contains a file list. When you select a file list, Data Validation Option processes all files in the list. Advanced Properties You can configure bad records, a parameter file, sampling, and caching for a single-table constraint. You must have the enterprise license to configure bad records and a parameter file. The properties vary based on the types of objects you select for the single-table constraint. The following list describes the advanced properties for a single-table constraint: Save All Bad Records for Test Execution Saves all bad records for value and unique tests for single-table constraints. Write Bad Records to: The location where Data Validation Option writes bad records. You can select one of the following options: Flat File. Data Validation Option writes bad records to the specified flat file. DVO Schema. Data Validation Option writes bad records to the Data Validation Option repository. File Name The location and name of the flat file to which Data Validation Option writes bad records. Default location is $INFA_HOME/server/infa_shared/BadFiles/<single table name /table pair name>/<file name>. Parameter File Name of the parameter file. Click Add to add a parameter file. Click Delete to delete the selected parameter file. Enable Data Sampling Indicator that determines if data sampling is enabled. Advanced Properties 87

101 Percentage of Rows Percentage of rows that you want to sample. Minimum value is Maximum value is less than 100. Seed Value The starting value used to generate a random number for sampling. You can configure the seed value for the following platforms: PowerCenter. Minimum seed value is 0. Maximum seed value is IBM DB2. Minimum seed value is 1-10³¹. Maximum seed value is 10³¹ - 1. Oracle. Minimum seed value is -2,147,483,648. Maximum seed value is 2,147,483,647. Use Native Sampling The database performs the sampling. If you do not enable this option, PowerCenter performs the sampling. Cache Size Total cache used to run PowerCenter sessions associated with the table pair. By default, the units are bytes. Append KB, MB, or GB to the value to specify other units. For example, you can specify , 1024 KB, or 1 MB. Select Automatic to enable PowerCenter to compute the cache size. Parameterization If you have the enterprise license, you can perform incremental data validation through parameterization. You can use parameters in the WHERE clause in Table Pairs and Single Tables. Before you use a parameter in the WHERE clause, you must enter the name of the parameter file and add the parameters on the Advanced tab of a table pair or single table definition. You must specify the data type, scale, and precision of the parameter. After you add the parameters, you can use them in a WHERE clause to perform incremental validation. You can validate the expression with the parameters that you enter in a table pair or single table. The parameter file must be in a location accessible to the Data Validation Option Client. The parameters in the parameter file must be in the format, $$<parameter name>=value. Ensure that the parameter that you add in a table pair or single table is available in the parameter file. When you run a test, Data Validation Option looks up the value of the parameter from the parameter file and runs the WHERE clause based on that value. If the Informatica Services run on a Windows machine, you can place the parameter files in a folder on the server. Ensure that the Data Validation Option Client can access the folder. In the Data Validation Option Client, enter the parameter file location as a network path to the parameter file in the server in the following format: \\<server machine host name>\<shared folder path>\parameter file name If the Informatica Services run on a UNIX machine, you can place the parameter files in a folder on the server. Install DVOCmd on the server. You can configure parameterization on the Data Validation Option Client and provide the absolute path of the parameter file and run the tests from the server with DVOCmd. Alternatively, you can ensure that the Data Validation Option Client can access folder and enter the parameter file location as a network path to the parameter file in the server. Run the tests from the Data Validation Option Client. 88 Chapter 10: Single-Table Constraints

102 Bad Records Configuration If you have the enterprise license, you can store up to 16,000,000 bad records per test to perform advanced analysis and up to 1000 bad records for reporting. If you have the standard license, you can store up to 1000 bad records for reporting and analysis. For reporting, you can set the maximum number of bad records in the mapping properties in the Preferences dialog box. The default number of bad records is 100. You can set up to 1000 bad records for reporting. For advanced analysis, you can set the maximum number of bad records for detailed analysis and the file delimiter. You set the properties in the Detailed Error Rows Analysis section in the mapping properties. The default number of bad records is You can set up to 16,000,000 bad records. You can store all the bad records from the tests for a table pair or single table. Select Save all bad records for test execution on the Advanced tab when you create or edit a table pair or single table. Data Validation Option stores the following constraint tests for single tables: Value Unique You must select whether you want to store the bad records in a flat file or a table in the Data Validation Option schema. If you store bad records in a flat file, you can optionally enter the name of the file. Data Validation Option appends test information to the name and retains the file extension. Note: If you modify the file delimiter in the preferences file, run the InstallTests command with the forceinstall option for the existing table pairs or single tables that you already ran. You can also edit and save the table pair or single table from the Data Validation Option Client before you run the test. If you modify the bad records value, you need not reinstall the tests. Bad Records in Database Schema Mode If you choose to save detail bad records in the Data Validation Option schema, the bad records are written into the detail tables. The following table describes the tables to which Data Validation Option writes the bad records based on the type of test: Test Type Table Columns Value, Outer Value, Value Constraint ALL_VALUE_RESULT_DETAIL TEST_RUN_ID, TEST_CASE_INDEX, KEY_A, VALUE_A, KEY_B, VALUE_B Unique Constraint ALL_UNIQUE_RESULT_DETAIL TEST_RUN_ID, TEST_CASE_INDEX, VALUE Set ALL_SET_RESULT_DETAIL TEST_RUN_ID, TEST_CASE_INDEX, VALUE_A, VALUE_B Note: You must ensure that the database table has enough table space to hold all the bad records. The test details of all the tests that you ran in Data Validation Option is available in the TEST_CASE table. The test installation details of the tests are available in the TEST_INSTALLATION table. You can obtain the TEST_ID of a test from the TEST_INSTALLATION table. You need TEST_ID of a test to query the complete details of a test from the TEST_CASE table. You can get the Test Case Index from the Properties tab and the Test Run ID from the results summary of the test in the detail area. You can get the Table Pair ID/Table ID from the Table Pair or Single Table properties. Bad Records Configuration 89

103 Foe example, you ran a table pair with a value test and outer value test. The following SQL query is a sample query to retrieve the information of bad records of a test with Test Case Index as 5. select ALL_VALUE_RESULT_DETAIL.*,TEST_CASE.* from ALL_VALUE_RESULT_DETAIL, TEST_RUN, TEST_INSTALLATION, TEST_CASE where ALL_VALUE_RESULT_DETAIL.TEST_RUN_ID=TEST_RUN.TEST_RUN_ID and TEST_RUN.TEST_INSTALLATION_ID=TEST_INSTALLATION.TEST_INSTALLATION_ID and TEST_INSTALLATION.TABLE_PAIR_ID=TEST_CASE.TEST_ID and ALL_VALUE_RESULT_DETAIL.TEST_CASE_INDEX=TEST_CASE.TEST_CASE_INDEX and ALL_VALUE_RESULT_DETAIL.TEST_RUN_ID=220 Bad Records in Flat File If you configure to store bad records in flat file, Data Validation Option creates the flat file in the machine that runs Informatica services. The flat files that Data Validation Option generates after running the tests are stored in the following folder:<powercenter installation directory>\server\infa_shared\tgtfiles You can edit the folder name in the Administration tool. Modify the $PMTarget variable in the PowerCenter Integration Service. Data Validation Option generates a folder for each of the single table. The name of the single table is in the following format: SingleTableName_TestRunID Data Validation Option creates flat files for each test inside the folder. The name of the flat file is in the following format: <user defined file name>_ TestCaseType_TestCaseColumn A_TestCaseIndex.<user defined file extension> Suppose you enter the file name as BAD_ROWS.txt when you configure the single table and you run an value constraint test on field SSID. The test-case index is 1. The bad records file after you run the test is of the format, BAD_ROWS_VALUE_CONSTRAINT_SSID_1.txt. Data Validation Option supports all the file delimiters that PowerCenter supports. When you enter non-printable characters as delimiters, you should enter the corresponding delimiter code in PowerCenter. When you import these files in PowerCenter, you have to manually create the different data fields since the code appears in the place of delimiter in the bad records file. Caution: Data Validation Option uses comma as a delimiter if there a multiple primary keys. Do not use comma as a file delimiter. When you run the tests for the single table, Data Validation Option stores the details of the bad records along with the following format: single table Name Table A Name Table A Connection Test Definition Test Run Time Test Run By User Key A[],Result A[] Data Sampling You can use data sampling to run tests on a subset of a dataset. You might use data sampling when the data set is large. You can perform data sampling on table pairs and single tables. 90 Chapter 10: Single-Table Constraints

104 When you run a test on a sample data set, Data Validation Option runs the test on a percentage of the data set. The sample percentage that you specify represents the chance that each data is included in the sample. You can use a seed value to repeat the same sample data set in a test. Data Validation uses the seed value as the starting value to generate a random number. If you do not enter a seed value, Data Validation Option generates a random seed value for each test run. You might use a seed value to replicate a Data Validation Option test. By default, PowerCenter performs the sampling. If you sample data from IBM DB2, Microsoft SQL Server, Oracle, or Teradata, you can perform native sampling in the database. Push sampling to the database to increase performance. If you add the WHERE clause and enable sampling, the order of operations depend on where you perform sampling and execute the WHERE clause. Generally, PowerCenter and the database performs sampling before executing the WHERE clause. However, when you configure the database to execute the WHERE clause and PowerCenter to perform sampling, the database executes the WHERE clause before PowerCenter performs the sampling. Rules and Guidelines for Data Sampling Data sampling is when you run tests on a subset of the dataset. You might use data sampling when the dataset is large. You can perform data sampling on table pairs and single tables. Consider the following rules and guidelines for data sampling: The sample percentage that you specify may differ from the actual sample percentage during a test run. The sample percentage that you specify represents the chance that each data is included in the sample. You cannot use native sampling if the underlying Oracle tables or views from which you are sampling do not have unique primary keys. Instead, use PowerCenter sampling. If you run sampling on a table pair, you select one table to sample. In a master-detail relationship, sample the detail table because it is the bigger table. When you sample the bigger table, the joiner processes a smaller set of data. Table A is the master table and table B is the detail table. PowerCenter Cache You can configure the total cache memory for a single-table constraint. If you enable caching, the PowerCenter Integration Service processes the Lookup, Joiner, Aggregator, and Sorter transformations faster. The PowerCenter Integration Service creates in-memory index and data caches to temporarily store data for each transformation while processing the transformation. If you enable caching and run a test for a single-table constraint, the PowerCenter Integration Service divides the specified cache size among all Aggregator, Joiner, Lookup, and Sorter transformations in the underlying PowerCenter session. The PowerCenter Integration Service allocates cache memory based on the transformation requirements, such as the number of rows processed, number and types of tests, and precision and scale of the table columns. For optimal session performance, configure the cache size so that the PowerCenter Integration Service processes each transformation in memory without paging to disk. Session performance decreases when the Integration Service pages to disk. To determine the required cache size for each transformation, review the cache size specified in the session log after the session completes. PowerCenter Cache 91

105 Adding a Single Table You can create a single table from the file menu or from the shortcut in the menu bar. 1. Select the folder to which you want to add the single table. 2. Click on the single table shortcut on the menu bar or click on File > New > Single Table. The Single Table Editor window appears. 3. Browse and select the data source that you want to use the single table. You can search for a data source by name or path. You can search for lookup views, sql views, and join views only with their names. 4. Click Edit and configure the connection properties for the table. 5. Enter the WHERE clause you want to execute on the data source. 6. Enter the description for the single table. 7. Enter the external id for the single table. You can use the external id to execute the single table from the command line. 8. If your data source is relational, you can choose whether to execute the WHERE clause within the data source. If you choose to execute the WHERE clause within the database, PowerCenter Integration Service passes the WHERE clause to the database for execution before data loading. 9. Select the database optimization level. The following options are available: Default. Data Validation Option creates a PowerCenter mapping based on the test logic and applies sorting on the data source. WHERE clause, sorting, and aggregation in DB. Data Validation Option pushes the WHERE clause and sorting to the database. Applicable for relational data sources. Already sorted input. Data Validation Option creates a PowerCenter mapping based on the test logic. If the data source is not sorted, the tests may fail. 10. Select the primary key for the table in the Key Column pane. Editing Single Tables To edit a single table, right-click it in the Navigator or on the Single Tables tab, and select Edit Single Table. You can also edit a single table by double-clicking it on the Single Tables tab. When you edit a single table, the Single Table Editor dialog box opens Deleting Single Tables To delete a single table, right-click it in the Navigator or Single Tables tab, and select Delete Single Table. You can also delete a single table by selecting it and pressing the Delete key. When you delete a single table, Data Validation Option deletes all of its tests. Data Validation Option does not delete lookup or SQL views used in the single table. 92 Chapter 10: Single-Table Constraints

106 Viewing Overall Test Results When you select a single-table constraint in the navigator, the Tests tab of the details area shows all tests for the selected single-table constraint. The Tests tab also shows information about the latest test run for each test. Select a test to view the details about that test in the properties area. The properties area contains multiple views. You can view the test properties in the Properties view, test results in the Results view, and bad records in the Bad Records view. Viewing Overall Test Results 93

107 C H A P T E R 1 1 Tests for Single-Table Constraints This chapter includes the following topics: Tests for Single-Table Constraints Overview, 94 Test Properties, 94 Adding Tests, 98 Editing Tests, 98 Deleting Tests, 99 Running Tests, 99 Bad Records, 99 Tests for Single-Table Constraints Overview Single-table constraints are tests are based on a single table. Data Validation Option allows you to run an aggregate test or a VALUE test on single tables. Note that there are no set tests nor an OUTER_VALUE test for single tables. However, there are some additional tests available for single tables that you cannot create for a table pair. Most single-table constraints allow you to enter a constraint value for the test. The constraint value defines the value or values to which you want to compare the values in a field. For example, you might want to verify that a SALARY field contains values greater than $10,000. Enter the minimum salary as the constraint value. Note: When you run tests, the target folder must be closed in the Designer and Workflow Manager. If the target folder is open, Data Validation Option cannot write to the folder, and the tests return an error. Test Properties When you select a test in the Navigator or on the Tests tab, the properties for that test appear in the Properties area. Most properties come from the values you enter in the Single Table Test Editor dialog box. Other properties apply to the most recent test run. Edit test properties in the Single Table Test Editor dialog box when you add or edit a test. 94

108 The following table describes the test properties: Property Function Field Operator Constraint Value Threshold Max Bad Records Case Insensitive Trim Trailing Spaces Condition Field is Expression Datatype Precision Scale Comments Description The test you run such as COUNT, COUNT_DISTINCT, VALUE, or NOT_NULL. The field that contains the values you want to test. The operator that defines how to compare each value in the field with the constraint value. The value or values you want to compare the field values to. The allowable margin of error for an aggregate or value test that uses the approximate operator. You can enter an absolute value or a percentage value. The number of records that can fail comparison for a test to pass. You can enter an absolute value or a percentage value. Ignores case when you run a test on string data. Ignores trailing spaces when you run a test that on string data. Data Validation Option does not remove the leading spaces in the string data. Filter condition for the test. Enter a valid PowerCenter Boolean expression. Allows you to enter an expression for the field. The datatype for the expression if the field is an expression. The precision for the expression if the field is an expression. The scale for the expression if the field is an expression. Information about a test. Data Validation Option displays the comments when you view test properties in the Properties area. Tests The following table describes the single table tests: Test COUNT COUNT_DISTINCT COUNT_ROWS MIN Description Compares the number of non-null values for the selected field to the constraint value. This test works with any datatype. Compares the distinct number of non-null values for the selected field to the constraint value. This test works with any datatype except binary. Compares the total number of values for the selected field to the constraint value. This test counts nulls, unlike the COUNT and COUNT_DISTINCT tests. This test works with any datatype. Compares the minimum value for the selected field to the constraint value. This test works with any datatype except binary. Test Properties 95

109 Test MAX AVG SUM VALUE FORMAT UNIQUE NOT_NULL NOT_BLANK Description Compares the maximum value for the selected field to the constraint value. This test works with any datatype except binary. Compares the average value for the selected field to the constraint value. This test can only be used with numeric datatypes. Compares the sum of the values for the selected field to the constraint value. This test can only be used with numeric datatypes. Examines the values for the field, row by row, and compares them to the constraint value. This test works with any datatype except binary. Determines whether the values in the field match the pattern in the constraint value. The PowerCenter Integration Service uses the REG_MATCH function for this test. This test cannot be used with binary datatypes. Confirms that the value in the field is unique. This test does not use a constraint value. This test cannot be used with binary datatypes. Confirms that the value in the field is not null. This test does not use a constraint value. This test cannot be used with binary datatypes. If the value in the field is a string value, this test confirms that the value in the field is not null or an empty string. If the value in the field is a numeric value, this test confirms that the value in the field is not null or zero. This test does not use a constraint value. This test cannot be used with datetime or binary datatypes. Field To create a single-table constraint, you must select the field that contains the values you want to test. The fields available in the single table appear in the Field drop-down list. Select a field from the list. Condition You can filter the values for the test field in a VALUE, FORMAT, NOT_NULL, or NOT_BLANK test. Data Validation Option does not test records that do not satisfy the filter condition. For example, you want to test rows in an ORDERS table only if the store ID number is not Enter STORE_ID <> 1036 in the Condition field. Data Validation Option does not check the condition syntax. Any valid PowerCenter expression, including expressions that use PowerCenter functions, is allowed. If the PowerCenter syntax is not valid, a mapping installation error occurs when you run the test. Enter the filter condition in the Condition field. Because the PowerCenter Integration Service processes the filter condition, it must use valid PowerCenter syntax. Do not include the WHERE keyword. Operator The operator defines how to compare the test result for the field with the constraint value. Enter an operator for aggregate, VALUE, and FORMAT tests. 96 Chapter 11: Tests for Single-Table Constraints

110 The following table describes the operators available in the Operator field: Operator Definition Description = Equals Implies that the test result for the field is the same as the constraint value. For example, SUM(field) is the same as the constraint value. <> Does not equal Implies that the test result for the field is not the same as the constraint value. < Is less than Implies that the test result for the field is less than the constraint value. <= Is less than or equal to Implies that the test result for the field is less than or equal to the constraint value. > Is greater than Implies that the test result for the field is greater than the constraint value. >= Is greater than or equal to Implies that the test result for the field is greater than or equal to the constraint value. ~ Is approximately the same as Implies that the test result for the field is approximately the same as the constraint value. The approximate operator requires a threshold value. It only applies to numeric datatypes Between Is between two values entered Implies that the test result for the field is between the two constants entered as the constraint value. This operator is generally used for numeric or datetime datatypes. Not Between Is not between two values entered Implies that the test result for the field is not between the two constants entered as the constraint value. This operator is generally used for numeric or datetime datatypes. In Not In Is included in a list of values entered Is not included in a list of values entered Implies that the test result for the field is in the list of constants entered as the constraint value. Implies that the test result for the field is not in the list of constants entered as the constraint value. Note: Data Validation Option compares string fields using an ASCII table. RELATED TOPICS: BIRT Report Examples on page 143 Constraint Value The constraint value represents a constant value to which you want to compare the field values. For example, you might want to verify that all values in the ORDER_DATE field fall between January 1, 2010 and December 31, Or, you might want to verify that the minimum ORDER_ID number is greater than The constraint value must be a string, numeric, or datetime constant. The datatype of the constraint value depends on the test. Test Properties 97

111 The following table lists the constraint value datatype allowed for each test: Test COUNT, COUNT_DISTINCT, COUNT_ROWS MIN, MAX, SUM, VALUE FORMAT AVG UNIQUE, NOT_NULL, NOT_BLANK Datatype Integer Same as the Field datatype. String Double These tests do not use a constraint value. Enter a constraint value in the Constraint Value field. Enter a constant or list of constants separated by commas. The number of constants you enter as the constraint value depends on the operator you use: Arithmetic operator such as =, <>, or ~ Enter a single constant. Between or Not Between operator Enter two constants separated by a comma. In or Not In operator Enter multiple constants separated by commas. Enclose each string value, datetime value, or format pattern within single quotes. Datetime values must match the PowerCenter standard datetime format of MM/DD/YYYY HH24:MI:SS. Remaining Controls on Test Editor The remaining controls on the Single Table Test Editor are used in the same manner that they are used for table pairs. Adding Tests To add a test to a single table, right-click the name of the table in the Navigator or on the Single Tables tab, or rightclick in the Tests tab, and select Add Constraint Test. The Single Table Test Editor dialog box opens. Editing Tests To edit a test, right-click the test in the Navigator or on the Tests tab, and select Edit Test. You can also double-click the test name. The Single Table Test Editor dialog box opens. 98 Chapter 11: Tests for Single-Table Constraints

112 Deleting Tests To delete a test, right-click the test in the Navigator or on the Tests tab, and select Delete Test. You can also use the Delete key. Running Tests Use one of the following methods to run tests: Select one or more table pairs and click Run Tests. Right-click a folder in the Navigator and select Run Folder Tests. Right-click a test on the Tests tab and select Run Selected Tests. Data Validation Option runs all tests for a single table together. You cannot run tests individually unless only one test is set up for the table. If you select an individual test, Data Validation Option runs all tests for the single table. After you run a test, you can view the results on the Results tab. Data Validation Option uses the following logic to determine whether a test passes or fails: An aggregate test is calculated as value <operator> constraint. If this relationship is true, the test passes. If the operator chosen is approximate, then the test is calculated as ABS(value-constraint) <= Threshold. A VALUE test must produce fewer or an equal number of records that do not match compared to the threshold value. If there is no threshold value and there are no records that do not match, the test passes. A FORMAT test is calculated as value <operator> constraint. If this relationship is true, the test passes. A UNIQUE, NOT_NULL, or NOT_BLANK test passes if the field value is unique, is not null, or is not blank, respectively. For string values, not blank means the string is not null or empty. For numeric values, not blank means the number is not null or 0. When you run tests, the target repository folders must be closed in the Designer or Workflow Manager. If the target folders are open, Data Validation Option cannot write to the folders, and the tests fail. Bad Records When you select a test on the Tests tab, the records that fail a test appear on the Bad Records view. Different columns appear on thebad Records view based on the test. Aggregate Tests Aggregate tests do not display bad records. The Results view displays the test result value. VALUE, FORMAT, NOT_NULL, and NOT_BLANK Tests These tests display the following columns for each bad record: The key or expression for the field The field value UNIQUE Tests UNIQUE tests display the field values that are not unique. Deleting Tests 99

113 C H A P T E R 1 2 SQL Views This chapter includes the following topics: SQL Views Overview, 100 SQL View Properties, 100 Adding SQL Views, 102 Editing SQL Views, 102 Deleting SQL Views, 102 SQL Views Overview SQL views facilitate the use of more complex functionality for single tables and table pairs. An SQL view allows you to use several tables and several calculations in a query to produce a set of fields that you can use as a table in a single table or table pair. This functionality is similar to the SQL override in PowerCenter or a view in a relational database. You can use any valid SQL statement to create an SQL view. SQL View Properties You can view SQL view properties by selecting an SQL view in the Navigator or on the SQL Views tab and viewing the properties. Most properties come from the values entered in the SQL View Editor. Other properties come from the tests set up for and run on the SQL view. Edit SQL view properties in the SQL View Editor dialog box when you add or edit an SQL view. The following table describes the SQL view properties: Property Description Table Definitions Connection Description SQL view description. Tables to create the SQL view. If you identify the table with an alias, enter the alias name with the table name. All tables you use in the SQL view must exist in the same database. PowerCenter connection for the tables. 100

114 Property Column Definition SQL Statement Comment Description The columns that make up the SQL view. Data Validation Option imports all columns from the tables you select. You can create, delete, and rearrange columns. SQL statement you run against the database to retrieve data for the SQL view. Information about an SQL view. Data Validation Option displays the comment when you view the SQL view in the Properties area. Description Enter a description so you can identify the SQL view. Data Validation Option displays the description in the Navigator and on the SQL Views tab. The description can include spaces and symbols. Table Definitions and Connection To provide Data Validation Option with the information it needs to create an SQL view, you must specify the tables that the SQL statement is based on and the corresponding database connection. When you provide the tables and connection information, Data Validation Option can access the metadata that is necessary for the view to function correctly. To add a table, click Add Table. The Choose Data Source dialog box opens. This dialog box displays all of the relational tables available in the repositories. You can sort information in this dialog box by clicking the column headers. You can reduce the number of items to select by typing one or more letters of the table, file, or view name in the Search field. Select a table and click Select. All of the tables you use in an SQL view must exist in the same database. Note: You cannot create an SQL view with self-join. Use a join view to create self-joins. If you identify the table with an alias in the SQL statement you use to create the view, enter the alias name next to the table name. When you finish adding tables, select the PowerCenter connection for the tables from the Connection list. Column Definition After you specify the tables on which the SQL view is based, you must specify the columns that make up the view. The number of columns you define for the view must match the SQL statement. To import the columns from the tables you select, click Populate. Data Validation Option imports all columns in the tables. Delete the columns that you do not want to use. You can rearrange the columns in the view. You can also create a column. To do this, open a column field in the Column Definition list and select Column. Enter the column name in the SQL View Column Definition Editor dialog box. You must also specify the datatype, precision, and scale for the column. The datatype, precision, and scale information must match the transformation datatype that Informatica uses for the column. For datetime, string, and integer datatypes, the scale must be zero. SQL Statement Enter an SQL statement to retrieve data for the SQL view. The statement that you enter runs as a query against the database, so it must use valid database syntax. Also, the columns that you enter in the SELECT statement must match the columns in the Column Definition list in number, position, and datatype. SQL View Properties 101

115 To avoid errors when you run tests, test the SQL statement in the database before you paste it into the SQL Statement field. Data Validation Option does not check the SQL statement syntax. You can call a stored procedure from an SQL view. The connection that you specify in the SQL view for the source must have permission on the stored procedure. Comment You can associate a comment with the view. Data Validation Option displays the comment when you view the SQL view in the Properties area. Adding SQL Views To create an SQL view, right-click SQL Views in the Navigator or right-click in the SQL Views tab, and select Add SQL View. The SQL View Editor dialog box opens. Editing SQL Views To edit an SQL view, right-click the SQL view in the Navigator or on the SQL Views tab, and select Edit SQL View. The SQL View Editor dialog box opens. Deleting SQL Views To delete an SQL view, right-click the SQL view in the Navigator or on the SQL Views tab, and select Delete SQL View. You can also select the SQL view and press the Delete key. When you delete an SQL view, Data Validation Option deletes all table pairs, single tables, and tests that use the SQL view. 102 Chapter 12: SQL Views

116 C H A P T E R 1 3 Lookup Views This chapter includes the following topics: Lookup Views Overview, 103 Lookup View Properties, 104 Adding Lookup Views, 105 Editing Lookup Views, 106 Deleting Lookup Views, 106 Lookup Views Example, 106 Joining Flat Files or Heterogeneous Tables using a Lookup View, 107 Lookup Views Overview Data Validation Option lookup views allow you to test the validity of the lookup logic in your transformation layer. Lookup views allow you to validate the process of looking up a primary key value in a lookup or reference table based on a text value from a source, and then storing the lookup table primary key in the target fact table. For example, a product name in the source system might be in a dimension that serves as the lookup table. The data transformation process involves looking up the product name and placing the primary key from the lookup table in the target fact table as a foreign key. You must validate the product name in the source table against the foreign key in the target table. The following table lists the keys used in the example: Source Table Lookup Table Target Table source_id product_name lookup_id product_name target_id source_id lookup_id The source table product name field is found in the lookup table. After the product name is found, the primary key from the lookup table is stored in the target table as a foreign key. To test the validity of the lookup table foreign key in the target table, complete the following tasks: 1. Create the lookup view. Add the source table and the lookup table to the lookup view. Then create a relationship between the product name in the source and lookup tables. 2. Create a table pair with the lookup view and the table that is the target of the data transformation process. Join the tables on the source table primary key, which is stored in the target table as a foreign key. 103

117 3. Create an OUTER_VALUE test that compares the primary key of the lookup table to the lookup ID that is stored as a foreign key in the target table. The OUTER_VALUE test checks the validity of the lookup table primary key stored in the target table against the contents of the source table. The test also finds any orphans, which are records in the target table that do not match any records in the lookup table. Lookup View Properties Lookup view properties describe the properties of source and lookup tables. You can view lookup view properties by selecting a lookup view in either the Navigator or on the Lookup Views tab and viewing the properties. Most properties come from the values entered in the Lookup View Editor dialog box. Other properties come from the tests set up for and run on the lookup view. Edit lookup view properties in the Lookup View Editor dialog box when you add or edit a lookup view. The following table describes the lookup view properties: Property Table Conn Override Owner Name Source Dir Source File File Type Description Source to Lookup Relationship Description Name of source table or lookup table. PowerCenter connection for the source table or lookup table. Overrides the schema or owner name for the source table. Source file directory if the table is a flat file. The path is relative to the machine that hosts Informatica Services. File name, including file extension, if the source table is a flat file. Type of flat file if the table is a flat file. Select one of the following values: - File Contains Source Data. Select if the flat file contains data. - File Contains a List of Files. Select if the flat file contains a file list. When you select a file list, Data Validation Option processes all files in the list. Lookup view description. The fields on which the source table and lookup table are joined. Selecting Source and Lookup Tables A lookup view consists of a source table and a lookup table. Use the Browse button and the Select Data Sources dialog box to select the source and lookup table in the same way that you select tables for table pairs. Selecting Connections Select the correct connections for the source and lookup tables in the same way that you select connections for table pairs. 104 Chapter 13: Lookup Views

118 Overriding Owner Name You can override the owner name for the source table, but not for the lookup table. To specify a different owner or schema name for the lookup table, create a connection in the Workflow Manager, and use that connection for the lookup table. Source Directory and File If the source table or lookup table is a flat file, specify the source directory, file name, and file type. The file name must include the file extension. Description Data Validation Option automatically generates the description for a lookup view based on the tables you select. You can change the description. Source to Lookup Relationship In the source and lookup tables, select the values you want to look up in the lookup table. Adding Lookup Views You can use a lookup view to validate data in a target table. The lookup view includes fields from the source table and lookup table. You join the tables based on fields in the tables or expressions. Data Validation Option precedes the source table field names with "S_." The lookup view stores the primary key of the lookup table as a foreign key. 1. Right-click Lookup Views in the Navigator and select Add Lookup View. The Lookup View Editor dialog box opens. 2. Select the source table, lookup table, and their connections. Select the lookup field to look up in the lookup table. You can use an expression for join fields in a lookup view. 3. Enter the description for the lookup view. 4. Create the source-to-lookup relationship. Select Expression to create an expression for the source or lookup field. If you enter an expression, you must specify the datatype, precision, and scale of the result. The datatype, precision, and scale of the source and lookup fields must be compatible. Use the PowerCenter expression syntax in the source field. Use databasespecific SQL in the lookup field. You can validate the PowerCenter expression syntax. 5. Click Delete Join to delete the selected source-to-lookup relationship. 6. Click Save. Adding Lookup Views 105

119 Editing Lookup Views To edit a lookup view, right-click the lookup view in the Navigator or on the SQL Views tab, and select Edit Lookup View. The Lookup View Editor dialog box opens. You cannot modify the sources in a lookup view. You can modify the lookup relationship. Deleting Lookup Views To delete a lookup view, right-click the lookup view in the Navigator or on the Lookup Views tab, and select Delete Lookup View. You can also select the lookup view and press the Delete key. When you delete a lookup view, Data Validation Option deletes all table pairs and tests that use the lookup view. Lookup Views Example Use a lookup view to test the validity of the foreign key stored in the target or fact table, and to confirm that there are no orphans. The following tables display sample data that is typical of data used to build a target table. Source Table ORDER_ID PRODUCT_NAME AMOUNT 101 ipod Laptop ipod 120 Product Lookup Table LKP_PRODUCT_ID LKP_PRODUCT_NAME 21 ipod 22 Laptop 106 Chapter 13: Lookup Views

120 Target Table TARGET_ID ORDER_ID LKP_PRODUCT_ID AMOUNT To test the validity of the lookup table foreign key in the target table, perform the following steps: Create the lookup view. Create a lookup view with the source and lookup tables. The lookup relationship uses the product name fields in both the source and the lookup tables. The fields that are now included in the lookup view are listed below: S_ORDER_ID S_PRODUCT_NAME S_AMOUNT LKP_PRODUCT_ID LKP_PRODUCT_NAME The columns that originate in the source have the prefix "S_." Create the table pair. Create a table pair using the lookup view and the target table. Create a join relationship between the primary key of the source table and the corresponding foreign key in the target table as follows: S_ORDER_ID and ORDER_ID Create an OUTER_VALUE test. Create an OUTER_VALUE test. Compare LKP_PRODUCT_ID in both the lookup table and the target table as follows: LKP_PRODUCT_ID and LKP_PRODUCT_ID Joining Flat Files or Heterogeneous Tables using a Lookup View The SQL view does not allow the use of flat files or heterogeneous database tables. However, you can join two heterogeneous sources with a lookup view. The source-to-lookup relationship is similar to an inner join between the two tables or files. Joining Flat Files or Heterogeneous Tables using a Lookup View 107

121 C H A P T E R 1 4 Join Views This chapter includes the following topics: Join Views Overview, 108 Join View Data Sources, 108 Join View Properties, 109 Adding a Join View, 112 Join View Example, 114 Join Views Overview A join view is a virtual table that contains columns from related heterogeneous data sources joined by key columns. Use a join view to run tests on several related columns across different tables. You can create a join view instead of multiple SQL views with joins. For example, the Employee table has employee details, Inventory table has sales details, and the customer table has customer details. If you create a join view with the tables, you can obtain a consolidated view of the inventory sold by the partner and the revenue generated by the employees associated with the partners. You can run tests with the join view to validate data across the tables. You can create a join view with different types of data sources. For example, you can create a join view with a flat file, SAP table, and an Oracle table. You can add a join view in a single table or a table pair. You can then create tests with the table pair or single table to validate the data in the join view. Add multiple data sources to the join view and add join conditions to define the relationship between data sources. Join View Data Sources You can create a join view with tables from multiple data sources such as relational, application, and flat files. You can use the following sources when you create a join view: Flat files IBM DB2 Microsoft SQL Server Netezza 108

122 Oracle PowerExchange for DB2/zOS Salesforce.com SAP SAS SQL view Join View Properties Join view properties include table definitions and join conditions for each of the data source. The following table describes the join view properties: Property Description Order Table Name Alias Join type Where Clause Left field Right field Description Description of the join view. Sequence of data sources in the join view. You can join a data source with any of the preceding data sources. Data source that you add in the join view. Alias name for the table. Type of join used to join the data sources. The WHERE clause to filter rows in the data source. Left field of the join condition. Left field is the key column of the data source to which you create the join. Right field of the join condition. Right field is the key column of the data source for which you create the join. Connection Properties Choose the connection properties based on the data source type. You must select a connection for all the data sources except for flat files. For flat files, you must provide the source directory and the file name. Connections are PowerCenter connection objects created in the Workflow Manager. Join View Properties 109

123 Relational Connection Properties Choose the relational connection properties for Microsoft SQL Server, Oracle, IBM DB2, Netezza, and PowerExchange for DB2 data sources. Configure the following properties when you select a relational data source: Property Connection Override Owner Name Description PowerCenter connection object to connect to the relational data source. Override the database name and schema of the source. For example, a Microsoft SQL Server table is identified by <database>.<schema>.<table>. To override the database and the schema, enter <new database name>.<to change the schema, enter <new schema name> in the text box. You cannot change only the database name. SAS and Salesforce Connection Properties You can use a SAS or Salesforce data source in Data Validation Option. Select the PowerCenter connection object when you select an SAS or Salesforce data source. Note: You cannot override the owner name for SAS and Salesforce data sources. SAP Connection Properties You must configure the SAP authentication information to use SAP data sources. Configure the following properties when you select an SAP data source: Property Connection SAP User Name SAP Password SAP Client SAP Language SAP Connect String Description PowerCenter connection object to connect to the SAP data source. SAP source system connection user name. Must be a user for which you have created a source system connection. Password for the user name. SAP client number. Language you want for the mapping. Must be compatible with the PowerCenter Client code page. Data Validation Option does not authenticate the value. Ensure that you enter the correct value so that the tests run successfully. Type A or Type B DEST entry in saprfc.ini. SAP Data Sources in Data Validation Option You cannot override the owner name for an SAP data source. Data Validation Option uses the stream mode for installation of the ABAP programs and cannot use the FTP mode. SAP data sources must not contain the backslash (/) character in the field names. 110 Chapter 14: Join Views

124 Flat File Connection Properties You can use the flat file data sources in the PowerCenter repository. Configure the following properties when you select a flat file data source: Property Source Dir Source File File Type Description Directory that contains the flat file. The path is relative to the machine that hosts Informatica Services. File name with the file name extension. Type of flat file. Select one of the following values: - File Contains Source Data. Select if the flat file contains data. - File Contains a List of Files. Select if the flat file contains a file list. When you select a file list, Data Validation Option processes all files in the list. Database Optimization in a Join View You can optimize the data sources in a join view for better performance. You can choose to select a subset of data in the data source and aggregate the rows for a relational data source. To improve read performance of the join view, you can provide a WHERE clause. The WHERE clause ensures that the data source uses a subset of data that satisfies the condition specified in the WHERE clause. Data Validation Option does not check the WHERE clause syntax. If the PowerCenter Integration Service executes the WHERE clause, any valid PowerCenter expression, including expressions that use PowerCenter functions, is allowed. If the PowerCenter syntax is not valid, a mapping installation error occurs. Use PowerCenter expression in cases where you do not push down the WHERE clause in to the data source. Use the following guidelines if the data source executes the WHERE clause: Relational data source. The WHERE clause must be a valid SQL statement. If the SQL statement is not valid, a runtime error occurs. SAP data source. The WHERE clause must be a valid SAP filter condition in the ERP source qualifier. Salesforce data source. The WHERE clause must be a valid SOQL filter condition. SAS data source. The WHERE clause must be a valid Where clause Overrule condition in the SAS source qualifier. You can choose one of the following optimization levels when you configure a data source in a join view: Default. Data Validation Option converts all test logic to a PowerCenter mapping and applies sorting to the data source. WHERE clause, Sorting, and Aggregation in DB. Data Validation Option pushes the WHERE clause, sorting logic for joins, and all aggregate tests to the database. Data Validation Option converts all other test logic to a PowerCenter mapping. You can choose this option with relational data sources. Already Sorted Input. PowerCenter mapping does not sort the input. Ensure that you sort the data so that the tests run successfully. Join Types You can choose join types to link fields in different data sources. You can create the following types of joins between two tables in a join view: Join View Properties 111

125 Inner join Inner join creates a result table by combining column values in two tables A and B based on the join condition. Data Validation Option compares each row in A with each row in B to find all pairs of rows that satisfy the join condition. Left outer join Left outer join between tables A and B contains all records of the left table A, even if the join condition does not find any matching record in the right table B. The resulting join contains all rows from table A and the rows from table B that match the join condition. Right outer join Right outer join between tables A and B contains all records of the right table B, even if the join condition does not find any matching record in the left table A. The resulting join contains all rows from table B and the rows from table A. Full outer join A full outer join contains all the rows from tables A and B. The resulting join has null values for every column in the tables that does not have a matching row. Alias in Join View Alias of the data source in a join view helps you identify data sources that share the same name. By default, Data Validation Option assigns the data source name as the alias name. If you select a data source with the same name as any other data source in the join view, Data Validation Option appends the alias name with a number. Note: If you edit alias name after you a create a join view, Data Validation Option deletes the join conditions. Create the join conditions again to save the join view. Join Conditions You must configure join conditions for all the data sources in the join view. When you configure a join condition, select the data source in the table definition list and specify the left and right fields of the join condition. Data Validation Option displays the alias names of the data sources. The left field of the join condition consists of the output fields from the alias name you choose. The right field consists of the output fields of the data source for which you configure the join condition. You can create a join condition with any of the data sources in the previous rows in a join view. You can create multiple join conditions for the same data source. You cannot save a join view unless you create at least one valid join condition for all the data sources. Adding a Join View You must configure all the table definitions and join conditions when you configure a join view. 1. Click File > New > Join View. The Join View Editor dialog box appears. 2. Enter a description for the join view. 112 Chapter 14: Join Views

126 3. Click Add in the Table Definitions pane. The Choose Data Source dialog box appears. 4. Select the data source. 5. Configure the table definition for the data source. 6. Optionally, click Output Fields and select the fields that you want to view when you create the join condition and when you configure the tests. 7. Configure multiple table definitions as required. You can change the sequence of data sources in the join view and delete table definitions. When you make any change to the data sources in the join view, you must recreate the join conditions. 8. Configure join conditions for all the data sources. You can join a data source with any of the preceding data sources in the Table Definitions pane. You need not specify the join condition for the first data source. Configuring a Table Definition Configure a table definition after you add a data source to the join view in the Join View Editor dialog box. 1. Select the data source in the Join View Editor dialog box. 2. Click Edit in the Table Definitions pane. The Edit Table dialog box appears. 3. Enter an alias name for the data source. By default, the data source name appears as the alias name. 4. Select the join type for the data source. Select join types for all the data sources except the first data source in the join view. 5. Configure the connection details for the table. The connection details vary depending on the data source type. 6. Optionally, enter the WHERE clause. Data Validation Option runs the WHERE clause when it fetches data from the table. 7. If the table is relational, you can choose to push down the WHERE clause in the database. 8. Select the database optimization level. 9. Click OK. Configuring a Join Condition Add join conditions to specify the relationship between data sources. You can create join conditions for a data source with any other data source in the previous rows. 1. Select the data source in the Join View Editor dialog box. 2. Click Add in the Join Conditions pane. The Join Condition dialog box appears. 3. Select the alias name of any of the data source in the previous rows. 4. Select the left field of the join condition. The left field of the join condition consists of the output fields from the alias name you choose. You can also configure and validate a PowerCenter expression as the field. When you enter a field name in the expression, append the alias name followed by an underscore. For example, if the alias name of the table is customer1 and you want to use the CustIDfield in an expression, enter the expression as customer1_custid > 100. Adding a Join View 113

127 5. Select the right field of the join condition. The right field consists of the output fields of the data source for which you configure the join condition. You can also configure and validate a PowerCenter expression as the field. 6. Click OK. Managing Join Views You can edit and delete join views that you create in Data Validation Option 1. Click Join Views in the Navigator. Join views appear on the right pane. 2. Edit or delete the join view. If you modify the join view, re-create the join conditions in the join view. If you delete a join view, you must recreate the table pairs or single tables that contain the join view. Join View Example You need to validate the inventory sales done by the employees and partners to cross-check with the annual sales report. Account table in an SAP system holds the information of an employee account. Partner table is a Salesforce table that contains the information of the inventory sold to a partner associated with an employee. Inventory is a flat file that contains the details of the inventory sold. Account_History is an Oracle table that contains the history of activities done by the account. Current requirement is to validate data across the tables based on the inventory sales of an account. You also need to validate the account details with the historic account details to check for discrepancies. You can create a join view with the tables so that you can run single table tests to validate data. Tables and Fields The following table lists the join view tables and their columns: Table Account (SAP) Partner (Salesforce) Columns Account table contains the following columns: - Account ID - Account Name - Collection - Inventory Partner contains the following columns: - Partner ID - Partner Name - Inventory - Cost - Associated Account ID 114 Chapter 14: Join Views

128 Table Inventory (Flat file) Account_History (Oracle) Columns Inventory table contains the following columns: - Inventory ID - Quantity - Associated Partner ID - Associated Account ID Account_History contains the following columns: - Historic Account ID - Account Name - Total Inventory - Total Collection Creating the Join View 1. Enter Account_Cumulative as the description. 2. Add Account as the first table in the join view. 3. Add Partner, Inventory, and Account_History tables in that order. 4. Configure the table definitions with the required join types. 5. Create join conditions for Partner, Inventory, and Account_History. Table Definition Configuration The following list describes the tables and their join types when you configure the table definitions: Partner You want to capture the details of partners associated with each account. Configure an inner join for the Partner table so that Data Validation Option adds the details of the partners for which there are corresponding accounts to the join view. Inventory You want to capture the details of the inventory sold by the partners. Configure an inner join for the Inventory table so that Data Validation Option adds the details of the inventory for which there are corresponding partners to the join view. Account_History You want to capture the historic details of an account. Configure a left outer join for the Account_History table so that Data Validation Option adds all the historic account details to the join view. Adding Join Conditions Configure the following join conditions for the tables: Partner Select Account as the join table. Select Account ID output field from the Account table as the left field and Associated Account ID output field from the Partner table as the right field of the join. Inventory Select Partner as the join table. Select Partner ID output field from the Partner table as the left field and Associated Partner ID output field from the Inventory table as the right field of the join. Account_History Select Account as the join table. Select Account ID output field from the Account table as the left field and Historic Account ID output field from the Account_History table as the right field of the join. Join View Example 115

129 The following figure illustrates the formation of the join view with the table relationship: After you create the join view, create a single table with the join view. Generate and run tests on the single table to validate the data in the join view. Removing Table from the Join View After you run the required tests for the join view with all the tables, you may want to remove the partner information and run tests solely for the account. If you remove the table Partner from the join view, the join condition for the table Inventory is no longer valid. You need to create another join condition for the table Inventory to save the join view. The following figure illustrates the broken join view: Add a join condition to the table Inventory with Account as the join table. Join Account ID field in the Account table with Associated Account ID field in the Inventory table. The following figure illustrates the join view without the Partner table: 116 Chapter 14: Join Views

130 C H A P T E R 1 5 Reports This chapter includes the following topics: Reports Overview, 117 Business Intelligence and Reporting Tools (BIRT) Reports, 117 Jasper Reports, 119 Jasper Report Types, 122 Dashboards, 124 Metadata Manager Integration, 125 Reports Overview Data Validation Option stores all test definitions and test results in the Data Validation Option repository. You can run reports to display test definitions and results. You can use the BIRT reporting engine to generate the reports. If you have the enterprise license, you can use JasperReports Server to generate the reports. If you use the JasperReports Server, you can also generate dashboards at different levels. You can also view the metadata properties of the data source in a test if you configure Metadata Manager integration for the Data Validation Option repository. Business Intelligence and Reporting Tools (BIRT) Reports You can use the BIRT reporting engine available in Data Validation Option to generate reports. You can generate a report for one or more table pairs or single tables. You can generate the following BIRT reports in Data Validation Option: Summary of Testing Activities Summary of Testing Activities report displays the number of table pairs or single tables, the number of tests for each table pair or single table, and the overall test results. 117

131 Table Pair Summary Table Pair Summary report lists each table pair or single table with the associated tests. Data Validation Option displays each table pair or single table on a separate page. The report includes a brief description of each test and result. Detailed Test Results Detailed Test Results report displays each test on a separate page with a detailed description of the test definition and results. If one of the test sources is a SQL or a lookup view, the report also displays the view definition. Note: Report generation can take several minutes, especially when the report you generate contains hundreds of tests or test runs. BIRT Report Generation You can generate a report for one or more table pairs or single tables. You can also generate a report for all table pairs and single tables in a folder or a test. Right-click the objects for which you want to generate a report, and select Generate Report. The Report Parameters dialog box appears. The following table describes the report options: Option User Table Pair Recency Result Type Run Dates Report Subtitle Description User that created the tests. By default, Data Validation Option generates a report for the tests that the current user creates and runs. Select All to display tests created and run by all users. Select a user name to display tests created and run by that user. Table pairs or single tables for which you want to run a report. You can generate a report on all table pairs and single tables, on the table pairs and single tables in a folder, or on a test. Data Validation Option gives you different options depending on what you select in the Navigator or details area. For example, you can generate a report on all table pairs and single tables or on the table pairs and single tables in the folder. Test runs for which you want to run a report. You can select the latest test runs or all test runs. Test results for which you want to run a report. You can select all results, tests that pass, or tests that do not pass. The test run dates. Enter the from date, to date, or both in the format MM/DD/YYYY. Data Validation Option displays the subtitle on each page of the report. Note: If you change the Data Validation Option repository after you configure BIRT reports, you must restart the Data Validation Option Client to generate accurate reports. SQL and Lookup View Definitions If a test contains either an SQL view or a lookup view as a source, Data Validation Option prints the view definition as part of the report. You cannot print a report showing the definition of the view by itself because each view is tied to a specific test definition. For example, you create an SQL view, use it in a table pair, and run a test. You then update the view by changing the SQL statement, and re-run the test. Each test result is based on a different view definition, which is why the view definition must be tied to a specific result. 118 Chapter 15: Reports

132 Custom Reports You can write custom reports against database views in the Data Validation Option schema. All Data Validation Option reports run against database views that are set up as part of the installation process. You can write custom reports based on the database views. Do not write reports against the underlying database tables because the Data Validation Option repository metadata can change between versions. RELATED TOPICS: Reporting Views Overview on page 152 Viewing Reports Data Validation Option displays reports in a browser window. Use the arrows in the upper right corner to scroll through the pages of the report. To display a specific page, enter the page number in the Go To Page field. You can display or hide the table of contents. To do this, click the table of contents icon in the upper left corner of the report. When you display the table of contents, you can click a heading to display that section of the report. You can also print a report or export it to a PDF file. Click the print icon in the upper left corner of the report. Jasper Reports You can generate and view reports in the JasperReports Server if you have the enterprise license. You can generate reports at the following levels: Table pairs and Single tables Folders Data Validation Option user PowerCenter repository Views When you generate reports, you can also select multiple table pairs and single tables, or folders. If you have a large data test data set use the report annotations to view the report details. Data Validation provides the following administrative reports: External IDs Used In Table Pairs Views Used in Table Pairs/Tables Sources Used In Table Pairs/Tables/Views The following table lists all the Jasper reports and the corresponding levels of reporting: Report Name Run Summary Table Pair Summary Report Level User, Folder, Table Pair/Single Table User, Folder, Table Pair/Single Table Jasper Reports 119

133 Report Name Detailed Test Results Table Pair Run Summary Last Run Summary Percentage of Bad Rows Percentage of Tests Passed Tests Run Vs Tests Passed Total Rows Vs Percentage of Bad Rows Bad Rows Most Recent Failed Runs Failed Runs Failed Tests Validation Failures External IDs Used In Table Pairs Views Used in Table Pairs/Tables Sources Used In Table Pairs/Tables/Views Report Level User, Folder, Table Pair/Single Table Table Pair/Single Table User, Folder, Table Pair/Single Table User, Folder, Table Pair/Single Table User, Folder, Table Pair/Single Table User, Folder, Table Pair/Single Table User, Folder, Table Pair/Single Table Folder, Table Pair/Single Table User, Folder Folder Folder User, Folder User, Folder, Table Pair/Single Table Views PowerCenter Repository You can export Jasper reports in the following formats: PDF DOC XLS XLSX CSV ODS ODT Status in Jasper Reports Reports display the status of a test, table pair, or single table based on the report type. The following status are available in Jasper reports for tests: Pass. The test has passed. Fail. The test has failed. Error. The test encountered a run error or the test has no result. 120 Chapter 15: Reports

134 The following status are available in Jasper reports for table pairs and single tables: Pass. Pass status for a table pair or single table can occur in the following scenarios: - All the tests have passed. - If there is at least one test with pass status and rest of the tests with no results. Fail. If one of the tests in a table pair or single table fail, reports display the status as fail. Error. If all the tests in a table pair or single table has an error or no result, reports display the status as error. Tests and table pairs or single tables with error status do not appear in the bar charts. Generating a Report You can generate a report at the table pair, folder, repository, or user level. 1. Select the object for which you want to generate a report. You can select multiple of objects of the same type. 2. Click Action > Generate Report. The Report Parameters dialog box appears. You can also right-click the object and select Generate Report if you want to generate a report at the table pair or folder level. 3. Select the report type. The following table displays the report options that you can select for each report: Report Name User Recency Run Date Run Summary All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Table Pair Summary All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Detailed Test Results All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Table Pair Run Summary N/A N/A Last 24 hours/last 30 days/ Custom date range Last Run Summary All Users/Current User/ Any User N/A Last 24 hours/last 30 days/ Custom date range Percentage of Bad Rows All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Percentage of Tests Passed All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Tests Run Vs Tests Passed All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Total Rows Vs Percentage of Bad Rows All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Bad Rows N/A All/Latest Last 24 hours/last 30 days/ Custom date range Jasper Reports 121

135 Report Name User Recency Run Date Most Recent Failed Runs All Users/Current User/ Any User All/Latest N/A Failed Runs N/A N/A Last 24 hours/last 30 days/ Custom date range Failed Tests N/A All/Latest Last 24 hours/last 30 days/ Custom date range Validation Failures All Users/Current User/ Any User All/Latest Last 24 hours/last 30 days/ Custom date range Table pairs/tables with External ID N/A N/A N/A Note: You can select the user only if you generate the report from the user level. 4. Optionally, enter the report subtitle. You can use the report subtitle to identify a specific report. 5. Click Run. The report appears in the browser. Jasper Report Types If you have the enterprise license, you can generate different Jasper reports in Data Validation Option. You can generate the following Jasper reports in Data Validation Option: Summary of Tests Run Summary of Testing Activities report displays the number of table pairs or single tables, the number of tests for each table pair or single table, and the overall test results. Table Pair/Table Summary Lists each table pair or single table with the associated tests. Data Validation Option displays each table pair or single table on a separate page. The report includes a brief description of each test and result. Detailed Test Result Displays each test on a separate page with a detailed description of the test definition and results. If one of the test sources is a SQL, join, or a lookup view, the report also displays the view definition. The following information is available in the Detailed Test Result report: Test details Table pair/single table details Runtime information Bad record details 122 Chapter 15: Reports

136 Table Pair/Table Run Summary Displays the summary of all the tests run for a table pair or single table for a given time period. You can click on the date in the report to view the Detailed Test Results report. Last Run Summary Displays the details of the last run of tests at the table pair or single table, folder, or user level. Reports lists the last test runs for the objects in the given time period. If you generate the report at the user level, the report displays the folder summary. You can click on the folder to view the Last Run Summary report for that folder. If you generate the report at the folder level or table pair/single table level, you can view the last run for all the table pairs or single tables for the given time period. You can click on a table pair or single table to view the Detailed Test Results report. Percentage of Bad Rows Displays the aggregate percentage of bad records in comparison with all the rows processed over a period of time. You can generate the report at the table pair or single table, folder, or user level. You can click on a bar for a particular date on the graph to view the Bad Rows report. The bars on the graph appear for the days on which you ran the tests. Percentage of Tests Passed Displays the percentage of passed tests in comparison with total number of tests over a period of time. You can generate the report at the table pair or single table, folder, or user level. The bars on the graph appear for the days on which you ran the tests. Tests Run Vs Tests Passed Displays the total number of tests run over a period of time as a bar chart. The number of tests passed is plotted across the bar chart. You can generate the report at the table pair or single table, folder, or user level. You can click on the test passed points on the graph to view the Validation Failure by Day report. The bars on the graph appear for the days on which you ran the tests. Total Rows Vs Percentage of Bad Records Displays the total number of rows tested over a period of time as a bar chart. The percentage of bad records is plotted across the bar chart. You can generate the report at the table pair or single table, folder, or user level. You can click on the percentage of bad records points on the graph to view the Bad Rows report. Bad Rows The bars on the graph appear for the days on which you ran the tests. Displays the number of bad records across test runs over a period of time in the form of a bar chart. You can click on a bar to view the Table Pair/Table Run Summary report. You can generate the Bad Rows report at the folder level or the table pair/single table level. Most Recent Failed Runs Displays the top ten most recent failed runs. You can run the Most Recent Failed Runs report at the folder or user level. If you run the report at the user level, the report displays the top ten failed runs across all the folders in the Data Validation Option repository in the given time period. If you click on a folder, you can view the Most Recent Failed Runs report for the folder. You can also click on the table pair or single table to view the Detailed Test Result report. If you run the report at the folder level, the report displays the top ten failed runs for that particular folder in the given time period. You can click on the table pair or single table to view the Detailed Test Result report. Jasper Report Types 123

137 Note: Select the recency as latest if you want to get the most recent state of the table pair or single table. Failed Runs Displays the number of failed runs for each table pair or single table over a period of time in the form of a bar chart. Each bar represents the number of failed runs for a table pair or single table. You can click on a bar to view the Table Pair/Table Run Summary report. You can run the Failed Runs report at the folder level. Failed Tests Displays the number of failed tests across test runs over a period of time in the form of a bar chart. Each bar represents the number of failed tests for a table pair or single table. If you click the Table hyperlink, you can view the report in a tabular format. You can click on a bar or the table pair/single table name to view the Detailed Test Result report. You can run the Failed Tests report at the folder level. Validation Failure by Folder Displays the number of failed table pairs or single table as a bar chart for a given time period. Each bar represents the folder in which the failure occurred. If you click the Table hyperlink, you can view the report in a tabular format. You can click on a bar or the folder name to view the Failed Tests report available as bar chart and tabular format. You can run the Validation Failure by Folder report at the user or folder level. External IDs Used In Table Pairs Displays the list of table pairs or single tables with external IDs across all users. You can run the External IDs Used In Table Pairs report at the user, folder, or table pair level. Sources Used In Table Pairs/Tables/Views Displays the list of table pairs, single tables, and views in which a data source is used. Right-click on the PowerCenter repository, a repository folder, or a data source in the repository and select Get Source Usage In Table Pairs/Tables/Views to generate this report. You cannot generate a report at the repository folder level or repository level if there are more than 100 sources. Views Used In Table Pairs/Tables/Views Displays the list of table pairs and single tables in which a view is used. Right-click on the view and select Get View Usage In Table Pairs/Tables/Views to generate this report. Note: Data Validation Option reports might display the details of table pairs or single tables that you previously deleted from the Data Validation Option Client. If you generate a report after you modify the description of a table pair or a single table, the reports might display two entries for the object with the object ID appended to the new and old description with same object ID or in the annotations of the bar chart. Dashboards You can generate Data Validation Option dashboards to get an overview of the testing activities and test results. Dashboards display multiple reports in a single page. You need not generate individual reports to get an overview of test results across a fixed time period. You can view the following dashboards in Data Validation Option: Home Dashboard Displays the following reports for the past 30 days: Tests Run Vs Tests Passed Total Rows vs Percentage of Bad Rows 124 Chapter 15: Reports

138 Percentage of Tests Passed Percentage of Bad Rows You can click through the Total Rows vs Percentage of Bad Rows report and Percentage of Bad Rows reports to view the Bad Rows report. You can click through Tests Run Vs Tests Passed to view the Validation Failures report. Repository Dashboard Displays the Validation Failures for Folders report for the past 24 hours and 30 days in both graphical and tabular formats. The dashboard also displays the Most Recent Failed Runs report for the repository. You can click through the Validation Failure by Folder report to view the Failed Tests report. You can click through the Most Recent Failed Runs report to view the Most Recent Failed Runs report for a folder and the Detailed Test Result report. Folder Dashboard Displays the Bad Rows report and Failed Tests report for the past 24 hours and 30 days in both graphical and tabular formats. You can click through the Failed Tests report to view the Table Pair/Table report and the Bad Rows report to view the Detailed Test Result report. Table Dashboard The Table dashboard displays the following reports for the past 30 days: Tests Passed Vs Test Failed Bad Rows Table Pair/Table Run Summary You can click through the Bad Rows report to view the contents to view the Detailed Test Result report. Metadata Manager Integration You can view the metadata about data sources if you configure Metadata Manager integration in Data Validation Option. You can analyze the impact of test results on data sources if you enable Metadata Manager integration. You can view the metadata about the data source in the PowerCenter repository. To view the metadata about data sources, you must setup a Metadata Manager Service in the Informatica domain. You must create a PowerCenter resource for the PowerCenter repository that contains the data source. Right-click on a data source in the repository and select Get Metadata to view the metadata of the data source. Metadata Manager Integration 125

139 C H A P T E R 1 6 Command Line Integration This chapter includes the following topics: Command Line Integration Overview, 126 Rules and Guidelines for Running DVOCmd Commands, 127 CopyFolder, 127 CreateUserConfig, 128 DisableInformaticaAuthentication, 129 ExportMetadata, 129 ImportMetadata, 130 InstallTests, 131 LinkDVOUsersToInformatica, 132 PurgeRuns, 133 RefreshRepository, 133 RunTests, 134 UpdateInformaticaAuthenticationConfiguration, 136 UpgradeRepository, 137 Command Line Integration Overview You can use the DVOCmd command line utility to invoke Data Validation Option tasks from the command line. For example, you can create and run tests without using the Data Validation Option Client. You can use DVOCmd command line utility to schedule tests to run. You can also embed a specific test as part of the ETL workflow or as part of another process. For example, you can create an ETL workflow that moves data from source to staging, runs validation, and then moves data into the target or an error table based on the validation results. The command line utility writes regular messages to the STDOUT output stream. It writes error messages to the STDERR output stream. You can use the redirection operator to write messages to a file. On Windows, the Data Validation Option command line utility is DVOCmd.exe. DVOCmd.exe exists in one of the following directories: 32-bit operating systems: C:\Program Files\Informatica<version>\DVO\ 64-bit operating systems: C:\Program Files (x86)\informatica<version>\dvo\ 126

140 Rules and Guidelines for Running DVOCmd Commands Review the following rules and guidelines to ensure that DVOCmd commands run successfully. Review the following rules and guidelines before you run DVOCmd commands: Run DVOCmd from the Data Validation Option installation directory. Use the following syntax when you run a DVOCmd command: DVOCmd Command [Argument] [--Options] [Arguments]. Use the proper case for all option names. Option names are case sensitive. Optionally, enclose command options and arguments in square brackets. Enter the command without any argument to get help about the command. For example, enter $HOME/DVO/DVOCmd RunTests. If you invoke DVOCmd from a Command task in a PowerCenter workflow, ensure that the environment variables are set for the user account that runs PowerCenter Services. Also, ensure that DVOCmd is installed under the same user account. You can run only a subset of DVOCmd commands on UNIX. CopyFolder Copies the contents of a folder in a user workspace to a different folder in the same workspace or to another user workspace. The target folder must be a new folder. The target folder cannot exist in the target workspace. You can run this command on Windows. The CopyFolder command copies the table pairs, single tables, and test cases that exist within the source folder. It does not copy test runs or the external IDs associated with table pairs or single tables. If the table pairs or single tables in the source folder use an SQL or lookup view, the CopyFolder command copies the SQL or lookup view to the target user workspace unless the workspace contains a view with the same name. Before Data Validation Option copies a folder, it verifies that all data sources associated with the objects being copied exist in the target workspace. If any data source is missing, Data Validation Option does not copy the folder. The CopyFolder command uses the following syntax: DVOCmd CopyFolder [--confdir conf_dir] [--username Username] [--password Password] [--namespace Namespace] -- fromuser source_user --fromfolder source_folder --touser target_user [--tofolder target_folder] [--reuseviews Yes] The following table describes CopyFolder options and arguments: Option Argument Description --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Because Windows directories often contain spaces, you must enclose the file path in quotes. --username User name Informatica domain user name for the domain to which you configured Informatica authentication. Rules and Guidelines for Running DVOCmd Commands 127

141 Option Argument Description Required if you configure Informatica Authentication. --password Password Password for the Informatica user name. Required if you configure Informatica Authentication. --namespace Namespace Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. --fromuser source_user Name of the source user. Data Validation Option copies the folder in this user workspace. --fromfolder source_folder Name of the source folder. Data Validation Option copies this folder. --touser target_user Name of the target user. Data Validation Option copies the folder to this user workspace. The source user and the target user can be the same user. --tofolder target_folder Name of the target folder. The target folder must be unique in the target workspace. If you do not specify a target folder, Data Validation Option creates a folder in the target workspace with the same name as the source folder. --reuseviews Yes Reuses an SQL or lookup view in the target workspace when the workspace contains a view with the same name as a source SQL or lookup view. If you specify this option, Data Validation Option does not copy the source view to the target workspace. If you do not specify this option, Data Validation Option prompts you for the action to take when views with the same name are found. CreateUserConfig Creates Data Validation Option users with the specified user names. You can run this command on Windows. The CreateUserConfig command uses the following syntax: DVOCmd CreateUserConfig user_name [user_name2 ] [--confdir conf_dir] --outputdir output_dir [--overwrite] Creates a preferences file called <username>-preferences.xml for each user in the output directory. The preferences file contains connection information for the Data Validation Option repository. Copy each preferences file from the output directory to the user configuration directory and rename it to preferences.xml. This allows each user to access the Data Validation Option repository. The following table describes CreateUserConfig options and arguments: Option Argument Description n/a user_name The name of the Data Validation Option user. To create multiple users, enter multiple user names separated with spaces. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Enclose the file path in quotes. --outputdir output_dir Directory in which to store user preferences files. 128 Chapter 16: Command Line Integration

142 Option Argument Description Enclose the file path in quotes. Use double-quotes if the path has space or special characters. --overwrite n/a Overwrites the configuration files. DisableInformaticaAuthentication Disables Informatica authentication in a Data Validation Option schema. You can run this command on Windows. The DisableInformaticaAuthentication command uses the following syntax: DVOCmd DisableInformaticaAuthentication [--confdir conf_dir] [--username User name] [--password Password] [-- namespace Namespace] The following table describes DisableInformaticaAuthentication options and arguments: Option Argument Description --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Enclose the file path in quotes. --username User name Informatica domain administrator user name for the domain to which you configured Informatica authentication. --password Password Password for the Informatica user name. --namespace Namespace Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. ExportMetadata Exports all Data Validation Option metadata to an XML file. You can run this command on Windows. The ExportMetadata command uses the following syntax: DVOCmd ExportMetadata file_name [--confdir conf_dir] [--username User name] [--password Password] [-- namespace Namespace] DisableInformaticaAuthentication 129

143 The following table describes ExportMetadata options and arguments: Option Argument Description n/a file_name The file to which you want to export metadata. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Because Windows directories often contain spaces, you must enclose the file path in quotes. --username User name Informatica domain user name for the domain to which you configured Informatica authentication. Required if you configure Informatica Authentication. --password Password Password for the Informatica user name. Required if you configure Informatica Authentication. --namespace Namespace Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. ImportMetadata Imports metadata into Data Validation Option from an export XML file. You can run this command on Windows. The ImportMetadata command uses the following syntax: DVOCmd ImportMetadata file_name [--confdir conf_dir] [--username Username] [--password Password] [--namespace Namespace] [--overwrite] The following table describes ImportMetadata options and arguments: Option Argument Description n/a file_name The file that contains metadata to be imported. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Because Windows directories often contain spaces, you must enclose the file path in quotes. --username User name Informatica domain user name for the domain to which you configured Informatica authentication. Required if you configure Informatica Authentication. --password Password Password for the Informatica user name. Required if you configure Informatica Authentication. --namespace Namespace Namespace of the LDAP domain. 130 Chapter 16: Command Line Integration

144 Option Argument Description Required if you configure Informatica Authentication with LDAP domain. --overwrite n/a Overwrites existing objects. InstallTests Prepares all tests for a single table or table pair. For each test, this command generates the PowerCenter mapping in the Data Validation Option target folder in the PowerCenter repository. You can run this command on Windows or UNIX. The InstallTests command uses the following syntax: DVOCmd InstallTests external_id [external_id ] [--confdir conf_dir] [--username User name] [--password Password] [--namespace Namespace] [--cachesize CACHESIZE] [--forceinstall] The following table describes InstallTests options and arguments: Option Argument Description n/a external_id The external ID for the single table or table pair. Separate multiple external IDs with spaces. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Enclose the file path in quotes. --username User name Informatica domain administrator user name for the domain to which you configured Informatica authentication. Required if you configure Informatica authentication. --password Password Password for the Informatica user name. Required if you configure Informatica authentication. -- namespace --cachesize -- forceinstall Namespace CACHESIZ E n/a Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. Memory allocation to generate transformations in PowerCenter mappings. Increase the cache size for tests that contain multiple joins and lookups. Default is 20 MB for the data cache and 10 MB for the index cache. Specify "Auto" to enable PowerCenter to compute the cache size. Creates new mappings for table pairs in the repository. DVOCmd uses the DTM buffer size value configured in preferences.xml. If you modify the DTM buffer size value in the preferences file, run the InstallTests command with the forceinstall option for existing table pairs before you run the RunTests command. InstallTests 131

145 Cache Settings You can configure the cache settings for the PowerCenter transformations that Data Validation Option generates for the tests. Data Validation Option generates a PowerCenter mapping with Joiner transformations and Lookup transformations for a test that contains joins and lookups. Joiner transformations and Lookup transformations require a large amount of memory. Configure the cache settings for a complex test that contains joins and lookups. The value that you specify as cache settings for a test persists until you modify the test. For information about optimal cache setting for the tests, see the PowerCenter Advanced Workflow Guide. LinkDVOUsersToInformatica Links the existing Data Validation Option users with Informatica domain users. You can run this command on Windows. Create a text file that contains a list of the Data Validation Option users and Informatica domain users in the following format: <dvo_user_name1>,<informatica_user_name1> <dvo_user_name2>,<informatica_user_name2>.. <dvo_user_namen>,<informatica_user_namen> The LinkDVOUsersToInformatica command uses the following syntax: DVOCmd LinkDVOUsersToInformatica [file_name] [--confdir conf_dir] [--username User name] [--password Password] [--namespace Namespace] The following table describes LinkDVOUsersToInformatica argument: Option Argument Description n/a file_name Name of the file that contains the mapping between Data Validation Option users and Informatica users. Enclose the file path in quotes. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Because Windows directories often contain spaces, you must enclose the file path in quotes. --username User name Informatica domain administrator user name for the domain to which you configured Informatica authentication. Required if you configure Informatica authentication. --password Password Password for the Informatica user name. Required if you configure Informatica authentication. --namespace Namespace Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. 132 Chapter 16: Command Line Integration

146 PurgeRuns Purges test runs from the Data Validation Option repository. You can purge deleted test runs or purge test runs by date. When you purge test runs by date, you can purge all test runs that occur on or after a specified date, before a specified date, or between two dates. You can run this command on Windows. The PurgeRuns command uses the following syntax: DVOCmd PurgeRuns [--confdir conf_dir] [--username User name] [--password Password] [--namespace] [--deleted] [--fromdate from_date] [--todate to_date] If you configure Informatica authentication for the Data Validation Option schema, enter --username and -- password. The following table describes PurgeRuns options and arguments: Option Argument Description --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Because Windows directories often contain spaces, you must enclose the file path in quotes. --username User name Informatica domain user name for the domain to which you configured Informatica authentication. Required if you configure Informatica authentication. --password Password Password for the Informatica user name. Required if you configure Informatica authentication. --namespace Namespace Namespace of the LDAP domain. --deleted n/a Purges deleted test runs. Required if you configure Informatica Authentication with LDAP domain. --fromdate from_date Purges test runs that occur on or after this date. --todate to_date Purges test runs that occur before this date. RefreshRepository Refreshes a source or target repository. You can run this command on Windows or UNIX. The RefreshRepository command uses the following syntax: DVOCmd RefreshRepository repo_name [--confdir conf_dir] [--username User name] [--password Password] [-- namespace] [--all] [--connections] [--folderlist] [--allfolders] [--folder folder_name] [--dryrun] Tip: Use the --folderlist and --folder options to get the sources and targets in a new PowerCenter repository folder. For example, if the repository name is "DVTgtRepo" and the new folder name is "NewOrders," enter the following command: DVOCmd RefreshRepository DVTgtRepo --folderlist --folder NewOrders PurgeRuns 133

147 The following table describes RefreshRepository options and arguments: Option Argument Description n/a repo_name Name of the repository you want to refresh. Note: This can take several minutes to several hours depending on the size of the repository. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Use double-quotes if the path has space or special characters. --username User name Informatica domain user name for the domain to which you configured Informatica authentication. Required if you configure Informatica authentication. --password Password Password for the Informatica user name. Required if you configure Informatica authentication. --namespace Namespace Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. --all n/a Refreshes all source, target, folder, and connection metadata for the repository. Note: This option can take several minutes to several hours depending of the sources and targets in the PowerCenter Repository Service, but Data Validation Option does not copy the objects into the Data Validation Option repository. You can use the option to checks whether the import from PowerCenter Repository Services works. ng on the size of the repository. --connections n/a Refreshes connection metadata for the target repository. --folderlist n/a Refreshes the folder list for the repository. --allfolders n/a Refreshes source and target metadata in all folders in the repository. --folder folder_name Refreshes source and target metadata for the named folder. --dryrun n/a Checks whether import from the PowerCenter Repository Services works. RefreshRepository reads the sources and targets in the PowerCenter Repository Service, but does not write the objects into the Data Validation Option repository. RELATED TOPICS: Repositories Overview on page 53 RunTests Runs all tests for a single table or table pair. You can run this command on Windows or UNIX. For example, to run tests for a table pair with the external ID "abc123," you might enter the following command: DVOCmd RunTests abc Chapter 16: Command Line Integration

148 If one test fails, the overall result also fails. The exit status code for a successful test is 0. A non-zero status code designates a failed test or an error. The following table describes the status codes: Status Code Description 0 The test is successful. 1 The test fails to install. 2 The test fails to run. 3 The test does not give any result. 4 The test fails. The RunTests command uses the following syntax: DVOCmd RunTests external_id [external_id ] [--confdir conf_dir] [--username User name] [--password Password] [--namespace] [-- _id,...] [--send NotPass] [--cachesize CACHESIZE] The following table describes RunTests options and arguments: Option Argument Description n/a external_id The external ID for the single table or table pair. Separate multiple external IDs with spaces. --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Enclose the file path in quotes. --username User name Informatica domain user name for the domain to which you configured Informatica authentication. Required if you configure Informatica authentication. --password Password Password for the Informatica user name. Required if you configure Informatica authentication. --namespace Namespace Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. -- _id The address to which Data Validation Option sends an when the tests are complete. You can provide multiple addresses separated by commas. The specifies whether the test has passed or failed and provides a link to the test results. Note: Configure the SMTP settings for the outgoing server on the PowerCenter Integration Service with the following custom properties: SMTPServerAddress, SMTPPortNumber, SMTPFromAddress, and SMTPServerTimeout. If you want to use Microsoft Outlook to send , enter the Microsoft Exchange profile in the MSExchangeProfile configuration property in the PowerCenter Integration Service. RunTests 135

149 Option Argument Description --send NotPass Limits Data Validation Option to sending an only if the test fails. --cachesize CACHESIZE Memory allocation to generate transformations in PowerCenter mappings. Increase the cache size for tests that contain multiple joins and lookups. Default is 20 MB for the data cache and 10 MB for the index cache. Specify "Auto" to enable PowerCenter to compute the cache size. When you run the RunTests command and set the cache size, Data Validation Option installs the tests in the repository before it runs the test. Cache Settings You can configure the cache settings for the PowerCenter transformations that Data Validation Option generates for the tests. Data Validation Option generates a PowerCenter mapping with Joiner transformations and Lookup transformations for a test that contains joins and lookups. Joiner transformations and Lookup transformations require a large amount of memory. Configure the cache settings for a complex test that contains joins and lookups. The value that you specify as cache settings for a test persists until you modify the test. For information about optimal cache setting for the tests, see the PowerCenter Advanced Workflow Guide. UpdateInformaticaAuthenticationConfiguration Updates the Informatica authentication properties in the Data Validation Option schema. You can run this command on Windows. The UpdateInformaticaAuthenticationConfiguration command uses the following syntax: DVOCmd UpdateInformaticaAuthenticationConfiguration [--confdir conf_dir] [--infahostname INFAHOSTNAME] [--infahttpport INFAHTTPPORT] [--issecureconnection ISSECURECONNECTION] [--username INFAADMINUSERNAME] [--password INFAADMINPASSWORD] [--namespace NAMESPACE] You can obtain these parameters from the nodemeta.xml file available in the following location: <InformaticaInstallationDir>/server/isp The following table describes UpdateInformaticaAuthenticationConfiguration options and arguments: Option Argument Description --confdir conf_dir The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. Because Windows directories often contain spaces, you must enclose the file path in quotes. --infahostname INFAHOSTNAME Host name of the Informatica gateway. 136 Chapter 16: Command Line Integration

150 Option Argument Description --infahttpport INFAHTTPPORT Port number to connect to the Informatica gateway. Enter the value of the port element in the nodemeta.xml file. -- issecureconnectio n ISSECURECONNE CTION Set the argument as true if TLS is enabled in the Informatica gateway. --username --password INFAADMINUSER NAME INFAADMINPASS WORD Informatica administrator user for the Informatica gateway. Password for the Informatica administrator user. --namespace NAMESPACE Namespace of the LDAP domain. Required if you configure Informatica Authentication with LDAP domain. UpgradeRepository Upgrades the Data Validation Option repository. Use this command when you upgrade from a previous version of Data Validation Option. You can run this command on Windows or UNIX. The UpgradeRepository command uses the following syntax: DVOCmd UpgradeRepository [--confdir CONFDIR] The following table describes the UpgradeRepository option and argument: Option Argument Description --confdir CONFDIR The user configuration directory. Specify the configuration directory if you have multiple Data Validation Option repositories on a client machine. If you have one Data Validation Option repository on a client machine and have not changed the configuration directory, you do not need to specify this option. If Windows directory contains spaces, enclose the file path in quotes. RELATED TOPICS: Data Validation Option Installation and Configuration on page 15 UpgradeRepository 137

151 C H A P T E R 1 7 Troubleshooting This chapter includes the following topics: Troubleshooting Overview, 138 Troubleshooting Initial Errors, 138 Troubleshooting Ongoing Errors, 139 Troubleshooting Command Line Errors, 140 Troubleshooting Overview When you run a test, Data Validation Option performs the following tasks: 1. Creates a mapping in the specified PowerCenter folder. 2. Creates the PowerCenter workflow. 3. Runs the PowerCenter session. Besides initial installation problems, Data Validation Option errors can occur in one of the following steps: Installation Error Data Validation Option cannot create a PowerCenter mapping. Run Error Data Validation Option cannot create or run a PowerCenter workflow. No Results The PowerCenter session runs but fails, or there are no results in the results database. Troubleshooting Initial Errors This section assumes that Data Validation Option has just been installed and no successful tests have been executed. It also assumes that the first test is a simple test that does not contain expressions or SQL views. 138

152 The following table describes common initial errors: Error Cannot connect to the Data Validation Option repository Cannot read the PowerCenter repository Installation Error Run Error No Results Possible Cause and Solution Database credentials are incorrect. Check the server, port, and database name specified in the URL line. If the problem persists, contact your database administrator. If the problem is database username and password, the error message explicitly states this. Check the repository settings up to Security Domain. (Informatica Domain names are not used until later.) If you cannot resolve the error, contact the Informatica administrator. Another way to troubleshoot this error is by trying to log into the repository through the pmrep command line utility. If you get an out-of-memory error when you access large repositories, increase the Java heap size of the Data Validation Option client from the command line. You can increase the Java heap size from the command line with the following command: DVOClient.exe -J-Xmx<heapsize value> Default is 1024 MB. Verify that the Data Validation Option folder is closed in the Designer, Workflow Manager and the Repository Manager. The Workflow Monitor can be open. Verify that the INFA_HOME environment variable is set. Verify that the Data Validation Option folder exists. Verify that the Data Validation Option folder is closed. Check the Informatica Domain name and Integration Service names. Verify that they are running. Verify that the PowerCenter connection name (connection to the Data Validation Option repository) is correct, and that the user has the privilege to use it. Open the session log and look for session errors. Most session failures are caused by an incorrect connection. If the error is Cannot get object class for dvo/infact/pivotpluginimpl, the dvoct.jar file cannot be read either because it is not on the server, because of its privileges, or because the information entered in the Administrator tool is incorrect. Verify that the user has the privilege to use the connection to the Data Validation Option repository specified in the Tools > Preferences > Data Validation Option database. This will also be apparent in the session log. If you install PowerCenter or earlier and the data source is SAP, install ABAP program on the mapping generated by the test in the Designer tool. Verify that there is data in the data set you are analyzing. Tables should have records, and filters and joins should not result in an empty set. Verify that the connection to the Data Validation Option repository specified in the Workflow Manager points to the Data Validation Option repository. Troubleshooting Ongoing Errors This section assumes that successful tests have been created and run before the error occurred. In general, you should always check for the following sources of errors: Incorrect connection The Data Validation Option folder is open in the Designer, Workflow Manager, or Repository Manager Troubleshooting Ongoing Errors 139

153 The following table describes common ongoing errors: Error Installation or run errors No results Inability to generate reports Inability to copy folders Possible Cause and Solution Verify that the Data Validation Option folder is closed in the Designer, Workflow Manager and Repository Manager. Verify that the PowerCenter environment is functioning correctly, for example, services are running and the repository is up. If the error occurred right after you created an expression either in a test editor dialog box or as a WHERE clause, check the syntax. Open the session log and verify that the PowerCenter connections are correct. Verify that there is data in the data set you are analyzing. Tables should have records. Filters or joins should not result in an empty set. Verify that you have read and write permissions on the Data Validation Option installation directory and subdirectories. Verify that the repository, data sources, and folders that contain the data sources have identical names in the source and the target workspaces. Object names in Data Validation Option are case sensitive. Verify that all data sources associated with the objects to copy exist in the target workspace in the same location and that the names match. Troubleshooting Command Line Errors I ran a DVOCmd command that got an error and used the redirection operator to write the messages to a file. The redirection operator does not redirect all messages to the output file. When you run a DVOCmd command, the command line utility writes regular messages to the STDOUT output stream and writes error messages to the STDERR output stream. You use the redirection operator to write messages to an output file. To merge messages from both output streams, enter "2>&1" after the output file name. For example, you encounter an error while refreshing folder "MyFolder" in Data Validation Option repository "DVORepo." To write all messages, including the error messages, to a text file called "Log.txt," use the following command: DVOCmd RefreshRepository DVORepo --folder MyFolder > C:\Log.txt 2>&1 To append messages to an existing log file called "DVOLog.txt," use the following command: DVOCmd RefreshRepository DVORepo --folder MyFolder >> C:\DVOLog.txt 2>&1 140 Chapter 17: Troubleshooting

154 A P P E N D I X A Datatype Reference This appendix includes the following topics: Test, Operator, and Datatypes Matrix for Table Pair Tests, 141 Test, Operator, and Datatypes Matrix for Single-Table Constraints, 142 Test, Operator, and Datatypes Matrix for Table Pair Tests Table pair tests use the following datatypes: s = string datatypes n = numeric datatypes d = date/time datatypes b = binary/other datatypes The following table describes the operators and datatypes for table pair tests: Operators Allowed Datatypes Allowed: Approx. Operator Datatypes Allowed: All Other COUNT All s,n,d,b s,n,d,b COUNT_DISTINCT All s,n,d s,n,d COUNT_ROWS All s,n,d,b s,n,d,b MIN All n s,n,d MAX All n s,n,d AVG All n n SUM All n n SET_AinB -- s,n,d s,n,d SET_BinA -- s,n,d s,n,d 141

155 Operators Allowed Datatypes Allowed: Approx. Operator Datatypes Allowed: All Other SET_AeqB -- s,n,d s,n,d VALUE All n s,n,d OUTER_VALUE All n s,n,d Note: SET tests do not use operators and allow string, numeric and date/time datatypes. Test, Operator, and Datatypes Matrix for Single-Table Constraints Single-table constraints use the following datatypes: s = string datatypes n = numeric datatypes d = date/time datatypes b = binary/other datatypes The following table describes the operators and datatypes for single-table constraints: Operators Allowed Datatypes Allowed Result Expression Datatype COUNT All s,n,d,b n COUNT_DISTINCT All s,n,d n COUNT_ROWS All s,n,d,b n MIN All s,n,d n MAX All s,n,d n AVG All n n SUM All n n VALUE All s,n,d s,n,d FORMAT =, <> s,n,d s UNIQUE -- s,n,d -- NOT_NULL -- s,n,d -- NOT_BLANK -- s,n,d Appendix A: Datatype Reference

156 A P P E N D I X B BIRT Report Examples This appendix includes the following topics: Summary of Testing Activities, 143 Table Pair Summary, 144 Detailed Test Results Test Page, 144 Detailed Test Results Bad Records Page, 145 Summary of Testing Activities The following figure shows an example of a Summary of Testing Activities report: 143

157 Table Pair Summary The following figure shows an example of a Table Pair Summary report: Detailed Test Results Test Page The following figure shows an example of a test page in the Detailed Test Results report: 144 Appendix B: BIRT Report Examples

158 Detailed Test Results Bad Records Page The following figure shows an example of a bad records page in the Detailed Test Results report: Detailed Test Results Bad Records Page 145

159 146 Appendix B: BIRT Report Examples

160 A P P E N D I X C Jasper Report Examples This appendix includes the following topics: Home Dashboard, 147 Repository Dashboard, 148 Folder Dashboard, 149 Tests Run Vs Tests Passed, 149 Total Rows Vs Percentage of Bad Records, 150 Most Recent Failed Runs, 150 Last Run Summary, 151 Home Dashboard Home dashboard displays the test details in the Data Validation Schema over the past 30 days. 147

161 Repository Dashboard Repository Dashboard displays the details of tests run in the Data Validation Option repository over the past 30 days and the past 24 hours. 148 Appendix C: Jasper Report Examples

162 Folder Dashboard Folder Dashboard displays the details of tests run in the repository over the past 30 days and the past 24 hours. Tests Run Vs Tests Passed The following figure shows an example of a Tests Run Vs Tests Passed report: Folder Dashboard 149