Code Page Configuration in PowerCenter

Transcription

1 Code Page Configuration in PowerCenter Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. All other company and product names may be trade names or trademarks of their respective owners and/or copyrighted materials of such owners.

2 Abstract To ensure that the character encoding for a data set remains true when data moves from source to PowerCenter or from PowerCenter to target, the code page in PowerCenter must match the code page in the source or target. This article explains how to configure code pages to ensure accurate character encoding when PowerCenter reads or writes data. Supported Versions PowerCenter 8.x PowerCenter 9.x Table of Contents Overview Code Page Compatibility for Relational Sources and Targets Configuring the Code Page in the Connection Object Configuring the Code Page in the Database Client Code Page Compatibility for Flat File Sources and Targets Configuring the Code Page in the Session Properties Troubleshooting Code Page Incompatibility Verifying Database Code Pages... 7 Overview A code page defines the character set used to encode data. To ensure accurate character encoding during data movement, the code page of the data must be compatible throughout the different stages of the data movement. When data passes from the source to PowerCenter, the code page that PowerCenter uses to read the data must match the code page used by the data source. Similarly, when data passes from PowerCenter to the target, the code page that PowerCenter uses to write the data must match the code page used by the target. Although PowerCenter internally uses the UCS-2 code page for data transformation, the PowerCenter Integration Service can read data from a source that uses a different code page and write data to a target that uses a different code page. When the PowerCenter Integration Service reads data from the source, the Integration Service changes the code page of the data to UCS-2 before it performs the transformation. When the PowerCenter Integration Service writes data to the target, the Integration Service changes the code page of the data from UCS-2 to the code page of the target. When the PowerCenter Integration Service reads and writes data, the following events occur: 1. The PowerCenter Integration Service spawns a reader thread to connect to the source and read the data. If the source is a relational database, the reader thread uses a relational connection object to connect to the database client. The database client reads data from the database and sends the data back through the source connection object. If the source is a flat file, the reader thread connects to the file directly and reads the data. 2. The reader thread converts the data to use the UCS-2 code page during processing. 3. The PowerCenter Integration Service processes the data and sends the target data to the writer thread. 4. The writer thread converts data from UCS-2 code page to the code page of the target. 2

3 5. The writer thread connects to the target to write the data. If the target is a relational database, the writer thread uses a relational connection object to send target data to the database client. The database client writes the data to the database. If the target is a flat file, the writer thread connects to the file directly and writes the data. Code Page Compatibility for Relational Sources and Targets PowerCenter requires a relational connection object to connect to a relational database as a source or target. The code page in the connection object must match the code page in the relational database client. Use the Workflow Manager to create a relational connection object. When you configure the connection object, set the code page to match the code page used by the database client. PowerCenter does not connect directly to the relational database. PowerCenter uses the connection object to connect to the client software of the relational database. If the code page set in the connection object is the same as the code page in the database client, PowerCenter reads or writes data to the database correctly. The following image shows how the code pages must match when PowerCenter connects to a database: Configure the source connection object with the same code page as the source database client. Configure the target connection object with the same code page as the target database client. Configuring the Code Page in the Connection Object Use the Workflow Manager to edit the connection object for the source or target and set the code page. 1. In the Workflow Manager, click Connections > Relational. The Relational Connection Browser dialog box appears. 2. Select a connection and click Edit. The Connection Object Definition dialog box appears. 3. In the Code Page property, select the code page that matches the code page of the database client. 3

4 The following image shows the Code Page property in the Connection Object Definition dialog box: 4. Click OK to close the Connection Object Definition dialog box. 5. Click Close to close the Relational Connection Browser dialog box. Configuring the Code Page in the Database Client PowerCenter uses a relational connection object to connect to the database client to read or write data to a relational database. The code page of the database client must match the code page of the relational connection object. Verify the code page of the database client before you set the code page of the relational connection object. Oracle Client The Oracle database client uses the language and character set defined in the Oracle environment variable NLS_LANG. Verify the code page in the NLS_LANG environment variable on the machine where the PowerCenter Integration Service runs. To verify the NLS_LANG settings used by the database client, run the following command: UNIX Windows SQL> HOST ECHO $NLS_LANG SQL> HOST ECHO %NLS_LANG% IBM DB2 Client To verify the code page used by the database client, view the settings in the DB2CODEPAGE registry variable on the machine where the Integration Service runs. 4

5 Microsoft SQL Server Client The sp_helpsort stored procedure displays the Microsoft SQL Server sort order and character set. Run sp_helpsort to determine the code page of the database client and database. For more information: Sybase Client To verify the code page of the Sybase database and database client, view the settings in the locales.dat file. The locales.dat file is located in the $SYBASEHOME/locales directory. Code Page Compatibility for Flat File Sources and Targets When PowerCenter connects to a flat file or an XML file, the code page for the session reader or writer must match the code page of the data in the file. To specify the code page to use when PowerCenter connects to a flat file or XML file, you set the code page in the session properties. Use the Workflow Manager to edit the properties of a session and set the code page. To set the code page that PowerCenter uses to read a file, set the code page for the session reader. To set the code page that PowerCenter uses to write to a file, set the code page for the session writer. Typically, you cannot determine the code page for the data in a file by viewing the content of the file. The application that creates or defines a file specifies the code page of the data in the file. For example, the WSDL defines the code page for data in an XML file. You must determine the code page of the data so that you can correctly set the code page for the reader and writer in the PowerCenter session. Note: Verify the code page used by the application that generated the source file and set the session reader code page to match. Likewise, verify the code page expected by the application that will use the target file and set the session writer code page to match. The following image shows how the code pages must match when PowerCenter connects to a file: Configuring the Code Page in the Session Properties Use the Workflow Manager to edit a session and set the code page for the source or target. 1. In the Workflow Manager, edit the session in the Task Developer and click the Mapping tab. 5

6 2. On the Transformations tab in the left pane, select the source or target for which you want to set the code page. If you select a source, the Readers, Connections, and Properties sections display on the right pane. If you select a target, the Writers, Connections, and Properties sections display on the right pane. 3. In the Properties section, click the Set File Properties link. Depending on your selection on the Transformations tab, the Flat Files - Sources or the Flat Files - Targets dialog box appears. 4. Click Advanced. The File Properties dialog box appears. 5. In the Code Page section, verify that the Select Codepage option is selected and select the code page for the file. The following image shows the dialog boxes that display when you configure the code page for a source file and the options that you select at each step: 6

7 The following image shows the dialog boxes that display when you configure the code page for a target file and the options that you select at each step: 6. Click OK to close the File Properties dialog box. 7. click OK to close the File Sources or File Targets dialog box. 8. click OK to close the Edit Task window and then save the changes to the repository. Troubleshooting Code Page Incompatibility If data must be converted from one code page to another and the code pages are incompatible, the character conversion can result in data error. When data is converted from one code page to another, the number of bytes allocated to the data can change. Depending on how data fits into the byte allocation in a code page, the conversion can result in unexpected characters or truncated data. If data is converted from a multibyte character encoding to a single-byte character encoding, there might not be enough bytes to correctly store the converted data. For example, the code page for the source is UTF-8 and the code page for the target is ISO UTF-8 uses multibyte character encoding and ISO uses single-byte character encoding. When the data is converted from UTF-8 to ISO , a smaller number of bytes are allocated to the data. If the single-byte code page cannot allocate enough bytes for a character and breaks within a code point, the error can appear as data corruption in the target. If the single-byte code page cannot allocate enough bytes for a string but breaks in between characters, the error can appear as data truncation in the target. To avoid data errors, verify that PowerCenter sources and targets are configured with compatible code pages. Verifying Database Code Pages Data corruption or truncation can occur when the database client and database use different code pages. To verify whether a data error is a result of a difference in the code pages of the database and database client, review the code page settings for the database and the database client. Use the following method to verify the code page used by the database and database client: 7

8 Oracle The Oracle environment variable NLS_LANG stores the code page of the database client. To verify the NLS_LANG settings used by the database client, run the following command: UNIX Windows SQL> HOST ECHO $NLS_LANG SQL> HOST ECHO %NLS_LANG% To verify the NLS_LANG settings used by the database, you can use the following SQL statement: SELECT * FROM V$NLS_PARAMETERS WHERE PARAMETER LIKE 'NLS_CHARACTERSET%' IBM DB2 To verify the code page used by the database client, view the settings in the DB2CODEPAGE registry variable. To verify the code page of the database, you can use the following command: db2 get db cfg for <dbname> You must be a DB2 instance owner to run the command. Sybase To verify the code page of the Sybase database and database client, view the settings in the locales.dat file. The locales.dat file is located in the $SYBASEHOME/locales directory. You can also view the code page of the Sybase database and database client in the LANG environment variable on the machine where the PowerCenter Integration Service runs. Microsoft SQL Server To determine the code page of the database client and database, run the sp_helpsort stored procedure. The sp_helpsort stored procedure displays the Microsoft SQL Server sort order and character set. For more information: Authors Sonali Verma Lead Product Specialist Marissa Johnston Documentation Editor 8