User Guide. Informatica PowerCenter Connect for MSMQ. (Version 8.1.1)

Transcription

1 User Guide Informatica PowerCenter Connect for MSMQ (Version 8.1.1)

2 Informatica PowerCenter Connect for MSMQ User Guide Version September 2006 Copyright (c) Informatica Corporation. All rights reserved. Printed in the USA. This software and documentation contain proprietary information of Informatica Corporation and are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright law. Reverse engineering of the software is prohibited. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without prior consent of Informatica Corporation. Use, duplication, or disclosure of the Software by the U.S. Government is subject to the restrictions set forth in the applicable software license agreement and as provided in DFARS (a) and (a) (1995), DFARS (c)(1)(ii) (OCT 1988), FAR (a) (1995), FAR , or FAR (ALT III), as applicable. The information in this document is subject to change without notice. If you find any problems in the documentation, please report them to us in writing. Informatica Corporation does not warrant that this documentation is error free. Informatica, PowerCenter, PowerCenterRT, PowerCenter Connect, PowerCenter Data Analyzer, PowerMart, SuperGlue, Metadata Manager, Informatica Data Quality and Informatica Data Explorer are trademarks or registered trademarks of Informatica Corporation in the United States and in jurisdictions throughout the world. All other company and product names may be trade names or trademarks of their respective owners. Portions of this software and/or documentation are subject to copyright held by third parties, including without limitation: Copyright DataDirect Technologies, All rights reserved. Copyright Sun Microsystems. All Rights Reserved. Copyright RSA Security Inc. All Rights Reserved. Copyright Ordinal Technology Corp. All Rights Reserved. Informatica PowerCenter products contain ACE (TM) software copyrighted by Douglas C. Schmidt and his research group at Washington University and University of California, Irvine, Copyright (c) , all rights reserved. Portions of this software contain copyrighted material from The JBoss Group, LLC. Your right to use such materials is set forth in the GNU Lesser General Public License Agreement, which may be found at The JBoss materials are provided free of charge by Informatica, as-is, without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Portions of this software contain copyrighted material from Meta Integration Technology, Inc. Meta Integration is a registered trademark of Meta Integration Technology, Inc. This product includes software developed by the Apache Software Foundation ( The Apache Software is Copyright (c) The Apache Software Foundation. All rights reserved. This product includes software developed by the OpenSSL Project for use in the OpenSSL Toolkit and redistribution of this software is subject to terms available at Copyright The OpenSSL Project. All Rights Reserved. The zlib library included with this software is Copyright (c) Jean-loup Gailly and Mark Adler. The Curl license provided with this Software is Copyright , Daniel Stenberg, <Daniel@haxx.se>. All Rights Reserved. The PCRE library included with this software is Copyright (c) University of Cambridge Regular expression support is provided by the PCRE library package, which is open source software, written by Philip Hazel. The source for this library may be found at ftp://ftp.csx.cam.ac.uk/pub/software/programming/ pcre. InstallAnywhere is Copyright 2005 Zero G Software, Inc. All Rights Reserved. Portions of the Software are Copyright (c) The OpenLDAP Foundation. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted only as authorized by the OpenLDAP Public License, available at This Software may be protected by U.S. and international Patents and Patents Pending. DISCLAIMER: Informatica Corporation provides this documentation as is without warranty of any kind, either express or implied, including, but not limited to, the implied warranties of non-infringement, merchantability, or use for a particular purpose. The information provided in this documentation may include technical inaccuracies or typographical errors. Informatica could make improvements and/or changes in the products described in this documentation at any time without notice.

3 Table of Contents Preface v About This Book vi Document Conventions vi Other Informatica Resources vii Visiting Informatica Customer Portal vii Visiting the Informatica Web Site vii Visiting the Informatica Developer Network vii Visiting the Informatica Knowledge Base vii Obtaining Technical Support vii Chapter 1: Understanding PowerCenter Connect for MSMQ Overview MSMQ Delivery Options Chapter 2: Working with MSMQ Sources and Targets Overview Creating MSMQ Source or Target Definitions Chapter 3: Creating and Configuring MSMQ Workflows Overview Working with MSMQ Workflows Message Count Real-time Flush Latency Idle Time Removing Messages from the Source Queue Configuring Delivery Options Pipeline Partitioning Configuring Session Properties Running MSMQ Workflows Index Table of Contents iii

4 iv Table of Contents

5 Preface Welcome to PowerCenter Connect, Informatica s family of packaged software products that helps you extract data and metadata from ERP and other third-party applications. PowerCenter Connect for MSMQ is a natural extension to the open architecture of PowerCenter, which supports data extraction from a wide variety of operational data sources. PowerCenter Connect for MSMQ lets you directly extract data from MSMQ messages, transform the data according to your business rules, and load data into MSMQ messages or data warehouses. v

6 About This Book The Informatica PowerCenter Connect for MSMQ User Guide provides information to build mappings to extract data from MSMQ messages and load data to MSMQ messages. It is written for the database administrators and developers who are responsible for extracting data from MSMQ messages and loading data into MSMQ messages. This book assumes you have knowledge of relational database concepts and database engines, PowerCenter, and the MSMQ system. You should also be familiar with the interface requirements for other supporting applications. For additional information about related MSMQ issues, refer to the MSMQ documentation. The material in this book is also available online. Document Conventions This guide uses the following formatting conventions: If you see italicized text boldfaced text italicized monospaced text Note: Tip: Warning: monospaced text bold monospaced text It means The word or set of words are especially emphasized. Emphasized subjects. This is the variable name for a value you enter as part of an operating system command. This is generic text that should be replaced with user-supplied values. The following paragraph provides additional facts. The following paragraph provides suggested uses. The following paragraph notes situations where you can overwrite or corrupt data, unless you follow the specified procedure. This is a code example. This is an operating system command you enter from a prompt to execute a task. vi Preface

7 Other Informatica Resources In addition to the product manuals, Informatica provides these other resources: Informatica Customer Portal Informatica web site Informatica Developer Network Informatica Knowledge Base Informatica Technical Support Visiting Informatica Customer Portal As an Informatica customer, you can access the Informatica Customer Portal site at The site contains product information, user group information, newsletters, access to the Informatica customer support case management system (ATLAS), the Informatica Knowledge Base, Informatica Documentation Center, and access to the Informatica user community. Visiting the Informatica Web Site You can access the Informatica corporate web site at The site contains information about Informatica, its background, upcoming events, and sales offices. You will also find product and partner information. The services area of the site includes important information about technical support, training and education, and implementation services. Visiting the Informatica Developer Network You can access the Informatica Developer Network at The Informatica Developer Network is a web-based forum for third-party software developers. The site contains information about how to create, market, and support customer-oriented add-on solutions based on interoperability interfaces for Informatica products. Visiting the Informatica Knowledge Base As an Informatica customer, you can access the Informatica Knowledge Base at Use the Knowledge Base to search for documented solutions to known technical issues about Informatica products. You can also find answers to frequently asked questions, technical white papers, and technical tips. Obtaining Technical Support There are many ways to access Informatica Technical Support. You can contact a Technical Support Center by using the telephone numbers listed the following table, you can send , or you can use the WebSupport Service. Preface vii

8 Use the following addresses to contact Informatica Technical Support: for technical inquiries for general customer service requests WebSupport requires a user name and password. You can request a user name and password at North America / South America Europe / Middle East / Africa Asia / Australia Informatica Corporation Headquarters 100 Cardinal Way Redwood City, California United States Toll Free Standard Rate United States: Informatica Software Ltd. 6 Waltham Park Waltham Road, White Waltham Maidenhead, Berkshire SL6 3TN United Kingdom Toll Free Standard Rate Belgium: France: Germany: Netherlands: United Kingdom: Informatica Business Solutions Pvt. Ltd. Diamond District Tower B, 3rd Floor 150 Airport Road Bangalore India Toll Free Australia: Singapore: Standard Rate India: viii Preface

9 Chapter 1 Understanding PowerCenter Connect for MSMQ This chapter contains information on the following topics: Overview, 2 1

10 Overview MSMQ (Microsoft Message Queuing) is a Windows-based message queuing application that can contain data in any format that is understood by both the sender and the receiver. PowerCenter Connect for MSMQ integrates with MSMQ to extract data from MSMQ sources and write data to MSMQ targets. With PowerCenter Connect for MSMQ, you can define MSMQ sources and targets in the Designer. You can use MSMQ source and target definitions in a mapping to read messages from MSMQ sources and write messages to MSMQ intermediate queues during a PowerCenter workflow. MSMQ then reads messages from the MSMQ intermediate queues and writes them to the targets. When extracting data from MSMQ or writing data to MSMQ, the Integration Service connects to a Microsoft Messaging Queue. The message queue can be a public message queue or a private message queue. The message queue can also be transactional or non-transactional. Figure 1-1 shows how PowerCenter Connect for MSMQ interacts with MSMQ: Figure 1-1. PowerCenter Connect for MSMQ Interaction with MSMQ Source Queues MSMQ Target Queues Intermediate Queue PowerCenter MSMQ Delivery Options MSMQ provides the following delivery options for sending messages: Recoverable messaging. MSMQ stores queue messages on disk during delivery. If any computer on which messages reside fails or shuts down during delivery, MSMQ can resend the messages once the Message Queuing service restarts. Recoverable messaging guarantees message delivery, but may decrease performance. Express messaging. MSMQ stores queue messages in memory during delivery until the target queue receives them. If a computer fails during delivery, MSMQ cannot recover any lost messages. However, if the network fails, MSMQ continues to store messages in memory until the network connection is restored. Express messaging may increase performance, but does not guarantee message delivery. 2 Chapter 1: Understanding PowerCenter Connect for MSMQ

11 For non-transactional queues, you can configure MSMQ to use recoverable messaging or express messaging. For transactional queues, you must use recoverable messaging. When you want to write messages to an MSMQ target queue during a PowerCenter session, you can select a delivery option in the session properties. For more information, see Configuring Delivery Options on page 13. Overview 3

12 4 Chapter 1: Understanding PowerCenter Connect for MSMQ

13 Chapter 2 Working with MSMQ Sources and Targets This chapter contains information on the following topics: Overview, 6 Creating MSMQ Source or Target Definitions, 7 5

14 Overview MSMQ source and target definitions represent metadata for MSMQ messages. When the Integration Service extracts messages from an MSMQ source during a workflow, it reads messages based on the message format defined in the source definition. When the Integration Service writes messages to an MSMQ intermediate queue during a workflow, it writes messages based on the format defined in the target definition. You create MSMQ source and target definitions in the Designer. When you create MSMQ sources and targets, the Designer displays a table with message fields and MSMQ datatypes. When the Integration Service extracts data from the MSMQ source, it converts the data based on the datatypes in the Source Qualifier transformation associated with the source. Every MSMQ source and target definition contains the following fields: MSG_BODY. Contains the MSMQ message body. LABEL. Describes the message label. CORRELATION_ID. Contains the correlation identifier of the message. Table 2-1 lists the MSMQ datatypes that PowerCenter Connect for MSMQ supports and the corresponding transformation datatype: Table 2-1. Transformation Datatypes MSMQ Datatypes Transformation Datatype Description VT_LPSTR String Variable length, null-terminated ASCII strings. VT_LPWSTR String Variable length, null-terminated Unicode strings. VT_BSTR Binary Variable length, Unicode strings that may or may not be null-terminated. Binary Binary Strings and numeric data. UniString String Variable length, null-terminated Unicode strings. Binary Binary Binary data. If you specify the datatype for the MSG_BODY field as VT_LPSTR, VT_LPWSTR, or VT_BSTR in a source definition, the Integration Service processes only those messages that match the specified datatype. If you specify one of these datatypes and enable the Remove Message on Read option when you configure MSMQ session properties, the Integration Service removes all messages even if they have not been processed. If you specify the datatype for the MSG_BODY field in the source definition as Binary, the Integration Service processes all messages regardless of datatype. The datatype for the LABEL field is UniString and cannot be edited. The datatype for the CORRELATION_ID field is Binary and cannot be edited. For more information on MSMQ datatypes, see 6 Chapter 2: Working with MSMQ Sources and Targets

15 Creating MSMQ Source or Target Definitions You manually create MSMQ source and target definitions in the Designer. To create a MSMQ source or target definition: 1. To create an MSMQ source definition, click Sources > Create in the Source Analyzer. To create an MSMQ target definition, click Targets > Create in the Target Designer. 2. Enter a name for the source or target definition. 3. If you are creating a source definition, enter a name in the Database Name field. 4. Select MSMQ as the database type. 5. Click Create. The Select Message Body Datatype dialog box appears. 6. Select one of the following message body datatypes: VT_LPSTR. Variable length, null terminated ASCII string. VT_LPWSTR. Variable length, null terminated Unicode string. VT_BSTR. Variable length, Unicode strings that may or may not be null-terminated. BINARY. Strings and numeric data. 7. Click OK. An MSMQ source or target definition appears. 8. To add another source or target definition, enter a new source or target name. Click Create. 9. Click Done. Once you create an MSMQ source or target definition, you can edit the source or target definition to change the precision for the MSG_BODY port. Creating MSMQ Source or Target Definitions 7

16 8 Chapter 2: Working with MSMQ Sources and Targets

17 Chapter 3 Creating and Configuring MSMQ Workflows This chapter contains information on the following topics: Overview, 10 Working with MSMQ Workflows, 11 Configuring Session Properties, 15 Running MSMQ Workflows, 17 9

18 Overview After you create mappings in the Designer, you can create a session and use the session in a workflow to extract, transform, and load data. You create sessions and workflows in the Workflow Manager. Before configuring an MSMQ workflow, you must complete the following tasks: Configure the Integration Service. Configure an Integration Service to run MSMQ workflows. For more information about configuring an Integration Service, see the PowerCenter Administrator Guide. Configure source and target connections. To read data from MSMQ or write data to MSMQ during a workflow, you need to configure a queue connection for MSMQ sources and targets in the Workflow Manager. For more information about configuring connections, see Managing Connection Objects in the PowerCenter Workflow Administration Guide. 10 Chapter 3: Creating and Configuring MSMQ Workflows

19 Working with MSMQ Workflows When you configure an MSMQ workflow, you define the session and scheduler properties that determine how the Integration Service reads messages from an MSMQ source or writes messages to an MSMQ target. When you configure an MSMQ session, you can set the following session properties: Message Count Real-time Flush Latency Idle Time Remove Message on Read Deliver Messages in Recoverable Mode You can also configure pipeline partitioning for the session. Message Count Use the Message Count session condition to control the number of messages the Integration Service reads from MSMQ before stopping. For example, when you specify 100 for Message Count, the Integration Service reads 100 messages from MSMQ. If you enter a Message Count value, and you configure the session to use pipeline partitioning, the session can run on a single node only. The Integration Service that runs the session cannot run on a grid or on primary and backup nodes. Real-time Flush Latency Use the Real-time Flush Latency session condition in a session with an MSMQ source definition in the mapping to run the session in real time. When you use the Real-time Flush Latency session condition, the Integration Service commits MSMQ source messages to the target at the end of a specified maximum latency period. Specify a Real-time Flush Latency value in seconds in the session properties. For example, if the value for Real-time Flush Latency is 5, the Integration Service commits all messages read from the source five seconds after the first message enters the source. The lower you set the interval, the faster the Integration Service commits messages to the target. When you specify a low Real-time Flush Latency interval, the session might consume more system resources. To use the Real-time Flush Latency session condition to run a session in real time: Configure the session for source-based commits in the session properties. When you configure the session to use source-based commits and add partitions to the pipeline, you must specify pass-through partitioning at each partitioning point. Configure a real-time session to run as a continuous workflow. Working with MSMQ Workflows 11

20 When you set the Real-time Flush Latency session condition and configure the session to use source-based commits, the Integration Service commits messages to the target using the source-based commit interval as well as the Real-time Flush Latency interval. For example, you use 5 seconds as the Real-time Flush Latency session condition and you set the sourcebased commit interval to 1,000 messages. The Integration Service commits messages to the target at two points: after reading 1,000 messages from the source and after each five-second Real-time Flush Latency interval. If you configure the session to use target-based commits, the Integration Service automatically runs the session using source-based commits. Also, it only commits messages to the target based on the Real-time Flush Latency interval. It does not commit messages to the target based on the commit interval. When you run the session, the Integration Service begins to read messages from the source. Once messages enter the source, the Real-time Flush Latency interval begins. At the end of each five-second Real-time Flush Latency interval, the Integration Service commits all messages read from the source. If the Integration Service reads 1,000 messages before the end of a five second interval, it issues a source-based commit. It issues another commit when it reaches the Flush Latency interval. For more information about commit types and intervals, see the PowerCenter Workflow Administration Guide. When you use the Real-time Flush Latency session condition, the following limitations apply: The pipeline cannot contain Aggregator, Rank, Sorter, or AEP transformations. The mapping cannot contain a flat file or XML target definition. The session cannot contain partitioning types other than pass-through at all partitioning points. The Integration Service ignores the Real-time Flush Latency session condition when you run a session in debug mode. Idle Time Use the Idle Time session condition to indicate how many seconds the Integration Service waits when no messages arrive before it stops reading from the queue. For example, if you enter 30 for Idle Time, the Integration Service waits 30 seconds after reading from queue. If no new messages arrive in MSMQ within 30 seconds, the Integration Service stops reading from the queue. Removing Messages from the Source Queue The Remove Message on Read option determines whether the Integration Service removes messages from the queue after reading. If the Remove Message on Read option is disabled, the Integration Service does not remove previously read messages from the queue. If there are multiple partitions in a pipeline, each partition may read duplicate messages. 12 Chapter 3: Creating and Configuring MSMQ Workflows

21 Configuring Delivery Options When you want to write messages to an MSMQ target queue during a PowerCenter session, you can use one of the following delivery options: Recoverable Messaging. MSMQ stores messages on disk until it writes them to an MSMQ target queue. MSMQ stores the messages on the disk of each machine through which MSMQ routes the messages. Express Messaging. MSMQ stores messages in memory until it writes them to an MSMQ target queue. Using Recoverable Messaging You can configure a PowerCenter session to use recoverable messaging. During the session, the Integration Service extracts messages from a source and transforms the data according to the mapping logic. It then writes the messages to an MSMQ intermediate queue. MSMQ stores the messages on disk and writes them to the target queue. Once MSMQ delivers messages to the target queue, it removes sent messages from disk at regular intervals. If the PowerCenter session fails during delivery, MSMQ can recover all messages that the Integration Service sent to the intermediate queue but did not reach the target queue. However, MSMQ cannot recover messages that the Integration Service did not send to the intermediate queue. If the target queue is transactional, you must use recoverable messaging. If you configure the PowerCenter session to use express messaging, MSMQ automatically delivers the messages using recoverable messaging. Using Express Messaging You can configure a PowerCenter session to use express messaging. During the session, the Integration Service extracts messages from a source and transforms the data according to the mapping logic. It then writes the messages to an MSMQ intermediate queue. MSMQ stores the messages in memory until it delivers them to the target queue. If the PowerCenter session fails during delivery, MSMQ cannot recover any messages that did not reach the target queue. If the target queue is non-transactional, you can use recoverable or express messaging. Pipeline Partitioning You can increase the number of partitions in a pipeline to improve session performance. Increasing the number of partitions allows the Integration Service to create multiple connections to sources and process partitions of sources and targets concurrently. When you create a session in a workflow, the Workflow Manager validates each pipeline in the mapping for partitioning. You can specify multiple partitions in a pipeline if the Integration Service can maintain data consistency when it processes the partitioned data. Working with MSMQ Workflows 13

22 Table 3-1 describes the partition types for partition points in MSMQ mappings: Table 3-1. Partition Types for Partition Points in MSMQ Mappings Partition Point Application Source Qualifier for MSMQ sources MSMQ target Partition Type Pass-through All For more information about partitioning and a list of all partitioning restrictions, see the PowerCenter Workflow Administration Guide. 14 Chapter 3: Creating and Configuring MSMQ Workflows

23 Configuring Session Properties Use the following procedure to configure MSMQ session properties. To configure session properties: 1. In the Workflow Manager, connect to a repository. 2. In the Task Developer, double-click an MSMQ session to open the session properties. 3. From the Connections settings on the Mapping tab (Sources node), select an MSMQ queue connection for Application Source Qualifiers connected to MSMQ sources. 4. From the Connections settings on the Mapping tab (Targets node), select an MSMQ queue connection for each MSMQ target. 5. From the General Options on the Properties tab, select the commit type. 6. Optionally, edit the commit interval. 7. Select the Mappings tab. 8. From the Properties settings on the Mapping tab (Sources node), enter a value for Message Count of 1 or greater. If you enter a value of -1, the session does not complete. You must abort the session. 9. Optionally, enter a value for Real-time Flush Latency. Configuring Session Properties 15

24 10. Optionally, enter a value for Idle Time. 11. Optionally, select the Remove Message on Read option. 12. Optionally, select Deliver Messages in Recoverable Mode to use MSMQ recoverable messaging. Or, clear Deliver Messages in Recoverable Mode to use MSMQ express messaging. 13. Click OK to exit the session properties. 16 Chapter 3: Creating and Configuring MSMQ Workflows

25 Running MSMQ Workflows During a session that reads messages from an MSMQ source, the Integration Service reads messages from a source queue. If you enable the Remove Message on Read option, the MSMQ server deletes the messages from the queue once the Integration Service reads them. If a mapping with an MSMQ target definition contains an Update Strategy transformation, the Integration Service only writes rows flagged as DD_INSERT to the target. It discards all other rows. Running MSMQ Workflows 17

26 18 Chapter 3: Creating and Configuring MSMQ Workflows

27 Index D datatypes (MSMQ) MSG_BODY field 6 MSMQ 6 transformation 6 delivery options (MSMQ) description 2 overview 13 recovery 16 E express messaging (MSMQ) description 2, 13 G grid (MSMQ) restriction 11 H high availability (MSMQ) restriction 11 I Idle Time (MSMQ) configuring 16 description 12 Integration Service (MSMQ) See also PowerCenter Administrator Guide configuring 10 M Message Count (MSMQ) configuring 15 description 11 high availability 11 MSG_BODY field (MSMQ) datatypes 6 MSMQ sessions (MSMQ) overview 10 MSMQ workflows (MSMQ) overview 10 N non-transactional queues (MSMQ) delivery options 3 19

28 P pipeline partitioning (MSMQ) description 13 PowerCenter Connect for MSMQ (MSMQ) overview 2 Q queue connections (MSMQ) See PowerCenter Workflow Administration Guide queues (MSMQ) removing source messages 12 R Real-time Flush Latency (MSMQ) configuring 15 description 11 recoverable messaging (MSMQ) description 2, 13 S session properties (MSMQ) configuring 15 source definitions (MSMQ) creating 7 MSG_BODY field 6 overview 6 source queues (MSMQ) removing messages 12 T target definitions (MSMQ) creating 7 overview 6 transactional queues (MSMQ) delivery options 3 W workflows (MSMQ) running Index