DataPower-MQ Integration Deep Dive

Transcription

1 DataPower-MQ Integration Deep Dive Robin Wiley (Robin Wiley Training) Your Presenter: Robin Wiley Senior Instructor, IBM Messaging Products MQ Administration & Application Development DataPower Administration & Service Development Integration Bus Administration & Message Flow Development IBM Certified: MQ Administrator MQ Solution Designer DataPower Solution Implementer Over 40 years IT Industry Experience Network Integration Managing Software Development Experience: 20 years with MQ; 10 with DataPower Technical Architecture, Project Management, Installation, Training Effective Instructor Over 35 years experience in corporate training and adult education Brings magic to the classroom (Member, Academy of Magical Arts) 2 1

2 Agenda Focus: MQ 9 & DP 7.6 Topics: Queue Manager Object MQ Front Side Handler MQ URL Message Headers Error Handling Transaction Management 3 Queue Manager Object 2

3 Queue Manager Object 5 Queue Manager Object Host Name (IPv4) address:port :1414 address(port) (1414) address default port 1414 Host Name (IPv6) [address]:port [2202::148:248]:1414 address(port) 2202::148:248(1414) address 2202::148:248 default port 1414 Host Name (DNS) hostname:port myserver:1414 hostname(port) myserver(1414) hostname myserver default port

4 Queue Manager Object 7 Queue Manager Object 8 4

5 Queue Manager Object Channel Name SVRCONN name as defined on the Queue Manager Channel Heartbeat (seconds) Approximate time between heartbeat flows on the channel 0 = no heartbeat flow exchanged Does not set the heartbeat on the channel Negotiates heartbeat value with channel definition -- greater is used Cache Timeout (seconds) How long the appliance keeps alive a dynamic connection in the connection cache Must be greater than the negotiated heartbeat interval but less than the Queue Manager keep alive interval (defined on the host) 9 Queue Manager Object User Name Supplied to Queue Manager at connection Maximum 12 characters Alternate User Enables or disables MQOD.AlternateUserId Off = use Message Descriptor User Identifier for queue authorization On (default) = use Object Descriptor Alternate User Identifier for queue authorization (need to create Object Descriptor) XML Manager Recommend using a dedicated XML Manager per QM object 10 5

6 Queue Manager Object Maximum Message Size Limit the size of the MQ message payload Units of Work Controls syncpoint processing (transaction management) Affects MQ Front Side Handlers only Two values: 0 or 1 Units of Work = 0 No syncpoint control Front Side Handler MQGET is immediate and irrevocable If error, message integrity is responsibility of DataPower Units of Work = 1 Enables syncpoint control Front Side Handler MQGET has an implied MQGMO_SYNCPOINT If error, message is rolled back via implied MQBACK 11 Queue Manager Object Automatic Backout Enable the use of BOTHRESH and BOQUEUE Backout Threshold Used if BOTHRESH not set on input queue Backout Queue Name Used if BOQUEUE not set on input queue 12 6

7 Queue Manager Connections Tab 13 Queue Manager Connections Tab Total Connection Limit: Connection pool size of the QM object Default value is 250 Can tune the total connection limit for performance Initial Connections: Maximum simultaneous initial connection requests Default value is 1 If too high, may flood the Queue Manager Local Address Local address for outbound connections Specific local interface and port For a range of ports, use (1414,1420) or x.x.x.x(1414,1420) 14 7

8 Queue Manager Connections Tab Automatic Retry On: Attempt to reconnect to the Queue Manager if connection dropped Off: Disable and re-enable the Queue Manager object to reconnect Retry Interval Time interval between attempts to retry failed connections Recommend 10 to 15 seconds (default is 1 second) Low value can spike CPU and memory usage Retry Attempts Number of attempts to retry the failed connections After threshold reached, the Long Retry Interval is used instead Default value of 0 (zero) disables Long Retry Interval Recommend non-zero value 15 Queue Manager Connections Tab Long Retry Interval Interval in seconds to retry connection after Retry Attempts Recommend 600 seconds (default is 1800) Must be greater than the Retry Interval (if less, ignored) Reporting Interval How often to log retries (seconds) Suppresses duplicate log entries Recommend setting this the same as Retry Interval 16 8

9 Queue Manager Connections Tab Sharing Conversations Maximum conversations sharing single TCP/IP connection Value is negotiated between SVRCONN SHARECNV setting and DataPower (lower value takes effect) Value of 0 means NO Shared Conversations Suppresses MQ V7+ features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Value of 1 means NO Shared Conversations Allows MQ V7+ features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Value > 1 means Shared Conversations permitted Allows MQ V7+ features (Administrator stop-quiesce; Heartbeating; Read ahead; Client asynchronous consume) Can impact performance of clients (unless V7+ features used) 17 Queue Manager Connections Tab 18 9

10 Queue Manager Connections Tab SSL Client Type: Client Profile Select the SSL Client Profile object to use from the pick list Must be used for connection to a z/os host SSL Client Type: Proxy Profile Deprecated recommend using Client Profile instead Select the SSL Proxy Profile object to use from the pick list SSL Key Repository Select the location of the key database file SSL Version 3 Support Permit SSL v3 or not SSL Cipher Specification Choose the Cipher Spec to use 19 Client Profile Object 10

11 Client Profile Main Tab 21 Client Profile Main Tab Protocols: Choose the protocols to be supported Ciphers: Choose the ciphers to be supported Use SNI: Send the Server Name Indication (SNI) TLS extension in the client hello message Permit connections to insecure SSL servers: Allow connection to potentially vulnerable servers Enable compression: Allow SSL compression Not recommended can allow CRIME or BREACH attacks Identification credentials: If mutual authentication requested by server Validate server certificate: Check the credentials presented by the server (Default: On) Validation credentials: Crypto Validation Credential object used for server certificate validation 22 11

12 Client Profile Session Caching Tab 23 Client Profile Session Caching Tab Enable session caching: Allow SSL session caching Session Cache Timeout: How long before cache is flushed Maximum: 86,400 seconds (24 hours) Session Cache Size: How many entries to be cached Maximum: 500,

13 Client Profile Advanced Tab Elliptical Curves: Build a list of acceptable Elliptical Curve algorithms (RFC 4492) Allows equivalent security to current cryptosystems like RSA but smaller key size Favored for mobile technology 25 Back to the Queue Manager Object 13

14 Queue Manager CCSI Tab Coded Character Set ID Presented to the SVRCONN channel during connection Same as setting MQCCSID Environment Variable Convert Input On: Ask the Queue Manager to convert messages using the CCSID (default) Off: No conversion 27 Queue Manager MQCSP Tab 28 14

15 Queue Manager MQCSP Tab Defines the MQCSP Data Structure Simulates passing MQCSP using MQCONNX MQCSP User ID Sent via MQCSP if present If blank (and Password Alias set to none ), no MQCSP is sent MQCSP Password Alias Points to the encrypted password stored within DataPower Password is sent in clear text in MQCSP after retrieval from the Alias 29 MQ Front Side Handler Object 15

16 MQ Front Side Handler 31 MQ Front Side Handler Get Queue Name of queue to get messages from Mandatory, unless Pub/Sub being used Put Queue Optional, because: May be "one-way" messaging (fire and forget) May be using Reply-To Queue May be dynamically allocated by Response Rule code The number of concurrent MQ conversations: Number of parallel active and pending MQGETs for the Get Queue Recommend value of 1 (in high throughput situations, may use up to 5) Regardless of this setting, multiple FSH threads will still use multiple connections If greater than 1, monitor Queue Manager for workload If greater than 1, Backout Threshold must be this value plus 1 If using message ordering (MQGMO_LOGICAL_ORDER), set it to

17 MQ Front Side Handler Get Message Options Allows the use of any MQGMO_Options parameters Overrides any specific parameters set elsewhere Default 1 (MQGMO_WAIT) Polling Interval How long to wait on an empty queue (seconds) Equivalent to Wait Interval with conventional MQ applications Low value increases network traffic Recommend default of 30 Retrieve Backout Settings Get BOTHRESH and BOQUEUE from the Get Queue Issues MQINQ before every MQGET potential performance hit Only relevant if queue parameters were set by MQ administrator Recommend set "off" and use Queue Manager Object settings 33 Some MQGMO Options 0 MQGMO_NONE 1 MQGMO_WAIT 2 MQGMO_SYNCPOINT 4 MQGMO_NO_SYNCPOINT 8 MQGMO_SET_SIGNAL 16 MQGMO_BROWSE_FIRST 32 MQGMO_BROWSE_NEXT 64 MQGMO_ACCEPT_TRUNCATED_MSG 128 MQGMO_MARK_SKIP_BACKOUT 256 MQGMO_MSG_UNDER_CURSOR 512 MQGMO_LOCK 1024 MQGMO_UNLOCK 2048 MQGMO_BROWSE_MSG_UNDER_CURSOR 4096 MQGMO_SYNCPOINT_IF_PERSISTENT 8192 MQGMO_FAIL_IF_QUIESCING MQGMO_CONVERT MQGMO_LOGICAL_ORDER MQGMO_COMPLETE_MSG MQGMO_ALL_MSGS_AVAILABLE MQGMO_ALL_SEGMENTS_AVAILABLE 34 17

18 MQ Front Side Handler Use Queue Manager in URL Defines the behavior of var://service/url-in when a QM Group is specified If on, the variable returns the name of the chosen Queue Manager If off, the variable returns the name of the Queue Manager Group Default off CCSI Sets the CCSID in the MQ Message Descriptor If blank or zero, default is ISO (latin-1) For MQCCSI_EMBEDDED enter For MQCCSI_INHERIT enter MQ Front Side Handler 36 18

19 MQ Front Side Handler Subscribe Topic String Pub/Sub topic string for subscription If Get Queue also defined, this is ignored Subscription Name Used to establish or resume a Durable Subscription Publish Topic String Pub/Sub topic string for response publication If Put Queue also defined, this is ignored Parse Properties Extracts MQ V7 (and above) Message Properties into Node Set Minor overhead, so leave off unless needed 37 MQ Front Side Handler Selector Allows selective retrieval of messages based on properties Forces sequential search of queue so may be inefficient Exclude Message Headers Strip off selected MQ header types Header to extract Content-Type Can obtain Content-Type from MQMD RFH RFH

20 MQ Front Side Handler Async Put Put message to queue without waiting for a response Do not use when Queue Manager units-of-work is 1 Recommend only use where performance is an issue Batch Size Number of messages to handled as a singe commit or rollback operation Recommend leave this as zero each message is a separate transaction 39 MQ Back-End URL 20

21 MQ Back-End URL General Syntax: dpmq://mqqueuemanagerobject/uri?<parameters> RequestQueue=requestQueueName Name of the backend MQ request queue ReplyQueue=replyQueueName Name of the backend MQ reply queue Sync=true Issues a Commit call when a message is put on Request Queue Transactional=true Begins a new transaction when getting a message from the ReplyQueue GMO=optionsValue MQGMO_Options parameter value when getting from Reply Queue PMO=optionsValue MQPMO_Options parameter value when putting to Request Queue 41 MQ Back-End URL ParseHeaders={true false} Specifies whether to parse and strip headers from message SetReplyTo={true false} Specifies whether to set MD ReplyToQ during Put AsyncPut={true false} Specifies whether to use Asynchronous Put Only valid when using MQ V7 (and above) Browse={first next current} Controls non-destructive retrieval of messages 42 21

22 MQ Back-End URL ContentTypeHeader=header Which MQ header identifies the content type of the message ContentTypeXPath=expression XPath expression to extract the content type of message ParseProperties={on off} Parse message properties PublishTopicString=string and SubscribeTopicString=string Specifies topic to use with Pub/Sub (MQ V7 and above) SubscriptionName=string Specifies name for a durable subscription (MQ V7 and above) Selector=expression SQL92 style query filtering on message properties Performance hit 43 Multi-Protocol Gateway Parameter Settings 22

23 MPGW Headers Tab MQ Headers 45 MPGW Headers Tab MQ Headers Using Header Injection (Header Tab) Example setting Format and Persistence: Direction: Front (for FSH MQPUT) Direction: Back (for Backend MQPUT) Header Name: MQMD Header Value: <MQMD><Format>MQSTR</Format><Persistence>1</Persistence></MQMD> 46 23

24 MQ Programmatic Control MQ Headers Programmatic Manipulation Using the Stylesheet method (page 1 of 2) 48 24

25 MQ Headers Programmatic Manipulation Using the Stylesheet method (page 2 of 2) 49 MQ Headers Programmatic Manipulation Context variable method to inject the MQMD header For the following code to work: Set Transform Action's OUTPUT context to "EVENTS" Set Result Action's INPUT context to "EVENTS" 50 25

26 MQ Headers Programmatic Manipulation JMS Headers as Message Properties Must set FSH Parse Properties to be on Must set Exclude RFH2 to be off Message Properties appear as MQMP header 51 MQ Headers Programmatic Manipulation JMS Headers as RFH2 Must set FSH Parse Properties to be off Must set Exclude RFH2 to be on Message Properties appear as MQRFH2 headers 52 26

27 MQ Error Handling MQ error handling example: 53 MQ Conversational Processing Backend application must copy MsgId to CorrelId DataPower Back-End retrieves reply using CorrelId MQPUT1 Simulation Create MQ Object Descriptor header with Queue Manager name in it Request Rule issues MQOPEN/MQPUT/MQCLOSE to back end Queue Manager ReplyToQ Usage If set, Response Rule sends message there ReplyToQmgr Usage Can be set to send to a different Queue Manager If destination is a Cluster, no need to supply ReplyToQmgr 54 27

28 MQ Conversational Processing XSL code snippet to set ReplyToQ and ReplyToQmgr in a Request Rule: 55 MQ Conversational Processing XSL code snippet to set ReplyToQ and ReplyToQmgr in a Response Rule: 56 28

29 Transactional Processing MQ Transactional Processing DataPower is a standard MQ Client It does NOT offer Extended Transactional Client functionality NO XA two-phase commit DataPower is considered an application by MQ Therefore, no inherent message integrity If the same Queue Manager at front and back: True message integrity Once and once-only delivery If different Queue Managers at front and back No possibility of two-phase commit Message integrity assured if DataPower configured properly Possibility of messages sent more than once 58 29

30 Scenario: Two Different Queue Managers No Transactional Control App 1 Units of Work = 0 Sync=false; Transactional=false App 2 FS.OUT FS.IN Queue Manager QM1 Queue Manager QM2 BE.OUT BE.IN DataPower Request Rule Response Rule 59 Scenario: Two Different Queue Managers Front Side Transactional Control only App 1 Units of Work = 1 Sync=false; Transactional=false App 2 FS.OUT FS.IN Queue Manager QM1 Queue Manager QM2 BE.OUT BE.IN DataPower Request Rule Response Rule 60 30

31 Scenario: Two Different Queue Managers Back End Transactional Control only App 1 Units of Work = 0 Sync=false; Transactional=true App 2 FS.OUT FS.IN Queue Manager QM1 Queue Manager QM2 BE.OUT BE.IN DataPower Request Rule Response Rule 61 Scenario: Two Different Queue Managers Front Side and Back End Transactional Control App 1 Units of Work = 1 Sync=false; Transactional=true App 2 FS.OUT FS.IN Queue Manager QM1 Queue Manager QM2 BE.OUT BE.IN DataPower Request Rule Response Rule 62 31

32 Scenario: One Queue Manager (Front & Back) No Transactional Control App 1 Units of Work = 0 Sync=false; Transactional=false App 2 FS.OUT FS.IN Queue Manager QM1 BE.OUT BE.IN DataPower Request Rule Response Rule 63 Scenario: One Queue Manager (Front & Back) Front Side Transactional Control only App 1 Units of Work = 1 Sync=false; Transactional=false App 2 FS.OUT FS.IN Queue Manager QM1 BE.OUT BE.IN DataPower Request Rule Response Rule 64 32

33 Scenario: One Queue Manager (Front & Back) Front Side Transactional Control only App 1 Units of Work = 1 Sync=true; Transactional=false App 2 FS.OUT FS.IN Queue Manager QM1 BE.OUT BE.IN DataPower Request Rule Response Rule 65 Scenario: One Queue Manager (Front & Back) Front Side and Back End Transactional Control App 1 Units of Work = 1 Sync=true; Transactional=true App 2 FS.OUT FS.IN Queue Manager QM1 BE.OUT BE.IN DataPower Request Rule Response Rule 66 33

34 Poison Message App 1 BOTHRESH = 3 BOQUEUE = FS.ERR FS.ERR FS.IN Units of Work = 1 Queue Manager QM1 DataPower FSH Request Rule 67 Poison Message App 1 BOTHRESH = 3 BOQUEUE = FS.ERR FS.ERR FS.IN Units of Work = 1 Queue Manager QM1 DataPower FSH Request Rule 68 34

35 Poison Message DataPower Knowledge Center Documentation om.ibm.dp.doc/retrieve-backout-setting_mqfsh.html If there are no backout settings, the backout function is disabled. 69 Questions & Answers 35

36 End of Session Thank You! Contact: Handouts: RobinWileyTraining.com/MQTC