Troubleshooting SCA Problems in WebSphere Process Server Open Mic

Transcription

1 IBM Software Group Troubleshooting SCA Problems in WebSphere Process Server Open Mic 4 January 2011 WebSphere Support Technical Exchange

2 Agenda Introduce the panel of experts Introduce Troubleshooting SCA Problems in WebSphere Process Server Answer questions submitted by (12 questions) Open telephone lines for questions Summarize highlights WebSphere Support Technical Exchange 2

3 Panel of Experts Panelist Pamela Fong Jeff Brent Alex Wood Paul Slater Lalitha Chandran Role at IBM Senior Software Engineer WebSphere Process Server Development Senior Engineer WebSphere Process Server SWAT Software Engineer WPS/WESB Development Level 3 Support Engineer Service Integration Bus Software Engineer WebSphere Process Server Technical Support Team WebSphere Support Technical Exchange 3

4 Introduction In this Open Mic we will discuss the following topics: 1. Frequently reported SCA problems 2. SCA behaviors like transactions, error handling and recovery 3. Best practices related to application design 4. Service Integration Bus 5. Specific queries related to MQ import/exports Topics do not include: 1. Business Process Choreographer and Human Tasks 2. Complex performance issues WebSphere Support Technical Exchange 4

5 Question 1 I have been told sync-over-async is an anti-pattern, how do I ensure I am not using it? WebSphere Support Technical Exchange 5

6 Answer to Question 1 1. Static analysis Check for components that have an asynchronous implementation, ie: Java Component (POJO) that implements the ServiceAsyncImpl interface Long running business process Human tasks JMS import MQ import Ensure the above components are invoked in a synchronous manner. To find the invocation style read here. WebSphere Support Technical Exchange 6

7 Answer to Question 1 In v7, you will find a warning in your SystemOut.log ac Core W CWSCA2011W: Service Component Architecture (SCA) is switching a synchronous call to an asynchronous call. Request will be sent in a new local transaction. Default timeout is milliseconds. WebSphere Support Technical Exchange 7

8 Question 2 We see a transaction that we know will not finish and as a result keep seeing transaction XA errors in the logs. We don't see any messages in our SIB queues and there are no failed events. We have been told not to delete the tranlogs. If that is the case, what is the best way to delete this transaction? WebSphere Support Technical Exchange 8

9 Answer to Question 2 1. Administrative console This information center link outlines the process. 2. Scripts Scripts can be used to delete such transactions as well. The script is available as part of the recovery redbook. The script can be downloaded here. WebSphere Support Technical Exchange 9

10 Question 3 What settings can I follow to ensure I don't end up with huge depth of messages in my SIB callback destinations? WebSphere Support Technical Exchange 10

11 Answer to Question 3 1. Make use of the SCA Response Expiration Qualifier on the reference calling the JMS or MQ Import. Messages on the SIB callback destinations used by the Import will expire and be removed in the time specified by this qualifier if set. 2. Avoid mixing calls to services with long and short response times via the same JMS or MQ import. Reponses to short running services will be particularly impeded when retrieving the correlation message from the SIB callback queue as SIB will have to sort through a large stack of older messages to find the correct one. WebSphere Support Technical Exchange 11

12 Question 4 What are the key parameters that need to be tuned for better performance with asynchronous SCA invocation in WebSphere Process Server? WebSphere Support Technical Exchange 12

13 Answer to Question 4 maxconcurrency Thread pool Performance Redbook WebSphere Support Technical Exchange 13

14 Question 5 Is there a big difference in terms of performance overhead in an asynchronous inter-module interaction vs asynchronous intra-module interaction. Should the former be avoided? WebSphere Support Technical Exchange 14

15 Answer to Question 5 No big difference. Consider the below scenario: Module1 WS export -> mediation -> SCA import Module 2 SCA export -> long running bpel WebSphere Support Technical Exchange 15

16 Question 6 I want to understand which queues are created in the context of SCA modules, and what is their purpose? WebSphere Support Technical Exchange 16

17 Answer to Question 6 SCA core asynchronous messaging: 1. A module queue 2. An external queue for each export and import SCA binding 3. An internal queue for each export and import SCA binding 4. A component queue for each SCA component In recent releases, category 3 and 4 have been eliminated. The queues in 3 and 4 will be created by the system if event sequencing or Store and Forward qualifiers are used. WebSphere Support Technical Exchange 17

18 Answer to Question 6 Specific SCA binding implementation: JMS binding Request and Response queue per JMS Export and Import binding. Callback queue to store correlation data. WS binding, if SOAP/JMS is desired, a JMS queue will be created for each binding. WebSphere Support Technical Exchange 18

19 Question 7 How do I interpret messages that are parked in queue and what do I do with them? WebSphere Support Technical Exchange 19

20 Answer to Question 7 SCA module queue Causes: When the message arrival rate is higher than the processing rate Error messages Expiry reports SCA internal system messages should not be manipulated by the user. Failed Event Manager To clean up stored messages or expiry reports, please contact IBM support WebSphere Support Technical Exchange 20

21 Question 8 What level of control do I have over MQMD header fields in an outgoing message from an MQ import. What fields can I set and how can I set them? WebSphere Support Technical Exchange 21

22 Answer to Question 8 Most MQMD fields can be propagated by setting them statically or dynamically using a mediation flow with a mapping primitive or MQ header setter primitive. Some require additional configuration to be set and some fields cannot be propagated as the MQ APIs do not allow them to set. There is a series of Developer Works articles which describe this in detail, read here. In v7.0 user context fields can be set with the correct destination configuration. WebSphere Support Technical Exchange 22

23 Answer to Question 8 Fields that require specific MQ import configuration to be propagated: MsgID - Request Message ID options must be set to copy from SCA message. MsgType - Set message type to MQMT_DATAGRAM or MQMT_REQUEST for request-response operation must be unticked. ReplyToQ - Override reply to queue of request message must be unticked. ReplyToQMgr - Override reply to queue of request message must be unticked. Field that cannot be propagated at all: BackoutCount, AccountingToken, PutDate, PutTime, Offset, OriginalLength WebSphere Support Technical Exchange 23

24 Question 9 We have a long running process (Module A) making synchronous calls to a SCA Import (Module B). We are aware of the fact that it is not a good practice to make synchronous calls from LR BPEL. However we would like to know the expected behavior for the invocation activity in case the related transaction timeout expires. We observe that if Module B provides a response after the transaction timeout of the invoke activity, then the BPEL successfully navigates to the next activity. This will happen after a small delay. No exceptions are raised for the BPEL to catch and there are no double messages towards Module B. Is this the expected behavior? WebSphere Support Technical Exchange 24

25 Answer to Question 9 Not an expected behavior Excepted behavior: The synchronous thread of execution should have passed through an qualifier boundary. At this point, the fact that the transaction was mark for time-out would have cause a ServiceRuntimeException to be raised and returned to the Long Running Process. Raise a PMR, capture trace with SCA.*=all:Transaction=all:com.ibm.bpe.*=all WebSphere Support Technical Exchange 25

26 Question 10 We have a long running process (Module A) making asynchronous calls to a SCA Import (Module B). If Module B provides the response after the transaction timeout of the invoke activity, we observe that Module B is unable to sent back the response to Module A. It seems to us that the SCA is having problems getting a connection to SIBus as a consequence of the transaction having timed out and the SCA code is retrying the connections for a long time before eventually throwing a CWSCA2001E error (SCA unable to establish connection to SIB/ME in xxx secs). The call/initial request afterwards is re-tried resulting in duplicate messages towards Module B. Is this the expected behavior? If so, what would you recommend us to do to overcome this issue? WebSphere Support Technical Exchange 26

27 Answer to Question 10 Transaction Time out will not directly cause the system to fail in getting a connection to the SI Bus. Not a duplicate message, but likely the re-delivery of the same message. To read about built-in retries in WPS, read here. WebSphere Support Technical Exchange 27

28 Question 11 I have set a "Request expiration" qualifier. After the expiration time, does the request get moved to the Failed Event Manager or any other exception queues? WebSphere Support Technical Exchange 28

29 Answer to Question 11 Setting request expiration would implicitly impose an expiration on the response. No failed event will be generated in this case. When any messages in the system expires, they turn into expiry reports. These reports are picked up by the calling module on a best effort basis. WebSphere Support Technical Exchange 29

30 Question 12 We see a huge depth in one of the SIB queues and suspect it might be causing performance problems. What is the fastest way to deal with such an issue? WebSphere Support Technical Exchange 30

31 Answer to Question 12 Consume them! Or if no longer needed delete Problems: Impact on GC Retrieving messages with a selector Out of memory errors. Timeouts retrieving messages from callback queue WebSphere Support Technical Exchange 31

32 Answer to Question 12 Recommendations: Use high message threshold Do not look at the messages in administrative console or SIB explorer WebSphere Support Technical Exchange 32

33 Open Lines for Questions WebSphere Support Technical Exchange 33

34 We Want to Hear From You! Tell us about what you want to learn Suggestions for future topics Improvements and comments about our webcasts We want to hear everything you have to say! Please send your suggestions and comments to: WebSphere Support Technical Exchange 34

35 Summary WebSphere Support Technical Exchange 35

36 Additional WebSphere Product Resources Learn about upcoming WebSphere Support Technical Exchange webcasts, and access previously recorded presentations at: Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: Join the Global WebSphere Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically: Sign up to receive weekly technical My Notifications s: WebSphere Support Technical Exchange 36