Debugging CSQJ160I Long-Running Unit of Work found Message

Transcription

1 IBM Software Group Debugging CSQJ160I Long-Running Unit of Work found Message Lauranette Wheeler WebSphere Support Technical Exchange

2 Agenda The purpose of this presentation is to identify reasons for long-running units of work, to provide hints on identifying resources associated with a long-running unit of work, and to recommend ways to prevent long-running units of work. WebSphere Support Technical Exchange 2

3 What is a Unit of Work (UOW)? Processing a set of updates between two syncpoints. Syncpoint - logical point for synchronizing data changes. Several MQGET and MQPUT calls can be part of a single unit of work. WebSphere Support Technical Exchange 3

4 What are logs? Active log - records all significant events for persistent messages and queue manager objects. Each log record is identified by a relative byte address (RBA), which is the offset of the record from the beginning of the log. Archive log - MQ copies the contents of the active log into an archive log. This process is called offloading. WebSphere Support Technical Exchange 4

5 What is a queue manager checkpoint? Records representing persistent message updates are first written to the log, then the queue is updated. Logs can become more up-to-date than the queue. A checkpoint is a point in time when the record described in the log is the same as the record in the queue. Checkpoint contains the beginning RBA for all units of work active at time of the checkpoint. WebSphere Support Technical Exchange 5

6 What is a Long-Running Unit of Work? A unit of work that has been active for more than three queue manager checkpoints. The changes have not been committed, which informs the queue manager that processing for that unit of work is complete. WebSphere Support Technical Exchange 6

7 Problems caused by Long-Running UOWs MQ V531, the origin RBA of the unit of work could be outside the scope of the current active logs. Restart could take a long time, because all the log records relating to the unit of work must be read, including those in the archive logs. MQ V6, when a unit of work is considered to be long, a representation of each log record is written further down in the active log. This is known as log shunting, and can help eliminate long queue manager restarts. WebSphere Support Technical Exchange 7

8 Problems caused by Long-Running UOWs The application may experience RC = 2024, MQRC_SYNCPOINT_LIMIT_REACHED. Reached the maximum number of uncommitted messages within a unit of work. WebSphere Support Technical Exchange 8

9 Problems caused by Long-Running UOWs Messages are not made available to other programs until the unit of work is committed. May cause unexpected RC=2033, MQRC_NO_MSG_AVAILABLE. The queue manager counts both committed and uncommitted messages when it assesses whether the conditions for a trigger event exist. WebSphere Support Technical Exchange 9

10 Problems caused by Long-Running UOWs Can cause storage problems, because of lock and storage pools needed for recovery. Recommend 10 messages per unit of work for long messages. WebSphere Support Technical Exchange 10

11 Identifying Long-Running Units of Work To help identify long-running units of work: MQ V531, issues message CSQJ160I if a longrunning unit of work is detected during active log offload. MQ V6, long-running units of work can be identified by the presence of message CSQR026I. This message indicates that a unit of work has existed for more than three queue manager checkpoints, and its log records have been shunted further down in the active log. WebSphere Support Technical Exchange 11

12 Identifying Long-Running Units of Work DISPLAY CHSTATUS displays the status of a specified channel. DISPLAY QSTATUS displays the status of a specified queue. DISPLAY THREAD displays information about active threads. DISPLAY CONN displays information about connections to queue manager and their associated unit of work. (MQ V6) WebSphere Support Technical Exchange 12

13 Identifying Long-Running Units of Work Active units of work To display active units of work in MQ V531 DISPLAY THREAD TYPE(ACTIVE) DISPLAY QSTATUS TYPE(HANDLE) WebSphere Support Technical Exchange 13

14 Identifying Long-Running Units of Work Active units of work To display active units of work in MQ V600 DISPLAY CONN(*) WHERE(UOWSTATE EQ ACTIVE). DISPLAY CONN(*) WHERE(QMURID EQ qmurid) where qmurid is the Relative Byte Address (RBA). WebSphere Support Technical Exchange 14

15 IMS Example CSQR026I Long-running UOW shunted to RBA=1E169D3040CF, URID=1E CEA DISPLAY CONN(*) WHERE (QMURID EQ 1E169D3040CF) CSQM201I DISPLAY CONN DETAILS. URTYPE(QMGR) APPLTAG(CSQ2QCP1) WebSphere Support Technical Exchange 15

16 Debugging Approach CSQR026I Long-running UOW shunted to RBA=1E169D3040CF, URID=1E CEA Run CSQ1LOGP specify URID. WebSphere Support Technical Exchange 16

17 Debugging Approach 01DFF140E528 URID(1E CEA) RM(RECOVERY) TYPE( START SHUNT ) SHUNTED FROM(1E CEA) 242.CSQ2QCP1SY 01DFF140E592 URID(1E CEA) RM(DATA) LRID( C703) TYPE( UNDO REDO ) 01DFF140E5C2 URID(1E CEA) RM(DATA) LRID( ) TYPE( UNDO REDO ) SUBTYPE( INSERT ) RECORD LENGTH(039C) MQXMIT * 1E CEA 01DFF140E642 URID(011E CEA) RM(RECOVERY) TYPE( END SHUNT ) F2F4F24B C3E2D8F2 D8C3D7F1 E2E8E2D6 D7D * 242.CSQ2QCP1SYSOPR WebSphere Support Technical Exchange 17

18 Debugging Approach DUMP of MSTR and CHIN. Using the LRID of the Insert Record from the CSQ1LOGP output, Support can find buffers containing messages associated with the UOW. DFS065 18:18:34 TRAN/LTERM STOPPED WebSphere Support Technical Exchange 18

19 Debugging Approach APPLTAG(CSQ2QCP1) MQ module associated with IMS reply-to-queues. IMS fix PK37783/UK23110 changed the way OTMA handles DFSxxx message. Apply fix for IMS PK45682/UK Restart IMS transaction to clear the CSQR026I message. WebSphere Support Technical Exchange 19

20 SVRCONN Example CSQJ160I CSQJLRUR LONG-RUNNING UOW FOUND,URID=0320CE10D970 Connection name=mq01chin Find the URID in DISPLAY THREAD output, CSQV402I ACTIVE THREADS NAME S T REQ THREAD-XREF USERID ASID URID MQ01CHIN T S E7E75C5C28758A30 MQ01CHIN 009C 0320CE10D970 WebSphere Support Technical Exchange 20

21 Debugging Approach Dump of QMGR and CHIN. Using information from DISPLAY THREAD, Support can find the channel name and Conn name associated with the unit of work. CSQV402I ACTIVE THREADS NAME S T REQ THREAD-XREF USERID ASID URID MQ01CHIN T S E7E75C5C28758A30 MQ01CHIN 009C 0320CE10D970 WebSphere Support Technical Exchange 21

22 Debugging Approach MQ V6, DISPLAY CONN command will provide information previously found in the dumps. WebSphere Support Technical Exchange 22

23 Debugging Approach CSQM201I +RTPH CSQMDRTC DISPLAY CONN DETAILS 825 CONN(C137E275D ) EXTCONN(C3E2D8C3D9E3D7C ) TYPE(CONN) CONNOPTS( MQCNO_STANDARD_BINDING ) UOWLOGDA( ) UOWLOGTI( ) UOWSTDA( ) UOWSTTI( ) UOWSTATE(NONE) URTYPE(QMGR) USERID(MQMTASK) APPLTAG(RTPHCHIN) ASID(0342) APPLTYPE(CHINIT) CHANNEL(SYSTEM.ADMIN.SVRCONN) CONNAME( ) WebSphere Support Technical Exchange 23

24 Debugging Approach Stop the SVRCONN channel with MODE(FORCE). Recycle the CHIN. WebSphere Support Technical Exchange 24

25 Debugging Approach To prevent manual intervention: Set TCPKEEP=YES SVRCONN definition, specify in the KAINT parameter. MQ V6, HBINIT and DISCONNECT INTERVAL was added for SVRCONN channels. WebSphere Support Technical Exchange 25

26 XA Client Example CSQJ160I CSQJLRUR LONG-RUNNING UOW FOUND, URID=04776E CONNECTION NAME=MQ01CHIN WebSphere Application Server XA Client interfacing with MQ and IMS, in a Shared Queue environment. WebSphere Support Technical Exchange 26

27 Debugging Approach Run CSQ1LOGP specify URID E URID(04776E707558) RM(RECOVERY) TYPE( START UR ) 04776E7076A6 URID(04776E707558) RM(D4) TYPE( XA_END ) 4776E707A12 URID(04776E707558) RM(RECOVERY) TYPE( END COMMIT1) XID obtained from CSQ1LOGP XA_END log record or from the MSTR/CHIN DUMP. WebSphere Support Technical Exchange 27

28 Debugging Approach WebSphere Appl Server and RRS needs the XA Client ID to determine the state of the unit of work DUMP of WebSphere Servant and Control Region com/ibm/ws390/tx/nativeglobaltransactioncontext.commitonephase. DUMP of RRS IMS transaction hung in back-out state. WebSphere Support Technical Exchange 28

29 Debugging Approach Recycle the WebSphere Application Server to clear the XA Client. WebSphere Support Technical Exchange 29

30 Batch Job Example CSQR026I Long-running UOW shunted to RBA=1E169D3041FC, URID=1E DBE WebSphere Support Technical Exchange 30

31 Debugging Approach DISPLAY CONN(*) WHERE (QMURID EQ 1E169D3041FC) CSQM201I +RTPH CSQMDRTC DISPLAY CONN DETAILS 826 CONN(C1386BC52E2E0001) EXTCONN(C3E2D8C3D9E3D7C ) TYPE(CONN) CONNOPTS( MQCNO_STANDARD_BINDING ) UOWLOGDA( ) UOWLOGTI( ) UOWSTDA( ) UOWSTTI( ) UOWSTATE(ACTIVE) QMURID(1E169D3041FC) URTYPE(QMGR) USERID(LAURANE) APPLTAG(LAURANEP) ASID(03DE) APPLTYPE(BATCH) END CONN DETAILS WebSphere Support Technical Exchange 31

32 Debugging Approach Cancel the Batch job, Updates will be backed out. If unit of work was successfully shunted, back outs will not require archive logs. If MQ V531, or not shunted, archive logs containing the beginning of the unit of work must be available to prevent a Queue Manager ABEND Reply y to CSQR021D message on qmgr restart MQ V6, PK24888 must be applied to ensure CSQR021D is issued, during Queue manager restarts. WebSphere Support Technical Exchange 32

33 Preventing Long- Running units of work Ensure application commits frequently enough. Ensure the SVRCONN channel is still active. Ensure the CICS transaction is not hung. Ensure the IMS transaction is not stopped. Ensure batch application is not swapped out. WebSphere Support Technical Exchange 33

34 Required Documentation MQ MSTR and CHIN Joblogs. Dump MQ MSTR and CHIN and dataspace. CSQ1LOGP for specific URID. Dump WebSphere Controller and Servant Region. Dump RRS. WebSphere Support Technical Exchange 34

35 Additional WebSphere Product Resources Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: Learn about other upcoming webcasts, conferences and events: Join the Global WebSphere User Group Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: View a Flash replay with step-by-step instructions for using the Electronic Service Request (ESR) tool for submitting problems electronically: Sign up to receive weekly technical My support s: WebSphere Support Technical Exchange 35

36 Questions and Answers WebSphere Support Technical Exchange 36