Troubleshooting Communication in WebSphere MQ

Size: px

Start display at page:

Download "Troubleshooting Communication in WebSphere MQ"

Gordon McCormick
5 years ago
Views:

1 IBM Software Group Troubleshooting Communication in WebSphere MQ Rick Armstrong Advisory Software Engineer 27 October 2010 WebSphere Support Technical Exchange

2 Agenda TCP/IP networking concepts Tracing at the network level Channel connection overview Common problems involving network and related layers WebSphere Support Technical Exchange 2 of 39

3 TCP/IP network overview WMQ Client IP Cloud z/os Queue Manager WebSphere Support Technical Exchange 3 of 39

4 TCP/IP Sockets Overview Client Connect Allows the client to open a connection to a server's port Server Bind Associates a socket to a port Listen Allows clients to connect to port Accept Accepts a client connection attempt Client and Server Socket Allocates a socket to read and write Write Sends data on a TCP connection Read Receive data on a TCP connection Close Terminate a connection WebSphere Support Technical Exchange 4 of 39

5 TCP/IP socket overview Simple example of application socket calls client/server Client Application Server Application Socket function Connection state Network flow Connection state Socket function Socket Connect synsent SYN SYN/ACK ACK listen Socket Bind Listen established established Accept Write 100 Read bytes ACK bytes ACK +100 Write 100 Close fin_wait1 FIN fin_wait2 ACK close_wait FIN timewait ACK last ack Close WebSphere Support Technical Exchange 5 of 39

6 TCP/IP Stack Application Application TCP UDP IP Network Interface WebSphere Support Technical Exchange 6 of 39

7 Gathering z/os TCPIP packet trace Start the Component Trace TRACE CT,ON,COMP=SYSTCPDA,SUB=(tcpip_name) Start TCPIP Packet Trace VARY TCPIP,tcpip_name,PKT,ON,IP=xx.xx.xx.xx Recreate the problem Stop TCPIP Packet Trace VARY TCPIP,tcpip_name,PKT,OFF TRACE CT,OFF,COMP=SYSTCPDA,SUB=(tcpip_name) Dump TCPIP and TCPIP dataspace DUMP COMM=('text') R xx,jobname=(tcpip_name),dspname=('tcpip_name'.*),end See Technote for additional information and options WebSphere Support Technical Exchange 7 of 39

8 MV23 MV23 PACKET :44: :44: Packet Trace Packet Trace From Interface From : GE1 : GE1 Device: QDIO Ethernet Device: Full=48 QDIO Ethernet Full=48 Tod Clock Tod Clock : 2010/10/06 : 2010/10/06 04:44: :44: Intfx: 9 Intfx: 9 Segment Source # : : 0 Flags: In Source Destination : : Destination Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Source Port : 3165 Dest Port: 1601 Asid: 009D TCB: This is a summary of the packet to follow WebSphere Support Technical Exchange 8 of 39

9 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 48 04:44: ID Number: Intfx: 9 79F8 Fragment Source : : DontFragment Offset: 0 TTL Destination : : 116 Protocol: TCP CheckSum: 0AAE FFFF Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Scoure Port : 3165 () Destination Port: 1601 () Sequence Number : Ack Number: 0 Header Length : 28 Flags: Syn Window Size : CheckSum: 0E41 FFFF Urgent Data Pointer: 0000 Option : Max Seg Size Len: 4 MSS: 1380 Option : NOP Option : NOP Option : NOP Option : NOP WebSphere Support Technical Exchange 9 of 39

10 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 44 04:44: ID Number: Intfx: Fragment Source : : Offset: 0 TTL Destination : : 64 Protocol: TCP CheckSum: BE Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 1601 () Destination Port: 3165 () Sequence Number : Ack Number: Header Length : 24 Flags: Ack Syn Window Size : CheckSum: 8240 A72C Urgent Data Pointer: 0000 Option : Max Seg Size Len: 4 MSS: 1456 WebSphere Support Technical Exchange 10 of 39

11 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 40 04:44: ID Number: Intfx: 9 79F9 Fragment Source : : DontFragment Offset: 0 TTL Destination : : 116 Protocol: TCP CheckSum: 0AB5 FFFF Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 3165 () Destination Port: 1601 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Window Size : CheckSum: 72CD FFFF Urgent Data Pointer: 0000 TCP 3-way handshake is finished at this point and the TCP connection is in established state WebSphere Support Technical Exchange 11 of 39

12 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : :44: ID Number: Intfx: 9 79FA Fragment Source : : DontFragment Offset: 0 TTL Destination : : 116 Protocol: TCP CheckSum: 09C8 FFFF Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 3165 () Destination Port: 1601 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Psh Window Size : CheckSum: 87AF FFFF Urgent Data Pointer: 0000 Continue on the next slide WebSphere Support Technical Exchange 12 of 39

13 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From Interface 79FA4000 : GE C8 Device: QDIO 0925F5A1 Ethernet Full= A47 Protocol Tod Header Clock : 2010/10/06 : 20 04:44: Port: Intfx: 3165, Offset: Source 0C5D0641 : 37ADABC CBC0F FFFF 87AF0000 Data Destination : : 236 Data Length: 236 Offset: Source Port EC : Dest Port: Asid: 009D... TCB: TSH B "...ID A F67F " @ FFC99A3B B 2E544F2E I......&....;RICK.TO.RTPH B D5F49...g...(....QM_I D5F (... BM_9ED43ACF C , A FF00FF FFFFFFFF FFFFFFFF... j FFFFFFFF FFFF A F40E B0 4D514D4D D5F49 (.((...(. MQMM QM_I 0000C0 424D5F F32.(.... BM_9ED43ACF522_2 0000D D 30372D31 335F3138 2E32312E _ E This is the initial data from the client initiating communication over the channel WebSphere Support Technical Exchange 13 of 39

14 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : :44: ID Number: Intfx: 9 796A Fragment Source : : Offset: 0 TTL Destination : : 64 Protocol: TCP CheckSum: A7 Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 1601 () Destination Port: 3165 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Psh Window Size : CheckSum: CC0 Urgent Data Pointer: 0000 Continue on the next slide WebSphere Support Technical Exchange 14 of 39

15 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From Interface 796A0000 : GE Device: QDIO 09147A47 Ethernet Full= F5A1 Protocol Tod Header Clock : 2010/10/06 : 04:44: Port: Intfx: 1601, Offset: Source 06410C5D : CBC0F ADACAF 50187F Data Destination : : 236 Data Length: 236 Offset: Source Port EC : Dest Port: Asid: 009D... TCB: TSH B "...ID F67F " @ FFC99A3B B 2E544F2E I......&....;RICK.TO.RTPH F f.4...&....RTPH to 00006F (X'000010' bytes) -- All bytes contain X'20' C , FF00FF FFFFFFFF FFFFFFFF FFFFFFFF FFFF A F40E B0 4D514D4D D5F49 (.((...(. MQMM QM_I 0000C0 424D5F F32.(.... BM_9ED43ACF522_2 0000D D 30372D31 335F3138 2E32312E _ E This is reply from the CHIN MCA WebSphere Support Technical Exchange 15 of 39

16 Let's do a little math The client sent data (236 bytes) with a TCP sequence number = The SRVCONN sent data with a TCP ACK number = ACK SEQ = 236 This is how you can ensure that TCP is receiving the data sent (i.e. via the Sequence and Ack numbers! WebSphere Support Technical Exchange 16 of 39

17 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From C4 Interface 79FB4000 : GE A17 Device: QDIO 0925F5A1 Ethernet Full= A47 Protocol Tod Header Clock : 2010/10/06 : 20 04:44: Port: Intfx: 3165, Offset: Source 0C5D0641 : 37ADACAF CBC0FA FF13 15E00000 Data Destination : : 156 Data Length: 156 Offset: Source Port C : Dest Port: Asid: 009D... TCB: TSH B "...ID F67F " @ FFC99A3B B 2E544F2E I......&....;RICK.TO.RTPH B D5F49...f...(....QM_I D5F (... BM_9ED43ACF C , A FF00FF FFFFFFFF FFFFFFFF... j FFFFFFFF FFFF More negotiation of the channel MCA from the client WebSphere Support Technical Exchange 17 of 39

18 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From C4 Interface 796B0000 : GE Device: QDIO 09147A47 Ethernet Full= F5A1 Protocol Tod Header Clock : 2010/10/06 : 04:44: Port: Intfx: 1601, Offset: Source 06410C5D : CBC0FA ADAD4B 50187F64 82D80000 Data Destination : : 156 Data Length: 156 Offset: Source Port C : Dest Port: Asid: 009D... TCB: TSH B ID F67F " @ FFC99A3B B 2E544F2E I......&....;RICK.TO.RTPH F f.4...&....RTPH to 00006F (X'000010' bytes) -- All bytes contain X'20' C , FF00FF FFFFFFFF FFFFFFFF FFFFFFFF FFFF More negotiation of the SRVCONN channel MCA WebSphere Support Technical Exchange 18 of 39

19 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From Interface 79FC4000 : GE B1 Device: QDIO 0925F5A1 Ethernet Full= A47 Protocol Tod Header Clock : 2010/10/06 : 04:44: Port: Intfx: 3165, Offset: Source 0C5D0641 : 37ADAD4B CBC0FA9D 5018FE77 484D0000 Data Destination : : 513 Data Length: 513 Offset: Source Port : Dest Port: F83AA4C Asid: 009D...c.< TCB: TSH L B D (......"...MSH D J......% C E514C...<...< XQH...LAURA.QL to 00005F (X'000010' bytes) -- All bytes contain X'20' &... RTPH to 00008F (X'000010' bytes) -- All bytes contain X'20' D (... MD A FFFFFFFF B B D (... "...MQSTR 0000C D D5F49...(...(....AMQ QM_I 0000D0 424D5F AA4C (...c.<..`. BM_9ED43%..L.y. 0000E F Continue to the next slide WebSphere Support Technical Exchange 19 of 39

20 MV23 PACKET :44: Packet... Trace From to 00011F Interface (X'000010' : GE1 Device: bytes) QDIO Ethernet -- All Full=48 bytes contain X'20' Tod Clock : 2010/10/06 04:44: D5F49 Intfx:...(. 9 QM_I Source 424D5F39 : (... BM_9ED43ACF Destination : Source Port : Dest Port: C Asid: 009D...<... TCB: LAUR E BA.+... ANETTE E06D260 3C EDBC2F K-...h;.B < B A B D...(...re M 0001B0 515C6A C6A 72655C62 696E5C6A.*././*...*..>*. Q\java\jre\bin\j 0001C E /./... avaw.exe D This 0001E D _.../.. is test message 0001F F E...?... 3 for the WSTE This packet contains the actual message sent from the client. WebSphere Support Technical Exchange 20 of 39

21 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 40 04:44: ID Number: Intfx: 9 796C Fragment Source : : Offset: 0 TTL Destination : : 64 Protocol: TCP CheckSum: BD Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 1601 () Destination Port: 3165 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Psh Window Size : CheckSum: 823C 9287 Urgent Data Pointer: 0000 This is the acknowledgement from z/os TCPIP that the message data arrived. Notice the Ack number. The previous Seq number was = Again, we know that TCPIP on z/os got the data sent from the client. WebSphere Support Technical Exchange 21 of 39

22 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From Interface 79FD4000 : GE A95 Device: QDIO 0925F5A1 Ethernet Full= A47 Protocol Tod Header Clock : 2010/10/06 : 04:44: Port: Intfx: 3165, Offset: Source 0C5D0641 : 37ADAF4C CBC0FA9D 5018FE77 1BB70000 Data Destination : : 28 Data Length: 28 Offset: Source Port C : Dest Port: F83AA4C Asid: 009D...c.< TCB: TSH...L B "... Client MCA doing a status check WebSphere Support Technical Exchange 22 of 39

23 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From Interface 796D0000 : GE Device: QDIO 09147A47 Ethernet Full= F5A1 Protocol Tod Header Clock : 2010/10/06 : 04:44: Port: Intfx: 1601, Offset: Source 06410C5D : CBC0FA9D ADAF FE Data Destination : : 28 Data Length: 28 Offset: Source Port C : Dest Port: Asid: 009D... TCB: TSH B SRVCONN replying...i'm OK WebSphere Support Technical Exchange 23 of 39

24 Ip Header 406 MV23 PACKET : :44: IP: Packet , Trace Offset: From Interface 7A : GE A89 Device: QDIO 0925F5A1 Ethernet Full= A47 Protocol Tod Header Clock : 2010/10/06 : 04:44: Port: Intfx: 3165, Offset: Source 0C5D0641 : 37ADAF CBC0FAB9 5018FE5B F16B0000 Data Destination : : 28 Data Length: 28 Offset: Source Port C : Dest Port: Asid: 009D... TCB: TSH B "... Client saying I'm done and closing the channel WebSphere Support Technical Exchange 24 of 39

25 TCP/IP socket overview Simple example of application socket calls client/server Client Application Server Application Socket function Connection state Network flow Connection state Socket function Socket Connect synsent SYN SYN/ACK ACK listen Socket Bind Listen established established Accept Write 100 Read bytes ACK bytes ACK +100 Write 100 Close fin_wait1 FIN fin_wait2 ACK close_wait FIN timewait ACK last ack Close WebSphere Support Technical Exchange 25 of 39

26 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 40 04:44: ID Number: Intfx: 9 7A0A Fragment Source : : DontFragment Offset: 0 TTL Destination : : 116 Protocol: TCP CheckSum: 0AA4 FFFF Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 3165 () Destination Port: 1601 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Fin Window Size : CheckSum: 6F0B FFFF Urgent Data Pointer: 0000 Client WMQ code has issued a close() for the socket. TCP is sending a packet with the Fin flag on. WebSphere Support Technical Exchange 26 of 39

27 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 40 04:44: ID Number: Intfx: Fragment Source : : Offset: 0 TTL Destination : : 64 Protocol: TCP CheckSum: D2 Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 1601 () Destination Port: 3165 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Psh Window Size : CheckSum: 823C 94C1 Urgent Data Pointer: 0000 Fin acked by z/os WebSphere Support Technical Exchange 27 of 39

28 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 40 04:44: ID Number: Intfx: Fragment Source : : Offset: 0 TTL Destination : : 64 Protocol: TCP CheckSum: D3 Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 1601 () Destination Port: 3165 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Psh Fin Window Size : CheckSum: 823C 94C2 Urgent Data Pointer: 0000 CHIN has issued a close on behalf of the SRVCONN because the client closed. Therefore a Fin flows from z/os. WebSphere Support Technical Exchange 28 of 39

29 IpHeader: 406 MV23 Version PACKET : :44: Header Packet Length: Trace 20 Tos From Interface : GE1 : 00 Device: QDIO Ethernet QOS: Full=48 Routine Normal Service Packet Tod Length Clock : 2010/10/06 : 40 04:44: ID Number: Intfx: 9 7A0B Fragment Source : : DontFragment Offset: 0 TTL Destination : : 116 Protocol: TCP CheckSum: 0AA3 FFFF Source Source Port : 3165 : Dest Port: 1601 Asid: 009D TCB: Destination : TCP Source Port : 3165 () Destination Port: 1601 () Sequence Number : Ack Number: Header Length : 20 Flags: Ack Window Size : CheckSum: 6F0A FFFF Urgent Data Pointer: 0000 Fin acked by client TCP stack. WebSphere Support Technical Exchange 29 of 39

30 (SESS output) TcpHdr 406 IO MV23 F PACKET Seq :44: Ack RcvWnd Packet Data Trace Delta Time TimeStamp S From I Interface : GE1 Device: QDIO 0 Ethernet Full= :44: A S Tod Clock O : 2010/10/06 04:44: Intfx: :44: A Source I u : :44: AP Destination I : :44: AP Source O Port : 3165 Dest Port: 1601 Asid: D TCB: :44: AP I :44: AP O :44: AP I :44: AP O d :44: AP I :44: AP O :44: A I d :44: AP I :45: A F I? :45: AP O a :45: AP F O? :45: A I a :45: WebSphere Support Technical Exchange 30 of 39

31 Domain Nameserver issues CSQX014E +RTPH CSQXSRSP Listener exceeded channel limit, TRPTYPE=TCP INDISP=QMGR. DISPLAY CHSTATUS(*) STATUS SUBSTATE CSQM422I RTPH CHSTATUS(TEST.CHAN) CHLDISP(PRIVATE) CONNAME( ) CURRENT CHLTYPE(RCVR) STATUS(BINDING) SUBSTATE(NAMESERVER) LSTMSGTI( ) LSTMSGDA( ) MSGS(0) WebSphere Support Technical Exchange 31 of 39

32 Domain Nameserver issues CHIN will attempt a getnameinfo() call to associate client IP address to a hostname Delays in DNS response can cause CSQX014E and other problems Task is single threaded This is a DNS problem... Check Resolver settings to see what DNS(s) are configured Check RESOLVERTIMEOUT setting. 30 sec default Bypass V6 PK79874/UK47580 V7 PK88882/UK47582 Requires Service parm WebSphere Support Technical Exchange 32 of 39

33 Channel connection reset +CSQX208E +RTPH CSQXRESP Error receiving data, channel TEST.CHAN, connection ( ) (queue manager????) TRPTYPE=TCP RC= (ECONNRESET) reason= CSQX206E RTPH CSQXRCTL Error sending data, channel TEST.CHAN, connection raleigh ( ) (queue manager RTPH) TRPTYPE=TCP RC= C reason= = JRCONNTCBNOTFOUND WebSphere Support Technical Exchange 33 of 39

34 Channel connection reset WMQ Client IP Cloud Problem shows here z/os Queue Manager WebSphere Support Technical Exchange 34 of 39

35 Channel connection reset The problem is reported on z/os The reset is not generated from the reporting host channel The reset could come from the other host The reset could come from any intermediate firewalls May need to trace on both sides of the channel connection May need to trace intermediate hops along the connection WebSphere Support Technical Exchange 35 of 39

36 Summary TCP/IP networking concepts How to trace at the network level Reviewed a Client channel connection via PKTTRACE Reviewed some common network related problems WebSphere Support Technical Exchange 36 of 39

37 Additional WebSphere Product Resources Learn about upcoming WebSphere Support Technical Exchange webcasts, and access previously recorded presentations at: Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: Join the Global WebSphere Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically: Sign up to receive weekly technical My Notifications s: WebSphere Support Technical Exchange 37 of 39

38 We Want to Hear From You! Tell us about what you want to learn Suggestions for future topics Improvements and comments about our webcasts We want to hear everything you have to say! Please send your suggestions and comments to: WebSphere Support Technical Exchange 38 of 39

39 Questions and Answers WebSphere Support Technical Exchange 39 of 39

CIS-331 Exam 2 Fall 2015 Total of 105 Points Version 1

CIS-331 Exam 2 Fall 2015 Total of 105 Points Version 1 Version 1 1. (20 Points) Given the class A network address 117.0.0.0 will be divided into multiple subnets. a. (5 Points) How many bits will be necessary to address 4,000 subnets? b. (5 Points) What is