Coupling Facility Dumps IBM. This document can be found on the web, Under the category of White Papers.

Transcription

1 s IBM This document can be found on the web, Under the category of White Papers. August Anna Clayton Nicole M. Fagen Daniel Rinck Page - 1 -

2 s Table of Contents Table of Contents...2 Overview Non-disruptive Serialized CFCC Service Levels 3 z/os Support for Non-disruptive Serialized Coupling Facility dumps 4 System Programmer Action.4 Initiating Non-Disruptive Serialized s.5 Impact of Non-disruptive Serialized 6 Coordinated z/os SVC Dumps...6 Sample z/os Message Flow...7 Recommendations Trademarks Feedback Acknowledgments Page - 2 -

3 s There are situations when a serialized dump of a coupling facility is required to diagnose an issue. The ability to disruptively collect a serialized coupling facility dump has always existed. A disruptive dump of a coupling facility requires the coupling facility to stop, dump and then restart. All of the structures in the coupling facility at the time of the disruptive dump were lost when the coupling facility restarted. With the z196, IBM enhanced first failure data capture (FFDC) with the introduction of the capability to take a non-disruptive serialized dump of a coupling facility (ND CF dump). For this type of dump, the CF image does not stop, is not restarted, and structures in the CF are not lost. ND CF dumps do not contain the user data written by the connectors. To dump the user data inside the LIST or CACHE structure use the z/os DUMP command or SLIP command specifying the desired STRLIST options. ND CF dumps are reviewed by IBM hardware support while SVC dumps with structure data are reviewed by IBM software support. This document contains information pertaining to: The CFCC service levels required to utilize the ND CF dump capability z/os maintenance which collects coordinated software information z/os maintenance which extends the reasons for which z/os will trigger the ND CF dumps System programmer response to ND CF dumps The hardware command to initiate the ND CF dumps The *NEW* z/os operator command to initiate the ND CF dumps Impact of collecting ND CF dumps Non-disruptive Serialized CFCC Service Levels z196 any coupling facility residing on a z196 has the capability to collect a ND CF dump. z10 at CFCC 16 Service Level 4.01 or higher - any coupling facility residing on a z10 at CFCC 16 service level 4.01 or higher has the capability to collect a ND CF dump. z10 CFCC 16 Service Level 4.04 and higher, z196 GA1 CFCC 17 Service Level 2.26 and higher, z196 GA2 CFCC 17 all levels will only collect the dumps for breakduplexing events if the coupling facility has at least one dedicated engine. At inception, the ND CF dumps were taken for ALL break-duplexing events. Many of these breakduplexing events were a result of a shared ICF configuration and the particular CF image not having a lot of weight. There is no benefit to IBM or IBM customers to collect the Page - 3 -

4 ND CF dumps for break-duplex events when there is no dedicated CF engine. ND CF dumps on CFs without a dedicated engine may still occur for other reasons. z/os Support for Non-disruptive Serialized Coupling Facility dumps OA z/os support to initiate ND CF dumps on a z196. PTFs are available on z/os V1R10 (UA56220), z/os V1R11 (UA56221) and z/os V1R12 (UA56222). The change is in the base of z/os V1R13. OA z/os support to initiate ND CF dumps on a z10 at CFCC 16 Service Level 4.01 or higher. PTFs are available on z/os V1R9 (UA57039), z/os V1R10 (UA57040), z/os V1R11 (UA57041) and z/os V1R12 (UA57042). The change is in the base of z/os V1R13. OA37802 z/os support to take ND CF dumps of both CFs involved when a duplex out of synch condition is detected. This APAR removed the need to enable the IXLDUPOUTOFSYNCH DIAG. PTFs are available on z/os V1R12 (UA63579) and z/os V1R13 (UA63580). OA37009 z/os support to initiate ND CF dump(s) when IXL044I indicating an excessive number of interface control checks (IFCCs) has occurred. PTFs are available on z/os V1R11 (UA63077), z/os V1R12 (UA63078) and z/os V1R13 (UA63079). OA35342 z/os support for a new z/os operator command to initiate a ND CF dump. PTFs are available on z/os V1R12 (UA65520) and z/os V1R13 (UA65521). System Programmer Action If a ND CF dump occurs and the CPC on which the Coupling Facility resides has phone home capabilities, a hardware PMH will be created and the IBM Remote Technical Support Center (RTSC) will be automatically notified and engaged. The IBM hardware support is the appropriate first level of support. IBM hardware support will review the logs and the ND CF dump then determine how best to proceed. Note: Only IBM Hardware Support can review coupling facility dumps. Neither customers nor z/os software support can review the CF dump. Depending upon the initial analysis, there may be a need to provide additional hardware or configuration information. The RTSC may engage a Customer Engineer to facilitate additional data collect. Similarly, it may be necessary to provide additional software information. If the coordinating z/os software dumps are required, IBM hardware support will engage the z/os software support center or request a support software record be opened. Should software support be engaged the desired documentation is the z/os Page - 4 -

5 SVC dumps, the logrecs spanning the error, the operlog spanning the error and SMF70-79 records with all subtypes from all systems spanning the time of error. Initiating Non-Disruptive Serialized s A ND CF dump can be initiated on the Hardware Management Console (HMC) or Service Element (SE) on the Operating System Console for a given coupling facility or by a z/os image which is physically connected to and using the coupling facility. The Operating System Console command which can be used on the HMC or SE to initiate the ND CF dump is CFDUMP. As mentioned above, z/os APAR OA35342 introduced the capability to initiate the ND CF dump by a z/os operator command. The command syntax to initiate the dumps from z/os: SETXCF DUMPCF, {CFNAME=cfname[,TYPE=NONDISRUPTIVE DISRUPTIVE][,UNCOND=NO YES]} {STRNAME=strname} CFNAME=cfname Allows the operator to specify the CF to be dumped. TYPE=DISRUPTIVE NONDISRUPTIVE Allows the operator to optionally request a disruptive CF dump. Default: Nondisruptive UNCOND=YES NO Allows the operator to bypass a confirmation if a disruptive CF Page - 5 -

6 dump is requested. Default: No STRNAME=strname Allows the operator to request a CF dump by specifying a structure name. The CF(s) in which the structure resides will be dumped and the dump requested will be non-disruptive. Please see for the complete documentation update associated with OA IBM support will leverage the z/os operator command in combination with SLIPs and automation to collect timely coupling facility documentation. There are times when CFCC will initiate ND CF dumps. ND CF dumps will be initiated when a break-duplexing event occurs, CFCC forced CHPID recovery or receiver CHPID went away. Link firmware can initiate a set of ND CF dumps if the DUPLEXCFDIAG function is enabled and the duplexed command takes longer than 200 microseconds. Similarly, there are situations where z/os will initiate ND CF dumps. z/os will initiate the dumps when duplex operations are detected to be out of sync, inconsistent lock data is detected (aka, missing record data entry) or when there is a high volume of IFCCs in a short amount of time. Impact of Non-disruptive Serialized The coupling facility is quiesced while the dump is collected. It takes approximately milliseconds to collect the dump. End users are unlikely to notice the millisecond delay while the dump is taken. After the dump is taken, it is written to the SE hard drive. It will take approximately 10 seconds to write the dump to the SE hard drive. The CF is not quiesced when the dump is written to the SE hard drive. For a given CF, only one ND CF dump can be taken in a given 5 minute interval. Additional dump requests during that interval will be rejected. If a second dump is requested via the SETXCF command within the 5 minute interval msgixc557i will be issued indicating that the CF bypassed the dump. Coordinated z/os SVC Dumps to go with Non-disruptive Serialized Coupling Facility Dumps When the coupling facility and the z/os images both reside on a z196 or newer machine, a *new* CRW will be passed from the CF to z/os when the ND CF dump is initiated. The *new* CRW is presented to z/os when the ND CF dump is triggered by (a) CFDUMP being issued on the HMC OPERMSG console, (b) the z/os operator command to take the CF dump is issued or (c) DUPLEXCFDIAG is the catalyst. z/os images physically connected to the CF will receive the CRW and initiate SVC dumps. ABEND026 RSN will be taken on all systems with connections to the CF Page - 6 -

7 which took the ND CF dump for break-duplex events, link diagnostic timeouts and CF command time outs. If z/os initiates the ND CF dump, z/os will initiate ABEND026s on all systems with connections to the structure which encountered a problem. The ABEND reason code will be RSN , , or for duplex out of synch; RSN 0C090020, 0C2A0020, or 0C5B0020 for record data entry not found. Sample z/os Message Flow The following is an example of the message flow when a ND CF dump is initiated by z/os operator command. Initiation of the dump Z :27:29.50 KHINDSSM SETXCF DUMPCF,CFNM=CF21,TYPE=NONDISRUPTIVE Z :27:29.56 IXC557I THE SETXCF DUMPCF REQUEST FOR CF CF21 WAS COMPLETED: A NON-DISRUPTIVE DUMP WAS INITIATED z/os issues messages indicating the dumps were captured on the CF. msgixl051e will be issued on the z/os system the SETXCF DUMPCF command originated on. msgixl010e could be issued on any system with physical connectivity to the coupling facility which took the ND CF dump. Most likely, msg IXL010E will be issued on a system in the sysplex using the CF. However, it is possible for msgixl010e to be issued on a system in a different sysplex which has physical connectivity to the CF which took the ND CF dump. Non-disruptive dumps will also be initiated on remote Coupling Facility image(s) if there is physical connectivity via Infiniband links. Note: this only applies to z196 and future. Automatic initiation of remote ND CF Dumps does not apply to z10. z/os message indicating ND CF dumps were taken. Z :27:29.56 *IXL010E NOTIFICATION RECEIVED FROM COUPLING FACILITY IBM FBAA6 PARTITION: 22 CPCID: 00 NAMED CF21 Page - 7 -

8 A NON-DISRUPTIVE DUMP WAS TAKEN BY THE CF. Z :27:29.56 IXL051E NON-DISRUPTIVE DUMP OF COUPLING FACILITY CF IBM FBAA6 PARTITION: 22 CPCID: 00 INITIATED BY THE OPERATOR DIAG DATA: SETXCF invoked DUMPCF by cfname Z :27:29.58 *IXL010E NOTIFICATION RECEIVED FROM COUPLING FACILITY IBM E15 PARTITION: 05 CPCID: 00 NAMED CF22 A NON-DISRUPTIVE DUMP WAS TAKEN BY THE CF. Z :27:29.59 *IXL010E NOTIFICATION RECEIVED FROM COUPLING FACILITY IBM E1A7 PARTITION: 03 CPCID: 00 NAMED CF23 A NON-DISRUPTIVE DUMP WAS TAKEN BY THE CF. Each system connected to the CF initiates z/os SVC Dumps Z :27:29.64 IEA045I AN SVC DUMP HAS STARTED AT TIME= DATE=07/17/2012 FOR ASIDS(0017,0006) ERROR ID = SEQ08079 CPU82 ASID0017 TIME QUIESCE = YES Z :27:29.65 IEA045I AN SVC DUMP HAS STARTED AT TIME= DATE=07/17/2012 FOR ASIDS(0017,0006) ERROR ID = SEQ10579 CPU83 ASID0017 TIME QUIESCE = YES Z :27:29.65 IEA045I AN SVC DUMP HAS STARTED AT TIME= DATE=07/17/2012 FOR ASIDS(0017,0006) ERROR ID = SEQ53272 CPU81 ASID0017 TIME QUIESCE = YES Z :27:29.68 IEA045I AN SVC DUMP HAS STARTED AT TIME= DATE=07/17/2012 FOR ASIDS(0017,0006) ERROR ID = SEQ09450 CPU85 ASID0017 TIME QUIESCE = YES SVC dumps successfully captured. Page - 8 -

9 Z :27:32.05 IEA794I SVC DUMP HAS CAPTURED: DUMPID=048 REQUESTED BY JOB (IOSAS ) DUMP TITLE=COMPON=XES,COMPID=5752SCIXL,ISSUER=IXLM1REC,MODULE=I XLE1CRW,ABEND=S0026,REASON= Z :27:32.85 IEA794I SVC DUMP HAS CAPTURED: DUMPID=004 REQUESTED BY JOB (IOSAS ) DUMP TITLE=COMPON=XES,COMPID=5752SCIXL,ISSUER=IXLM1REC,MODULE=I XLE1CRW,ABEND=S0026,REASON= Z :27:33.94 IEA794I SVC DUMP HAS CAPTURED: DUMPID=003 REQUESTED BY JOB (IOSAS ) DUMP TITLE=COMPON=XES,COMPID=5752SCIXL,ISSUER=IXLM1REC,MODULE=I XLE1CRW,ABEND=S0026,REASON= Z :27:40.57 IEA794I SVC DUMP HAS CAPTURED: DUMPID=007 REQUESTED BY JOB (IOSAS ) DUMP TITLE=COMPON=XES,COMPID=5752SCIXL,ISSUER=IXLM1REC,MODULE=I XLE1CRW,ABEND=S0026,REASON= Trademarks A full list of U.S. trademarks owned by IBM may be found at: Feedback Please send comments or suggestions for changes to rinck@us.ibm.com or nfagen@us.ibm.com. Acknowledgments The authors would like to thank all the contributors and reviewers of Coupling Facility Dumps. A special thanks to William Neiman for his contributions. Page - 9 -