Diagnosing z/os Communications Server TCP/IP storage growth and storage abends

Transcription

1 IBM Software Group Diagnosing z/os Communications Server TCP/IP storage growth and storage abends Donna Fitzgerald WebSphere Support Technical Exchange

2 Topics to be covered: AGENDA Overview Storage definitions Monitoring storage utilization Limiting TCP/IP storage utilization Storage messages Abends Problem determination Collecting documentation Common problems INFO APARs and TECHNOTES Reference materials WebSphere Support Technical Exchange 2

3 OVERVIEW The key to the successful resolution of most TCP/IP storage problems is to first determine whether the storage problem you are experiencing is related to common, private or communication storage manager (CSM) storage. This presentation will outline steps you can use to determine the type of storage you are having a problem with and the steps to take to diagnose the storage problem. WebSphere Support Technical Exchange 3

4 Storage definitions Common storage Common storage is shared across the system and can be accessed from any address space. TCP/IP's usage of common storage for the most part is for extended common storage area (ECSA). Private storage Private storage is storage that is unique to an address space. Private storage is also referred to as pool storage. CSM Communications Storage Manager (CSM) enables TCP/IP, VTAM, and other applications to use CSM buffers to reduce data movement. CSM interfaces with the z/os operating system to provide the storage for the buffers. Buffers are maintained in both common storage and in the CSM dataspace. WebSphere Support Technical Exchange 4

5 Monitoring storage utilization Storage is a resource that many users monitor very closely. Common methods to monitor TCP/IP storage utilization are: TCP/IP and CSM display commands Use of Netview CLISTs that are triggered at specific time intervals Resource Measurement Facility (RMF) reports The output can be used to establish a history of storage utilization. WebSphere Support Technical Exchange 5

6 Monitoring storage utilization TCP/IP Monitor TCP/IP common and private storage utilization by issuing the console command: Display TCPIP,,STOR d tcpip,,stor TCPIP STORAGE TCPCS STORAGE CURRENT MAXIMUM LIMIT TCPCS ECSA 14M 28M 120M TCPCS POOL 52M 62M NOLIMIT DISPLAY TCPIP STOR COMPLETED SUCCESSFULLY WebSphere Support Technical Exchange 6

7 Monitoring storage utilization CSM TCP/IP's CSM storage utilization is not included in the TCP/IP display command output. TCP/IP's CSM storage utilization can be monitored with the following console commands: To display information about storage managed and used by CSM for all owners, issue the following console command: d net,csm,ownerid=all To display information about CSM utilization for TCP/IP: d net,csm,ownerid=tcpipasid WebSphere Support Technical Exchange 7

8 Monitoring storage utilization CSM d net,csm,ownerid=1f6 IVT5508I DISPLAY ACCEPTED IVT5549I PROCESSING DISPLAY CSM COMMAND - OWNERID SPECIFIED IVT5530I BUFFER BUFFER IVT5551I SIZE SOURCE STORAGE ALLOCATED TO OWNER IVT5532I IVT5553I 4K ECSA 256K IVT5554I TOTAL ECSA 256K IVT5532I IVT5553I 4K DATA SPACE K IVT5554I TOTAL DATA SPACE K IVT5532I IVT5554I TOTAL DATA SPACE 128K IVT5532I IVT5556I TOTAL FOR OWNERID 384K IVT5557I OWNERID: ASID = 01F6 JOBNAME = TCPIP IVT5599I END WebSphere Support Technical Exchange 8

9 Limiting TCP/IP common and private storage utilization The amount of common and private storage that TCP/IP can consume can be limited with TCP/IP profile parameters: GLOBALCONFIG ECSALIMIT POOLLIMIT ECSALIMIT - the maximum amount of common storage that TCP/IP can use. This limit does not include the CSM storage used by TCP/IP. POOLLIMIT - the maximum amount of TCP/IP private storage that TCP/IP can use. Valid limits for ESCALIMIT and POOLLIMIT are in the range 10240k and k or 10m and 2047m. The default is 0. Tip: Care should be taken when coding ECSALIMIT and POOLLIMIT parameters. Setting them too low can cause TCP/IP to terminate prematurely. ECSALIMIT and POOLLIMIT parameter values can be increased dynamically with a VARY TCPIP,,OBEYFILE command. - WebSphere Support Technical Exchange 9

10 Limiting CSM storage utilization CSM storage limits are located in the SYS1.PARMLIB member IVTPRMxx. The values you can allocate are: ECSA MAX - the maximum amount of ECSA storage that CSM can allocate. FIXED MAX - the maximum amount of fixed storage that CSM can allocate. This includes both fixed CSM ECSA and CSM data space storage. If no values are specified in SYS1.PARMLIB member IVTPRMxx, the system uses the default values of 100m ECSA and 100m FIXED. If the limit specified by these values is reached, results are unpredictable. TCP/IP might not be able to continue. CSM ECSA and FIXED values can be modified dynamically with the command: MODIFY net,csm,ecsa=value,fixed=value where value is in the range 1024KB MB WebSphere Support Technical Exchange 10

11 Storage messages Common storage The following messages are issued for TCP/IP common storage shortages. Common storage messages: EZZ4360I jobname ECSA CONSTRAINED EZZ4361I jobname ECSA CRITICAL EZZ4362I jobname ECSA EXHAUSTED EZZ4363I jobname ECSA SHORTAGE RELIEVED Specifying a nonzero ECSALIMIT value enables warning messages EZZ4360I, EZZ4361I and EZZ4362I to be issued when a storage shortage occurs. Note: These messages may be an indication that the limits need to be raised. WebSphere Support Technical Exchange 11

12 Storage messages Private Storage The following messages are issued for TCP/IP private storage shortages. Private storage messages: EZZ4364I jobname POOL CONSTAINED EZZ4365I jobname POOL CRITICAL EZZ4366I jobname POOL EXHAUSTED EZZ4367I jobname POOL SHORTAGE RELIEVED Specifying a nonzero POOLLIMIT enables warning messages EZZ4364I, EZZ4365I, and EZZ4366I to be issued when a storage shortage occurs Note: These messages may be an indication that the limits need to be raised. WebSphere Support Technical Exchange 12

13 Storage messages CSM The following messages are issued for CSM ECSA and fixed storage shortages. CSM ECSA storage messages: IVT5560I CSM ECSA storage limit exceeded IVT5562I CSM ECSA storage at critical level IVT5564I ESCA storage shortage relieved CSM FIXED storage messages: IVT5561I CSM FIXED storage limit exceeded IVT5563I CSM FIXED storage at critical level IVT5565I CSM FIXED storage shortage relieved Note: Some of these messages may indicate that the limits for the particular type of CSM storage should be increased. Issue the Display CSM command to get more details on current CSM allocations and limits. CSM limits can be increased using the Modify CSM command. WebSphere Support Technical Exchange 13

14 Storage messages SYSPLEX Problem Detection and Recovery (SPDR) Sysplex Problem Detection and Recovery (SPDR) detects critical storage shortages for ECSA, private or CSM storage. When TCP/IP profile statement GLOBALCONFIG statement parameters ECSALIMIT or POOLLIMIT are not coded, storage failures are detected only when SYSPLEX code fails. EZD1170E tcpstackname WAS NOT ABLE TO GET TCP/IP storagetype STORAGE When TCP/IP profile GLOBALCONFIG statement ECSALIMIT or POOLLIMIT parameter is NOT coded message EZD1170E is issued. In this situation, the storage request fails because ECSA or private storage is exhausted. EZD1187E tcpstackname WAS NOT ABLE TO GET TCP/IP storagetype STORAGE When TCP/IP profile GLOBALCONFIG statement ECSALIMIT or POOLLIMIT is coded, EZD1187E is issued. In this situation the storage request fails because ECSA or private storage is critical. EZZ9679E tcpstackname DETERMINED THAT CSM WAS CRITICAL FOR AT LEAST timevalue SECONDS SPDR issues EZZ9679E when CSM storage problems are detected. This message is issued when CSM storage has been critical for the configured value specified on the GLOBALCONFIG SYSPLEXMONITOR TIMERSECS parameter or the default value of 60 seconds if the parameter is specified. WebSphere Support Technical Exchange 14

15 ABENDS Storage shortages typically result in the following abends: Common ABEND878 RC04 or RC08 ABEND80A RC04 or RC08 ABEND4C5 rsn xxxx2500 Private ABEND878 RC0C or RC10 ABEND80A RC0C or TC10 ABEND4C5 rsn xxxx2500 WebSphere Support Technical Exchange 15

16 Problem determination Steps to take when reviewing a storage problem: 1. Issue D TCPIP,,STOR and D Net,CSM,Ownerid=All commands to track storage usage. 2. Look in the system log for messages related to storage. 3. Run EREP (Environmental Record Editing and Printing Program) against SYS1.LOGREC and review software records, looking for any storage related abends. 4. Review the messages and abends you found to determine whether they indicate a common, private, or CSM storage problem. 5. Review your storage settings for any identified problem area (for example, common, CSM, or private storage). 6. Compare your current storage usage to your previous usage. If usage has increased, do you know of any situations that would cause an increased storage usage (for example, new applications that use common storage or increased connections)? 7. Review your response to step 6. Can the storage problem be resolved by increasing your storage limits? 8. If you have a dump that was created as a result of the storage problem, proceed to step If no dump was taken, collect documentation to submit to the IBM Support Center before proceeding to step 10. If you are unable to obtain a dump of the problem, a stand-alone dump may be the only method to gather documentation for the IBM Support Center. 10. Call the IBM Support Center for assistance in reviewing the documentation. WebSphere Support Technical Exchange 16

17 Collecting documentation for the IBM Support Center If your storage problem is caused by TCP/IP common, CSM or private storage usage, ensure that you have a dump of TCP/IP; IBM service will need to review it. If you did not get a system dump for the abend, or if you want to obtain a dump of TCP/IP to perform a storage analysis, the following commands can be used to gather a dump. WebSphere Support Technical Exchange 17

18 Collecting documentation for the IBM Support Center SLIP Dump of the abend SL SET,COMP=xxx,ACTION=SVCD,JOBNAME=tcpipprocname, SDATA=(NUC,RGN,CSA,SQA,LSQA,TRT),DSPNAME=( tcpiprocname.*), END SLIP DUMP of the abend if the storage problem is related to TCP/IP s CSM storage usage SL SET,COMP=xxx,ACTION=SVCD,JOBNAME=tcpipprocname, SDATA=(NUC,RGN,CSA,SQA,LSQA,TRT),DSPNAME=( tcpipprocname.*, 1.CSM*),END Where xxx equals the abend code you are receiving (for example, 878) WebSphere Support Technical Exchange 18

19 Collecting documentation for the IBM Support Center Console dump of TCP/IP DUMP COMM=(tcpip storage growth) r xx,jobname=(tcpipprocname),cont r xx,sdata=(nuc,rgn,csa,sqa,lsqa,trt),cont r xx,dspname=( tcpipprocname.*),end Message ID SLIP can be used to take a dump of TCP/IP when a particular message is issued SL SET,MSGID=zzzzz,ACTION=SVCD, JOBLIST=(tcpipprocname),DSPNAME=( tcpipprocname.*,) SDATA=(NUC,RGN,CSA,SQA,LSQA,TRT),END Where zzzzz is the message id WebSphere Support Technical Exchange 19

20 Collecting documentation for the IBM Support Center Console dump when the storage problem is related to CSM storage usage Console dump of TCP/IP DUMP COMM=(tcpip storage growth) r xx,jobname=(tcpipprocname),cont r xx,sdata=(nuc,rgn,csa,sqa,lsqa,trt),cont r xx,dspname=( tcpipprocname.*,1.csm*),end WebSphere Support Technical Exchange 20

21 INFO APARs and TECHNOTES INFO APAR II12658 / II13951 TCP/IP maintenance for CSM problems TECHNOTES CSM Storage growth and D NET,CSMUSE Command CSM usage for TCP/IP increases after OW52291 is applied or migrating to z/os for OSA QDIO and IQDIO devices ABEND106 RC28 or RC0C at TCP/IP initialization CSA/ECSA storage exhausted due to inbound UDP traffic rate ECSA shortages WebSphere Support Technical Exchange 21

22 Reference Materials z/os Communications Server: IP System Administrator's Commands, SC z/os Communications Server: IP Configuration Reference, SC z/os Communications Server: IP Configuration Guide, SC z/os Communications Server: IP Messages Volume 2 (EZB, EZD), SC z/os Communications Server: IP Messages Volume 4 (EZZ, SNM). SC z/os Communications Server: CSM Guide, SC z/os Communications Server: SNA Operations, SC z/os Communications Server: SNA Diagnosis Vol 2, FFST Dumps and the VIT, GC z/os Communications Server: SNA Messages, SC z/os MVS Initialization and Tuning Reference, SA z/os MVS Initialization and Tuning Guide, SA z/os MVS Diagnosis and Tools and Service Aids, GA WebSphere Support Technical Exchange 22

23 Additional WebSphere Product Resources Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at: Learn about other upcoming webcasts, conferences and events: Join the Global WebSphere User Group Community: Access key product show-me demos and tutorials by visiting IBM Education Assistant: Learn about the Electronic Service Request (ESR) tool for submitting problems electronically: wf.html Sign up to receive weekly technical My support s: WebSphere Support Technical Exchange 23