z/os Performance HOT Topics Share, Winter 2008 Session: 2500

Transcription

1 IBM Advanced Technical Support z/os Performance HOT Topics Share, Winter 2008 Session: 2500 Kathy Walsh IBM Corporation Advanced Technical Support Trademarks and Disclaimers AIX* AIX 5L* BladeCenter Chipkill DB2 DB2 Universal Database Domino Enterprise Storage Server* e-business logo* GDPS* Geographically Dispersed Parallel Sysplex * Registered trademarks of IBM Corporation HiperSockets IBM* IBM eserver IBM logo* iseries Lotus* OnForever* Parallel Sysplex* POWER POWER5 Predictive Failure Analysis* pseries* S/390 ServerProven* Tivoli* TotalStorage* TotalStorage Proven Virtualization Engine X-Architecture xseries* z/os* z/vm* zseries* System z9* IBM System z The following are trademarks or registered trademarks of other companies. Intel is a trademark of Intel Corporation in the Unites States, other countries or both. Java and all Java-based trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries Linux is a registered trademark of Linux Torvalds in the United States other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Microsoft, Windows and Windows NT are registered trademarks of Microsoft Corporation in the United States, other countries, or both. * All other products may be trademarks or registered trademarks of their respective companies. Notes: Performance is in Internal Throughput Rate (ITR) ratio based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput improvements equivalent to the performance ratios stated here. IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply. All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending on individual customer configurations and conditions. This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the product or services available in your area. All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Information about non-ibm products is obtained from the manufacturers of those products or their published announcements. IBM has not tested those products and cannot confirm the performance, compatibility, or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. Prices subject to change without notice. Contact your IBM representative or Business Partner for the most current pricing in your geography. WSC IBM Corporation,

2 Agenda New Performance Support z/os 1.10 z/os 1.9 Performance and Capacity Planning Topics WSC Performance Flashes White Papers and Technical Documents on Performance zpcr zaap and ziip Enhancements Parallel Sysplex LPAR Group Capacity APARs of Interest SRM SDUMP Addendum Older APARs or Performance Information WSC IBM Corporation,

3 z/os 1.10 Performance Items Contention Management Phase 3 Promote units of work identified by exploiters for longer periods of time Promote to priority of highest-priority unit of work waiting for the held resource Info on promoted workloads recorded in RMF CPU Management of ziips WLM algorithms for adjusting dispatch priorities extended to ziip workloads Manage selected components in service class SYSTEM Prevent inadvertent misclassification of system address spaces regardless of the WLM Policy and protect system at high utilization XCFAS, GRS, SMSPDSE, SMSPDSE1, CONSOLE, IEFSCHAS, IXGLOGR, SMF, and CATALOG Support for 10 additional PB delay types Extract the WLM service definition in XML format Install and activate a WLM service definition in XML format via a CIM Server z/os 1.10 RMF Enhancements RMF Monitor III provides new reports about spin and suspend locks Suspend Lock report will display the address spaces which hold locks and which are suspended Spin Lock report will display how often global locks are held and who is spinning Reporting of lock statistics is intended to help analyze lock contention in the system Selection ===> RMF Overview Report Selection Menu Enter selection number or command for desired report. Basic Reports 1 WFEX Workflow/Exceptions (WE) 2 SYSINFO System information (SI) 3 CPC CPC capacity Detail Reports 4 DELAY Delays (DLY) 5 GROUP Group response time breakdown (RT) 6 ENCLAVE Enclave resource consumption and delays (ENCL) 7 OPD OMVS process data 8 ZFSSUM zfs summary (ZFSS) 9 ZFSACT zfs activity (ZFSA) 10 SPACEG Storage space (SPG) 11 SPACED Disk space (SPD) 12 LOCKSP Spin locks (LSP) 13 LOCKSU Suspend locks (LSU) WSC IBM Corporation,

4 z/os 1.9 Performance Items 1. Provide automatic CPU promotion for canceled jobs Canceled job may be holding resources needed elsewhere, (storage, ENQ, latch), but if dispatch priority is not high enough Cancel processing cannot run 2. Provide throughput to blocked workloads Dispatch low priority workloads from time to time Helps resolve resource contention for workloads without resource management implemented Getting CPU Not Getting CPU IMP 3 IMP 3 System IMP 1 IMP 2 IMP 3 IMP 5 Needs resource x Has resource x High Priority work is now blocked by lower priority work Blocked Workloads New IEAOPTxx parameters BLWLTRPCT Percentage of the CPU capacity of the LPAR to be used for promotion In tenths of a percent (0.1%) Range: 0 to 200 (0.1% to 20%) Default: 5 BLWLINTHD Starvation threshold in seconds. Amount of time when an address space or enclave has not received CPU service within this time and is considered blocked Range: 5 seconds to seconds (18+ hours) Default: 60 seconds - default changing to 20 seconds RMF Reports updated with information CPU Activity Workload Activity WSC IBM Corporation,

5 WSC FLASH Blocked Workload Support BLOCKED WORKLOAD ANALYSIS BLWLTRPCT (%) 0.5 PROMOTE RATE: DEFINED 14 WAITERS FOR PROMOTE: AVG BLWLINTHD 60 USED (%) 4 PEAK 1 PROMOTE RATE: DEFINED - Number of blocked work units which may be promoted in their dispatching priority per second PROMOTE RATE: USED (%) - The utilization of the defined promote rate during the reporting interval It demonstrates how many trickles were actually given away (in percent of the allowed maximum) for the RMF interval APAR OA22443 New recommended defaults of BLWLINTHD=20 and BLWLTRPCT=5 Applies to z/os 1.9, 1.8 and 1.7 Recommended for the IEAOPTxx member of SYS1.PARMLIB to not code parameters specifying default values. Survey on Blocked Workload Support How many installations are at a release or maintenance level where they have the blocked workload support? How many installations are using the support? In production? In test? How many installation have disabled the support? WSC IBM Corporation,

6 z/os 1.9 New WLM Routing Support (IWMSRSRS) Weight recommendations returned by WLM to route work are based on GCP capacity only Introduction of zaaps and ziips meant the utilization of this capacity was ignored in routing decisions WLM Routing Services will now provide: Individual weights for all processor types, GCP, zaap, ziip Combined server weight based on usage Up to 300 servers per system will be returned (previous limit of 96 servers) Requires all systems in the sysplex to be at z/os 1.9 or later WSC Flash10598 Performance Improvements When Managing Large Real Environments For systems with data spaces which define more than 100MB of virtual significant performance improvements can be attained by limiting the impact of UIC updating Recommends the use of IEAOPTxx parameters UICFRAMESCHECKTIME and MAXRSMDISABLEDMS Once migrate to z/os 1.8 or later no need to specify this parameter Method of UIC updating has changed IEAOPTxx Keyword Description Values UICFRAMESCHECKTIME Specifies, in thousands, the number of frames RSM processes on one address space queue before checking for the timeout value MAXRSMDISABLEDMS Range: Default: 525 Recommend Setting 5 MAXRSMDISABLEDMS Specifies, in milliseconds, the time RSM stays disabled during UIC Update and Steal Range: Default: WSC IBM Corporation,

7 WSC Flash after APAR OA18452 Low Capture Ratio due to SRM Timing Routines SRM timing routines were running too frequently causing a lower z/os capture ratio Discovered after migration to a z9 Impact depends upon number of LPARs and number of address spaces Provide design improvements changing frequency for less critical SRM functions to reduce uncaptured time Period switching, time slicing MTTW, swap analysis For z/os 1.7 and later releases: Default value for RMPTTOM is changed to 3000 Recommend to not code the value in the IEAOPTxx member For z/os 1.6 and earlier release with per CP speed greater than 100 MIPS: Raise RMPTTOM to 2000 on z9 processors For non-production LPARs Higher values of RMPTTOM may be possible but installation need to evaluate the importance of SRM functions which run less often WP z/os Large Memory Performance Studies Studies done to evaluate the benefits of very large processor memory sizes to improve performance Reduced DB2 transaction and query response times Reduced system CPU time Decreased database I/O Larger Java heaps with minimal garbage collection times Very large processor storage sizes supported System z9 supports up to 512GB of real storage z/os 1.8 and later releases support processor maximum and theoretically up to 4TB Workloads evaluated included: DB2 OLTP Large and small DB2 data query Java heap Processor Storage sizes from 4GB to 256GB Session 2530: z/os 1.8 and 1.9 Performance Update - Tue, 8:00 AM WSC IBM Corporation,

8 WP z/os SMF Recording with MVS Logger Introduced in z/os 1.9 it allows SMF data to be recorded to z/os System Logger Merge SMF data from multiple systems in a parallel sysplex in real time Separate SMF data by record type into separate streams, allowing recording to DASD by parallel processes Supports much higher data rates Can use either CF structures or DASD-only logstreams White paper discusses a sample customer environment and describes the process followed to move SMF recording to logstreams No MANx datasets needed New SMF exit IEFU29L New SMF Utility program - IFASMFDL Session z/os 1.9: Record your SMF Data to the System Logger - Wed, 11:00 AM Session z/os 1.9: SMF System Logger User Experiences - Wed, 6:00 PM TD Large Stand-Alone Dump Handling V2 Completely updated document with "Best Practices" for managing large dumps Recommends: Multi-volume dump datasets with 4-6 volumes Use of IPCS COPYDUMP and not IEBGENER Creation of a smaller subset dump containing ASIDs 1-20 It is this dump which should be sent to IBM for analysis Use of Terse Method for transmitting dumps to IBM Method for remote viewing support Testing stand-alone dump in a DR setting, for all new releases, and when the server is changed Stresses the importance of remote viewing for fastest dump analysis WSC IBM Corporation,

9 zpcr Enhancements zpcr V4.5d generally available at end of August, 2007 Some changes have workload metric and algorithm changes which may cause capacity results to differ slightly from those of previous zpcr versions New features available starting with zpcr V4.5 Reworked the method in which we characterize specialty CPs Provides more realistic expectations for asymmetrical configurations Does not change the capacity expectations for the LPAR environment CFCC algorithms are adjusted to better reflect capacity when such partitions are defined on very large z9 EC and z990 processors (i.e., greater than 16 CPs). Updated the n-way capacity expectations for: VSE, z/vm, Linux on System z, and CFCC Better reflection of the n-way capabilities of these environments Functions Previously Available RMF Reader Application Automatically reads RMF report to create zpcr configurations Nonstandard ziip and zaap configurations supported WP Running IBM System z at High Utilization System z can run a diverse mix of customer applications and exploit the full capability of the underlying hardware while meeting customer SLA WLM, IRD, PR/SM, Parallel Sysplex, etc. Can Any Single Workload Be Run At 100% CPU Busy? The simple answer is no Queuing theory shows under many conditions described by arrival rate and service time distribution, as a server approaches 100% the queue of work waiting to be processed grows exponentially as does response time Homogeneous, transaction processing workloads behave like this Is there a growth in CPU time per transaction at higher utilization? The simple answer is yes Transactions processed on a system share the physical hardware resources (CPs, caches, memory buses) and software resources (z/os, subsystems) on the system One might presume if a transaction rate of X drives a processor at 50% busy, then at 100% the processor would be able to support 2X the transaction rate, however, this will not be the case WSC IBM Corporation,

10 IEAOPTxx Specialty CP Parameters APAR OA Provide IIPHONORPRIORITY= support for ziips Provides support for only z/os 1.8 NEW! APAR OA21993 provides IIPHONORPRIORITY support for z/os 1.7 IFAHONORPRIORITY GCP zaap YES Process non-zaap work in priority order including zaap work in priority order when the zaap needs help NO Process no zaap work. Do not process zaap work even if the zaap needs help IIPHONORPRIORITY GCP ziip YES Process non-ziip work in priority order including ziip work in priority order when the ziip needs help NO Process no ziip work. Do not process ziip work even if the ziip needs help Process only zaap work in priority order Process only zaap work in priority order Process only ziip work in priority order Process only ziip work in priority order Queuing Impacts of Server Busy 1 CP 2 CP 3 CP 4 CP Response Time Utilization 1 CP 2 CP 3 CP Honorpriority=NO is a valid method of running Specialty CPs Arrival rates and Server busy will influence IIPCP/AAPCP time Can run Specialty CPs very busy IF there are multiple classes of work with different response time objectives WSC IBM Corporation,

11 Benefits of Parallel Sysplex and Data Sharing MIPS Thousands Industry Leading Availability Single Point of Control Single System Image Enhanced Systems Performance x30 vs 4x8 1,859 additional MIPS if able to grow horizontally 15% faster per CP speed Significantly better availability 0 Unplanned outage Impact 1x30 2x15 3x10 4x8 100% LPARs x CPs 50% 33% 25% Improved vertical growth WLM - OA22160 WLM service macro IWMEQTME returns enclave processor service time for GCPs and ziips Updated to return enclave service time for zaaps ZAAPTIME=zaaptime Optional output parameter which contains total accumulated zaap time for the enclave associated with the current dispatchable work unit Value is not normalized to the speed of regular CPs ZAAPONCPTIME=zaaponcptime Optional output parameter, which contains total accumulated time spent on a GCP for zaap eligible work for the enclave associated with the current dispatchable work unit ZAAPNFACTOR=zaapnfactor Optional output parameter which contains the normalization factor for zaap CPs If zaaps are running at a different speed, multiply zaap times with this factor and divide the result by 256 to normalize the values to the speed of regular CPs WSC IBM Corporation,

12 WLM - OA22607 Due to increased size of the IWMPB control block (256 bytes to 768 bytes) additional CPU service is being consumed with CSECT IRASASRV Most evident in environment with many CICS regions and DB2 subsystems Review the number of PBs created for these environments Impacts: z/os 1.8 and above z/os 1.7 with OA12935 installed Increased CPU time because: During PB sampling, the PB is copied to private storage of sampling module using MVCSK (Move with Source Key) Increased size require 3 MVCSK instructions rather than 1 The PBs are also not aligned for optimal cache performance Changes: Additional bytes in the PB are used by EWLM and not needed during PB sampling so only first 256 bytes will be moved (1 MVCSK) Align the PBs to a 256 byte boundary for optimal cache performance TCB1 TDUMP SDUMP - OA HIPER Data Capture to Data Space All Other TCBs in A/S nondispatchable Write to Dump data set Free Data Space Storage TCB1 TDUMP TCB2 TDUMP TCB3 TDUMP Data Capture to Data Space All Other TCBs in A/S nondispatchable Data Capture to Data Space All Other TCBs in A/S nondispatchable Write to Dump data set Free Data Space Storage Write to Dump data set Free Data Space Storage Data Capture to Data Space All Other TCBs in A/S nondispatchable Write to Dump data set Free Data Space Storage Multiple instances of TDUMP (Transaction Dump) can lead to central and aux shortages and potentially a WAIT03C IEAVTDMP is changed to issue an ENQ on SYSIEA01.TDUMPENQ to serialize TDUMPS within an address space Reject requests (RC08 RSN3B) while the ENQ is unavailable WSC IBM Corporation,

13 Lightly Used Partitions in a Parallel Sysplex Care needs to be exercised for LPARs with a low weight to provide sufficient capacity to ensure the LPAR can participate in a parallel sysplex Responsive Handle peak times, error situations When running IRD in a cluster where more important work is running in other LPARs in the cluster ensure enough capacity is still provided Minimum weight equal to the importance 0 (SYSTEM and SYSSTC) workloads Use VARYCPUMIN controls to prevent uni-processor situations with care Strive for each LPAR in a production parallel sysplex to have access via LPAR weight to 20-50% of a CP Depends upon amount of system workloads LPAR Group Capacity Basics Now can manage CPU for a group of z/os LPARs on a single CEC Limit is set to total usage by all LPARs in group Members which don't want their share will donate to the other members Independent of sysplex scope and IRD LPAR cluster Works with defined capacity limits on an LPAR Target share will not exceed defined capacity Works with WLM LPAR CPU management (IRD) Can have more than one group on a CEC but an LPAR may only be a member of one group LPARs must share engines and specify WAIT COMPLETION = NO Minimum z/os 1.8 image running on z9 EC GA3 or z9bc GA2 processor (announced 4/18/07) Capacity groups are defined on the HMC Change LPAR Group Controls panels Specify group name, limit in MSUs, and LPARs in the group Members can be added or removed dynamically WSC IBM Corporation,

14 LPAR Group Capacity Basics Uses WLM rolling 4 hr avg in MSUs to manage the group capacity limit Cap enforced by PR/SM if group rolling 4 hr avg exceeds limit Each member is aware of other members' usage and determines its share based on its weight as a percentage of total weight for all members in group NOTE: When using IRD the weights can change and therefore the target MSU value can change The defined capacity limit, if also specified, is never exceeded Until members "learn" about the group and build a history, the cap not enforced May take up to 4 hours (48 measurements at 5 minute intervals are maintained for rolling 4 hour average) for capping to start Similar to the bonus period with defined capacity When new member joins the group, it has to build up its history and during this time the group usage may exceed the capacity limit Capping is removed when group rolling 4 hour average drops below group limit Example shows how many MSUs each LPAR would get if they all wanted their share. Target MSUs based on a group limit of 200. Total group weight is 500. LPAR WEIGHT SYSPLEX CAPACITY GROUP TARGET MSU LPAR1 150 PLEX1 GROUPA 60 LPAR2 300 PLEX2 GROUPA 120 LPAR3 500 PLEX1 n/a n/a LPAR4 50 PLEX1 GROUPA 20 Example use of Group Capacity Processor is an 2094-S54 with 18 GCPs Two LPARs, 18-way and 10-way (logical:physical is 1.6:1) Customer uses WLC and pays software based on rolling 4 hr avg Now run a CBU test by adding 4 CBU CPs Processor now has 22 physical CPs in the shared pool Logical:Physical is now 1.3 to 1 and the additional 4 CPs allow the rolling 4 hr average to increase (all 22 CPs can be dispatched) Software bills go up accordingly because of the CBU test Implement a group capacity equal to the MSU of the 18 GCPs The 2 LPARs would be in the group and the total capacity used cannot exceed 18 CPs, even when 22 are in the shared pool Weights would still be honored and if either LPAR did not use their full share the other LPAR could use the capacity. WSC IBM Corporation,

15 Addendum Older flashes which should still be understood, or make you go Hmmmm. APARs which are still causing issues, even though they are old. ziip Assisted IPSec IBM is enhancing z/os Communications Server to allow the IPSec processing to take advantage of ziips The ziip Assisted IPSec function moves most z/os IPSec processing from general purpose processors to ziips Performs the encryption processing Handles cryptographic validation of message integrity, and IPSec header processing z/os Communication Server (z/os CS) is designed to interact with z/os Workload Manager to have all of its enclave Service Request Block (SRB) work made eligible to run on the ziip Availability on z/os 1.8 and later WSC IBM Corporation,

16 WSC FLASH10550 z/os DFSMS 1.8 provides support for page spaces with a maximum size of 16,777,215 slots z/os ASM provides support for 1,048,576 slots Though large page spaces can be defined, z/os ASM will not take advantage of the extended format Ensure there are enough addressable local page slots z/os APAR OA20749 taken to provide support Performance Enhancements in DFSMS Large storage groups take up more CPU time when storage pools have volumes This CPU time is uncaptured in the SMF72 records Recorded in SMF30 records in field SMF30ICU In z/os 1.8 new support called 'fast' volume selection is provided See SMS Volume Selection for Data Set Allocation in the DFSMS Storage Administration Reference For non-best-fit allocations using fast volume selection, SMS will perform volume selection from the prioritized list until 100 volumes have been rejected by DADSM for insufficient space When that occurs, SMS will exclude, based on the volume statistics in the SMS configuration, all volumes with insufficient free space Fast volume selection can greatly reduce the number of candidate volumes, and thus the number of retries Activate fast volume selection by using the FAST_VOLSEL(ON) parameter in IGDSMSxx or SETSMS FAST_VOLSEL( ON) command WSC IBM Corporation,

17 Page Replacement Enhancements Pre z/os 1.8 Process is run periodically to update Unreferenced Interval Counts (UIC) Replacement is an LRU algorithm, based on UIC, on an address space basis Available Frame Queue thresholds (LOW/OK) are triggers to start stealing Issues Very large address spaces drives the cost of UIC Updating too high UIC updating holds address space locks which can impact other requests for service such as page fault processing z/os 1.8 RSM will use a Global LRU algorithm for page replacement Use a referenced bit for each page as an indicator of which pages have been recently referenced When a page steal routine is called it cycles through a page frame table If page is unreferenced and is stealable, it is stolen and put in AFQ If page is referenced it can't be stolen, but the reference bit is reset, "aging" the page so it may potentially be stolen the next time page steal runs UIC Changes In z/os 1.8 UIC now represents the time for one steal cycle through the whole storage area Reported in RMF as a range from (18 hours) Higher the UIC, the lower the storage contention Lower the UIC, the higher the storage contention New UIC Metrics Current UIC - calculated every second Minimum UIC - Lowest UIC during the last walk through the storage area Maximum UIC - Highest UIC during the last walk through the storage area Old form of UIC (0-2054) is still kept for compatibility by both SRM and RMF WSC IBM Corporation,

18 Changes to Physical Swap Address spaces are incrementally trimmed then fixed storage is swapped to Aux Technique is mainly used to manage pageable storage shortages Issue: Amount of fixed storage owned by an address space is getting too large to swap Swappable Address Space Pageable Storage Fixed Frames } } } 1. Trim Request 2. Trim Request 3. Swap the Fixed Frames AUX z/os 1.8 will no longer physically swap address spaces to Aux Review the CPU Activity Report to understand impacts of paging (Check Out and Ready and OUT and Wait counts) Review Fixed Frames for swapped in users to determine capacity required especially if large fixed frame users are swappable (Use RMF Mon III STORF report) Pageable Storage Shortage Previously managed by a physical swap z/os 1.8 will now do an "In Real" Swap where frames in the shortage area are exchanged with other frames For Pageable Storage Shortage below 16M frames will be exchanged with frames above 2G or above 16M For a Pageable Storage Shortage between 16M-2G frames will be exchanged with frames above 2G Pageable storage shortage above 2G indicates there are too many fixed frames and additional central storage is needed Message IRA404I lists the five largest users of fixed frames in the shortage area 4T 2G 16M WSC IBM Corporation,