z Processor Consumption Analysis Part 1, or What Is Consuming All The CPU?

Transcription

1 z Processor Consumption Analysis Part 1, or What Is Consuming All The CPU? Peter Enrico z/os Performance Education, Software, and Managed Service Providers Enterprise Performance Strategies, Inc rd Avenue North, #145 Bradenton, FL Voice: Mobile: Creators of Pivotor Enterprise Performance Strategies, Inc. z/os Proc Consumption Analysis - 1

2 Contact, Copyright, and Trademark Notices Questions? Send to Peter at or visit our website at or Copyright Notice: Enterprise Performance Strategies, Inc. All rights reserved. No part of this material may be reproduced, distributed, stored in a retrieval system, transmitted, displayed, published or broadcast in any form or by any means, electronic, mechanical, photocopy, recording, or otherwise, without the prior written permission of Enterprise Performance Strategies. To obtain written permission please contact Enterprise Performance Strategies, Inc. Contact information can be obtained by visiting Trademarks: Enterprise Performance Strategies, Inc. presentation materials contain trademarks and registered trademarks of several companies. The following are trademarks of Enterprise Performance Strategies, Inc.: Health Check, Reductions, Pivotor The following are trademarks of the International Business Machines Corporation in the United States and/or other countries: IBM, z/os, zseries WebSphere, CICS, DB2, S390, WebSphere Application Server, and many others. Other trademarks and registered trademarks may exist in this presentation z/os Proc Consumption Analysis - 2

3 Abstract and Reports Offer Abstract The first step to any processor analysis is to understand your processor configuration and settings. The second step is to understand what workloads, address spaces, and transactions are consuming the fixed processor resource. It is only after understanding what and how the processor is being consumed can you conduct any sort of processor tuning or optimization exercise. During this presentation Peter Enrico will show you how to conduct a processor resource consumption analysis. You will be provided with a top down approach to better understand processor measurements available to help you gain a drilldown insight into how the CPU resource is being consumed, and by what LPARs, Workloads, and transactions. Shown is what is known as a drill down approach for a processor performance analysis. z/os Proc Consumption Analysis - 3

4 Other EPS presentations this week What Who When Where zsystems Processor Consumption Analysis Getting Started Part 1 zsystems Processor Consumption Analysis The Next Steps Part 2 zsystems Processor Consumption Analysis Getting Started Part 1 zsystems Processor Consumption Analysis The Next Steps Part 2 Peter Enrico Peter Enrico Peter Enrico Peter Enrico Tuesday, 11:30am Tuesday, 13:45pm Wednesday, 15:00pm Wednesday, 16:30pm z/os WLM Revisiting Goals Over Time Peter Enrico Thursday, 10:00am Sydney Sydney Sydney Sydney Sydney What Who When Where Memory Management in the TB Age More / Slower vs Fewer Faster CPUs: Practical Considerations The MLC Cost Reduction Cookbook Scott Chapman Scott Chapman Scott Chapman Monday, 16:30am Wednesday 10:00am Friday 8:45am Sydney Sydney Partenkirchen z/os Proc Consumption Analysis - 4

5 Performance Workshops Available During these workshops you will be analyzing your own data! WLM Performance and Re-evaluating of Goals Instructor: Peter Enrico and Scott Chapman October 3 7, 2016 Chicago, IL, USA September 12 16, 2016 Stockholm, Sweden Parallel Sysplex and z/os Performance Tuning (Web / Internet Based!) Instructor: Peter Enrico and Scott Chapman August 16 18, 2016 Essential z/os Performance Tuning Workshop Instructors: Peter Enrico, Tom Beretvas, and Scott Chapman z/os Capacity Planning and Performance Analysis Instructor: Ray Wicks z/os Proc Consumption Analysis - 5

6 CPU Measurement Reports Processing/Discussion Offer!!! Special Reports Offer! See your Processor measurements in chart and table format Please contact me, Peter Enrico for instructions for sending raw SMF data Send an to Deliverable: Dozens of processor based reports (charts and tables) CPU - Machine Level Analysis CPU - LPAR Level Analysis CPU - LPAR SMT Analysis CPU - HiperDispatch CPU Activity CPU - SMF 113 Processor Counters WLM - Workload Utilization Analysis And much more! One-on-one phone call to explain your measurements z/os Proc Consumption Analysis - 6

7 Presentation Overview Many areas need to be examined when decomposing CPU consumption This presentation just discussions some of the many areas Basic Processor Consumption Analysis Decomposing CPU Consumption At Machine Level At LPAR Level At WLM workload Level Looking at CPU Dispatching Priorities Looking at Latent Demand Remember, there is much more to be looked at z/os Proc Consumption Analysis - 7

8 Breakdown of General Purpose Processor We always needed to understand the break down of CP CPU consumption Total Physical Processor (CEC) Time Total LPAR Dispatch Time (Partition 1) Total LPAR Dispatch Time (Partition N) *PHYSICAL* Effective LPAR Dispatch Time MVS Captured Time MVS Uncaptured Time Partition LPAR Mgmt Time etc... Effective LPAR Dispatch Time MVS Captured Time MVS Uncaptured Time Partition LPAR Mgmt Time LPAR Time to Manage LPAR Workload CPU Service Class CPU Service Class Period CPU Workload CPU Service Class CPU Service Class Period CPU Address Space or Enclave CPU (TCB,SRB,RCT,IIT,HST) Address Space or Enclave CPU (TCB,SRB,RCT,IIT,HST) z/os Proc Consumption Analysis - 8

9 Breakdown of ziip Engine Time We need to understand how PR/SM allocates the ziip processor resource In all measurements ziips Total Physical ziip Processor Time ziip Qualified But not Eligible (so on CP) ziip CPU Time (Partition 1) ziip Captured Time ziip Uncaptured Time etc... ziip CPU Time (Partition N) ziip Captured Time ziip Uncaptured Time ziip Qualified But not Eligible (so on CP) ziip Eligible But Overflowed To CPs needs help ziip Eligible on ziip ziip Eligible on ziip ziip Eligible But Overflowed To CPs needs help Workload ziip Service Class ziip Service Class Period ziip Address Space or Enclave ziip z/os Proc Consumption Analysis - 9

10 Breakdown of zaap Engine Time We now need to understand where the zaap CPU time is consumed Total Physical zaap Processor Time zaap CPU Time (Partition 1) zaap CPU Time (Partition N) zaap Captured Time zaap Uncaptured Time etc... zaap Captured Time zaap Uncaptured Time zaap Eligible put on General CP zaap Eligible on zaap zaap Eligible on zaap zaap Eligible put on General CP Workload zaap Service Class zaap Service Class Period zaap Address Space or Enclave zaap z/os Proc Consumption Analysis - 10

11 Decomposing CPU Consumption - Machine Level Analysis Which LPARs are using the physical CPUs? Utilization MSUs Look at LPAR Management Busy% to ensure it is within guidelines Was there LPAR weight enforcement? z/os Proc Consumption Analysis - 11

12 Machine Busy CP Percent Busy Machine CP CPUs 100% Busy nearly all day z/os Proc Consumption Analysis - 12

13 Could measure CPU Consumption in MSUs Another way of measuring CPU consumption is MSUs z/os Proc Consumption Analysis - 13

14 Machine Busy ziip Percent Busy Machine ziip CPU relative high utilization during peaks. z/os Proc Consumption Analysis - 14

15 Assigned LPAR Weights for CP Engines LPAR weights set very well, but is there any opportunity to bump SYSPROD up just a little against SYSDVL? Note: Sum of weights: = 100 LPAR Guaranteed Percentages of CEC: LPARCLMS: 8% (SYSC) SYSDVL: 32% (SYSD) SYSPROD: 57% (SYSA) SYSTEST: 3% (SYSB) z/os Proc Consumption Analysis - 15

16 Weight Enforcement Weight enforcement from 7:00am till about 7:00pm z/os Proc Consumption Analysis - 16

17 LPAR Management Time LPAR Management time - Guideline: Combined LPAR Management% for CPs, ziips, zaaps (combined) should be less than 2% z/os Proc Consumption Analysis - 17

18 HiperDispatch Parked and Unparked CPs Due to LPAR weights and activity in all LPARs, SYSA usually has 2 CP CPUs in pool. If bump SYSA weight up slightly relative to other LPARs might be able to get a 3 rd shared processor during the day. z/os Proc Consumption Analysis - 18

19 HiperDispatch Parked and Unparked CPs Based on the weight of this LPAR, 2 CPs are regularly unparked, and 1 CP is usually parked, but gets unparked when needed. z/os Proc Consumption Analysis - 19

20 Decomposing CPU Consumption - LPAR Level Analysis CPU Consumed by an LPAR The LPAR utilization trinity (LPAR Busy%, Workload Busy%, and MVS Busy % Capture ratios Work Unit distribution to gain insights to latent demand Host Effect CPU Consumption z/os Proc Consumption Analysis - 20

21 z/os CPU Times Capture Ratios used to understand the stability and cost of system overhead Effective Dispatch Time Time that the z/os and the workloads were executing on the CPU MVS Capture Time Time that can be accounted for towards specific workloads MVS Un-captured Time System overhead Capture Ratio Ratio of MVS Capture Time to Effective Dispatch Time Total LPAR Dispatch Time (Partition 1) Effective LPAR Dispatch Time MVS Captured Time MVS Uncaptured Time Workload CPU Partition LPAR Mgmt Time Service Class CPU Service Class Period CPU Address Space or Enclave CPU (TCB,SRB,RCT,IIT,HST) z/os Proc Consumption Analysis - 21

22 Causes for Uncaptured Time Many causes for uncaptured time. Common causes are as follows: High page fault rates Full preemption Suspense lock contention Spin lock contention Getmain/Freemain activity (recommend cell pools) SRM time-slice processing Interrupts SLIP processing Long queues being processed in uncaptured processing Affinity processing (such as need for a specific CPU or crypto facility) z/os Proc Consumption Analysis - 22

23 LPAR Busy%, Workload%, MVS%, and Capture Ratio MVS Busy % is what utilization relative to MVS Waits. Basically what MVS wanted. Delta between LPAR Busy % and MVS Busy % is usually due to capping Delta between LPAR Busy% and Workload % is Un-captured %. Capture Ratio is ratio of LPAR Busy% and Workload Busy % z/os Proc Consumption Analysis - 23

24 Understanding Dispatching to Gain Insight to MVS Busy % Dispatch Time Time logical processor is associated with a physical processor Logical Processor associated with a physical processor Dispatch Interval Logical Processor disassociated with a physical processor MVS Time Time z/os was busy before voluntarily giving up a processor XXXXXXXXXXX Voluntary Wait - z/os voluntarily gives up the processor - MVS time equals dispatch time Involuntary Wait (mostly on vertical mediums) - z/os does not give up the processor voluntarily - Instead PR/SM un-dispatches the partition - MVS time will be greater than dispatch time z/os Proc Consumption Analysis - 24

25 Decomposing CPU Consumption - WLM Workload Level Analysis CPU Consumption at the importance level CPU Consumption at the WLM Service Class and Service Class Period Level Commentary about Report Classes Other CPU consumption measurements CPU consumed at promotion Did lower importance work not consume CPU due to lack of demand or due to lack of CPU? z/os Proc Consumption Analysis - 25

26 Workload Utilization by Importance Level In this example, notice that little CPU is being used by low importance work. Workload utilization is calculated as (sum Service Class CPU Tim) (#Logical CPs * Interval Time) Is it because there is little to no low importance work? Or is it because low importance work cannot run due to lack of capacity? z/os Proc Consumption Analysis - 26

27 APPL% by Importance Level Profile of this chart is the same as that of workload utilization, but scale is based on percentage of 1 CPU. So 180 means 1.8 CPUs of capacity. z/os Proc Consumption Analysis - 27

28 Delay Samples by Importance Level Take note of delay samples: High CPU delay in lower importance service class periods indicates latent demand z/os Proc Consumption Analysis - 28

29 Workload Utilization by Service Class Period Top consuming service classes z/os Proc Consumption Analysis - 29

30 Delay Samples by Service Class Period Take note of delay samples: High CPU delay in lower importance service class periods indicates latent demand z/os Proc Consumption Analysis - 30

31 CP CPU Delay Samples by Service Class Period Also take note of just CP CPU delay samples: High CP CPU delay in lower importance service class periods indicates latent demand z/os Proc Consumption Analysis - 31

32 Workload Utilization for Top Report Classes Top consuming report classes Effective usage of report classes can help an analysis immensely. z/os Proc Consumption Analysis - 32

33 Workload Utilization as a Percentage of 1 CPU (APPL%) - By CPU Type Also make sure you understand the CPU cross over from ziip and zaap engines to CP engines. Are you crossing over during peak periods of the month and possibly affecting your monthly WLC software bill? z/os Proc Consumption Analysis - 33

34 Processor Speed Information SMF113_2_CPSP CPU speed in cycles per microsecond Recorded for each logical CPU (but is really the physical CPU speed) Example: z10 : 4404 Cycles/Mic (i.e. 4.4 GHz) z196: 5208 Cycles/Mic (i.e. 5.2 GHz) zec12: 5500 Cycles/Mic (i.e. 5.5 GHz) z13: 5000 Cycles/Mic (i.e. 5.0 GHz) For knee capped processors Will reflect the reduced speed But ziips and zaap on the machine will show full speed numbers z/os Proc Consumption Analysis - 34

35 Example of CPU Speed 16 full speed CPs with 3 full speed ziips 7 slower speed CPs with 3 full speed ziips Chart created at z/os Proc Consumption Analysis - 35

36 Workload Utilization as a Percentage of 1 CPU (APPL%) - By CPU Type Also make sure you understand the CPU cross over from ziip and zaap engines to CP engines. Are you crossing over during peak periods of the month and possibly affecting your monthly WLC software bill? z/os Proc Consumption Analysis - 36

37 Looking at CPU Dispatching Priorities (an approximation) z/os Proc Consumption Analysis - 37

38 Average CPU Dispatching Priorities for Address Spaces Can use the SMF 30 to gain insights as to the average CPU dispatching priority. - Not a real accurate measurement, but can gain insight into where in the DP order work is. z/os Proc Consumption Analysis - 38

39 Average CPU Dispatching Priorities for Address Spaces If examined up close, we can get a feel for where WLM is placing the work from a CPU dispatching priority point-of-view. z/os Proc Consumption Analysis - 39

40 Insights into Latent Demand Dispatched Work Accumulating CPU Using Samples Queued Work - waiting at priority Accumulating CPU delay samples Dispatcher Queue Processors z/os Proc Consumption Analysis - 40

41 Min / Max / Avg Work Unit Queuing z/os Proc Consumption Analysis - 41

42 Work Unit Distribution Showing Latent Demand z/os Proc Consumption Analysis - 42

43 Delay Samples by Importance Level Take note of delay samples: High CPU delay in lower importance service class periods indicates latent demand z/os Proc Consumption Analysis - 43

44 Older Style In-Ready Distribution Less Accurate Latent Demand z/os Proc Consumption Analysis - 44

45 Session summary Many areas need to be examined when decomposing CPU consumption This presentation just discussions some of the many areas Basic Processor Consumption Analysis Decomposing CPU Consumption At Machine Level At LPAR Level At WLM workload Level Looking at CPU Dispatching Priorities Looking at Latent Demand Remember, there is much more to be looked at z/os Proc Consumption Analysis - 45

46 CPU Measurement Reports Processing/Discussion Offer!!! Special Reports Offer! See your Processor measurements in chart and table format Please contact me, Peter Enrico for instructions for sending raw SMF data Send an to Deliverable: Dozens of processor based reports (charts and tables) CPU - Machine Level Analysis CPU - LPAR Level Analysis CPU - LPAR SMT Analysis CPU - HiperDispatch CPU Activity CPU - SMF 113 Processor Counters WLM - Workload Utilization Analysis And much more! One-on-one phone call to explain your measurements z/os Proc Consumption Analysis - 46