Why did my job run so long?

Transcription

1 Why did my job run so long? Speeding Performance by Understanding the Cause John Baker MVS Solutions

2 Agenda Where is my application spending its time? CPU time, I/O time, wait (queue) times What am I waiting for? Various flavors of queue time What can/should I do about delays? Real world comparison Q/A Stay tuned! Conclusions and wrap up 2

3 Distribution of Elapsed Time Elapsed time = CPU time + I/O time + wait times CPU time = TCB + SRB I/O = IOSQ + PEND + CON + DISC Wait (queue) times Initiator Allocation (ENQ contention) System services (HSM recall) CPU Delay LPAR dispatch 3

4 Sample Job A JOB RUNTM CPUTM IOTIME JOBA 4:23:53 0:48:21 0:09:12 Elapsed time over 4 hours CPU time almost 1 hour I/O time under 10 minutes = Focus on CPU time 4

5 Reducing CPU time Recompile Many improvements in OS updates Tune application Application Performance Tools (e.g. Strobe, FreezeFrame) Identify CPU use by area of source code Make friends with your developer 5

6 Sample Job B JOB RUNTM CPUTM IOTIME JOBB 2:41:30 0:21:20 1:37:44 Elapsed time under 3 hours CPU time 20 minutes I/O time over 1.5 hours = Focus on I/O time 6

7 Identify patterns Reducing I/O time sequential vs random; read vs write Buffers For VSAM consider NSR vs LSR Give SORT memory but not too much! Block size System-determined generally works well but check! Half track for sequential; No smaller than 2K for random Compression zedc sounds promising! Include Storage Subsystem in your capacity planning 7

8 Wait/Queue time 8

9 High Utilization = High wait time Time vs Utilization 30 Response/Elapsed Time Elapsed time grows exponentially with utilization. Increasing priority doesn t make the CPU any faster Wait Execution 5 0 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95% 100% Percent Utilization At high utilization levels, wait time is much greater than service time 9

10 Flavors of Queue (wait) Time Wait for server : initiator / CICS AOR / IMS MPR CPU delay (wait for logical CPU) I/O delay (iosq, pend, disconnect) Capping delay (LPAR capped vs actual delay) Resource Group maximum enforced Wait for LPAR (logical CPU) to be dispatched PR/SM weight Demand from other LPARs CPC/CEC capacity 10

11 Initiator Queue SMF: R723CQDT TOTAL queue time (divide for average per job) Just start more inits? Not necessarily a good idea Tuning to reduce the number of simultaneously active address spaces to the proper number needed to support a workload can reduce RNI and improve performance ibm.com/servers/resourcelink/lib03060.nsf/pages/lsprwork?OpenDocument&pathID= 11

12 Automated Initiators: Less is More Benchmark: TM vs WLM concurrent Jobs TM WLM TM jobs ahead Active Jobs Time (hours) Concurrency based on performance and utilization 12

13 CPU Delay Wait for logical CPU SMF: R723CCDE Work is ready to run but is delayed access to CPU Related to Service Class / goal / importance Dispatching priority There is almost always some CPU delay Tolerance is subjective Are goals/sla s being met? Priorities are relative overloading leads to thrashing Consider discretionary for MTTW 13

14 Utilization vs CPU delays At 100% busy, throughput degrades significantly 100 Utilization vs CPU Delay % delay may be acceptable HIDLY MDDLY LODLY UTIL 14

15 I/O Delay IOSQ: HyperPAV Pend: CMR = overloaded controller DB = volume contention (reserve?) Any remaining = likely channels Disconnect Random read misses Synchronous remote copy 15

16 Revisit: sample Job B JOB RUNTM CPUTM IOTIME SMF30AID SMF30AIW EXCPS JOBB 2:41:30 0:21:20 1:37:44 0:40:31 0:04: Disconnect time of 0:40:31 = 40% of total 1:37:44 40:31 (2431 seconds) divided by I/O s =.263 ms average disconnect time Likely not unreasonable for random reads (consider SSD) Could also be replicated writes Become familiar with your typical application response times 16

17 Capping Delay Possible when caps present SMF70NSW WLM caps the logical CPUs Delays LPAR dispatch SMF70NCA Work is actually delayed for CPU due to capping Consider TM automation 17

18 Capping vs Delay MSU Demand vs R4HA LPAR is capped (SMF70NSW) Work is delayed SMF70NCA 0 LPAR_C LPAR_B LPAR_A CPCR4HA CAP MSU/HR 0:00:00 0:15:00 0:30:00 0:45:00 1:00:00 1:15:00 1:15:00 1:30:00 1:45:00 2:00:00 2:15:00 2:30:00 2:45:00 3:00:00 3:15:00 3:30:00 3:45:00 4:00:00 4:15:00 4:30:00 4:45:00 5:00:00 5:15:00 5:30:00 5:45:00 6:00:00 6:15:00 6:30:00 6:45:00 7:00:00 7:15:00 7:30:00 7:45:00 8:00:00 8:00:00

19 Capping can impact all Workloads 100 STC_H: Importance 1 Batch is not the only workload suffering. Even the most critical workloads are unable to meet their goal Velocity Machine CPU reaches 100% Capping begins Goal Velocity

20 Resource Group (max) Also a form of capping (same WLM algorithms) Pro: Useful to control problem applications Con: Static. Not flexible R723CCCA Resource Group maximum enforced Will override Service Class goals 20

21 LPAR Dispatch Delay Ratio of logical processor busy to physical processor busy Not always as obvious but very common! Term Short CPs introduced by Kathy Walsh (IBM WSC) Share, Aug MXG: PLCPRDYQ Improved with Hiperdispatch and IRD 21

22 LPAR Dispatch Delay 40% delay 100 LPAR Dispatch Delay INITS CPUBSY MVSBSY More initiators and/or higher dispatching priority will not resolve this problem 22

23 What does this look like in the real world? Let s take a trip to the deli 23

24 How many in the store at one time? INITIATORS 24

25 Who s next in line? Dispatching Priority (Service Class) 25

26 How long til I can give my order? Logical Processor (CP) Busy 26

27 How long til I m done! Physical Processor (CP) Busy 27

28 Forcing one step only stresses the next 28

29 29

30 Balance 30

31 About MVS Solutions MVS Solutions Inc. Installed in over 200 datacenters worldwide IBM Partner in Development ThruPut Manager Automated Workload Balancing Automated Batch Prioritization Automated Capacity Management Contact Join our Blog at 31

32