iseries Performance Configuration and Tools September 13, 2005 Jim Oberholtzer Senior Technical Architect INFORMATION S O L U T I O N S THAT WORK FOR BUSINESS GREEN BAY MILWAUKEE MINNEAPOLIS Agenda Work Management Objects Used in Tuning Initial Setup Setting a Performance Plan The Performance Tools iseries Navigator Performance Explorer Conclusions
Work Management Objects Subsystem Descriptions Work Station Entries Prestart Job Entries Job Queues Classes Shared Pools Activity Levels Job Descriptions System Values Job Table entries have a large affect on IPL performance, but also affect operational performance The CHGIPLA command will set how the entries are handled at IPL CPRJOBTBL *NORMAL will compress the table on a normal IPL CPRJOBTBL *ABNORMAL will compress only on an abnormal IPL Set the number of jobs correctly to start with to avoid some of the problems
CHGIPLA System Values DSPJOBTBL command will show the state of the table entries (no equivalent in iseries Nav) The number of entries The size in bytes The actual number of entries The number of entries available The number of entries used The number of entries not available
DSPJOBTBL DSPJOBTBL
Threads vs. Jobs Thread support was added for JAVA and other multithread languages Every job on the system has at least one thread A job can have more than one thread Each thread requires an activity level The API QWTCHGJB will allow changes to a thread directly The only place a job is referred to now is on a job queue Initial Setup Information Center, System Management, Work Management http://publib.boulder.ibm.com/iseries/v5r3/ic2924/info/rzaks/rzaks1.htm Understand some basic terms and concepts Threads can be in several transition states Active Wait to Ineligible Active to Wait Active to Ineligible
Short Wait Wait States Up to 2 Seconds waiting for resources Waiting for a break Message Restore Display *YES Still in the Activity Level If the system waits more than 2 seconds, The thread is in Extended Short Wait, Loses Activity Level, and goes into Long Wait (Paging) Long Wait Wait States No Activity Level assigned Types: Key/Think (User Generated) Record Locks DDM Requests There are more. If memory is tight, the thread is purged from main storage (paging)
Ineligible threads Wait States Goes Ineligible when an activity level is not available when the thread needs to process When a thread goes ineligible, it is placed on a FIFO queue (most of the time) to get the next available activity level The transition from ineligible to active is VERY SLOW Time Slice A time slice is the amount of time a thread can have the resources of the machine at one time - Not how long the thread runs At end of time slice: If a thread with > = priority in queue, purge from main storage Move a thread in from Ineligible queue Otherwise Keep Going for another time slice On Line threads may use Time Slice End Pool
Initial Setup Determine: How many interactive subsystems How many batch subsystems What communications there will be How much ODBC traffic Operations Navigator usage Note: HPT (Hardware Page Table) for LPAR systems is critical to get right at i5 Machine Size Example V4R4M0(V5R3M0) No Recommendation on current web site S = Initial Machine Pool Size S=M+J+L+F+D + Guest M = Minimum Machine pool size J = Job Space L = Communications Space F = Functional Space D = Disk Space Guest = Total memory given to Guest Partitions
Minimum Machine Pool Size Determined by Main Storage Size Changes with operating system version Multiply main storage by.04 Add 12mb (12288K) So, a 2048 MB machine: 2097152 *.04 + 12288 = 96175k 97 MB Is minimum Job Space Add 4k for each job on the system (I usually use 6k, don t forget system and group jobs) 500 jobs on the system = 3000k Note the terminology: job vs. thread Work Management guide uses jobs here
Communications Space Based on the amount of communications that will be done Exclusive of the communication subsystems 125kb/Comm line 100kb/Protocol (SDLC, BSC, TCP/IP, ETC.) 25kb/Remote Workstation Controller 1.5meg per LAN Card 3600 kb/ipcs* with one LAN card *Now called ixs or ixa Functional Space Additional functions require additional memory: Save/Restore 68kb LAN (Token Ring) 250kb X.25 48kb Double Byte 50kb
Drive Space Use 64k for each of the first 64 drives on the system Add 20k for each additional drive System with 80 disk drives: (64 * 64k) + (16 * 20k) = 4416k A note re: Drive Space, Remember Disk Arms Hosted Guest Partitions Memory given to guest partitions taken from machine pool If two partitions running Linux 256 mb given to each For AIX 5L, consult the AIX manuals Add 512 mb to Machine pool minimum more is better Virtualization happens there HPT resources at i5
Memory allocation for LPAR With Power4 and sstar somewhat straightforward Power 5 changes the world Memory assigned in blocks of 16, 32, 64, 128, & 256Mb The Hardware Page Table is critical to LPAR performance Processor and memory affinity are critical for performance The default ratio is 1/64 but that my not be correct for your environment Only a skilled experienced (certified) technician will be able to determine the best configuration Initial Setup The next step is to set up the Shared Pools I suggest: *INTERACT for interactive jobs *SPOOL for Spoolers *SHRPOOL1 for Batch *SHRPOOL2 for Com., SNADS *SHRPOOL3 for Pgmr Interactive *SHRPOOL4 for Pgmr Batch
Initial Setup Memory Use the WRKSHRPOOL command to set share pool size and activity level Make sure that all subsystems have at least two memory pools: The first pool should be *BASE The second pool should be the shared pool for that subsystem Ensure that all routing entries use Pool 2 for the job memory pool Initial Setup Time slice Interactive jobs should have a time slice that is three times the average CPU usage for each transaction. IE: If performance tools show CPU/Transaction is 150 milliseconds, then timeslice should be about 450 milliseconds Round to 500 Found on the System Report in Performance tools Leave Batch timeslice at 5000 milliseconds under normal use This includes ODBC type transactions using SQL
Initial Setup Spoolers *SPOOL Size is set by number of active writers, as is activity level Start at 1500kb and activity level 1 For each writer, add 200kb and an activity level up to 3 writers Over 3 writers, add 2000k per job plus activity level Watch the paging and printers to fine tune the activity level Never have any Ineligible Add memory if paging is >25 or Faults >.5 Interactive Memory The books says: (main storage Machine pool QSPL) *.7 Activity level = 25% of QINTER active jobs Recommendation: Set Machine Pool *BASE should be at least 30% (interactive formula) Activity level for *BASE is 30 for 1.5gb Systems (5) Set Batch up If not enough memory for Interactive is left reduce batch load (or get memory)
Initial Setup - Batch Memory Compiles take about 16000kb Most batch jobs will run better the more memory you give them (min 2000kb each) Set up activity level to match the number of threads that will run in the Shared Pool Be careful with multi-thread batch programs WebSphere Server/Commerce Suite, any Java system Domino Watch Paging levels and adjust as needed Consider Using routing entries to classify jobs change run priority and timeslice Setting a Performance Plan and Objectives Define Good Response Time Many users don t need sub-second response Usually 1 to 2 second response time is acceptable Batch cycle times are defined by the amount of batch window you have Save while active will shorten save downtime Consider using system managed journals
Performance Objectives Key/Think time is the time the user spends thinking between transactions This time is found on the transaction report from the performance tools If the Key/Think time is long, good response time is not needed as badly Sequence jobs so several short jobs run against one longer running job Performance Objectives All iseries can collect performance data Use iseries Navigator, Collection Services PM/400 is installed on all machines IBM will analyze your performance for you, Business Partner is your best asset Sign PM/400 permission form for your partner or register your partner on the web http://www-1.ibm.com/servers/eserver/iseries/pm400/existing_customers/authorization.html
iseries Navigator & PM400
This is the Management Central Collection services panel
Note Create Database files You control when the cycle starts How long is data kept? Disk space can be used up Extended Cache PEX Data
Analyze Performance Data Only available if you create the files from the collection System report, option 1 from print performance report menu Page 2 shows Response times CPU/Transaction DB-I/O per transaction Analyze Performance Data Too many I/O = bad response time Limit size/numbers of cursors in SQL programs Page 6 of system report shows the same data as the WRKSYSSTS panel does for the entire observation Adjust memory and activity levels to minimize faulting and paging
Other Performance Reports Component report Transaction report I use these reports most often Find problems Begin drill down Other Performance Reports Lock report Shows seizes and locks Produces an output file to report against Pool report Shows paging and faulting by collection period Zeros in when the problem is occurring
Other Performance Reports Batch job trace report Job report Detail data reported for each job Resource report iseries Navigator When loading iseries Nav, use the manual install Management Central is where the action is Determine which machine will be the Central System
Start a multisystem Monitor job!
Single Machine Collection Single Machine Collection
Performance Explorer Remember TPSD, (Timing and Paging Statistics) At V3R6 Performance Explorer became part of Performance Tools Option 5, from the PERFORM menu Performance Explorer Watch DASD closely, uses A GREAT DEAL OF SPACE 10 minute trace = 3 million records spool file = 100,000 pages! Reserve 500 meg or so for collection Keep observations short 5 minutes is usually enough Use idoctor in conjunction with PEX
Add Definition Sets up the name of the collection The type of collection Statistical Trace Profile Which jobs or programs to observe Organization of the data Statistical collection Collection Definition Used mainly to find problem areas for further study Why is the application behaving the way it does Don t prejudge the software
Statistical collection Collection Definition Use to establish control points When used with *HIER option shows program flow Trace The most detailed collection Profile Collection Definition Used to zero in on specific problems Most useful when you know which programs are the problem
Print from Performance Explorer Option 6 from the Performance Explorer menu Creates organized and sorted reports of the data collected Use the select criteria to zero in on the problem
Performance Explorer Breaks down CPU usage and paging MI programs (user and system) MISVLa (Supervisor Linkages) MI Processes Licensed Internal Code Tasks Does not produce detail for Licensed internal code or low level MI Performance Explorer MI-SVL (MI Supervisor Linkage) Link between HLL programs and Licensed internal codes Insulates HLL from the SLIC QDBPUT is the module that puts a record to the database By watching these modules, you can find your problems
HVLPTASK Used on Power4 to help report CPU and other statistics DOES NOT REALLY USE CPU!!!!! Not applicable to i5 boxes Conclusion The Performance tools are a good investment only if you will run the reports and spend the time to analyze them Requires a better than average programmer or a systems integrator to do a good job
Conclusion Performance tools do not help batch processing much because of the lack of a transaction boundary. Manual analysis of the job streams is needed here The idea is to try and make the machine last as long as the lease Where do you find me? Computech on the Web: www.compures.com Jim Oberholtzer joberholtzer@compures.com Milwaukee: 262/785-8111 250 Sunnyslope Rd Suite 300 Brookfield, WI 53005