ASG-TriTune Started Task Monitor Invocation via ASG-TMON-generated z/os Commands Best Practices Document

Transcription

1 1 ASG-TriTune Started Task Monitor Invocation via ASG-TMON-generated z/os Commands Best Practices Document

2 2 Introduction Utilizing the ability of an ASG-TriTune monitor to be initiated immediately via an external global monitoring tool provides the additional application performance information to assist in determining the root cause of system and applications performance issues at the time of the exception, rather than waiting for the condition to occur a second or third time. ASG-TriTune provides application program performance profiling either as part of a testing and implementation quality improvement methodology or as part of an event-driven methodology. The use of ASG-TriTune described in this document covers its use in an event-driven methodology. It is recommended to determine that the exception event is persistent, rather than a one-time anomaly, before utilizing ASG-TriTune for further analysis. What is determined to be a persistent exception event will vary from customer to customer. Customers who have already invested in Mainframe system monitoring and/or message automation solutions have previously defined the events they consider exceptions, that is, those events recognized by their system monitors or message automation solutions which require additional investigation to determine what actions will need to be taken to analyze and resolve the exception. Those actions may include notifications to the CICS or DB2 systems programmer, initiation of mainframe processes to help further identify the exception, and even automated calls to the personnel responsible to resolve the particular exception. In most cases, the first occurrence of an exception event may not be considered a persistent problem, but the exception event continuing across successive monitoring intervals would be considered a persistent exception. This document is intended to be a guide to assist in using this ASG-TriTune feature. The recommendations that are listed are suggestions none of the recommendations have to be implemented to have the ASG-TriTune started task invoke a monitor.

3 3 Prior to Implementing the ASG-TriTune Started Task Monitor Invocation Feature Before using the ASG-TriTune Started Task Monitor Invocation Feature, we recommend that Global Profiles are created for the Monitors that will be invoked via this feature. Remember, that all monitors that are invoked via this feature will be Global Profiles. The following documents our recommendations for these monitor definitions. Monitor Dataset Naming Conventions We recommend that before using the ASG-TriTune Started Task Monitor Invocation Feature, a review of the name that will be used to create the Global monitor datasets is completed. All personnel who will be analyzing the ASG-TriTune Global monitor files and the ASG-TriTune Started Task must have UPDATE authority to these datasets. To review the name that will be formed by ASG-TriTune for the Global Monitor Datasets, select option G (Global) from the ASG-TriTune Primary Option Menu, and then select Option 0 (Parameters). The following screen will appear : Looking at Monitor Dataset Options, the name that will be used to create the Global Monitor dataset in this example is ESWUSER.TUN.&PROFILE.&SYSDATE.&SYSTIME. Based on this example, to access the Global Monitor files, all personnel who will use the ASG-TriTune Started Task Monitor Invocation Feature and the ASG-TriTune Started Task would need to have UPDATE access to datasets beginning with the high-level qualifiers ESWUSER.TUN. Contact your site security personnel to verify that UPDATE access is defined.

4 4 Contact your ASG-TriTune administrator to make any changes needed to the monitor dataset name so that it conforms to site-specific naming conventions and can be updated by ASG- TriTune as well as personnel using the ASG-TriTune Global Monitor facility. By design, Global Monitor datasets are intended to be accessible by all personnel who utilize ASG-TriTune. It is therefore recommended to avoid using names beginning with &USERID when naming the Global Monitor Data Sets, such as the following : &USERID.&PROFILE.&SYSDATE.&SYSTIME In most cases, datasets with an individuals high-level qualifier will not be accessible by other personnel.

5 5 Monitor Profile Naming Conventions There are many ways you can choose to name your monitor profiles. The following are some recommendations for monitor profile naming conventions. We will comment about the advantages and disadvantages of each approach at the end of each example. Example 1 Our first example is quite simple, yet effective. For the global profile name, we will use the name of the batch job or the started task for which an exception condition has been triggered. This information is available to all system monitors as well as automated messaging products, so there is no problem with determining the name of the profile. For online systems, we would want to predefine the global profiles that will be used. However, for batch jobs it would be rather tedious to define a profile for every batch job that will possibly have a monitor request issued directly to ASG-TriTune. At this time, note that ASG-TriTune, when there is a monitor request with no profile name defined, will use the job name as the profile name. If the profile is not defined to ASG-TriTune, ASG-TriTune will create the global profile. As an example, we will be planning to automatically issue ASG-TriTune measurement requests for CICS regions CICSPRD1 and CICSPRD2, IMS region IMSPAY01, and batch jobs beginning with the letters PAYRL. We would predefine global profiles for the online systems, CICSPRD1, CICSPRD2, and IMSPAY01. We would not predefine the profiles for all batch jobs beginning with PAYRL, as we will choose to let ASG-TriTune create the global profile if it does not exist. Since ASG-TriTune will be creating the global profiles for batch jobs, it is recommended to review the default parameters for global profiles to insure that the defaults will provide the values required for the monitors invoked directly by the ASG-TriTune started task. Example 1 Advantages The profile name is easy to form, and all monitors and automation products have access to this name. By using the job name only on the ASG-TriTune monitor invocation request, we will reduce the number of characters for the z/os command that is generated, which should comply with any restrictions on the length of z/os commands that can be generated by your monitor or automated messaging solutions. With using the job name as the profile name, only one measurement of an online region or batch job is possible at a time. This will be fine, as any second request for an ASG-TriTune measurement of an online region or batch job, when there is already a measurement in progress, would be unnecessary as the first measurement would capture the application performance information needed to identify any application performance issues.

6 6 Example 1 Disadvantages For this method of forming the profile name, the profiles for any batch jobs that potentially will have an ASG-TriTune measurement invoked via a z/os command will not be predefined. For online regions as well as batch jobs, the ASG-TriTune profile name will not further identify the exception condition that triggered the measurement. An analysis of the ASG-TriTune measurement will have to be done to determine additional details as to why the measurement request was initiated. Example 2 We have mission-critical transactions that we have defined exception alerts to invoke ASG- TriTune measurements. We want to immediately be able to find these monitors for analysis, as well as identify which of our mission-critical transactions is causing the exception condition. We could use the transaction name as part of the profile name. We recommend that you choose a letter that will be unique as the first letter of the monitor profiles that are invoked via this example, with the second letter being a T for transaction exception. This will allow the filtering of Global Profile names by the first two letters to find all of the invoked monitors that have been initiated due to mission-critical transaction exceptions. For this example, we will choose X as the first character of the monitor profile names (X for exception profiles). The second character would be a T. The next four characters would be the transaction name that triggered the exception condition. For this example, if the CICS transaction EC01 triggered a response time exception, the name of the monitor profile that will be used would be XCEC01. Example 2 Advantages With the naming conventions chosen, any ASG-TriTune measurements triggered for missioncritical transactions can be easily identified, located, and thus analyzed quickly to determine the cause of the transaction exception. Example 2 Disadvantages Two or more measurements could be invoked at the same time for the same online region. This could lead to additional stress on your online region as well as skewed analysis data for the measurements which were started while the initial measurement for the online region was executing.

7 7 Example 3 We recommend that you choose a letter that will be unique as the first letter of the monitor profiles that are invoked via this feature. This will allow the filtering of Global Profile names by the first letter to find all of the invoked monitors that have been initiated via this facility. For this example, we will choose X as the first character of the monitor profile names (X for exception profiles). For the second character, we would recommend a letter which designates the target of the monitor request. Examples of the second character are : B Batch Job C CICS Subsystem D DB2 Subsystem I IMS Subsystem The next four characters of the profile name can be used to further define the exception condition that was issued. Examples of the types of names that might be useful are listed below : ELPS Elapsed Time exception ARSP Average Response Time exception TRSP Transaction Response Time exception STRG Amount of Storage Used Exception ACPU Average CPU Time Exception MAXI Average I/O s Exception SRVU Service Units Exception PCPU Percent CPU Exception (over defined threshold) PWAT Percent Wait Exception (over defined threshold) TRUN Runaway Transaction Due to the restriction in some ASG-TMON components of allowing a console command of up to 56 characters, it is recommended to use a PROFILE name of no more than 6 characters. This will insure that the console command that will be issued by ASG-TMON to start a TriTune monitor will not exceed the 56 character limit. Using our example, if I wanted to view all monitors that were invoked by the TriTune started task for CICS regions, I would use the filter XC to filter the profile list to view only these monitors. To further filter this list, I could use the filter XCTRUN to view all monitors that were invoked by the TriTune stated task for CICS regions which had exceptions identified as runaway transactions.

8 8 Example 3 Advantages The ASG-TriTune monitors invoked via the z/os command interface will be easily identified. The monitor names also indicate as to what exception condition was met which initiated the ASG- TriTune measurement request. The global profiles can be predefined, as the names of the profiles can be predetermined based on the exception conditions for which you choose to initiate an ASG-TriTune monitor request. Example 3 Disadvantages Depending on the exception conditions you have defined, it may be possible to miss an important ASG-TriTune measurement. For example, if we had a response time exception condition for transaction ABCD in CICS region CICSPRD1, and at the same time had a response time exception condition for transaction MNOP in CICS region CICSPRD2, the profile names that would be formed are exactly the same, XCTRSP. Also, the username that will be associated with the monitor request will be GLOBAL as this is a global monitor. ASG-TriTune will not invoke a second monitor request issued with the same profile name from the same user. In this case, the first XCTRSP monitor request will be invoked, the second request will be rejected. Potentially important transaction performance information would not be available due to the formation of the same profile name. Again, these are just suggestions, and there are many more combinations that can be created. The key is to have these defined so that when viewing the list of Global Monitors defined, you can identify by viewing the profile name the type of exception that was identified.

9 9 Monitor Parameters specified in Profile The object of the invocation of a TriTune monitor via the TriTune started task is, in most cases, to get a measurement of the application that is involved in the current exception. The measurement should be short, but timely. We recommend the following values to be specified for all Profiles which will be invoked via the ASG-TriTune Started Task Monitor Invocation Feature : Batch Reports = N (will be looking at the results online) Job = DUMMYJOB (the jobname or online region will be specified on the INVOKE command to the TriTune started task until the exception occurs, it is not known which batch job or online region to monitor. By specifying a non-existent job for jobname, if someone does invoke the default profile by mistake, it will not run as there is no job to measure. ) Multi-Step Monitoring = NO (Only want to measure currently active step) System= <yourplex> (the Sysplex where the batch job or online system is executing if there is no Sysplex, leave this blank) Monitor Duration = 60S (monitor for one minute) Monitor Entire Step = NO (want to monitor what is happening and then view results, if you specify monitor entire step = YES, you will not get the monitor results until the job or online region ends a bonus Best Practice NEVER specify Monitor Entire Step = YES for an online region) Observations = 6000 (the usual default) Even if you are invoking an ASG-TriTune measurement for an online region, do not specify transaction names for CICS or IMS. You will want to analyze the region as a whole at the time of the exception, to determine which transactions are causing the exception condition.

10 10 Invoking an ASG-TriTune Monitor via the ASG-TriTune Started Task The details for how to invoke an ASG-TriTune Monitor via the ASG-TriTune Started Task can be found in the ASG-TriTune User s Guide, Version 4.8.1, in Chapter 7 : TriTune Open Application Program Interface. The sub-section to review is Invoking a Monitor by Communicating with the TriTune Started Task this is found under the section TUNCALL Verbs and Keywords. We will walk through each of the Monitor Profile Naming recommendations. As defined in each recommendation, monitor profiles will be predefined prior to the exception. At the time of the exception, the Batch Job name or Online Region name associated with the exception will be known. For our example, an exception condition has been identified in our CICSP1 region. The exception is a Transaction Response Time Exception for transaction EC01. Knowing the TriTune Started task name (we will use TRIPROD) with our naming conventions above, the knowledge of the CICS region where the exception occurred, and the instructions on how to invoke an ASG- TriTune Monitor via the ASG-TriTune Started Task, the z/os modify command to invoke the monitor will be : For Monitor Profile Naming Convention Example 1: Where : F TRIPROD,MONI,INVOKE,JOBNAME(CICSP1),DESC(MONITOR) TRIPROD is the Started Task Name of the TriTune Server MONI,INVOKE are required parameters JOBNAME(CICSP1) specifies that the job, or as in this case the started task, CICSP1 is to be monitored with ASG-TriTune. Since no PROFILE name is specified, ASG-TriTune will use the JOBNAME specified (CICSP1) as the profile name. As per our recommendations for Example 1, a global profile named CICSP1 should already be defined. DESC(MONITOR) specifies a 1 to 20 character description for the monitor, which in our example is MONITOR. DESC is an accepted abbreviation for the DESCRIPTION keyword. What we would recommend for the DESC field is the name of your monitoring software. This description can be used to filter global monitor names to determine the list of global monitors that were invoked via the ASG-TriTune command interface. All parameters must be separated by a comma.

11 11 For Monitor Profile Naming Convention Example 2: Where : F TRIPROD,MONI,INVOKE,PROFILE(XTEC01),JOBNAME(CICSP1) TRIPROD is the Started Task Name of the TriTune Server MONI,INVOKE are required parameters PROFILE(XTEC01) specifies that XTEC01 is the pre-defined ASG-TriTune profile to be used for this monitoring session JOBNAME(CICSP1) specifies that the job, or as in this case the started task, CICSP1 is to be monitored with ASG-TriTune All parameters must be separated by a comma. For Monitor Profile Naming Convention Example 3: Where : F TRIPROD,MONI,INVOKE,PROFILE(XCTRSP01),JOBNAME(CICSP1) TRIPROD is the Started Task Name of the TriTune Server MONI,INVOKE are required parameters PROFILE(XCTRSP01) specifies that XCTRSP01 is the pre-defined ASG-TriTune profile to be used for this monitoring session JOBNAME(CICSP1) specifies that the job, or as in this case the started task, CICSP1 is to be monitored with ASG-TriTune All parameters must be separated by a comma. Any ASG-TMON product exception monitor can be used to trigger an ASG-TriTune monitoring session. For demonstration purpose we have included examples from ASG-TMON for CICS TS for z/os as well as from ASG-TMON for z/os.

12 12 ASG-TMON for CICS TS for z/os In preparation for the examples below, we should review some of the important TMON metrics involved in exception events. Exception events are of two types, exception-driven and interval-driven. Exceptiondriven exception events are acted on immediately. Interval-driven events are acted on at the next recording interval. For exception-driven exceptions, you should review the value of the CYCLE parameter. The CYCLE parameter controls the frequency at which the ASG-TMON Alert Monitor collects samples. To view the current setting of the CYCLE parameter, we start at the CICS JOBLIST DISPLAY and select the name of the job we are reviewing. In this example, we are reviewing CICS region ESWDC31S, so we will select that job name and press ENTER : The following display will appear :

13 13 We will choose option 10 to review the CYCLE parameter under the System Administration section. We will then select Option 3 to review the Problem/Alert controls.

14 14 And finally Option 2 to review our CYCLE setting for Threshold Alerts. We now want to select the job we are interested in reviewing, which in our example is our CICS region ESWDC31S.

15 15 We see that CYCLE is specified as 60. To validate what unit that number represents, we can position our cursor under CYCLE and press the PF1(HELP) key. So, we find that our CYCLE time is 60 seconds for exception-driven alerts. For interval-driven exceptions, you should review the value of the INTERVAL parameter. The INTERVAL parameter controls the frequency at which the ASG-TMON Alert Monitor collects interval statistics and evaluates interval-driven exceptions. To view the current setting of the INTERVAL parameter, we will navigate to our System Administration Menu.

16 16 Then we will select Option 2 to review Interval Data Collection Controls. We will again select our CICS region, ESWDC31S.

17 17 Now we will look at the TI record controls. Here you can see when the next interval is due to expire, the last time the TI records were logged, and the information we are looking for, the collection interval time. To verify the units this value represents, we would position the cursor under the field name TI RECORD COLLECTION INTERVAL and press the PF1(HELP) key.

18 18 In this case, the units are HH:MM, so our record collection interval is one minute, or 60 seconds. Also note that the interval always occurs on the minute, so any interval would occur at second 00, or as in the previous screen, the next interval will occur at 10:34:00. To summarize, we have determined our CYCLE time for exception-driven exception events is 60 seconds, and our INTERVAL time for interval-driven exception events is 1 minute, or 60 seconds. In ASG-TMON for CICS TS for z/os, the Alert Metrics Record captures exception conditions at a CICS region level. As the Data Guide document indicates in Chapter 4, TMON for CICS increments a counter each time an event or state is recognized for a given CICS subsystem. The document also indicates that if any of these event/state counters are greater than 1, a serious condition exists. It is at this point, when a counter is 2 or greater, we recommend issuing an ASG-TriTune monitor request for the region identified as having the event/state. For the following event/state conditions identified by ASG-TMON for CICS, we would recommend that ASG-TMON for CICS issues the command to invoke an immediate TriTune monitor of the CICS region when the count is greater than 1 : Alert Metrics (AM) Record AMLOOPCT the number of consecutive intervals during which ASG-TMON for CICS TS has observed a CICS region consuming excessive amounts of CPU time AMMAXTCT - the number of consecutive intervals during which ASG-TMON for CICS TS has observed a CICS region running at MAXTASK

19 19 AMSOSCNT - the number of consecutive intervals during which ASG-TMON for CICS TS has observed a CICS region in a short-on-storage condition For the ASG-TMON for CICS TS evaluated targets, also contained in the Alert Metrics Record, we recommend you review your Smart Target definitions in your ASG-TMON for CICS TS installation and determine which targets should be used to trigger ASG-TMON for CICS TS exceptions that will issue the TriTune command to start an immediate TriTune monitor of the CICS region. As a guide, we recommend reviewing the following targets: AM203OBS - the number of ended transactions during the evaluation cycle for example, if this number goes below a certain value during prime shift, this could be an issue AM209OBS - the amount of CPU per minute being utilized by the CICS region, during the evaluated cycle if this number goes above a certain value, this could be an issue By monitoring these targets and having ASG-TMON for CICS TS invoke an immediate TriTune measurement of the CICS region, a major problem can be identified before it becomes a major problem. On a CICS transaction level, we recommend reviewing the metrics available in the ASG-TMON for CICS TS for z/os Data Guide in the Transaction Performance (TA) Record. Another method would be to monitor Transaction Interval (TI) records testing for transactions that exceed, for instance, 1-2 seconds response time (Field=TIRPS12). For any TriTune monitor requests generated by ASG-TMON for CICS TS for a transaction exception condition, it is recommended to monitor the CICS region as a whole, rather than just the transaction which is experiencing the exception condition. This is to get a complete view of the CICS region as close to the exception condition as possible. At times, the transaction which is experiencing the exception condition is the victim of another transaction. By issuing the TriTune monitor request for the entire CICS region, you will see all the transaction that are executing at the time of the monitor request and determine which transaction or transactions may be contributing to the exception event.

20 20 Example 1 ASG-TMON for CICS TS issuing a ASG-TriTune monitor request for a specific transaction which is exceeding it s SLA response time Step 1 : Pre-define the TriTune profile to be used for monitoring this exception condition As in our recommendations, the first step to invoking a ASG-TriTune measurement request when this exception condition is identified by ASG-TMON for CICS is to predefine the ASG- TriTune profile to be used when monitoring this exception condition. For this example, we will use the profile naming conventions recommended in Example 1, that is to use the name of the job, or in this case the CICS started task, as the Profile Name. For this example, we will want to monitor CICS region ESWDC31S if this exception occurs, as it is the region where our transaction, TTL1, will execute. If the region cannot be determined, do not predefine the ASG-TriTune profile. ASG-TriTune will define the profile at the time the measurement request is invoked, if there is no profile previously defined. To define the profile ESWDC31S to ASG-TriTune, you would utilize the ASG-TriTune ISPF interface. From the ASG-TriTune Primary Option Menu, select Option G Global Monitoring and Listing Registration. If you have not previously verified that you as well as the ASG-TriTune started task has UPDATE authority to the Global Monitor dataset name that will be created, please review the Monitor Dataset Naming Conventions section above. From the Global Monitoring Menu, select Option 1 Monitor Create and Analyze Monitor Sets. We will first filter the list of existing monitors to see if the profile we want to use, ESWDC31S, has already been defined. We will filter the profile list by initial characters ESWDC :

21 21 When we press enter, the remaining list of profile definitions currently defined will be those that start with the letters ESWDC : As we see, our profile is already defined, so we will select the profile to verify that the parameters specified are what we expect for this monitor request. As in the previous screen, we selected the profile by specifying an S next to the profile name on the prior screen. We can then review the current definition for profile ESWDC31S :

22 22 For this profile, the parameters are already defined as per our recommendations, with the following exceptions. Note that the Job Name is specified, as this profile will be used exclusively to monitor CICS region ESWDC31S. Also, we have specified a very short, but identifiable, Description for this monitor, indicating that the monitor request was invoked via TMON for CICS TS. Since the profile is defined and no changes are required, we have completed our ASG- TriTune profile definition for this exception condition. Now that the ASG-TriTune monitor profile is defined, we will define the ASG-TMON for CICS TS exception condition for which we will want to invoke an immediate ASG-TriTune monitor using the profile ESWDC31S for transaction TTL1. To begin this process, we will access the ISPF interface for ASG-TMON for CICS TS and access the CICS JOBLIST DISPLAY panel :

23 23 The CICS subsystem, ESWDC31S, is already defined, so we will select that JOBNAME by entering a S next to the JOBNAME and pressing ENTER. The Primary Selection Menu will be displayed : We will select Option 11 as shown above to access the Exception Monitor Main Menu :

24 24 We will select Option 2 to see the list of currently defined exceptions : As you see under the Grp/Job column, we are going to filter the Exception Definitions to display only those exceptions defined for the CICS region ESWDC31S :

25 25 Note that according to the descriptions, the defined exception RESPTTL1X may already provide the exception conditions we are looking for. It is recommended if you do not own the exception listed, make a copy of the exception and then modify it for your conditions. In this case we will copy RESPTTL1X. Once you request the copy, you will be at the following screen : To created the new member, type in the new name over the existing name, and then position the cursor under Add New at the top right corner :

26 26 When you press ENTER, the member XRESPTTL1 will now be created. We will now select the new member for editing : The following screen is now displayed :

27 27 As in prior examples, if we position the cursor under a field and press PF1(HELP), we will get additional information regarding the data contained in the field. We positioned the cursor under Visibility Threshold and pressed PF1 : Note that there is more help information available by scrolling down. Looking at our value for Visibility Threshold, we will leave it set to 1, which indicates as soon as the exception condition is identified, the exception condition is visible. The Description and the Message field are user-defined. Note that in the message field we are substituting the TMON-supplied value of TAPTRAN in the message by adding character in

28 28 front of the TMON-defined variable We will see more examples of substitutions throughout the examples. Since we copied an existing definition, we will want to check the metrics defined. In this case, we have TARSPTM and TAPTRAN defined. To review what these fields represent and what other fields are available, position the cursor under Data Source and press ENTER : To review the metrics already defined, we first type in the name of the first metric, TARSPTM, in the yellow box under the metric column to find the metric :

29 29 We find that this field is the response time of the transaction, and is already selected as part of our criteria. When we follow the same procedure for TAPTRAN, we find that this is the field for the Primary Transaction ID, and is also selected. Enter PF3(END) to return to the previous screen. The field TAPTRAN is a character field, so setting the criteria is straightforward. For our exception condition, it is also set up correctly, that is, to specify the transaction for this exception to be met has to be TTL1. The field TARSPTM represents the response time. By now we have seen what to do to determine how the value is defined. We will start by positioning the cursor under the value in the Comparison Value column and press PF1(HELP) : Which then provides the information on how the comparison values are defined. If we scroll down to the last page of help, we find the following :

30 30 The value currently specified, , represents 2 seconds. So our criteria for the Exception Condition is if the transaction name is TTL1 and the response time is greater than 2 seconds. For this example, we will modify the time to be 5 seconds, so that the exception condition will be met if the transaction TTL1 has an over 5 second response time. To do this, simply overtype the with and press ENTER : The data type is EVENT, which indicates that the Exception Condition is event-driven. Also, as previously discussed, event-driven exceptions are immediate and do not wait for the next interval.

31 31 To determine how long the exception condition will be active, we will use the following values : CYCLE : 60 seconds (discussed earlier) Visibility Threshold : 1 Sampling Frequency : 5 minutes For an event-driven exception, ASG-TMON for CICS TS will sample at the cycle rate, 60 seconds. Samples are taken on a minute boundary. The visibility threshold is set to 1, which indicates as soon as the exception condition is met, an exception event is activated. With a visibility threshold of 1, for each sampling cycle, either the exception condition has been met or not. The first cycle after the exception condition will meet the condition. The sampling frequency controls when ASG-TMON reviews the exception event. For our example, if no additional occurrence of the exception condition has occurred in the 5 minutes (sampling frequency) since the first cycle after the exception event was activated, the exception count is decremented by 1. Since our count was 1, the exception count would be 0, which ends the exception event until the next occurrence of the exception condition. As an example, let s review the following two scenario based on our example : Scenario 1: 10:25:25 initial exception condition (TTL1 over 5 second response time 4 times) 10:26:00 first cycle exception condition has been met 10:27:00 next cycle no additional exception 10:28:00 next cycle no additional exception 10:29:00 next cycle no additional exception 10:30:00 next cycle no additional exceptions 10:31:00 next cycle no additional exceptions sample frequency met (5 minutes) no exceptions in the last 5 cycles - visibility count decremented by one, now zero, so exception event closed Note that the number of TTL1 transactions which meet the exception condition during the same cycle do not influence the longevity of the exception event. This is because our visibility count is 1.

32 32 Also note that with the sampling frequency at 5 minutes, there must be 5 complete cycles (60 second cycles) without an exception condition for the visibility count to be decremented by one and the exception event cleared. Scenario 2 : 10:25:25 initial exception condition (TTL1 over 5 second response time 1 time) 10:26:00 first cycle exception condition has been met 10:27:00 next cycle no additional exception 10:28:00 next cycle no additional exception 10:29:00 next cycle additional exception condition (TTL1 over 5 second response time 2 times) 10:30:00 next cycle no additional exceptions 10:31:00 next cycle no additional exceptions sample frequency met (5 minutes) exception occurred again prior to 10:29:00 cycle exception event remains opened next check in 5 minutes (sampling frequency) 10:32:00 next cycle no additional exception 10:33:00 next cycle no additional exception 10:34:00 next cycle no additional exception 10:35:00 next cycle no additional exceptions 10:36:00 next cycle no additional exceptions sample frequency met (5 minutes) no exceptions in the last 5 cycles - visibility count decremented by one, now zero, so exception event closed We will now check the actions we would like to take when this exception condition is met and the exception event is activated. To do this, we position the cursor under the action column and press ENTER :

33 33 We are now at the following screen : Note that the actions only occur on the second occurrence of the exception condition. For our example, if the response time for TTL1 exceeds 5 seconds only once, and then for five consecutive cycles does not exceed 5 seconds, the actions will never be taken as a second occurrence of the exception did not take place before the exception event was closed. Reviewing our scenarios from before to determine at what point the actions would be taken (and adding Scenario 3) :

34 34 Scenario 1: 10:25:25 initial exception condition (TTL1 over 5 second response time 4 times second observation met actions taken) Scenario 2 : 10:26:00 first cycle exception condition has been met 10:27:00 next cycle no additional exception 10:28:00 next cycle no additional exception 10:29:00 next cycle no additional exception 10:30:00 next cycle no additional exceptions 10:31:00 next cycle no additional exceptions sample frequency met (5 minutes) no exceptions in the last 5 cycles - visibility count decremented by one, now zero, so exception event closed 10:25:25 initial exception condition (TTL1 over 5 second response time 1 time) 10:26:00 first cycle exception condition has been met 10:27:00 next cycle no additional exception 10:28:00 next cycle no additional exception 10:29:00 next cycle additional exception condition (TTL1 over 5 second response time 2 times second observation met actions taken) 10:30:00 next cycle no additional exceptions 10:31:00 next cycle no additional exceptions sample frequency met (5 minutes) exception occurred again prior to 10:29:00 cycle exception event remains opened next check in 5 minutes (sampling frequency) 10:32:00 next cycle no additional exception 10:33:00 next cycle no additional exception 10:34:00 next cycle no additional exception 10:35:00 next cycle no additional exceptions

35 35 Scenario 3: 10:36:00 next cycle no additional exceptions sample frequency met (5 minutes) no exceptions in the last 5 cycles - visibility count decremented by one, now zero, so exception event closed 10:45:25 initial exception condition (TTL1 over 5 second response time 1 time) 10:46:00 first cycle exception condition has been met 10:47:00 next cycle no additional exception 10:48:00 next cycle no additional exception 10:49:00 next cycle no additional exception 10:50:00 next cycle no additional exceptions 10:51:00 next cycle no additional exceptions sample frequency met (5 minutes) no exceptions in the last 5 cycles - visibility count decremented by one, now zero, so exception event closed Note that since the exception condition was only met once, the actions were never taken as the observation count on which the action would have been taken is the second observation. For both the WTO message and the MVS Command, note the use of the substitution of the ASG-TMON variables in the message as well as the MVS command. Our MVS command generated will read : F ESDTUN48,MONI,INVOKE,JOBNAME(ESWDC31S),DESC(TMONCICS) We have finished our review of the newly created ASG-TMON exception XRESPTTL1. To save our changes, we will position our cursor under the work Update and press ENTER :

36 36 If the update was successful, we should see the following : Note the message that the record was successfully updated. We can now back out to the Exception Monitor Menu Screen. However, we forgot to do two items, so we will once again select Option 2 Exception Definitions :

37 37 The message Definition Updates Are Pending indicates that we did not do a REINIT of the ASG- TMON Exception Definitions after we created and changed our new definition. However, we also forgot one more important change. We will want to edit the Exception Definition we just created, XRESPTTL1 : The item we forgot was to enable the exception. To do this, we will type an E in the status field for Enable, position the cursor under Update, and press ENTER to update the exception definition :

38 38 As before, we should see the message indicating that the record was successfully updated : If we now press PF3 to end our edit, we will return to the Exception Definition Screen. To reinitialize the Exception Monitor definitions, place the cursor under the word Reinit at the top right corner and press ENTER :

39 39 The screen should now show the following : To verify that the correct Exception definitions are now enabled, back out to the Exception Monitor menu and select Option 2 for Exception definitions. Filter on Jobname ESWDC31S (CICS started task ID) and press ENTER. Verify that the Exceptions that are Enabled are the correct ones :

40 40 For our example, note that Exception XRESPTTL1 is now enabled. We have now completed our Exception Definition for ASG-TMON for CICS TS. Now let s see how this exception will help us diagnose CICS application issues. We have just been notified that TTL1 had a response time exception in region ESWDC31S the time of the exception based on the console message was approximately 12:10 P.M. We know, that based on the way we defined our exception condition to ASG-TMON for CICS TS, there should also be an ASG-TriTune monitor of the CICS region available from this approximate time which may give us an indication of the root cause of the performance issue with TTL1. We will now access the ISPF interface for ASG-TriTune, select Option G for Global Montiors, and then select Option 1 Monitor. This will display the list of Global Monitors that have been defined and/or executed by ASG-TriTune :

41 41 We know by the design of our ASG-TriTune invoke command from ASG-TMON that our profile name will be the CICS region name, ESWDC31S and that the Description field will have TMONCICS. We now use both of those values to filter the Monitor Definition screen, which leaves us the following display : Note that there have been 17 runs of the ESWDC31S profile. To find our profile that is associated with the TTL1 performance issue reported, we will type in H next to the ESWDC31S profile name in the LC (Line Command) column and press ENTER :

42 42 To filter further, we will now select the date we are interested in looking at. In our example, we are looking for today, which is July 5,2012. We filter on this date and press ENTER, which leaves us with the following screen : We know we have the correct date do we have the correct time? If we scroll to the right, we will eventually see two fields named Invocation Date and Invocation Time which reconfirm the date of the ASG-TriTune measurement and shows that the time of the measurement is the timeframe of our current issue with the TTL1 response time. Wouldn t it be more efficient not to have to scroll to see these values? So let s customize the monitor history screen. To do this, we first type in CUST on the COMMAND line :

43 43 We are now in the screen customization menu for the Monitor History screen : We can see that there are a number of fields that can be displayed that are not included in the current display. For our example, we simply want to move the two fields, Invocation Date and Invocation Time, to display in the columns immediately after start date. To do this, we find the two fields by scrolling down until we find the fields. The only fields that look reasonable for these values are LDATE and LTIME. To verify the value of the fields, we will request help for each field. For this example, we will type an H in the S column next to LTIME and press ENTER to get the definition of the field :

44 44 If we did this for LDATE also, we would find that these two fields, LDATE and LTIME, are the fields we want to move. To return to the previous screen we press PF3(END) and then select the fields to move. To do this, we can use the M command to move one field at a time we will start by moving the LDATE field, specifying M in the S column and press ENTER :

45 45 Note in the upper right corner, we now see Move Pending we now must scroll up to find the location to move this field after, the SDATE field. We type in A(after) next to SDATE : Once we press ENTER, we now see the following :

46 46 Note that LDATE now is immediately after SDATE. We will follow the same procedure for LTIME, which leaves us at the following screen : To exit customization, press the PF3(END) key. This returns us to the Monitor definition screen we scroll all the way to the left and we now see our customized changes available on the initial screen without scrolling :

47 47 The next time we exit the Monitor History panel, we will be asked if we want to save the customization changes we made during this session. When we press PF3(END), we will see the following screen : In our case, we would like to save the customized Monitor History panel view, so we will press ENTER and see the following message in the upper right-hand corner indicating that the changes were saved :

48 48 One last note about customization. Almost every ASG-TriTune panel is customizable, so feel free to enter the CUST command and change your view. Also, many panels have additional data fields available that are not shown by default. For example, DB2View has 77 fields available, of which only a handful are shown by default. In this example, we were looking for the monitor profile ESWDC31S on July 5, 2012 at 12:10 P.M. we see this is the remaining monitor on our Monitor History list. To analyze this monitor to determine the delays to the TTL1 transaction, we can position the cursor in the LC column next to ESWDC31S and press ENTER which defaults to an Analyze Normal. For viewing purposes, we will type in an A(Normal) next to ESWDC31S and hit ENTER to start the analysis process :

49 49 ASG-TriTune will indicate it is loading the analysis data : Once the data is loaded, you will see the ASG-TriTune primary option menu :

50 50 In any ASG-TriTune analysis, all Options should point to the largest delay to the application measured. In this case, we measured the ESWDC31S CICS region, so we are looking for the largest delay to that region, but even more specifically, what is causing the response time problem for transaction TTL1. Our first approach will be to access Option 6 TranView to specifically look for transaction TTL1 : We see that for this measurement of CICS region ESWDC31S, transaction TTL1 is causing the highest delay among the transactions that were running. To drill deeper and access the Delay

51 51 Details panel, we can use the auto-navigation function of ASG-TriTune. Simply position the cursor in the LC column next to TTL1 and press ENTER : We will now access the DelayView Detail Screen : Note that we ve already positioned the cursor next to the Program Active row. Our largest delay to the TTL1 transaction is a program that is running. We will again let ASG-TriTune s autonavigation function do the work by pressing ENTER, which accesses the Delay Locations Detail screen :

52 52 Note that our largest delay to TTL1 is the activity in IBM Module IGZCPAC, Csect IGZCXDI, at offset +20C. Note this location and also the fact that the Csect name, IGZCXDI, is highlighted. We will get back to this shortly. Our second approach to analyzing this monitor will be by accessing Option 2 DelayView : Program Active is our largest delay (should sound familiar). Again, we will position the cursor next to the Program Active row, and press ENTER to let ASG-TriTune s auto-navigation function lead us to the source of all the program activity :

53 53 As in the analysis following the TranView path, DelayView points us to our largest delay to TTL1, which again is the activity in IBM Module IGZCPAC, Csect IGZCXDI, at offset +20C. Our third approach to analyzing this monitor will be by accessing Option 3 CodeView : We see that the largest delay to this application, and therefore to transaction TTL1 as it was the most active transaction during this measurement, is again IBM Module IGZCPAC, Csect IGZCXDI. To see if this is the same location as we found via TranView and DelayView, we again use autonavigation, position the cursor next to the row with the highest % delay (29.66) and press ENTER :

54 54 We see that the most activity in Module IGZCPAC, Csect IGZCXDI, is between offset +200 and offset The offset we found previously via TranView and DelayView, offset +20C, is certainly within this range. Also, the message you see, STMT not available, indicates that the listing for Csect IGZCXDI is not available. We will get back to this a bit later. As we mentioned earlier, all ASG-TriTune analysis options should point to the largest delay to the application measured. In our example, TranView, DelayView, and CodeView all point to IBM module IGCZPAC, Csect IGZCXDI, at offset +20C. Let s now return to following our TranView example we were at the following screen :

55 55 We mentioned earlier to remember that the Csect name, IGZCXDI, was highlighted. During ASG- TriTune analysis, when you see a highlighted field such as IGZCXDI, this indicates that there is context-sensitive help available from ASG-TriTune which may shed some additional light into what is the exact problem. To review the context-sensitive help for Csect IGZCXDI, position the cursor under the highlighted Csect name, and press PF1(HELP) : The plus in the upper right hand corner of the help screen indicates there is additional help available if you press PF8(DOWN) we will show the remaining help panels and then discuss the context-sensitive help provided :

56 56 We note that the ASG-TriTune context-sensitive help recommends that the use of IGZCXDI should be avoided. We also note that the definition of the variables can be the cause of the use of IGZCXDI. Now we need to find out which statements that we can control in our applications are causing the use of IGZXCDI. To do this, first we need to find out which statements are calling IGZXCDI. To do this, we will request ASG-TriTune to give us Caller-ID information to indicate which programs caused IGZCXDI to be executed. To do this, we type in a C in the LC column next to the line with IGZCXDI (the first one) and press ENTER : Note that application program TTL1B04 is the first application module listed at this time we will see that later. We now are looking at the CallerID detail screen :

57 57 The IBM LE module CEECCICS obviously we cannot change, but we certainly have access to our TTL1B02 program. Note that the STMT column is blank this indicates that the listing for TTL1B02 (as well as TTL1B01 and TTL1B03) is not registered to ASG-TriTune. To register the listing of TTL1B02 at this time, we can type in the REG (register) option on the command line and press ENTER : The ASG-TriTune Program Listing Registration panel will now be displayed as we know our TTL1xxx application modules are written in COBOL, we will select that option and press ENTER:

58 58 We will now add the listings for TTL1B01 through TTL1B04. To do this, we specify the add command on the Registered Listings : Cobol panel : We will now enter the name of the partitioned dataset where our listings are located, and then press ENTER to select the listings we would like to register :

59 59 We now filter on TTL : We will select TTL1B01 to register and press ENTER :

60 60 Note the messages Successfully Registered and Successfully Completed. We will now proceed to register TTL1B02, TTL1B03, and TTL1B04 (not shown). We now press PF3(END) until we return to the CallerID screen and now see a difference :

61 61 There are now statement numbers for the TTL1 modules indicating that they are registered and we can access the listing to get to the exact line of code causing the delay. For example, we will type in L in the LC column next to the TTL1B01 at offset 128A and press ENTER we will now see the line of code which is statement 449 : If we remember our context sensitive help recommendation, we should check the definition of BIGNUM1, BIGNUM2, and BIGNUM4 we do this as we would normally in ISPF BROWSE by entering the command F BUGNUM1 FIRST on the command line, which shows us the following :

62 62 If you remember the help recommendation to avoid the use of IGZCXDI, it was to redefine variables defined with more than 8 significant digits to COMP-2 (floating point). In our example above, we would redefine the variables : 01 BIGNUM1 PIC S9(16) VALUE +333 COMP. 01 BIGNUM2 PIC S9(16) COMP. 01 BIGNUM3 PIC S9(16) VALUE COMP. 01 BIGNUM4 PIC S9(16) COMP. to be the following : 01 BIGNUM1 VALUE E+3 COMP BIGNUM2 COMP BIGNUM3 VALUE E+7 COMP BIGNUM4 COMP-2. We would then recompile the program, reregister the changed listing to ASG-TriTune, and then retest. One other recommendation at this time if you are working with a series of programs, such as TTL1B01 through TTL1B04, there is a good chance if there is a performance issue in one program, there may be the same issue in more programs in that series.

63 63 Earlier we had mentioned that TTL1B04 had the highest delay when we were looking at the Delay Locations Detail panel. Now that the listings are registered for the TTL programs, we return to the Delay Locations Detail panel and notice the view has changed : Now that the listings are registered, the delays are combined for each program statement, rather than for an individual offset. If we looked at the listings for each TTL program listed on this panel, we would see a DIVIDE statement (we will look at TTL1B04 by typing an L in the LC column and pressing ENTER): If we looked at TTL1B01, TTL1B02, and TTL1B03 in the same manner, we would also see that the statement number in those listings is also a DIVIDE statement. This ends Example 1.

64 64 Example 2 ASG-TMON for CICS TS issuing a TriTune monitor request for a short-on-storage region-level condition Step 1 : Pre-define the TriTune profile to be used for monitoring this exception condition As in our recommendations, the first step to invoking a ASG-TriTune measurement request when this exception condition is identified by ASG-TMON for CICS is to predefine the ASG- TriTune profile to be used when monitoring this exception condition. Based on our proposed naming conventions, Example 3, the profile which will be used will be named XCSTRG. To define the profile XCSTRG to ASG-TriTune, you would utilize the ASG-TriTune ISPF interface. From the ASG-TriTune Primary Option Menu, select Option G Global Monitoring and Listing Registration. If you have not previously verified that you as well as the ASG-TriTune started task has UPDATE authority to the Global Monitor dataset name that will be created, please review the Monitor Dataset Naming Conventions section above. We will choose the naming conventions outlined in naming convention Example 3 for this section. From the Global Monitoring Menu, select Option 1 Monitor Create and Analyze Monitor Sets. We will first filter the list of existing monitors to see if the profile we want to use, XCSTRG, has already been defined. We will filter the profile list by our initial character, X : When we press enter, the remaining list of profile definitions currently defined will be those that start with the letter X :

65 65 Our profile has not been previously defined, so we will use the ADD line command to define the profile : Now we will define the parameters for the XCSTRG profile. As recommended previously, we will use mostly the defaults. We will add the fictional jobname DUMMYJOB and also add a description for the profile. Our screen now looks like the following :

66 66 To save this profile definition, press your END key or type in END on the command line. You should now see the profile XCSTRG defined : Now that the ASG-TriTune monitor profile is defined, we will define the ASG-TMON for CICS TS exception condition for which we will want to invoke an immediate ASG-TriTune monitor using the profile XCSTRG. Step 2 : Pre-define the ASG-TMON for CICS TS exception condition that will trigger an immediate ASG-TriTune monitor using profile XCSTRG In this case, we have pre-defined an exception, SOSTRIT1, which we will review :

67 67 Note that this is an interval-driven exception. Follow the instructions from Example 1 to verify the interval setting if you do not know the frequency of your current interval setting. The metric we are using is AMSOSCNT which is the count of successive intervals this CICS region has been in an SOS condition. What we are indicating is that ASG-TMON for CICS TS should open an exception event at the first interval an SOS condition is encountered. We now review the actions that will be taken and when:

68 68 On the first occurrence of the exception, the following TriTune monitor request will be executed : F ESDTUN48,MONI,INVOKE,PROFILE(XCSTRG),JOBNAME(@EMJOBNM) is the name of the started task for this CICS Region Note that by using the profile name of XCSTRG, you limit yourself to one ASG-TriTune monitor per interval as ASG-TriTune will not invoke a monitor which has the same profile name and user as a monitor that is currently executing. You may want to adjust this command, to be able to handle the condition that two or more CICS regions have a SOS condition during the same interval: F ESDTUN48,MONI,INVOKE,JOBNAME(@EMJOBNM) With the above definition, an ASG-TriTune monitor request will be invoked using the CICS region name as the profile name. In this case, monitors can be started during the same interval for as many regions as there are SOS conditions. Insure that the exception definition is enabled for the exception condition to be identified by ASG-TMON for CICS TS. Once the ASG-TriTune monitor file is available, you would analyze the ASG-TriTune monitor. Follow the steps in Example 1 to analyze the monitor file. Understand that the cause of the issue will vary from exception condition to exception condition, but having an ASG-TriTune measurement immediately available for the CICS region which has the exception will go a long way to identifying the root cause of the exception condition.

69 69 Example 3 ASG-TMON for CICS TS issuing a TriTune monitor request for transaction-level response time condition (all transactions) Step 1 : Pre-define the TriTune profile to be used for monitoring this exception condition As in our recommendations, the first step to invoking a ASG-TriTune measurement request when this exception condition is identified by ASG-TMON for CICS is to predefine the ASG- TriTune profile to be used when monitoring this exception condition. Based on our proposed naming conventions, Example 3, the profile which will be used will be named XCTRSP. To define the profile XCSTRG to ASG-TriTune, you would utilize the ASG-TriTune ISPF interface. From the ASG-TriTune Primary Option Menu, select Option G Global Monitoring and Listing Registration. If you have not previously verified that you as well as the ASG-TriTune started task has UPDATE authority to the Global Monitor dataset name that will be created, please review the Monitor Dataset Naming Conventions section above. From the Global Monitoring Menu, select Option 1 Monitor Create and Analyze Monitor Sets. We will first filter the list of existing monitors to see if the profile we want to use, XCTRSP, has already been defined. For this example, we will filter the profile list by the first two characters of our recommended naming conventions, to view only the profiles that have been created for exception conditions identified by ASG-TMON for CICS TS : When we press enter, the remaining list of profile definitions currently defined will be those that start with the letters XC :

70 70 Our profile has not been previously defined, so we will use the ADD line command to define the profile : Now we will define the parameters for the XCTRSP profile. As recommended previously, we will use mostly the defaults. We will add the fictional jobname DUMMYJOB and also add a description for the profile. Our screen now looks like the following :

71 71 To save this profile definition, press your END key or type in END on the command line. You should now see the profile XCTRSP defined : Now that the ASG-TriTune monitor profile is defined, we will define the ASG-TMON for CICS TS exception condition for which we will want to invoke an immediate ASG-TriTune monitor using the profile XCTRSP. Step 2 : Pre-define the ASG-TMON for CICS TS exception condition that will trigger an immediate ASG-TriTune monitor using profile XCTRSP

72 72 To define the ASG-TMON for CICS TS exception condition that will trigger an immediate ASG- TriTune monitor when a high response time is identified for any transaction. To define the exception in ASG-TMON for CICS TS we will access the CICS Joblist Display in ASG-TMON for CICS TS : We will select the CICS region for which we want to define the exception condition, that being ESWDC31S : The Primary selection menu for ESWDC31S will appear we will select Option 11 (Exception Monitor) :

73 73 For this example, we have an exception definition defined, RESPTRIT2, that we will use. We will verify that the definition will trigger an immediate ASG-TriTune monitor if the response time as indicated in the TA record is greater than 2 seconds : Please insure that your TMON for CICS TS implementation is recording TA records for this exception condition to be effective. Now we will define the actions that will be taken by TMON for CICS TS when this exception condition is met :

74 74 This exception condition will trigger the following actions for the second occurrence of this exception for a given exception event : Send a WTO Message with offending transaction name and CICS region name Invoke a TriTune monitor to immediately monitor the CICS Region which triggered the transaction response time exception Insure that the exception definition is enabled for the exception condition to be identified by ASG-TMON for CICS TS. Once the ASG-TriTune monitor file is available, you would analyze the ASG-TriTune monitor. Follow the steps in Example 1 to analyze the monitor file. Understand that the cause of the issue will vary from exception condition to exception condition, but having an ASG-TriTune measurement immediately available for the CICS region which has the exception will go a long way to identifying the root cause of the exception condition.

75 75 Example 4 ASG-TMON for CICS TS issuing a TriTune monitor request for transaction-level response time condition (selected transactions based on a filter) For this example, we will first define the ASG-TMON for CICS TS metrics which trigger the exception condition : Transaction response time GT Tran Name EQ VC** We now define the actions that will be taken for this exception condition:

76 76 Actions that will be taken by ASG-TMON for CICS TS based on the second occurrence of this exception condition for a given exception alert : Send a WTO Message with offending transaction name Invoke a TriTune monitor to immediately monitor the CICS Region which triggered the transaction response time exception for any transaction beginning with the letters VC monitor the entire region, not just the transaction Jump to the CICS Active Tasks screen (Opt 1.1) with a FIND command for the CICS TASKID (if still active). Insure that the exception definition is enabled for the exception condition to be identified by ASG-TMON for CICS TS. Once the ASG-TriTune monitor file is available, you would analyze the ASG-TriTune monitor. Follow the steps in Example 1 to analyze the monitor file. Understand that the cause of the issue will vary from exception condition to exception condition, but having an ASG-TriTune measurement immediately available for the CICS region which has the exception will go a long way to identifying the root cause of the exception condition.

77 77 Example 5 ASG-TMON for z/os issuing a TriTune monitor request for a job using a high percentage of CPU For this example, we just want to show how an ASG-TriTune monitor can be immediately invoked from any TMON subsystem that can issue a z/os modify command. We will define an exception condition for TMON for z/os to invoke an immediate ASG-TriTune monitor for any job that uses more than.5% of CPU. Although this is not practical, it is a good test case. We will navigate in TMON for z/os to the exception definition screen. We select the exception field we want to monitor, JDCPUP, and the qualifier, which in this case is ESTN00X, and then review the exception definition : Based on the exception definition above, for any job that starts with the letters ESTN00Xand uses more than.5% of CPU, TMON for z/os will issue the following operator command : F ESDTUN48,MONI,INVOKE,JOBNAME(@JDJANME) As in previous examples, the ASG-TriTune profile name will be the Jobname, with ASG-TriTune for z/os substituting the Jobname into the z/os modify command. This will be issued only on the first occurrence of a given exception event. Insure that the exception definition is enabled for the exception condition to be identified by ASG-TMON for z/os. Once the ASG-TriTune monitor file is available, you would analyze the ASG-TriTune monitor. Follow the steps in Example 1 to analyze the monitor file, specifically the DelayView and the CodeView paths. Understand that the cause of the issue will vary from exception condition to exception condition, but having an ASG-TriTune measurement immediately available for the