Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications

Transcription

1 IBM WebSphere Software Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Mike Polan Lewis Lo Winfield Choi Kimberly Mungal

2 Copyright International Business Machines Corporation US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 CONTENTS Contents...3 List of Figures...4 List of Tables Overview Introduction IBM Tivoli Monitoring Structure ITM and Agent Structure Monitoring Environment Operations Team Use Case Out Of Memory / Memory Leak Response Rate Degradation SME Team Use Case Out Of Memory / Memory Leak Deep Dive Heap dump and Javacore dump analysis Situation Action and Policy Automation Agent Overhead Cost Summary...32

4 LIST OF FIGURES Figure 1: IBM Tivoli Monitoring Structure...9 Figure 2: Monitoring Environment Setting...10 Figure 3: Sample of historical data collected for requests analysis...12 Figure 4: Collected JVM Memory in Chart representation...13 Figure 5: JVM Memory Model Situation...14 Figure 6: Situation...15 Figure 7: Raised Situations...18 Figure 8: IBM Support Assistant IBM Thread and Monitor Dump Analyzer for Java...21 Figure 9: IBM Thread and Monitor Dump Analyzer for Java...22 Figure 10: Failed to allocate memory for object...23 Figure 11: Launch Activity drop down list...25 Figure 12: IBM Support Assistant - Memory Analyzer...25 Figure 13: Memory Analyzer - Histogram...26 Figure 14: Memory Analyzer - Outgoing References...27 Figure 15: Memory Analyzer - Incoming References...27 Figure 16: Action and Policy for Server down situation...28 Figure 17: Policy log of copying dump files and restart server...31

5 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications LIST OF TABLES Table 1: Situation Conditions...16 Table 2: Correlating Situations...18 Table 3: Situations for Response Rate Performance...20 Table 4: System with agent overhead cost analysis

6 1 Overview In many organizations the teams responsible for managing applications in production are distinct from the teams that developed that application. Each team may have a slightly different set of priorities and focus on how the application should be managed. Naturally this leaves gaps between how an operations team may be able to manage an application and how the owner of the application would like to see it managed. For example, operational monitoring focuses on the health of the infrastructure and deals with the following questions: Are the servers running? Is the network operational? Are the databases available? Are the application servers running? while the application owner is most interested in the key performance indicators for the business application itself such as: Is the application running? Is the application responding to requests within the agreed service objectives? This paper examines the typical scope and skills of the teams within a typical organization and the gaps left in an ideal monitoring process of a production application. Here you will be given best practice extensions when using IBM Tivoli Monitoring (ITM) and IBM Tivoli Composite Application Manager (ITCAM) products to monitor applications running on IBM's WebSphere Application Server family. 2 Introduction In this article, we address typical gaps exposed by organizations in application management by using the IBM solution software stack including IBM Tivoli Monitoring with WebSphere Application Agent, IBM Monitoring and Diagnostic Tools for Java Memory Analyzer, and IBM Thread and Monitor Dump Analyzer for Java.

7 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications The goal is to provide additional best practices for monitoring and to focus on the gap between the operations monitoring and the application monitoring by utilizing the IBM monitoring software stack. When managing applications in a production environment, the management tasks are divided into use cases often handled by different teams: Operations: Monitor the data center assets using monitoring tools. They may manage the application as well, depending on the capability of those monitoring tools. The operations team looks for issues such as crashed servers, corrupted disks, a broken network, out of memory conditions, or server hardware problems. Problem determination: When the application or application server is misbehaving, Web application experts (SMEs) look for issues such as memory leaks, poorly behaving transactions, improper tuning, full logs, and other operational aspects specific to web application servers. Development: Developers will determine the reasons of application failures, such as programming errors, environmental issues, memory leaks, etc, by analyzing the behavior of the application's business logic at the code level. Operating application servers with multiple applications running often experience the following pain points: Transaction correlation is difficult Proactive detection is complex using existing tooling Baseline analysis, detect and alert divergences requires considerable skill and experience Service Level Objective (SLO) based management of individual applications is not a capability available to the operations team Much of the effort is spent doing leak analysis (memory analysis). Improved techniques for understanding load and heap growth would reduce this effort Memory leaks are a typical defect of many production applications, often requiring considerable effort to work around, find, and fix. This white paper will focus on the use case of leak detection and heap usage monitoring to illustrate the techniques. Memory leaks manifest themselves as instability in production applications; out of memory errors result in failure of the application and loss of in flight transactions, and the recovery is post error or proactive restarts of the applications or the application servers. It can take a lot of time and effort to identify the cause of such errors, and for the repair to be made available in the production environment. Each team involved will find the following set of processes useful in managing the production application until a repair is available: Operations team responsibilities: Collect everyday data of the application operations 7

8 Determine the baselines or thresholds for the memory attributes Define alerts and situations which are raised when the baselines or thresholds are met Take required actions when the alerts are raised, including restarting the systems WebSphere experts (SME team) responsibilities: Perform deep dive analysis with the data generated Work with application development to deliver fixes Developers responsibilities: Conduct real code changes in business logic in the application in order to deliver the fix for the application An example of Out Of Memory (OOM) will be provided, the Operations team will need to define situations, alerts, and follow up actions; as well as monitor the systems. This is shown in the Operations Team Use Case section. The SME team will need to perform deep dive analysis using additional toolsets, this will be shown in the SME Team Use Case section of this white paper. The remainder of this paper describes how to use the IBM Tivoli monitoring products to simplify this task. 3 IBM Tivoli Monitoring Structure IBM Tivoli Monitoring (ITM) provides a framework and offers facilities for other monitoring products to process, archive, and present performance and availability information of the monitored systems. When working with the agents, including HTTP Server agent, Websphere Application Server agent, Operation System OS agent, and other agents (DB2 agent, Oracle agent, SAP agent, NetApp agent), different aspects of information can be gathered for different interfaces. The example here uses a WebSphere stack which is installed on a Windows environment, and the HTTP server agent, WebSphere agents, and Windows OS agent are used for monitoring purposes. 3.1 ITM and Agent Structure The IBM Tivoli Monitoring framework consists of several individual components, Tivoli Enterprise Monitoring Server (TEMS), TEMS Agent (TEMA), Tivoli Enterprise Portal (TEP) Server, and the TEP Client.

9 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Figure 1: IBM Tivoli Monitoring Structure TEMS: Tracks all agents availabilities by sending and receiving regular heartbeats from the agents. It stores short-term data from agents and starts and tracks all situations and policies. TEMS can be split into separate servers, namely hub and remote TEMS. o HUB TEMS is the centralized monitoring server which collects the information from remote TEMS. o Remote TEMS control a subset of agents and connects to the HUB TEMS. TEMA: Runs on the monitored system to collect performance and availability data to the TEMS to process. TEPS and TEP Client: Tivoli Enterprise Portal manages the presentation of ITM data. Users access the information using TEP clients. TEP clients can be a Java client process running locally, or through a web browser to the HTTP server on the TEPS. TDW: Tivoli Data Warehouse archives information for long term use, including historical data. 9

10 3.2 Monitoring Environment The current monitoring environment consists of a WebSphere Application Server configured with a cluster which has two members in it, a HTTP server, a TEMS server with ITM framework, and a grid backend for the SpecJ application. SpecJ applications are installed and configured in the two members of the cluster. Figure 2: Monitoring Environment Setting WAS server (wasmonitor01): It contains the WebSphere Application Server with deployment manager. A cluster, SpecJCluster, is defined in the wasmonitor01 server. WAS server (wasmonitor02): It is a member of the cluster, SpecJCluster. It is managed from wasmonitor01. HTTP server (wasmonitor03): An HTTP server is installed. It is configured to run against the cluster in the round robin fashion. ITM TEMS server (wasmonitor06): The entire software stack of ITM is installed and configured in wasmonitor06. GRID (gridmonitor): This is the backend of the SpecJ application that provisions the data for the SpecJ transactions. Client Drivers (gridmonitor): This is installed in the same server of the GRID, it provides a script to spawn multiple http requests to the HTTP server (i.e., wasmonitor03).

11 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications 4 Operations Team Use Case Out Of Memory / Memory Leak When an application causes memory leaks, certain behaviors can be recognized before out of memory occurs and brings down the system: The garbage collection process is running all the time Garbage Collector (GC) struggles to free up memory High CPU usage is caused as a result of the frequent GC cycles The number of completed requests decrease as the application server cannot process incoming requests fast enough Out Of Memory (OOM) occurs and the JVM stops With the use of ITM and TEMS, we first start with collecting historical data in order to identify the behavior and thus the baseline or threshold of a specific application (i.e., SpecJ). Historical data can be collected for requests, garbage collection, and JVM information for specific nodes and servers. 11

12 Figure 3: Sample of historical data collected for requests analysis A graphical representation on collected historical data on JVM memory is shown as follows. It shows the total, free, and used JVM memory in one of the cluster members, i.e., wasmonitor01.

13 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Figure 4: Collected JVM Memory in Chart representation The historical view offers a model situation feature to help define the threshold and create a situation based on the historical data collected. The following diagram shows the model situation of the JVM memory and the collected historical data. The threshold or baseline can be drawn using different functions. 13

14 Figure 5: JVM Memory Model Situation A situation can be created with the condition defined in the model situation. The resulting situation can be enabled in TEMS and an alert will be raised if the situation is evaluated to be true. The alert will remain open until the situation is evaluated to false. In the current scenario, we find that the top 20% of the transactions before the OOM occurs are performed with 256MB and 228MB for the total and used JVM memory respectively. A situation, JVM Used and Total Memory, will be created with the findings for the thresholds. With the historical data, percentage changes to the previous data can be calculated and utilized in the condition of a situation. The following situation will be

15 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications raised when the amount of Kbytes freed is 20% less than the previous data collected. Figure 6: Situation The following table list all the situations defined to help identify the memory leak and the out of memory issue. 15

16 Table 1: Situation Conditions Situation Name Kbytes Used in GC Kbytes Used in GC Percent decreases in total Kbytes Freed Percent decreases in total Kbytes Freed Average Heap Average Object Moved during Compaction Real Time ms > 50% Real Time ms > 50% Object Freed decreased by 30% Object Freed decreased by 30% Slow Response Time > 200% Condition ( Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND AVG(Kbytes Used) >= ) ( Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND AVG(Kbytes Used) >= ) ( Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND PCTCHANGE(Kbytes Free) <= -20) Target wasmonitor01 ( Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND PCTCHANGE(Kbytes Free) <= -20) wasmonitor02 ( AVG(Heap Used (%)) >= 70.0 AND Node Name == 'WASMONITOR01' AND Server Name == 'server1') OR AVG(Heap Used (%)) >= 70.0 AND Node Name == 'WASMONITOR02' AND Server Name == 'server2' ( Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND AVG(Objects Moved) >= ) OR Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND AVG(Objects Moved) >= ( Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND PCTCHANGE(Real Time (ms)) >= 50) ( Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND PCTCHANGE(Real Time (ms)) >= 50) ( Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND PCTCHANGE(Objects Freed) <= -30) ( Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND PCTCHANGE(Objects Freed) <= -30) ( Server Name == 'server1' AND Node Name == 'WASMONITOR01' AND PCTCHANGE(Average Response Time (ms) ) >= 200) wasmonitor01 and wasmonitor02 wasmonitor02 wasmonitor01 wasmonitor01 and wasmonitor02 wasmonitor01 wasmonitor02 wasmonitor01 wasmonitor02 wasmonitor01

17 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Slow Response Time > 200% Completed count dropped Completed count dropped High CPU Usage on GC and Application JVM Memory Free under JVM Memory Free dropped 50% JVM Memory Free dropped 50% JVM Used and Total Memory ( Server Name == 'server2' AND Node Name == 'WASMONITOR02' AND PCTCHANGE(Average Response Time (ms) ) >= 200) ( Server Name == 'server1' AND Node Name == 'WASMONITOR01' AND PCTCHANGE(Completion Count) <= -30) ( Server Name == 'server2' AND Node Name == 'WASMONITOR02' AND PCTCHANGE(Completion Count) <= -30) ( AVG(CPU Used (%)) >= 70.0 AND Node Name == 'WASMONITOR01' AND Server Name == 'server1') OR AVG(CPU Used (%)) >= 70.0 AND Node Name == 'WASMONITOR02' AND Server Name == 'server2' ( JVM Memory Free (bytes) < AND Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND JVM Memory Free (bytes) > 0) OR JVM Memory Free (bytes) < AND Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND JVM Memory Free (bytes) > 0 ( Node Name == 'WASMONITOR01' AND Server Name == 'server1' AND PCTCHANGE(JVM Memory Free (bytes)) <= -50) ( Node Name == 'WASMONITOR02' AND Server Name == 'server2' AND PCTCHANGE(JVM Memory Free (bytes)) <= -50) ( JVM Memory Total (bytes) >= AND JVM Memory Used (bytes) >= AND Node Name == 'WASMONITOR01' AND Server Name == 'server1') OR JVM Memory Total (bytes) >= AND JVM Memory Used (bytes) >= AND Node Name == 'WASMONITOR02' AND Server Name == 'server2' wasmonitor02 wasmonitor01 wasmonitor02 wasmonitor01 and wasmonitor02 wasmonitor01 and wasmonitor02 wasmonitor01 wasmonitor02 wasmonitor01 and wasmonitor02 The situations are started automatically given that the Run at startup option is enabled during the creation of the situation. With a proper workspace view, the operator can monitor different systems at the same time. Alerts will be raised when 17

18 the situations are satisfied. In our example, we will run the driver client to generate http requests. All the requests are browsing the catalogs for vehicle information for the dealers. Additional logic is added so that objects are created without being freed, thus inducing a memory leak as a result. The situation log view will show the alerts being raised after the batch of requests are run for some time. Figure 7: Raised Situations During the creation of the situation, different severity options are available. Depending on the nature of the situation, it can range from informational to fatal. The view can be filtered to show situations with specific severities. Situations can be evaluated from the outcomes of other situations. It is called correlation. A correlating situation is evaluated true if and only if the other situations that it defines upon are evaluated to true. In the current use case, some correlated situations are defined. They are shown in the following table. Table 2: Correlating Situations Situation Name GC Stressed and Struggle to free space JVM Stressed Request Slow Down Correlated Situations Kbytes Used in GC Average Object Moved during Compaction Real Time ms > 50% Object Freed decreased by 30% Percent decreases in total Kbytes Freed Average Heap High CPU Usage on GC and Application JVM Memory Free under JVM Used and Total Memory Slow Response Time > 200% Completed count dropped Target wasmonitor01 and wasmonitor02 wasmonitor01 and wasmonitor02 wasmonitor01 and wasmonitor02

19 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications The operations team will need to monitor the servers running the applications by keeping track of the raised alerts from the situations. Once an alert is raised, the operations team will inform the WAS SME to analyze the problem and determine if it can be fixed. If the problem is traced to the application, the WAS SME will forward the findings and data to the development team. 4.1 Response Rate Degradation During the analysis of OOM, we usually find that response rates degrades rapidly on the verge of OOM. However, OOM is sufficient but not necessary for response rate degradation, since response rates may degrade on other aspects like I/O or network traffic. The following attributes can be monitored for response rate performance: Average response rate Number of completed requests count Request rate per second When the performance of response rates decline, it will show the value of average response rates increase, both the number of completed requests count and request rates per second decrease. The following situations can be used for detecting response rate degradation. Notice that for each of the situations defined below, ITM offers the capability to monitor the situation so that an alert will be raised if the condition remains true for consecutive intervals. This can be used to ensure the occurrence of the response rate degradation. 19

20 Table 3: Situations for Response Rate Performance Situation Name Request Rate dropped 50% Response Rate degrade 100% Completed request count dropped 5 Condition ( Node Name == 'wasmonitor01' AND Server Name == 'server1' AND PCTCHANGE(Request Rate (per sec)) <= ) ( Server Name == 'server1' AND Node Name == 'WASMONITOR01' AND PCTCHANGE(Average Response Time (ms)) >= 100) Remarks Request rate dropped by 50% relative to previous sampling data. The same situation can be defined for the other target (i.e., wasmonitor02) Response rate degrades by 100%. That is the response time increases by 100% relative to the previous sampling data. The same situation can be defined for the other target (i.e., wasmonitor02) Monitor the system if the completed count has dropped by 30%. ( Server Name == 'server1' AND Node Name == 'WASMONITOR01' AND PCTCHANGE(Completio The same situation can be defined for the other n Count) <= -30) target (i.e., wasmonitor02) SME Team Use Case Out Of Memory / Memory Leak Once the situations and alerts are raised, actions and policies may be triggered as a result. The tasks performed usually constitute a short term solution. Communication will be made between the operations team and the WAS SME team to conduct further analytical investigation. It may require the application development team to assist for a solution to the problem. 5.1 Deep Dive Heap dump and Javacore dump analysis To provide further assistance to the WAS SME team, we will complement ITM with another toolset, namely IBM Monitoring and Diagnostic Tools for Java Memory Analyzer, and IBM Thread and Monitor Dump Analyzer for Java. This solution provides an end-to-end analysis from monitoring to deep dive analysis.

21 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications IBM Thread and Monitor Dump is used for analyzing javacore dump; whereas IBM Monitoring and Diagnostic Tools for Java Memory Analyzer is used for analyzing heap dump. In the current use case, when an application is running out of heap, the JVM will eventually stop. As a result, javacore dump and heap dump will be generated in the %WAS_HOME%/profiles/AppSrv01 that provides tremendous information on JVM processes, threads running, and what objects are in the memory when the problem occurs. The toolset, IBM Monitoring and Diagnostic Tools for Java Memory Analyzer and IBM Thread and Monitor Dump Analyzer for Java, runs on ISA (IBM Support Assistant Workbench). The ISA can be downloaded from Step 1) After the ISA is installed, expand the Update menu and select Find New > Tools Add-ons to find the add-ons toolset. Step 2) Launch the IBM Support Assistant 4.1, then click Analyze Problem workbench. Figure 8: IBM Support Assistant IBM Thread and Monitor Dump Analyzer for Java 21

22 Step 3) Select IBM Thread and Monitor Dump Analyzer for Java (TMDA) to open up the Analyzer adds-on. Figure 9: IBM Thread and Monitor Dump Analyzer for Java Step 4) Open Thread Dumps and select the javacore dump files, e.g., javacore txt Step 5) Select the file in the list, its content will be displayed. Step 6) From the content, it shows that it fails to allocate memory for object creation.

23 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Figure 10: Failed to allocate memory for object The javacore dump also provides information on Memory Segment Analysis, as well as the free java heap size. The object represents the heap memory; class represents the off java heap memory which is not subject to GC; JIT Code Cache and JIT Data Cache are for byte code that is converted to assembler and stored in the memory for runtime execution. 23

24 The Thread Status Analysis shows the different states of the threads. The Thread Aggregation Analysis shows the type of threads in the dump. Step 7) Open up the Memory Analyzer from ISA by launching Analyze Problem and then IBM Monitoring and Diagnostic Tools for Java Memory Analyzer 64bit

25 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Figure 11: Launch Activity drop down list Figure 12: IBM Support Assistant - Memory Analyzer 25

26 Step 8) Click File > Open Heap Dump Step 9) Select the heapdump file, e.g. heapdump phd Next to the Overview, there will be a report generated using the build-in algorithms to attempt to find the leak suspect. This tends to find DynaCache as the leak suspect as the cache is one of the largest structures in the heap. This is not true in this case. Step 10) Click the Histogram icon,, to launch the histogram view. Some definitions of the terms are provided as follows: Objects Number of objects (count). Shallow heap Memory consumed by the object without including the objects referenced by it. Retained heap Memory that would be freed if all the objects for the class were garbage collected. In this example, it shows that the object MemoryObject and the SpecServletAction both have a very high retained heap. Figure 13: Memory Analyzer - Histogram

27 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Step 11) Select the MemoryObject, click Show object by class > by outgoing references to show the classes that the MemoryObject is using. Incoming references Shows the classes that use this class. Outgoing references Shows the classes that are referenced by this class (i.e. MemoryObject). Figure 14: Memory Analyzer - Outgoing References Step 12) Click Show object by class > by incoming references to show the classes that use the MemoryObject. It shows that SpecServletAction uses the MemoryObject class. Figure 15: Memory Analyzer - Incoming References If you go back to the Histogram view, you will find that the MemoryObject actually contributes almost all the retained heap of the SpecServletAction, as we just find 27

28 out that SpecServletAction is using MemoryObject. This indicates SpecServletAction is holding a list of MemoryObject which occupies lots of memory. Now, we can hand this set of information to the application developer to find out how the MemoryObject is being used in the SpecServletAction and why it is holding so many instances of MemoryObject without freeing them. 5.2 Situation Action and Policy Automation When a situation is raised, ITM offers a capability to define a follow-up action and policy automation. An action and policy is an automation workflow that consists of multiple tasks which will be performed on the target machine. The tasks can be a script to attempt to fix the problem, or as in the current case, to copy the javacore dump and heapdump file over to the TEMS server for further analysis. A policy is created and is triggered when a monitored WebSphere server is disconnected. An attempt will be made to copy the javacore and heapdump files from the remote monitored server to ITM TEMS server. Figure 16: Action and Policy for Server down situation The action runs the following script on the TEMS server (i.e. wasmonitor06), and copies the files from the monitored servers (wasmonitor01 and wasmonitor02) using SCP. #core and heap dump transfer script for winscp #run with following cmd: #"C:\Program Files (x86)\winscp\winscp.exe" /console /script=c:\was01dumpfiles\dumpxferscript.txt option batch abort option confirm off option transfer binary #connection to wasmonitor01 open wasuser:think4me@wasmonitor01.torolab.ibm.com:24

29 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications #change remote directory cd "Program Files (x86 )\ibm\websphere\appserver\profiles\appsrv01" #change local directory to javacore lcd C:\was01dumpfiles\javacore\ #download javacore text files option include "*.txt" synchronize local #change local directory to heapdump lcd C:\was01dumpfiles\heapdump\ #download heapdump phd files option include "*.phd" synchronize local close #connection to wasmonitor02 open #change remote directory cd "WAS80Install\IBM\WebSphere\AppServer\profiles\AppSrv 01" #change local directory to javacore lcd C:\was02dumpfiles\javacore\ #download javacore text files option include "*.txt" synchronize local #change local directory to heapdump lcd C:\was02dumpfiles\heapdump\ #download heapdump phd files option include "*.phd" synchronize local close exit 29

30 In the following scenario, we want to restart the server when the average heap usage is over 70% for a period of 3 minutes. This requires situations and policies to be created. Two situations with the following conditions must be created; 1. Condition of situation for wasmonitor01: ( AVG(Heap Used (%)) >= 70.0 AND Node Name == 'WASMONITOR01' AND Server Name == 'server1') 2. Condition of situation for wasmonitor02: ( AVG(Heap Used (%)) >= 70.0 AND Node Name == 'WASMONITOR02' AND Server Name == 'server2') The two situations are set with the sampling interval of 3 minutes and run at startup Two policies are created with the action to restart the monitored servers 1. Policy for situation on wasmonitor01 where the scripts to stop and start WAS are: YN:StopAppSvr <wasadminuser> <password> YN:StartAppSvr <wasadminuser> <password> 2. Policy for situation on wasmonitor02 where the script to stop and start WAS are: YN:StopAppSvr <wasadminuser> <password> YN:StartAppSvr <wasadminuser> <password>

31 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications To view the policies in action, the agent operations log view can be launched from the Tivoli Enterprise Monitoring Server > Agent Operations Log > Agent Operations Log query. The table below shows the transactions of the policies of copying dump files from the monitored system to TEMS, and restarting the server. Figure 17: Policy log of copying dump files and restart server 31

32 5.3 Agent Overhead Cost With an agent installed on the monitored system, an overhead cost to run the agent is measured. A set of data for each of the cases with and without the agent running on the monitored system is recorded respectively. Table 4: System with agent overhead cost analysis Agent Overhead Cost Analysis Total # of requests Total time in seconds Average throughput Coefficient of variance Delta (%) Without Agent With Agent* % % -3.56% Remarks HTTP requests, transaction type: browse (i.e. read only) System with agent run 3.56% slower than the system without agent *Agents installed on the monitored system: ITCAM Agent for WebSphere applications, Monitoring agent for Windows OS, ITCAM agent for HTTP servers 6 Summary We have illustrated the use of ITM and TEMS, javacore dump, and heapdump to perform the tasks from monitoring to deep dive analysis. The approach includes: Collecting the historical data from the monitored servers Identifying baselines and thresholds from the historical data Creating situations with appropriate severity Configuring actions and policies for automation when situations are raised Analyzing javacore dump and heapdump data using IBM Monitoring and Diagnostic Tools for Java Memory Analyzer, and IBM Thread and Monitor Dump Analyzer for Java This article attempts to fill in the gap between the operations team and the WebSphere SME team. The same approach shown in this use case can be adapted for other aspects of the systems. All that is needed is to choose the right set of TEMA agents. It is important to apply the right set of agents as they will feed in the right set of data for analytic purpose. Thus, the right set of the situations can be custom made to address the issues in questions.

33 Bridging Organizational Gaps with Monitoring: ITM/ITCAM for WebSphere Production Applications Copyright IBM Corporation 2012 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation 4205 South Miami Boulevard Research Triangle Park, NC U.S.A. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. If you are viewing this information softcopy, the photographs and color illustrations may not appear. 33

34 Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at Java and all Java-based trademarks and logos are trademarks or registered trademarks of Oracle and/or its affiliates. Other company, product, or service names may be trademarks or service marks of others. 34