IBM Tivoli Monitoring Agent Management Services Performance Considerations in a Virtualized Environment

Transcription

1 IBM Tivoli Monitoring Agent Management Services Performance Considerations in a Virtualized Environment This document can be found on the web at Search for author s name under the category of White Papers. Version Date: May 2017 John Harris IBM Software Group Solutions Architect harrisjo@us.ibm.com & Mike Sine IBM Advanced Technical Skills I/T Specialist sine@us.ibm.com

2 Special Notices This document reflects the IBM Advanced Technical Skills organizations understanding of the IBM Tivoli Monitoring Agent Management Services. It was produced and reviewed by the members of the IBM Advanced Technical Skills organization. This document is presented As-Is and IBM does not assume responsibility for the statements expressed herein. It reflects the opinions of the IBM Advanced Technical Skills organization. These opinions are based on the authors experiences. If you have questions about the contents of this document, please contact the authors at or Trademarks The following are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. IBM, the IBM logo, Candle, DB2, developerworks, iseries, Passport Advantage, pseries, Redbooks, Tivoli Enterprise Console, WebSphere, z/os, xseries, zseries, z Systems, z/vm. A full list of U.S. trademarks owned by IBM may be found at NetView, Tivoli and TME are registered trademarks and TME Enterprise is a trademark of Tivoli Systems, Inc. in the United States and/or other countries. Microsoft, Windows, Windows NT, Internet Explorer, and the Windows logo are registered trademarks of Microsoft Corporation in the United States and/or other countries. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. UNIX is a registered trademark in the United States and other countries licensed exclusively through The Open Group. Intel and Pentium are registered trademarks and MMX, Pentium II Xeon and Pentium III Xeon are trademarks of Intel Corporation in the United States and/or other countries. Other company, product and service names may be trademarks or service marks of others. Special thanks to the following people who contributed information to this effort: Mike Bonett

3 IBM Tivoli Monitoring Agent Management Services IBM Tivoli Monitoring introduced a new feature in version known as Agent Management Services. While this feature offers many benefits, proper tuning is needed for large virtualized environments, to ensure its resource usage is not excessive and is in line with local requirements and expectations. This paper will briefly introduce this new feature, provide a basic understanding of the performance considerations, and discuss the options available to administrators. IBM Tivoli Monitoring provides a feature known as Agent Management Services (AMS). This service enhances availability of the ITM components using the following capabilities: Monitor the availability of the ITM agent as defined in user policy. Automated start, stop, manage, and unmanage actions for the ITM agent per policy. Monitor ITM agent via the Tivoli Enterprise Portal (TEP) The following is a description of the components of Agent Management Services: Agent Watchdog: The agent watchdog performs specific availability monitoring actions against the ITM agent based on the policy in the agent s common agent package (CAP) file. This component runs inside the OS agent process as a logical component. Other than the OS agent itself, the agent watchdog watches any monitoring agent that has an XML file in the CAP directory of the OS agent installation. Agent Management Services Watchdog: The AMS watchdog is the same programmatically as the agent watchdog within the OS, but it is used only to watch the OS agent. It is included as a stand-alone executable file with the OS agents and runs as a process kcawd on Linux and UNIX operating systems and as a process kcawd.exe on Windows. Performance Considerations in a Virtual Environment: It is important to recognize that the components of Agent Management Services are designed to provide the optimum availability of the ITM components running on a given system. It is also important to recognize the ITM components run very efficiently without the AMS feature. There is a trade-off between optimizing for availability with the AMS feature and resource usage. For a given system this is minimal, but aggregated across a large virtual environment, the default policy must be assessed and customized to avoid taking more CPU (e.g. an additional 2-4 percent as observed by the authors) than may be desired. In a dedicated system environment, this increase will most often go unnoticed. However, in a large virtual environment, this increase multiplied across all virtual machines can require more CPU resource and therefore its usage and setup must be planned. To better understand the potential performance impact in a virtual environment, let s consider a realistic Linux on z Systems environment. A z/vm system hosting 50 Linux on z Systems virtual machines running the ITM Linux OS agent deploying the AMS feature with a policy configuration requiring 2% additional CPU resources would consume a single IFL (System z CPU for Linux). In a CPU constrained

4 system, this subtle increase of 2% in a given guest may be difficult to notice in a test environment, yet the impact once deployed across these 50 virtual machines may be significant. Note: The example above is Linux on z Systems. The same principles apply to Windows, UNIX, and/or Linux systems virtualized on other platforms. AMS in a Virtual Environment: The AMS feature and the associated application support files are automatically installed with the Linux, UNIX, and Windows OS agents on many platforms. With the deployment of the ITM agent on these operating systems in a virtual environment, it is important to understand the potential impact of the AMS feature. If the AMS feature is desired, it is important to optimize the feature to the specific needs of the installation. If the AMS feature is not desired or the resources needed are not readily available, it may be necessary to disable the AMS feature. The AMS feature may be restarted when the needs of the feature arise and/or the system resources are available. Optimizing the AMS Feature: The monitoring behavior of AMS towards a particular agent is governed by settings in an XML-based policy file. This file is referred to as a common agent package (CAP) file. Every agent that can be managed by the AMS installs a CAP file. The naming convention of this CAP file is kpc_default.xml, where pc is the product code, and resides in the directory defined by the KCA_CAP_DIR environment variable in the OS monitoring agent configuration file. Example directories follow: Windows 64-bit: Install_dir\TMAITM6[x64]\CAP Linux or UNIX: Install_dir/config/CAP It is possible to minimize the impact of the AMS feature by modifying the policy settings of this file. The CAP file installed by the agent is configured to be read only and should not be directly modified. To customize the policy settings of a given CAP file, create a copy of the file and name it with the convention kpc.xml. There are several customizable elements of a CAP file: checkfrequency: Multiples of 5 seconds, up to a maximum of 3600 seconds (1 hour), Default: 30 cputhreshold: A positive integer from memory Threshould: A string of characters between 0 and 9. managertype: NotManaged or ProxyAgentServices, Default: NotManaged maxrestarts: Positive integer, Default: 4 Changing the customizable values in the CAP file may provide a desired amount of availability monitoring while minimizing the overhead of the AMS feature. Please see the IBM ITM Monitoring Administrator s Guide, Agent Management Services section for more details on the CAP file and its customizable elements.

5 Disabling the AMS Feature: For many customers the overhead of the AMS feature in a virtual environment outweighs the benefits and they will elect to disable the feature. On Windows Use the Services Applet to start/stop/disable the Watchdog. Permanently disabling AMS Feature on Windows: 1) Stop the agent 2) Open Manage Tivoli Enterprise Monitoring Services (MTEMS) GUI 3) Right-click on Monitoring Agent for Windows and Select Advanced->Edit Variables 4) Click on Add 5) Choose the KCA_CAP_DIR from the list 6) with empty 7) Select OK 8) Start Agent On Linux/UNIX: Use the itmcmd execute command to disable the AMS watchdog, examples are Linux for Unix the product code is ux: Install_Dir/itmcmd execute lz disarmwatchdog.sh There are also additional options that can be applied to the itmcmd command which are self explanatory:

6 Install_Dir/itmcmd execute lz startwatchdog.sh Install_Dir/bin/itmcmd execute lz stopwatchdog.sh Install_Dir/bin/itmcmd execute lz statuswatchdog.sh Substitute Install_Dir with your the installation directory, in most cases /opt/ibm/itm Please see the IBM ITM Monitoring User s Guide, for more details on using Agent Management Services. To permanently disable the AMS Feature on Linux/Unix: 1) Stop the agent 2) Edit configuration file of OS agent (lz.ini/ux.ini) 3) Comment out the line which starts with KCA_CAP_DIR=$CANDLEHOME$/config/CAP:/opt/IBM/CAP 4) Add line KCA_CAP_DIR= 5) Restart agent All Systems May Not be Equal: When performing a cost benefit analysis of the AMS feature in a virtual environment, it is important to realize not all virtual machines are equal in importance. In fact, many customers host systems of dissimilar resource needs and dissimilar importance in the same virtual environment in order to maximize the use of the hardware, without creating contention for the same resources at the same priority. Therefore, it may be that different systems require different levels of the AMS features for the ITM agents monitoring these systems. ITM administrators may find it appropriate to have different CAP files on different systems while disabling AMS on other systems altogether. It is important not to approach the use of AMS, or the lack thereof, with an all-or-nothing approach, but match the system to the level of monitoring and the availability features of that monitoring agent. Note: It is also possible to use situations in the Tivoli Enterprise Portal (TEP) to monitor the availability of the ITM agents. While it is not possible to auto-restart the agent via this method, alerting operators to the outage may be an attractive alternative if the AMS feature needs to be disabled. Examples using the TEP to monitor ITM agents for availability are outside the scope of this paper. Summary: IBM Tivoli Monitoring s new Agent Management Services feature provides high availability of the ITM agents on the Windows, Linux, and UNIX platforms. This new feature comes at a minimal cost on dedicated servers, however, may have a negative performance impact in large virtual server deployments. ITM administrators should evaluate the cost benefits against the feature s availability benefits. Customization and even disabling of the feature must be considered in large virtual environments based on business needs and system resources. It is important to recognize that different levels of AMS features may be appropriate for different systems in the same virtual environment.