ITCAM for Transactions: updating Web Response Time agent configuration to address Transaction Tracking overload

Transcription

1 ITCAM for Transactions: updating Web Response Time agent configuration to address Transaction Tracking overload Preface This document records the technical challenges encountered during a particular Agentless Monitoring deployment and the techniques and strategies used to overcome them. Authors: wshort@us.ibm.com William Lanny Short, Certified Process Specialist, IBM robert.cheung@au1.ibm.com Robert Cheung, ITCAM for Transactions Developer, IBM The problem A business has a mission critical application that spans many systems and components, and creates a lot of network traffic. When the deployment of ITCAM for Transactions: Web Response Time (KT5) was planned, the intention was to configure each agent to run in appliance mode. This deployment strategy would mean that only a small number of agents would be required and they could be deployed to monitor the application-generated web traffic. To allow an KT5 agent to monitor many hosts at once, network switches will need to enter Port Spanning mode and gave the KT5 agent host a copy of the network traffic. In this case, the network team could not support these hardware changes and port spanning could not be implemented. This resulted in the need to install KT5 agents on every host that monitoring was desired. For example, if the application consists of six IBM HTTP servers and six Websphere Application Servers, a KT5 agent is needed on each of those servers. The KT5 agent, by default, monitors all network traffic seen by its host's network interfaces. This has the potential for generating a lot of noise that can significantly impact the performance of the KT5 agent and the ITCAM infrastructure that it points to. The Application Management Console agent (KT3), which is critical for the management and maintenance of the ITCAM for Transactions solution, could use up a lot of CPU cycles or crash. This can happen because the KT3 agent is configured, by default, to report all of the applications that each KT5 agent finds. The CPU cycle consumption or crashing of the KT3 agent could be the result of seeing duplicate applications or other unnecessary network noise. So how do you mitigate this problem and prevent such a problem from occurring in future application on-boarding exercises? This white paper addresses these questions.

2 What are the underlying issues? The underlying issue is the performance of the Transaction Reporter (KTO) and the Application Management Console agent (KT3) agents. Due to a large number of KT5 agents deployed, these agents have a lot of incoming data which causes each agent to consume a lot of CPU cycles or to crash. The KT3 agent has the primary responsibility for configuring all the ITCAM for Transactions data and filtering it based on applications that have been created as part of its configuration settings. It also consolidates all the applications detected from the various ITCAM for Transactions agents, such as the KT5 agent, the Robotic Response Tracking agents, and other ITCAM for Transactions agents, into one pane. The KTO agents have the primary responsibility for displaying the ITCAM for Transactions data as it has been collected. They read through all the collected data and create transaction overlays based on the configurations created in the KT3 and the server and client sources and destinations of the collected data. If the data collected is extremely noisy, the agents spend a significant amount of time trying to sort through the noise and that is the primary cause of CPU cycle consumption or agent crashes. What is the solution? The solution to this problem has two phases. The first phase is to correct the immediate problem. The steps for this phase address the current problem with the KTO and KT3 agents getting bogged down in all the noisy ITCAM for Transactions data and help get the agents back to doing their jobs with no performance problems. The second phase is to prevent future problems when on-boarding applications. The steps for this phase include creating a sand-box that can be used for initial testing of the KT5 agents deployed onto the new application's components so that filtering configuration can be done as part of the on-boarding process. Phase 1: Correcting the immediate problem The aim of this phase is to limit the amount of data that KT5 agent creates by configuring it to only monitor traffic of interest. Using above example, the T5 agents are configured to monitor only HTTP(S) and Websphere Application Server traffic, and ignore other traffic. Furthermore, the KTO is checked to ensure that the number of entities it is monitoring has not exceeded its capacity. If it has, additional KTO agents has to be deployed to spread the load. This second step is important particular in scenario where dozens or even hundreds of KT5 agents has been deployed. First, stop the KTO agent from retrieving data from all the KT5 agents that is connected to the TEMS but instead restricted to a relevant subset. The KTO agent should be talking only to the KT5 agents that are deployed onto the application in question. This is done by setting the Aggregation Agent List configuration parameter of the KTO agent, documented in this knowledge center page. Second, using the Application Manager Console Editor (in the Tivoli Enterprise Portal) configure both the KT5 data sources and the Transaction Collector (KTU) data sources to clearly restrict the KT5 agents to only report on traffic related to the application at hand. Detailed instruction on this can be found in the best practices guide ITCAM for Transactions V7.3 Customization: Transaction Tracking Filtering and reporter. Third, confirm that the number of nodes and edges seen in the KTO transaction overlay diagrams is manageable. To get an idea of the number of nodes, edges, and interactions, in the Tivoli Enterprise Portal, complete the following steps:

3 1. In the navigator, select Transaction Reporter. 2. Right-click Transactions and select the Transaction Aggregate Topology workspace. 3. Click on the Table/Topology view toggle button to switch the view into a table row. 4. Check the number of rows returned. Figure 1: TEP Transaction Aggregate Topology Workspace Figure 1 documents what would be seen in the TEP Transaction Aggregation Topology Workspace. Alternative Methods: Check the Total displayed (circled in Figure 1). This is a coarser estimate because some of the nodes and interactions can be hidden by various conditions Perform a more detailed investigation of the Transaction Reporter logs. During each collection interval (configured to be 2 minutes by default), the KTO gathers all aggregates and interactions and logs how many of each were gathered and from which Transaction Tracking agents. To find this information, search for "collectionperiods()" in the latest KTO log. Count the number of RecordIdentityxxx.xml files that are contained in the <ITMHOME>/todata directory for the KTO. Each file represents a node that the KTO has seen at some stage. For example, on UNIX or Linux run the following command: > find <todata directory> -name "RecordIdentity*.xml" wc -1

4 Example output: 5628 A manageable overlay diagram should be less than 5,000 nodes and edges. If there are more nodes than that number, add more KTO agents to reduce the load, and then test the outcome. Repeat this process interactively. That is, activate only a small number of KT5 agents at a time, and complete the filtering and reporting steps for those agents before activating additional KT5 agents. Phase 2: Preventing future problems when on-boarding applications After phase 1, the KTO agent began working but introduction of additional KT5 agents cause it to again be overloaded. So how can future applications be on-boarded problem free? Ideally, create a sand-box in your Production environment where you can ensure that the on-boarding steps for the Production version of the new application do not cause any problems for the current deployment. More importantly, in the event that overload occurs, you can reset the KTO by flushing filtered transaction tracking nodes and edges. The only required component of the sand-box is a KTO agent that is used as part of the on-boarding process. The KTO agent's Aggregation Agent List should contain only those KT5 agents that have been deployed onto the new application's components. Perform the same reporting and filtering steps that you used when correcting the original problem. After you have completed those steps, note how many nodes and edges are in the sand-box KTO transaction overlay diagram. If the current main KTO's number of nodes and edges plus the sand-box KTO's number of nodes and edges is more than 5,000, complete the following steps: 1. Make sure that the main KTO's Aggregation Agent list includes only those KT5 agents that were already deployed. 2. Deploy and configure an additional main KTO agent and add the KT5 agents that were pointing to the sandbox KTO to the new main KTO's Aggregation Agent list. 3. Remove the new KT5 agents from the sandbox KTO's Aggregation Agent list. 4. Reset the sandbox KTO agent by deleting the <agent_home>\todata directory and restart that agent. 5. Repeat steps 1-4 until all the new application's KT5 agents have been added. Note: Do not turn on data warehousing for the new main KTO until the filtering is complete. Never turn on data warehousing for the sandbox KTO, there is no benefit and the agent's performance will be impacted. Conclusion: This white paper discussed addressing Transaction Tracking overload problems that occur when there is a need to deploy a significant number of ITCAM for Transactions: Web Response Time (KT5) agents to monitor web traffic for an application.

5 Document History Date Revision Notes June Initial version June Added sandbox concept, added diagram 17 June Added Preface, more edits. Thanks to Alexander Thornton for editorial review. First published