Integrating IT Monitoring and Business Activity Monitoring

Transcription

1 Integrating IT Monitoring and Business Activity Monitoring February 2008 Integrating IT Monitoring and Business Activity Monitoring Document version 1.0 John W. Alcorn IBM Software Group David E. Cox IBM Software Group Janet Andersen IBM Software Group Joachim (Jim) H. Frank IBM Software Group Vivian Huang IBM Software Group

2 Copyright International Business Machines Corporation All rights reserved. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

3 CONTENTS List of Figures... vi List of Tables...vii Revision History...viii 1 Introduction Overview Installation and Configuration Prerequisites Installing and Configuring WebSphere Business Monitor Installing and Configuring IBM Tivoli Monitoring Installing and Configuring ITCAM for SOA Installing and Configuring the EIF to WS-Notification Converter Installing the EIF to WS-Notification Converter on Windows Systems Installing the EIF to WS-Notification Converter on a Linux or UNIX system Installing the WS-Notification Consumer on IBM WebSphere Application Server Configuring the Tivoli Enterprise Console product to Forward Events to the EIF to WS-Notification Converter ITCAM for SOA Situation Events Situation Event Background Information ITCAM for SOA Situations...22 Page 3 of 59

4 4.3 Correlating ITCAM for SOA Events Converting Events from EIF to CBE Format Introduction to the Sample Scenarios WebSphere Business Monitor Concepts Situation Monitor Event Subscriptions for the Situation Monitor Keys and Metrics for the Situation Monitor Counters and Stopwatches for the Situation Monitor Triggers for the Situation Monitor KPIs for the Situation Monitor Example Dashboard Views of the Situation Monitor Operation Monitor Event Subscriptions for the Operation Monitor Triggers and Counters for the Operation Monitor Metrics for the Operation Monitor Dimensional Model for the Operation Monitor Example Dashboard Views of the Operation Monitor Test Monitor...50 Appendix...52 Page 4 of 59

5 References...55 Notices...57 Page 5 of 59

6 LIST OF FIGURES Figure 1: ITCAM for SOA Service Monitoring View...10 Figure 2: A WebSphere Business Monitor Dashboard...11 Figure 3: Event integration flows when Tivoli Monitoring sends events directly to the EIF to CBE Converter...15 Figure 4: KPIs on a WebSphere Business Monitor Dashboard...26 Figure 5: ITCAM for SOA Situation Monitor (Monitor Details Model)...27 Figure 6: Stopwatch Controls for Time_Since_Last_Closed...34 Figure 7: KPI Counting Extended Time Situations...36 Figure 8: Example Monitoring Contexts for ITCAM for SOA Situations (instance view)...37 Figure 9: Extended Time Situation Count KPI...38 Figure 10: ITCAM for SOA Operation Monitor (model overview)...39 Figure 11: Dimensional Model for the Operation Monitor...46 Figure 12: Parent and Child Monitoring Contexts for ITCAM for SOA Operation Model (instance view)...47 Figure 13: Server Dimension View for the Parent Monitoring Context...48 Figure 14: Service Operation Dimension View for the Parent Monitoring Context...48 Figure 15: Service Requester Dimension View for the Child Monitoring Context...49 Figure 16: Operation Monitor Alerts View...49 Figure 17: Test Monitor Model for ITCAM for SOA Situation Events...50 Figure 18: Monitoring Contexts for ITCAM for SOA Test Monitor (instance view)...51 Page 6 of 59

7 LIST OF TABLES Table 1: Situation attributes present for each situation type...21 Table 2: Event Subscription Filter Conditions for the Situation Monitor...28 Table 3: Event Subscription Correlation Predicates for the Situation Monitor...29 Table 4: Keys and Metrics for the Situation Monitor...30 Table 5: Triggers for the Situation Monitor...35 Table 6: Event Subscription Filter Conditions for the Operation Monitor...39 Table 7: Correlation Predicates for Operation Monitor...40 Table 8: Triggers for the Operation Monitor...42 Table 9: Counters for the Operation Monitor...43 Table 10: Metrics for the Operation Monitor...45 Page 7 of 59

8 REVISION HISTORY Date Version Revised By Comments 02/11/ John Alcorn, Janet Andersen, David Cox, Joachim Frank, Vivian Huang Initial version Page 8 of 59

9 1 Introduction In the business world, people want to understand the health of their systems, and to gain insight into how well they are actually delivering value to their customers, so they can take action if some desired goal is not being met. Different people are interested in different levels of detail; for example, an operations manager may care about service invocation throughput and whether additional hardware is needed for a cluster to meet service response time requirements. However, a vice president is more likely to care about the hiring or reassigning of people or the altering of plans or budgets to meet their business goals related to costs, sales, and customer satisfaction. IBM provides a spectrum of products to address this range of monitoring needs, from fine-grained IT monitoring to aggregate level Business Activity Monitoring (BAM). These products can be combined to provide a wealth of information and insight into the performance of, and improvements needed in, all parts of a business. IBM provides a comprehensive suite of monitoring and management tools through its Tivoli software offerings. Tivoli software provides the monitoring and management of IT resources such as systems, application servers, and applications through its IBM Tivoli Monitoring (Tivoli Monitoring) product family and IBM Tivoli Composite Application Manager (ITCAM) product family. IBM Tivoli Business Service Manager can bring together views of various parts of the IT environment into a complete picture and can relate the health of individual IT resources to the overall business objectives. IBM Netcool /OMNIBus products can consolidate and correlate event information from across an enterprise to quickly isolate the root cause of a problem and to provide automatic actions or notifications. One monitoring and management product is IBM Tivoli Composite Application Manager for SOA (ITCAM for SOA) [1], which provides monitoring and management of the services layer in a Service Oriented Architecture (SOA) environment. An example of an ITCAM for SOA monitoring view is shown in Figure 1. Page 9 of 59

10 Figure 1: ITCAM for SOA Service Monitoring View IBM also provides IBM WebSphere Business Monitor [2], a BAM software product that subscribes to electronic messages that report business events. Based on such real-time electronic events, WebSphere Business Monitor can deliver up-to-date views of business performance using business metrics and key performance indicators (KPIs). Although the events that are processed by WebSphere Business Monitor can originate from any source, products in the WebSphere Business Integration family [3] facilitate their emission through built-in configuration options. For example, an insurance claim process running on WebSphere Process Server [4] can be configured to send events reporting its progress in real time as in addition to sending important information about the claim being processed. Based on this information, WebSphere Business Monitor can help an insurance company track its claim processes using business-level criteria such as geographic region, company division, and total claim value. WebSphere Business Monitor also provides the ability to detect and react to business situations in real-time, and includes business intelligence capabilities such as multi-dimensional analysis of historical business data. An example of a WebSphere Business Monitor dashboard is shown in Figure 2. Page 10 of 59

11 Figure 2: A WebSphere Business Monitor Dashboard Business performance in an SOA environment can critically depend on the health of the underlying systems and the services they provide. For example, system outages can prevent the completion of business processes and service response time degradation can result in customer dissatisfaction and the violation of service-level agreements. Consider the case of a business SLA, such as a promise to gold customers that their transactions will be processed within 4 hours. If an automated system to check their credit score is unavailable or requests to it are timing out, then it might not be processed in time if it has to be routed to a person to make phone calls or do other manual tasks necessary to determine if their credit rating is sufficient. In this case, an unrealized IT problem could lead to a business SLA violation, potentially impacting the bottom-line. Therefore, up-to-date information about the health and performance of services can be important input to a business monitoring scenario. Integration of IT status with Business Activity status on the same dashboard can also speed the process of problem isolation. If the WebSphere Business Monitor dashboard shows a slow process, the problem might be related to a business process problem such as too many approval steps, or the problem may be related to a slow IT resource such as a service or application server. The up-to-date information about the underlying services that implement the business process activities can lead the user down the right path toward solving the problem. To enable such synergies between service monitoring and business level monitoring, a communication link between ITCAM for SOA and WebSphere Business Monitor was developed. It allows events reporting situations that are detected by ITCAM for SOA to be Page 11 of 59

12 transmitted to WebSphere Business Monitor, which then receives, correlates, parses, and interprets the transmitted information. This paper explains how to set up this intercommunication. It also explains how events reporting situations detected by ITCAM for SOA can be post-processed by WebSphere Business Monitor, and how it can add value by aggregating information, detecting threshold violations, and rendering information about monitored services on a business dashboard. Page 12 of 59

13 2 Overview The first part of this paper describes how to set up a communication link and configure an event format converter between ITCAM for SOA and WebSphere Business Monitor. The second part describes three sample scenarios for processing ITCAM for SOA situations in WebSphere Business Monitor: 1. A "Situation Monitor," which demonstrates the basic techniques for receiving ITCAM for SOA situation events, correlating them, extracting data, post-processing the extracted information, aggregating it, and visualizing results on a dashboard. 2. An "Operation Monitor," which aggregates information about ITCAM for SOA situations by monitored service operation, and within each operation by service requester. It demonstrates some more advanced techniques for data aggregation and event post processing in WebSphere Business Monitor. 3. A "Test Monitor," which merely extracts all data fields that are available in ITCAM for SOA situation events. It was written to test the communication link and the event format converter; it shows how to access any of the transmitted data fields. Page 13 of 59

14 3 Installation and Configuration 3.1 Prerequisites The following products and components are used with this solution: IBM Tivoli Composite Application Manager for SOA V6.1 [1], V6.1 Fixpack 1 [5], or V7.1 [1] WebSphere Business Monitor V6.1 (including its prerequisites) [2] IBM Tivoli Monitoring V6.1 Fixpack 5, or a later fixpack [6] EIF to WS-Notification Converter (also called the Converter in this paper) [7] (Optional) Tivoli Enterprise Console V3.9 [8] (Optional) IBM Tivoli Netcool/OMNIbus V7.1 and the IBM Tivoli Event Integration Facility (EIF) Probe Version 1 [9] One of the following products: WebSphere Integration Developer V6.1 [10] Rational Application Developer V [11] with WebSphere Application Server 6.1 Fix Pack 13 [12] The EIF to WS-Notification Converter is available from the Tivoli Open Process Automation Library [7]. The Converter transforms EIF events received from the Tivoli Monitoring, Tivoli Enterprise Console, or Netcool/OMNIbus products into Common Base Events (CBEs) that are processed by WebSphere Business Monitor. For more information on the supported event flows, see the "Tivoli Event Flows Best Practices Integration for SOA" document that is available with the EIF to WS-Notification Converter. The installation and configuration sections in this paper describe how to configure event forwarding directly from Tivoli Monitoring to the Converter. This is the simplest configuration to get events from ITCAM for SOA to WebSphere Business Monitor. Page 14 of 59

15 Web service and IT data; situation events Web service and IT data Tivoli Enterprise Portal ITCAM for SOA and ITM EIF events EIF to CBE Converter CBEs WebSphere Business Monitor Business dashboards Figure 3: Event integration flows when Tivoli Monitoring sends events directly to the EIF to CBE Converter If you already have Tivoli Monitoring in your environment, Tivoli Monitoring might already be configured to forward EIF events to a supported Tivoli Enterprise Console or Netcool/OMNIbus event server. Tivoli Monitoring can only send events to one destination. When an event server is used, the event server is configured to forward a copy of the EIF events to the EIF to WS-Notification Converter. See the EIF to WS-Notification Converter package for more information on configuring the Tivoli Enterprise Console or Netcool/OMNIbus products to forward EIF events to the Converter. ITCAM for SOA V6.1 Fixpack 1 or V7.1 is required if you plan to use the Operation Monitor model described in section 7 on page 38 to show situation event data by service requester identity in WebSphere Business Monitor. 3.2 Installing and Configuring WebSphere Business Monitor Install and configure WebSphere Business Monitor V6.1 and the development toolkit using the installation and configuration documentation provided with the product [2]. 3.3 Installing and Configuring IBM Tivoli Monitoring Install and configure IBM Tivoli Monitoring V6.1 Fixpack 5 or a later fixpack using the installation and configuration documentation provided with the product [6]. Note: The remaining instructions in this section assume that Tivoli Monitoring is configured to forward EIF events directly to the Converter. The Tivoli Monitoring component that forwards EIF events is the Hub Tivoli Enterprise Monitoring Server. If Page 15 of 59

16 the monitoring server is configured to forward events to the Tivoli Enterprise Console or Netcool/OMNIbus products, follow the instructions in the Converter package to configure your event server to forward EIF events to the Converter. When configuring the Tivoli Enterprise Monitoring Server, enable the Tivoli Enterprise Console Event Integration Facility to send EIF events directly from Tivoli Monitoring to the Converter and provide the following information: Tivoli Enterprise Console Server Location: Specify the host name of the computer where the EIF to WS-Notification Converter will be installed. Tivoli Enterprise Console Port Number: Specify the port number that the Converter will use to listen for events. By default, all EIF events are then sent by Tivoli Monitoring to the EIF to WS-Notification Converter. To configure Tivoli Monitoring to only forward ITCAM for SOA events to the Converter, edit the EIF configuration file, which is the following file on the server where Tivoli Enterprise Monitoring Server is installed: On a Windows system: C:\IBM\ITM\cms\TECLIB\OM_TEC.CONFIG On a UNIX or Linux system: /opt/ibm/itm/tables/host name/teclib/om_tec.config where /opt/ibm/itm and C:\IBM\ITM are the default locations where Tivoli Monitoring is installed (if a different installation directory was chosen, change as appropriate) and host name is the value supplied during the Tivoli Enterprise Monitoring Server configuration. Add or modify the following parameters in the EIF configuration file: FilterMode=IN Filter:Class=ITM_Generic FilterCache:Class=ITM_Generic Filter:Class=ITM_Services_Inventory_610 FilterCache:Class=ITM_Services_Inventory_610 Filter:Class=ITM_Services_Inventory_ReqID_610 FilterCache:Class=ITM_Services_Inventory_ReqID_610 This configures Tivoli Monitoring to forward EIF events having one of the following three event types: ITM_Generic ITM_Services_Inventory_610 ITM_Services_Inventory_ReqID_610 Note: Page 16 of 59

17 If the EIF configuration file already contains values for any of these parameters, you should replace the current values with the values shown here. The last two lines for the ITM_Services_Inventory_ReqID_610 class are only required if you plan to process ITCAM for SOA situation events by service requester identity, as illustrated in section 7 "Operation Monitor" on page 38. After the configuration changes are complete, restart the Tivoli Enterprise Monitoring Server. 3.4 Installing and Configuring ITCAM for SOA Install and configure ITCAM for SOA V6.1 [1], V6.1 Fixpack 1 [5], or V7.1 [1] using the installation and configuration documentation provided with the product. If you plan to process ITCAM for SOA situation events by service requester identity for the Operations Monitor, configure monitoring by Web service requester using the configuration instructions in the ITCAM for SOA V6.1 Fixpack 1 readme file or V7.1 documentation [1]. 3.5 Installing and Configuring the EIF to WS-Notification Converter The EIF to WS-Notification Converter package is available from the Tivoli Open Process Automation Library [7] and contains multiple.tar files and a.zip file. For this solution, install and configure the components described in the following sections Installing the EIF to WS-Notification Converter on Windows Systems The EIF_WSN_CONVERTER_WIN.zip file contains the EIF to WS-Notification Converter installation and configuration package for Windows systems. Use this file if you plan to install the Converter on a Windows system. The EIF_WSN_Converter.readme file describes how to install and configure the Converter. When you configure the EIF to WS-Notification Converter, you must specify values that match your Tivoli Enterprise Management Server configuration and WebSphere Business Monitor installation as described in the following list: The Server 1 Name (also called the Channel 1 Name) should match the Tivoli Enterprise Console Server Location that you specified when configuring Tivoli Enterprise Monitoring Server. The Server 1 Port (also called the Channel 1 Port) should match the Tivoli Enterprise Console Port Number that you specified when configuring Tivoli Enterprise Monitoring Server. The Endpoint address should specify the host name and HTTP port number of the WebSphere Application Server hosting the Common Event Infrastructure (CEI) used by WebSphere Business Monitor. Page 17 of 59

18 You must also configure the Converter to use the kd4.baroc file, which defines the ITCAM for SOA event class types. The kd4.baroc file is located in the KD4\TEC directory on the ITCAM for SOA installation media. To configure the Converter to process the kd4.baroc file, perform these steps: 1. Copy the kd4.baroc file from the ITCAM for SOA installation media to the directory where the EIF to WS-Notification Converter is installed. 2. Edit the loadclasses file which is located in the directory where the Converter is installed. Add the following line to the end of the file: kd4.baroc 3. Restart the EIF to WS-Notification Converter using the instructions in the EIF_WSN_Converter.readme file Installing the EIF to WS-Notification Converter on a Linux or UNIX system The EIF_WSN_CONVERTER_NONWIN.tar file contains the EIF to WS_Notification Converter installation and configuration package for operating systems other than a Windows system. Use this file if you plan to install the Converter on a supported system that is not running the Windows operating system. The EIF_WSN_Converter.readme file describes the supported platforms and operating systems and contains the installation and configuration instructions for the Converter. When you configure the EIF to WS-Notification Converter, you must specify values that match your Tivoli Enterprise Monitoring Server configuration and WebSphere Business Monitor installation as described in the following list: The Server 1 Name (also called the Channel 1 Name) should match the Tivoli Enterprise Console Server Location that you specified when configuring Tivoli Enterprise Monitoring Server. The Server 1 Port (also called the Channel 1 Port) should match the Tivoli Enterprise Console Port Number that you specified when configuring Tivoli Enterprise Monitoring Server. The Endpoint address should specify the host name and HTTP port number of the WebSphere Application Server that hosts the Common Event Infrastructure (CEI) used by WebSphere Business Monitor. You must also configure the Converter to process the kd4.baroc file, which defines the ITCAM for SOA event class types. The kd4.baroc file is located in the kd4/tec directory on the ITCAM for SOA installation media. To configure the Converter to process the kd4.baroc file, perform the following steps: 1. Copy the kd4.baroc file from the ITCAM for SOA installation media to the directory where the EIF to WS-Notification Converter is installed. 2. Edit the.loadclasses file which is located in the directory where the Converter is installed. Add the following line to the end of the file: kd4.baroc 3. Restart the EIF to WS-Notification Converter using the instructions in the EIF_WSN_Converter.readme file. Page 18 of 59

19 3.5.3 Installing the WS-Notification Consumer on IBM WebSphere Application Server The WS_Notification.tar file contains the WS_Notification.readme file, scripts, and a properties file for enabling a WS-Notification consumer in WebSphere Application Server V6.1. The WS-Notification consumer receives CBE events from the EIF to WS-Notification Converter and passes them to the Common Event Infrastructure (CEI) server in WebSphere Application Server. Install and configure the WS-Notification Consumer on the WebSphere Application Server that is monitored by WebSphere Business Monitor. You can skip the step that enables the Common Event Infrastructure because it was already enabled when you installed and configured WebSphere Business Monitor Configuring the Tivoli Enterprise Console product to Forward Events to the EIF to WS-Notification Converter The TEC_CEI.tar file contains a readme file, scripts, and a configuration file for enabling the Tivoli Enterprise Console product to forward EIF events to the EIF to WS-Notification Converter. Use this file if Tivoli Enterprise Monitoring Server is configured to forward events to the Tivoli Enterprise Console product. Page 19 of 59

20 4 ITCAM for SOA Situation Events 4.1 Situation Event Background Information IBM Tivoli Monitoring provides a Situation Editor in the Tivoli Enterprise Portal. A situation is defined using the following information: A condition expression that checks the values of one or more attributes The attribute group that contains the attributes, which can be used in a condition expression The list of managed systems or managed system types that should be monitored for the situation condition For example, you can define a situation to detect when the response time for a Web service operation is greater than 5 seconds and check for this condition on application servers that are monitored by the ITCAM for SOA monitoring agent. When a situation condition evaluates to true, the Tivoli Enterprise Monitoring Server creates a situation event that signals the occurrence ("opening") of this situation. It is displayed in the Tivoli Enterprise Portal and also sent as an EIF event if the monitoring server has been configured to forward events. The EIF event contains the following items: The event class name, which defines the set of attributes that the event contains The name of the situation whose occurrence is being reported A situation status attribute set to Y to designate an opening of a situation event, that is, a new occurrence of the situation condition The name of the Tivoli Monitoring managed system whose data matched the situation condition (the system where the situation condition was detected) An optional situation display item attribute, which can be used to uniquely identify the resource whose status is being reported by this event The attributes from the attribute group that is referenced in the situation definition Additional attributes that provide information about the Tivoli Enterprise Monitoring Server that detected that the situation condition was true and the Hub Tivoli Enterprise Monitoring Server that sent the event When a situation condition evaluates to true for a monitored resource and an open event is generated, the situation remains open for the resource until one of the following scenarios occur: The situation condition evaluates to false for the resource: In this case, Tivoli Enterprise Monitoring Server closes the opened situation and also sends an EIF event reporting this. The EIF event situation status attribute is set to N to designate the closing of the situation ( false evaluation of the situation condition) for the resource. Page 20 of 59

21 An operator stops the situation from being monitored: In this case, Tivoli Enterprise Monitoring Server stops monitoring the situation condition on all managed systems and sends an EIF event that reports this. The EIF event situation status attribute is set to P to designate a stop event. The Tivoli Enterprise Monitoring Server is restarted: In this case, Tivoli Enterprise Monitoring Server sends an EIF event that indicates all open situations have been closed. Also, after a situation condition evaluates to true for a resource, only one open situation event is generated. Another open situation event for the same situation condition and resource is not generated until both of the following things occur: The situation condition evaluates to false, or monitoring of the situation has been stopped and restarted, or Tivoli Enterprise Monitoring Server is restarted The situation condition evaluates to true again. Table 1 shows what type of information is included in an EIF event for the situation types previously described. Table 1: Situation attributes present for each situation type Situation Type Event Class Name Situation Name Attribute Open X X X Situation Status Attribute Managed System Name Attribute Situation Display Item Attribute Attributes from the attribute group referenced by the situation definition X X X X Monitoring Server Attributes (set to Y) Close X X X X X X (set to N) Stop X X X X (set to P) All events closed because the Monitoring Server was restarted X X (set to *) X (set to N) X Note: Page 21 of 59

22 EIF events are also sent when operators acknowledge situations that have been opened or start the monitoring of a situation condition. The situation status attribute in these EIF events indicates what action has been performed. Situation status values other than open, close, and stop are ignored by the examples described in this paper. For a complete list of the situation status values, see the IBM Tivoli Monitoring V6.1 Information Center [6]. 4.2 ITCAM for SOA Situations ITCAM for SOA includes a predefined set of situation definitions. An operator can also use the Tivoli Monitoring Situation Editor to define custom situation definitions for ITCAM for SOA attribute groups. The ITCAM for SOA monitoring agent supports multiple attribute groups. However, the solution described in this paper uses the following two attribute groups because they contain metrics about Web service operations: Services Inventory_610: This attribute group contains attributes that identify an operation name and namespace, its port name and namespace, and the application server environment where the operation is monitored. The attribute group also contains metrics for the operation such as the average response size, average message count, and number of messages observed during the monitoring interval. Services Inventory ReqID_610: This attribute group contains the same attributes as the Services Inventory_610 attribute group plus an additional attribute that identifies the user or business partner that originated a Web service request for the operation. The ITCAM for SOA Information Center [1] describes the attributes in the Services Inventory_610 attribute group. Tivoli Enterprise Monitoring Server also adds additional attributes to the EIF event to identify the monitoring server that detected the event and the Hub Tivoli Enterprise Monitoring Server that sent the event. These attributes are described in the IBM Tivoli Monitoring Information Center [6]. The Page 22 of 59

23 Test Monitor described in section 8 on page 50 shows how to extract each of these attributes from an event. ITCAM for SOA maintains metric data for requesters of a Web service operation and providers of a Web service operation. The service type attribute in a situation event identifies whether the event contains data associated with a service requester or a service provider. The monitor models described in this paper only define the processing of events for a service provider. In addition, they define subscriptions to "open" events whose interval status attribute indicates that the metric data is from a completed monitoring interval and whose situation display item attribute is not set to an empty string. Therefore, if you use the Tivoli Monitoring Situation Editor to define custom situation definitions for the Services Inventory_610 or Services Inventory ReqID_610 attribute groups, you must configure a display item for the situation to ensure the event is processed by the examples described in this paper. The situation display item for the Services Inventory_610 attribute group is a hash code that is calculated using the operation name and namespace, port name and namespace, and the service type attributes. The situation display item for the Services Inventory ReqID_610 attribute group is also a hash code that is calculated using the same attributes plus the requester identity attribute. 4.3 Correlating ITCAM for SOA Events ITCAM for SOA events can be correlated using different sets of event attributes. If you want to correlate open and close events for the Services Inventory_610 attribute group, you must use the following attributes: Situation name Managed system name Situation display item The same attributes can be used to correlate open and close events for the Services Inventory ReqID_610 attribute group. The operation name and namespace, port name and namespace attributes from the Services Inventory_610 and Services Inventory ReqID_610 attribute groups, which also uniquely identify the service operation whose status is being reported, cannot be used when correlating open and close events because these attributes are not included in a situation close event. Therefore, the situation display item value must be used for correlation because it is included in both types of events. If you want to correlate open events that are generated using either the Services Inventory_610 attribute group or the Services Inventory ReqID_610 attribute group and that pertain to the same Web service operation, you must use the following attributes: Operation name and namespace Port name and namespace Managed system name Page 23 of 59

24 Note: In this case, the situation display item value cannot be used for correlation. The situation display item for a Web service operation that is included in the Services Inventory_610 attribute group is calculated differently than the situation display item for the same Web service operation in the Services Inventory ReqID_610 attribute group. 4.4 Converting Events from EIF to CBE Format The EIF to WS-Notification Converter receives EIF events from Tivoli Monitoring or an event server and converts them to Common Base Events (CBEs) [13]. Then, it sends the CBEs to a WS-Notification consumer defined in WebSphere Application Server V6.1. After a CBE is received by the WS-Notification Consumer, it is passed to the Common Event Infrastructure (CEI) server in WebSphere Application Server and can be processed by WebSphere Business Monitor. The EIF_WSN_Converter.readme file included in the EIF to WS-Notification Converter package describes how EIF event attributes (also called slots) are mapped to CBE attributes. The Converter uses the BAROC file for a Tivoli monitoring agent (such as the ITCAM for SOA monitoring agent) when performing the conversion. The BAROC file defines the event classes (which correspond to an attribute group), the attributes available in each event class, and the attribute types. A Common Base Event can contain the payload of a user-defined structure in context data elements and extended data elements [13]. The names and types of these data elements are described in a CBE file. WebSphere Business Monitor uses CBE files to validate expressions referencing event content and to determine the data types of user-defined payload elements. The CBE files provide a type definition for the user-defined part of a CBE, just as the BAROC files provide a type definition for EIF events. Therefore, CBE files describe the output of the EIF to WS-Notification Converter and are derived from the BAROC files which describe the input to the Converter. Three CBE files are included in the project interchange file provided with this solution: 1. ITM_Services_Inventory_610.cbe: This file defines attributes for open, close, and stop events generated from the Services Inventory_610 attribute group. It also includes the base attributes that Tivoli Monitoring adds to each EIF event. 2. ITM_Services_Inventory_ReqID_610.cbe: This file defines attributes for open, close, and stop events generated from the Services Inventory ReqID_610 attribute group. It also includes the base attributes that Tivoli Monitoring adds to each EIF event. 3. ITM_Generic.cbe: This file defines the attributes for the event that is sent when the Tivoli Enterprise Monitoring Server is restarted. Page 24 of 59

25 5 Introduction to the Sample Scenarios This paper explains how WebSphere Business Monitor can process ITCAM for SOA events that have been converted to CBE format using the EIF to WS-Notification Converter that was introduced in section 3.5 "Installing and Configuring the EIF to WS-Notification Converter" on page 17. Three sample scenarios are presented, which also illustrate general techniques for defining "monitor models" that are used to configure WebSphere Business Monitor. To introduce these scenarios, it might be helpful to review some of the basic concepts of WebSphere Business Monitor. If you are familiar with these concepts, you can skip section WebSphere Business Monitor Concepts To monitor any real or abstract entity, WebSphere Business Monitor instantiates a monitoring context. A monitoring context is an "observer object", which provides a context for receiving events, correlating them, calculating metrics, and detecting business situations. A monitoring context is instantiated for each observed entity, usually when WebSphere Business Monitor receives an event that signals its creation. The context then remains dedicated to monitoring the associated entity until it is terminated. Monitored entities can be business processes (for example monitoring execution time, current state, accumulated cost), products (for example, monitoring inventory levels, sales volume, production cost), devices (for example, monitoring failure rates, filling levels, need for maintenance), or anything else that can have its state tracked by receiving and analyzing events. The attributes of a monitoring context are called metrics. They hold the information that WebSphere Business Monitor has acquired about the entity. Their values are set using maps, which can depend on data in inbound events and other metrics. A monitoring context subscribes to inbound events that carry information about the state of the monitored entity: it receives the events, extracts any interesting data, and feeds it into maps which then update its metrics. Business conditions are detected in a monitoring context using triggers: A trigger has a gating condition and one or more evaluation criteria. The evaluation criteria define when the gating condition is evaluated (for example, when an event arrives, when a metric is updated, or periodically). If it evaluates to true the trigger fires. As a result, maps can be executed, and outbound events can be sent for human notification or to initiate action. In addition to metrics, triggers, and inbound/outbound event data, a monitoring context can contain counters and stopwatches (timers), which are useful auxiliary constructs in defining monitor models. A variant of a monitoring context is a KPI context. It holds KPIs (key performance indicators), which can aggregate information from several monitoring contexts, and contain triggers to detect situations based on such aggregates, such as an average dropping below a threshold. The aggregation of one metric (for example, order value) can be qualified by restricting the values of other metrics (for example, product type or sales Page 25 of 59

26 region). Using the terminology of dimensional analysis, the metrics that are being aggregated are called facts, and metrics qualifying the aggregation are called dimensions. A sample rendering of KPIs on a dashboard is shown in Figure 4. Figure 4: KPIs on a WebSphere Business Monitor Dashboard Page 26 of 59

27 6 Situation Monitor The Situation Monitor model is used for tracking ITCAM for SOA situations. The complete model can be downloaded from [14]. The Monitor Details Model view is shown in Figure 5. Figure 5: ITCAM for SOA Situation Monitor (Monitor Details Model) The model contains a single monitoring context definition (ITCAM for SOA Situation). It is instantiated when an Open event arrives, indicating that an ITCAM for SOA situation condition evaluated to true for some service operation. Each combination of the following gives rise to a new instance:. (managed system name, port name, operation name): designating a deployed service operation (situation name): designating an ITCAM for SOA situation definition Page 27 of 59

28 Each monitoring context instance monitors the occurrences of this situation for a particular (deployed) service operation The termination of this context is triggered by either a Close event (indicating that the situation condition is no longer true for the service operation), a Stop event (indicating that this situation is no longer monitored), or an event indicating a Tivoli Enterprise Monitoring Server restart (Close_All). Context termination is not immediate, but occurs after a configurable delay, so that ITCAM for SOA situations can be viewed on the WebSphere Business Monitor dashboard for some time after they are closed. Nested within the monitoring context definition in Figure 5, there are definitions for the following model elements: 3 keys, which together identify an ITCAM for SOA situation for a particular monitored service operation 23 metrics, which contain further information about this ITCAM for SOA situation and the service operation where it occurred 5 triggers, which control the termination of the monitoring context and the detection of conditions that might require attention 4 inbound event subscriptions 1 counter 2 stopwatches (or timers) Each model element is described in detail in the following sections. 6.1 Event Subscriptions for the Situation Monitor There is one event subscription for each kind of situation event (as introduced in section 4.1 "Situation Event Background Information" on page 20): Open, Close, Stop, and Close_All (indicating Monitoring Server restart). Each inbound event subscription has a filter condition, which characterizes the events it subscribes to, and a correlation predicate, which correlates the event with the monitoring contexts to which it should be delivered. Note that the Stop and Close_All events can pertain to multiple situations and different service operations, and therefore might have to be delivered to more than one monitoring context. Finally, each event subscription defines a CBE extension name, which serves two functions: It is part of its filter condition It references a type definition in a CBE file, which describes the event's payload. The following two tables summarize the characteristics of the four subscriptions. Table 2: Event Subscription Filter Conditions for the Situation Monitor Id Description Extension name Filter condition Open Event signaling that the ITM_Services_ fn:exists(open/extendeddata/situation_displayitem) and Page 28 of 59

29 situation condition has been detected Inventory_610 udf:removequotes(open/extendeddata/situation_displayitem) ne '' and Open/extendedData/service_type = 1 and Open/extendedData/interval_status = 2 and udf:removequotes(open/extendeddata/situation_status) = 'Y' Close Event signaling that the situation condition is no longer true ITM_Services_ Inventory_610 fn:exists(close/extendeddata/situation_displayitem) and udf:removequotes(close/extendeddata/situation_displayitem) ne '' and udf:removequotes(close/extendeddata/situation_status) = 'N' Stop Event signaling the end of monitoring for this situation ITM_Services_ Inventory_610 udf:removequotes(stop/extendeddata/situation_status) = 'P' Close_All Event signaling that ITM has been restarted and all open events have been closed ITM_Generic udf:removequotes(close_all/extendeddata/ master_reset_flag) = 'S' or udf:removequotes(close_all/extendeddata/ master_reset_flag) = 'R' Table 3: Event Subscription Correlation Predicates for the Situation Monitor Id Correlation predicate Zero matches One match Multiple matches Open udf:removequotes(open/extendeddata/situation_displayitem) = Unique_Identifier_for_Operation_and_Service_Port and udf:removequotes(open/extendeddata/situation_origin) = Managed_System_Name and udf:removequotes(open/extendeddata/situation_name) = Situation_Name Close udf:removequotes(close/extendeddata/situation_displayitem) = Unique_Identifier_for_Operation_and_Service_Port and udf:removequotes(close/extendeddata/situation_origin) = Managed_System_Name and udf:removequotes(close/extendeddata/situation_name) = Situation_Name Stop udf:removequotes(stop/extendeddata/situation_name) = Situation_Name create new instance ignore ignore deliver to the instance deliver to the instance deliver to the instance Close_All Situation_Status = 'Opened' ignore deliver to the instance treat as error treat as error deliver to all instances deliver to all instances Comparing the extension name and filter condition columns, you see that a Monitor Server Restart event (Close_All) is distinguished by its extension name, while Open, Close, and Stop events have the same extension name but different situation_status values. The filter Page 29 of 59

30 conditions for Open and Close events also verify the existence of a situation_displayitem value (which is needed for correlation), and the filter for Open events ensures that the event has service_type = 1 (for service provider) 1 and interval_status = 2 (completed monitoring interval). A user-defined function of udf:removequotes is applied to any string values retrieved from an inbound event in order to remove enclosing quotation marks. A source code listing of the implementation is provided in the Appendix. The correlation criteria shows that Open and Close events are correlated using the three key metrics. They are set from the situation_displayitem, situation_origin, and situation_name fields in the Open event when a context is created. If no matching context is found for an Open event, a new one is created, while a Close event for which no matching context is found is ignored. If correlation processing returns one matching context for an Open or Close event, the event is delivered to that context, while multiple matches are considered an error (it would mean that several monitoring contexts have identical keys). A Stop event is correlated with any monitoring context for the ITCAM for SOA situation whose monitoring is discontinued. There can be more than one if the situation has occurred for different service operations. The event is delivered to all matching monitoring contexts. A Close_All event is delivered to any monitoring context for an ITCAM for SOA situation that is currently open. This is appropriate when all events are produced by a single Tivoli Monitoring installation so that a Tivoli Enterprise Monitoring Server Restart implies closing any and all situations that are reported open. In scenarios where several Tivoli Monitoring installations report ITCAM for SOA situations, a more selective correlation mechanism is needed. The monitoring server identification must then be retained in the monitoring context and tested in the correlation predicate for Close_All. 6.2 Keys and Metrics for the Situation Monitor The following table shows all metrics defined in our monitor model (including the three keys) with their data types and map expressions. Table 4: Keys and Metrics for the Situation Monitor Metric id [type] Map expression Managed_System_Name (key) [xs:string] Situation_Name (key) [xs:string] Unique_Identifier_for_ udf:removequotes(open/extendeddata/situation_origin) udf:removequotes(open/extendeddata/situation_name) udf:removequotes(open/extendeddata/situation_displayitem) 1 While in our test setup all services are monitored at the provider side, monitoring at the service requester side might be necessary, for example, when monitoring a workflow engine where data collection might be feasible at the process server but not at the invoked services (especially if those are remote). Page 30 of 59

31 Operation_and_Service_Port (key) [xs:string] Application_Server_Cell_Name [xs:string] Application_Server_Cluster_Name [xs:string] Application_Server_Environment [xs:string] Application_Server_Hostname [xs:string] Application_Server_IP_Address [xs:string] Application_Server_Name [xs:string] Application_Server_Node_Name [xs:string] Average_Message_Length [xs:integer] Average_Response_Time [xs:integer] Event_Open_Time [xs:datetime] Event_Severity [xs:string] Extended_Time_Alert [xs:string] udf:removequotes(open/extendeddata/application_server_cellname_u) udf:removequotes(open/extendeddata/application_servercluster_u) if (Open/extendedData/application_serverenv = 1) then 'WebSphere Application Server' else if (Open/extendedData/application_serverenv = 2) then '.NET' else if (Open/extendedData/application_serverenv = 3) then 'BEA WebLogic Server' else if (Open/extendedData/application_serverenv = 4) then 'JBoss' else if (Open/extendedData/application_serverenv = 5) then 'CICS' else if (Open/extendedData/application_serverenv = 6) then 'SAP' else if (Open/extendedData/application_serverenv = 7) then 'WebSphere Application Server Community Edition' else if (Open/extendedData/application_serverenv = 10) then 'WebSphere Message Broker' else 'Unknown' udf:removequotes(open/extendeddata/local_hostname_u) fn:concat(udf:removequotes(open/extendeddata/local_ipaddress_u), ':', Open/extendedData/port_number) udf:removequotes(open/extendeddata/application_servername_u) udf:removequotes(open/extendeddata/application_server_nodename_u) Open/extendedData/avg_msg_length Open/extendedData/avg_elapsed_time if (fn:exists(open/predefineddata/creationtime)) then Open/predefinedData/creationTime else fn:current-datetime() if (Open/predefinedData/severity = 20) then 'Harmless' else if (Open/predefinedData/severity = 30) then 'Warning' else if (Open/predefinedData/severity = 40) then 'Minor' else if (Open/predefinedData/severity = 50) then 'Critical' else if (Open/predefinedData/severity = 60) then 'Fatal' else 'Unknown' Start_Extended_Time_Alert => 'Situation event has been open for more than 10 minutes' Stop_Extended_Time_Alert => '' Page 31 of 59

32 Extended_Time_Alert_On [xs:integer] Fault_Count [xs:integer] High_Request_Volume_Alert [xs:string] Max_Message_Length [xs:integer] Max_Response_Time [xs:integer] Message_Count [xs:integer] Monitoring_Interval_In_Minutes [xs:integer] Service_Operation_Qualified_Na me [xs:string] Service_Port_Qualified_Name [xs:string] Situation_Status [xs:string] Time_of_Last_Update [xs:datetime] Start_Extended_Time_Alert => 1 Stop_Extended_Time_Alert => 0 Open/extendedData/fault_count Start_High_Request_Volume_Alert => 'High Service Request Volume' Stop_High_Request_Volume_Alert => '' Open/extendedData/max_msg_length Open/extendedData/max_elapsed_time Open/extendedData/msg_count Open/extendedData/interval_length fn:concat('{', udf:removequotes(open/extendeddata/operation_namespace_u), '}', udf:removequotes(open/extendeddata/operation_name_u)) fn:concat('{', udf:removequotes(open/extendeddata/port_namespace_u), '}', udf:removequotes(open/extendeddata/service_name_u)) if (udf:removequotes(open/extendeddata/situation_status) = 'Y') then 'Opened' else 'Unknown' if (udf:removequotes(close/extendeddata/situation_status) = 'N') then 'Closed' else 'Unknown' if (udf:removequotes(stop/extendeddata/situation_status) = 'P') then 'Stopped' else 'Unknown' if (udf:removequotes(close_all/extendeddata/master_reset_flag) = 'S' or udf:removequotes(close_all/extendeddata/master_reset_flag) = 'R') then 'Closed' else 'Unknown' if (fn:exists(open/predefineddata/creationtime)) then Open/predefinedData/creationTime else fn:current-datetime() if (fn:exists(close/predefineddata/creationtime)) then Close/predefinedData/creationTime else fn:current-datetime() if (fn:exists(stop/predefineddata/creationtime)) then Stop/predefinedData/creationTime else fn:current-datetime() if (fn:exists(close_all/predefineddata/creationtime)) then Close_All/predefinedData/creationTime else fn:current-datetime() Most metrics, including the three keys and all metrics containing measurements (such as Average_Message_Length and Average_Response_Time) are simply set from an attribute Page 32 of 59

33 in the Open event (the user-defined XPath function for quotation mark removal is again applied to all strings). The copied values can be viewed on a dashboard, aggregated via KPIs, and used in maps for higher level metrics and trigger conditions. Other metrics have more sophisticated maps: The maps for Application_Server_Environment and Event_Severity translate numeric indicators into human readable strings. The maps for Application_Server_IP_Address, Service_Operation_Qualified_Name, and Service_Port_Qualified_Name use string concatenation to populate their target metrics with the IP address and port number of the application server, or with the fully qualified name (namespace name and local part) of a service operation or port. The maps for Event_Open_Time and Time_of_Last_Update have conditional expressions which return the event's creation time if present, or else the system time of the monitor server. The maps for Extended_Time_Alert, Extended_Time_Alert_On, and High_Request_Volume_Alert, are trigger controlled, which in the previous table is indicated using the notation Trigger_Name => map-expression: While maps without triggers are executed each time an input value changes (for example, when an event arrives on which the map depends, or when a metric on which it depends is updated) a trigger-controlled map only runs when the associated trigger fires. The Start_Extended_Time_Alert and Stop_Extended_Time_Alert triggers fire when a situation that is open for an extended amount of time is detected, and when it is closed, respectively. They set and reset, respectively, the Extended_Time_Alert and Extended_Time_Alert_On metrics. The first metric only highlights a situation that is open for a long time on the dashboard; the second is used to aggregate the number of extended time situations in a KPI, which is discussed later. The start and stop triggers for a high request volume alert set and clear the High_Request_Volume_Alert metric, which serves as a high request volume indicator on the dashboard. The Situation_Status and Time_of_Last_Update metrics have several maps each, which are triggered by the different kinds of events subscribed to. The maps for Situation_Status also provide another example of translating single-character indicators into human readable strings. 6.3 Counters and Stopwatches for the Situation Monitor A counter is like an integer metric that can be incremented, decremented, and set to zero via triggers or events. The Number_of_Times_Open counter is incremented by Open events and thus shows how many times the monitored situation was opened or reopened for the monitored port and operation. A stopwatch (timer) is a special kind of metric that shows elapsed time. It can be started, stopped, and reset by events or triggers. As we see in Figure 6, the Time_Since_Last_Closed stopwatch is started by Close, Stop, or Close_All events. Each Page 33 of 59

34 of these events signals that either the situation condition is no longer true or is no longer being monitored. Therefore, the stopwatch shows the time since the end of the situation's last occurrence. It is used to trigger the termination of the monitoring context at a configurable time after that. (The trigger that causes the termination is described later.) If a new Open event arrives before the monitoring context times out, the stopwatch is stopped and reset. Any Close, Stop, or Close_All event that will end the new situation occurrence will re-start it. Figure 6: Stopwatch Controls for Time_Since_Last_Closed The Total_Time_Opened stopwatch is used to track the total time that a situation is in the open state for a monitored service operation. It is started or restarted when an Open event is received for a service operation and is stopped by Close, Stop, or Close_All events. It is never reset. Page 34 of 59

35 6.4 Triggers for the Situation Monitor Triggers are like "watchdogs" in a monitoring context. They have two defining aspects, an evaluation time and a gating condition. The evaluation time determines when the trigger is evaluated, and can be periodic or given by some occurrence in the monitoring context (the arrival of an event, a value change of a metric, or the firing of another trigger). The gating condition then determines if the trigger fires. The definitions of the five triggers are shown in the following table. Table 5: Triggers for the Situation Monitor Trigger id Evaluated when Gating condition Context_Termination_ Trigger Start_Extended_Time_ Alert Stop_Extended_Time_ Alert Start_High_Request_ Volume_Alert Stop_High_Request_ Volume_Alert Periodically (once per day) periodically (once per minute) Close event, Stop event, or Close_All event Open event Close event, Stop event, or Close_All event Time_Since_Last_Closed > xs:daytimeduration('p30d') (fn:current-datetime() - Event_Open_Time) > xs:daytimeduration('pt10m') and Situation_Status = 'Opened' and (fn:empty(extended_time_alert) or (Extended_Time_Alert = '')) fn:not(fn:empty(extended_time_alert) or (Extended_Time_Alert = '')) (Message_Count div Monitoring_Interval_In_Minutes) > 4 and Situation_Status ='Opened' and (fn:empty(high_request_volume_alert) or (High_Request_Volume_Alert = '')) fn:not( fn:empty(high_request_volume_alert) or (High_Request_Volume_Alert = '')) The Context_TerminationTrigger is evaluated once per day and fires if the Time_Since_Last_Closed timer shows that more than 30 days have passed since the Situation last ended. It then terminates the monitoring context. The Start_Extended_Time_Alert trigger is evaluated every minute. It fires if more than 10 minutes have passed since the monitored situation has been opened (and an Extended_Time_Alert has not been signaled already). The situation is then considered an "extended time situation," and the Extended_Time_Alert metric is set to highlight this on the dashboard. The threshold of 10 minutes was used for test purposes; different values might apply in a production environment. The Start_High_Request_Volume_Alert trigger is evaluated when an Open event arrives. It fires if the event reports more than 4 requests per minute and a High_Request_Volume_Alert has not already been signaled. The threshold of 4 minutes was used for testing and might be different in a production version. Page 35 of 59

36 The Stop_Extended_Time_Alert and Stop_High_Request_Volume_Alert triggers are evaluated when any event arrives that indicates the end of the situation (Close, Stop, or Close_All). They fire if an Extended_Time_Alert or a High_Request_Volume_Alert has been raised for this situation and they reset the indicator metric. 6.5 KPIs for the Situation Monitor The model for the Situation Monitor also includes a KPI, which shows the total number of extended time situations. If it is high, indicating that many situations stay open for a long time, some general improvement of the service infrastructure is advised. If it is low, then most situations are resolved rather quickly and there is less reason to be concerned about a general degradation of service quality. The KPI definition is shown in Figure 7. The settings in the KPI Value and KPI Details sections show a metric on which the KPI is based (Extended_Time_Alert_On) and an aggregation function (sum). As previously stated, this metric is set to 1 when an extended time situation is detected, and reset to 0 when it ends. Therefore, adding up those metrics across all monitoring contexts gives results for the number of currently open extended time situations. Figure 7: KPI Counting Extended Time Situations Page 36 of 59

37 In the middle of Figure 7 there are definitions for three KPI ranges, Excellent [0... 5), Normal [ ) and Critical [ ). You also see that the KPI has a target value of 2. These settings govern the KPI visualization on the dashboard, and they can be used in trigger conditions that cause outbound events when those limits are crossed. 6.6 Example Dashboard Views of the Situation Monitor This completes the description of the monitoring context for ITCAM for SOA situations. You saw how it is instantiated when a situation occurrence is first reported, gets information about the occurrence from the Open event, and detects when the situation remains open for an extended amount of time or when there are excessive service requests. The following figures show several dashboard views of this monitor model, created by events from a test harness that simulates continuous requests on a number of monitored Web service operations. Figure 8: Example Monitoring Contexts for ITCAM for SOA Situations (instance view) Page 37 of 59

38 Figure 9: Extended Time Situation Count KPI 7 Operation Monitor The Situation Monitor model explained in the previous section was "situation-centric." An "operation-centric" point of view is now explained; you want to monitor a service operation by subscribing to events that are reporting any ITCAM for SOA situation occurring for this operation. You also want to use the ITCAM for SOA function that detects situations by service requester identity, and to monitor situations occurring for an operation for each requester identity separately. The Operation Monitor model defines a monitoring context for a service operation with nested (child) contexts for service requester identities. The child contexts subscribe to events reporting ITCAM for SOA situations that carry requester identity information. They gather information about all situations caused by a particular requester identity for a particular operation. Aggregate information (at the operation level) is then captured in the parent context. An overview of this model is shown in Figure 10. The complete model can be downloaded from [14]. Page 38 of 59

39 Figure 10: ITCAM for SOA Operation Monitor (model overview) This model has two nested monitoring context definitions (ITCAM for SOA Operation and ITCAM for SOA Requester Operation). The effect of this on the model logic and the inbound event subscriptions instantiating parent and child contexts are described in the following sections. 7.1 Event Subscriptions for the Operation Monitor The three keys defined for a parent monitoring context, Managed_System_Name, Service_Operation_Qualified_Name, and Service_Port_Qualified_Name, uniquely identify a service operation and where it is deployed. The key of a child monitoring context, Requester_Identity, uniquely identifies a requester identity. The properties of the inbound event subscriptions are shown in the following tables. Table 6: Event Subscription Filter Conditions for the Operation Monitor Id Description Extension name Filter condition Open [parent] Event signaling that the situation condition was ITM_Services_ Inventory_610 fn:exists(open/extendeddata/situation_displayitem) and udf:removequotes(open/extendeddata/situation_displayitem) ne '' Page 39 of 59

40 detected. and Open/extendedData/service_type = 1 and Open/extendedData/interval_status = 2 and udf:removequotes(open/extendeddata/situation_status) = 'Y' Requester _Open [parent] Event signaling that the situation condition was detected based on service requests originating from a particular requester identity ITM_Services_ Inventory_ReqID_ 610 fn:exists(requester_open/extendeddata/situation_displayitem) and udf:removequotes(requester_open/extendeddata/ situation_displayitem) ne '' and Requester_Open/extendedData/service_type = 1 and Requester_Open/extendedData/interval_status = 2 and udf:removequotes(requester_open/extendeddata/situation_status) = 'Y' Requester _Open [child] Event signaling that the situation condition was detected based on service requests originating from a particular requester identity ITM_Services_ Inventory_ReqID_ 610 fn:exists(requester_open/extendeddata/situation_displayitem) and udf:removequotes(requester_open/extendeddata/ situation_displayitem) ne '' and Requester_Open/extendedData/service_type = 1 and Requester_Open/extendedData/interval_status = 2 and udf:removequotes(requester_open/extendeddata/situation_status) = 'Y' Table 7: Correlation Predicates for Operation Monitor Id Correlation predicate Zero matches One match Multiple matches Open [parent] udf:removequotes(open/extendeddata/situation_origin) = Managed_System_Name and fn:concat('{', create new instance deliver to the instance treat as error udf:removequotes(open/extendeddata/operation_namespace_u), '}', udf:removequotes(open/extendeddata/operation_name_u)) = Service_Operation_Qualified_Name and fn:concat('{', udf:removequotes(open/extendeddata/port_namespace_u), '}', udf:removequotes(open/extendeddata/service_name_u)) = Service_Port_Qualified_Name Requester _Open [parent] udf:removequotes(requester_open/extendeddata/situation_origin) = Managed_System_Name and fn:concat('{',udf:removequotes( create new instance deliver to the instance treat as error Requester_Open/extendedData/operation_namespace_u),'}', udf:removequotes(requester_open/extendeddata/operation_name_u)) = Service_Operation_Qualified_Name and Page 40 of 59

41 fn:concat('{', udf:removequotes( Requester_Open/extendedData/port_namespace_u), '}', udf:removequotes(requester_open/extendeddata/service_name_u)) = Service_Port_Qualified_Name Requester _Open [child] udf:removequotes(requester_open/extendeddata/situation_origin) =../Managed_System_Name and fn:concat('{',udf:removequotes( Requester_Open/extendedData/operation_namespace_u), '}', udf:removequotes(requester_open/extendeddata/operation_name_u)) =../Service_Operation_Qualified_Name and fn:concat('{', udf:removequotes( Requester_Open/extendedData/port_namespace_u), '}', udf:removequotes(requester_open/extendeddata/service_name_u)) =../Service_Port_Qualified_Name and udf:removequotes(requester_open/extendeddata/requester_identity_u) = Requester_Identity create new instance deliver to the instance treat as error The Requester_Open events are received in both the parent and child contexts. WebSphere Business Monitor delivers events that are received in a hierarchy of monitoring contexts to the higher level contexts first, so that an instance can be created and initialized before a child context can be created. This is important so that a new child context can be associated with the correct parent. Open events without a requester identification are also received, but only in the parent context. This was done to track the full tally of operations for which ITCAM for SOA situations occur. Much less information is captured in this case, however. The filter conditions for the three subscriptions are the same as for the Open event subscription described in section 6 "Situation Monitor" on page 27. The three explicit filter conditions are identical, and the Requester_Open events are distinguished by their extension name, which is implicitly tested as part of event filtering. Correlation settings are shown in the second table. All event subscriptions match the three key metrics of the parent context with the corresponding event attributes, so that situation events are always delivered into the monitoring context hierarchy for the service operation where the ITCAM for SOA situation was detected. Note the abbreviated reverse steps '../' used in the right hand side expressions to reference parent context metrics. The child context subscription for Requester_Open events has an additional correlation condition, comparing the requester identity of the event with the child context key. Thus, these events are correlated with the child context for the correct requester. Requester_Open events are processed when an event arrives that matches the filter and extension name of the Requester_Open subscription. One of the following conditions applies when the event is processed: 1. A parent context for the service operation does not exist Page 41 of 59

42 2. A parent context for the service operation exists, but no child context exists for the requester 3. Both a parent context for the service operation and a child context for the requester exist In the first case, the parent context subscription is processed first, and its zero-matches setting instructs WebSphere Business Monitor to create a new instance and populate its metrics from the event. The child context subscription is processed next. Under the parent just created, WebSphere Business Monitor finds no child context for the requester identity and the zero-matches setting for the child subscription causes the creation of a new instance. The monitoring context identified by the first part of the correlation predicate (comparing the parent context keys) becomes its parent. The event is then delivered to the new child context. The processing sequence for cases 2 and 3 simply skips the parent context creation or the child context creation, as appropriate Triggers and Counters for the Operation Monitor There are various triggers and counters in this model, which are summarized in the following two tables. Table 8: Triggers for the Operation Monitor Trigger id Evaluated when Repeating? Gating condition Requester_Critical_Situation_ Exceeded_Trigger [parent] Total Critical Situations counter updated no Total_Critical_Situations > 2 Critical_Situation_Trigger [child] Harmless_Situation_Trigger [child] Warning_Situation_Trigger [child] New_Requester [child] Valid_Average_Message_Length_Trigger [child] Requester_Open event arrives Requester_Open event arrives Requester_Open event arrives Requester_Open event arrives Requester_Open event arrives yes Requester_Open/predefinedData/severity = 50 yes Requester_Open/predefinedData/severity = 20 yes Requester_Open/predefinedData/severity = 30 no yes Requester_Open/extendedData/avg_msg_length > 0 Valid_Average_Response_Time_Trigger Requester_Open yes Requester_Open/extendedData/avg_elapsed_time 2 Nested monitoring context definitions are similar to Java inner classes [15]. The instances of the nested (class or context) definitions form a tree structure at runtime. Page 42 of 59

43 [child] event arrives > 0 Table 9: Counters for the Operation Monitor Counter id Trigger / event Resulting action Requesters_Causing_Situations [parent] Total_Critical_Situations [parent] Total_Situations [parent] New_Requester [child] Critical_Situation_Trigger [child] Open [parent] Requester_Open [parent] add one add one add one add one Critical_Situations [child] Harmless_Situations [child] Warning_Situations [child] Events_Processed [child] Events_Processed_for_Valid_Average_Message_Length [child] Events_Processed_for_Valid_Average_Response_Time [child] Critical_Situation_Trigger [child] Harmless_Situation_Trigger [child] Warning_Situation_Trigger [child] Requester_Open [child] Requester_Open [child] Requester_Open [child] add one add one add one add one add one add one The Critical, Harmless, and Warning situation triggers in a child context signal that a situation event with a certain severity arrived, and increment a corresponding counter. The New_Requester trigger, also defined for child contexts, is evaluated when Requester_Open events arrive. It has no gating condition, but also is non-repeating. A repeating trigger fires each time its gating condition evaluates to true. A non-repeating trigger fires when its gating condition changes from false to true. After it fires, it can only fire again after the gating condition becomes false and then turns true again. As a result, the New_Requester trigger in each child context fires only once, when the first Requester_Open event arrives and creates this context. It increments the Requesters_Causing_Situations counter in the parent context, which counts the number of requester identities causing situations for the service operation. The Valid_Average_Message_Length_Trigger and Valid_Average_Response_Time_Trigger triggers (also in a child context), fire when a Requester_Open event is received that report either an average message length or Page 43 of 59

44 average response time that is greater than 0. When the Valid_Average_Message_Length_Trigger trigger fires, the Events_Processed_for_Valid_Average_Message_Length counter is incremented by 1. When the Valid_Average_Response_Time_Trigger trigger fires, the Events_Processed_for_Valid_Average_Message_Length counter is incremented by 1. These triggers and counters are necessary to calculate accurate averages for message length and response time, given that not every event reports these measurements. There are three remaining counters: Total_Critical_Situations [parent] is incremented by a critical situation trigger firing in any child context. It thus shows the total number of critical situations caused by any requester identity for the monitored service operation. Its value is equal to the sum of the critical situation counters in all child contexts. 3 Thus, the trigger (or event) in a child context can have effects in a parent context. The reverse flow (events or triggers defined for a parent context having effects in subordinate contexts) is not permitted in order to prevent uncontrolled control fanout. Depending on the number of child contexts at runtime, a trigger firing in the parent context could cause updates to a huge number of counters (or stopwatches, or metrics, or outbound events,) in subordinate monitoring contexts; each of these could again cause updates in multiple subordinate contexts, and so on, with unpredictable performance implications. Total_Situations [parent] is incremented by Open or Requester_Open events arriving in that context, and counts the total number of ITCAM for SOA situations reported for the service operation that this context monitors (regardless of situation type, or requester identity). This example shows that a counter can be controlled by more than one event or trigger. Events_Processed [child] counts the number of Open events received for this monitoring context. The Requester_Critical_Situation_Exceeded_Trigger is evaluated each time the Total_Critical_Situations counter is updated, and fires when it exceeds a threshold (set to 2 in this model, for testing purposes). The trigger causes the emission of the outbound Critical_Situation_Exceeded_Alert, which can be received by the action manager of WebSphere Business Monitor to trigger notifications or some other automated response to an excessive number of critical situations. 3 Strictly speaking, you would have to add a trigger that fires for critical situations reported by way of Open events in the parent context, to make sure we capture them all. This was left out for simplicity. As a result, the Total_Critical_Situations counter shows only the number of critical situations caused by a known requester. Page 44 of 59

45 7.3 Metrics for the Operation Monitor The parent context metrics in this model are initialized from attributes of an Open or Requester_Open event; their maps are similar to what is in section 6 "Situation Monitor" on page 27. These metrics hold the characteristics of the monitored service operation. Similarly, the Requester_Identity and Requester_Name metrics in the child monitoring context store characteristics of the service requester identity. The remaining six child metrics are shown in the following table. They contain aggregate information about message length and response time, across all ITCAM for SOA situations reported for a given requester identity and service operation. Table 10: Metrics for the Operation Monitor Metric id [type] Map expression Average_Message_Length [xs:decimal] Average_Message_Length_aux [xs:integer] Average_Response_Time [xs:decimal] Average_Response_Time_aux [xs:integer] Maximum_Message_Length [xs:integer] Maximum_Response_Time [xs:integer] Average_Message_Length_aux div Events_Processed_for_Valid_Average_Message_Length Average_Message_Length_aux + Requester_Open/extendedData/avg_msg_length Average_Response_Time_aux div Events_Processed_for_Valid_Average_Response_Time Average_Response_Time_aux + Requester_Open/extendedData/avg_elapsed_time if (Requester_Open/extendedData/max_msg_length > Maximum_Message_Length) then Requester_Open/extendedData/max_msg_length else Maximum_Message_Length if (Requester_Open/extendedData/max_elapsed_time > Maximum_Response_Time) then Requester_Open/extendedData/max_elapsed_time else Maximum_Response_Time Because the average message length and average response time attributes in an event can be set to 0 or -1, the Valid_Average_Message_Length_Trigger and Valid_Average_Response_Time_Trigger triggers are used to ensure that the average metrics in this monitoring context are only updated if the event contains average values greater than 0. The Average_Message_Length_aux and Average_Response_Time_aux metrics add up the message length and elapsed time values reported by Requester_Open events with a valid average value. The Average_Message_Length and Average_Response_Time metrics are calculated by dividing these sums by the number of events received with a valid, average value. The Maximum_Message_Length and Maximum_Response_Time metrics have conditional maps, which update these metrics when the inbound event reports a value that exceeds the maximum reported thus far. Page 45 of 59

46 7.4 Dimensional Model for the Operation Monitor You can use the WebSphere Business Monitor dashboard to perform dimensional analysis of metric values. The Operation Monitor includes a dimensional model that defines both measures and dimensions. Measures are calculated by aggregating metrics, and five types of aggregation are supported: Minimum, Maximum, Sum, Average, and Count. Dimensions define the domain of an aggregation (that is, the monitoring context instances from which values are aggregated). The dimensional model for the Operation Monitor is shown in Figure 11. Figure 11: Dimensional Model for the Operation Monitor Two cubes are defined, one for each monitoring context definition. For the Operation_Cube, which aggregates metrics across parent monitoring context instances, you see its Measures and Dimensions definitions. For example, the Most Critical Situations measure is defined as the maximum of the Total Critical Situations counters in all ITCAM for SOA Operation monitoring contexts. In other words, it shows the highest critical situation count of any operation being surveyed. Dimensions are used to restrict the domain of an aggregation. For example, the Server dimension, which comprises a Host and a System level, can be used to look at the maximum critical situation count for each server. Page 46 of 59

47 7.5 Example Dashboard Views of the Operation Monitor The monitoring contexts in this model have no termination logic, and thus exist indefinitely. In a real production scenario, termination logic similar to what is in section 6 "Situation Monitor" on page 27 could be added, using terminating events and timeouts or both (for example, terminating a context that has not received an event for more than 30 days). The following figures show a dashboard view of this monitor model, based on events from a test harness that simulates continuous requests on a number of monitored Web service operations. Figure 12: Parent and Child Monitoring Contexts for ITCAM for SOA Operation Model (instance view) Users can drill down to child monitoring contexts by clicking the Drill Down arrow in a parent context instance view, and navigate back to the parent by clicking the Drill Up button in a child context instance view. The remaining figures in this section show several dashboard views leveraging the dimensional model, as well as the alerts view, which holds a record of outbound events reporting high counts of critical situations. The three dimensional views leverage the three defined dimensions of Server, Service Operation, and Requester. Page 47 of 59

48 Figure 13: Server Dimension View for the Parent Monitoring Context Figure 14: Service Operation Dimension View for the Parent Monitoring Context Page 48 of 59

49 Figure 15: Service Requester Dimension View for the Child Monitoring Context Figure 16: Operation Monitor Alerts View Page 49 of 59

50 8 Test Monitor The purpose of this model is simple event validation: testing that a CBE produced by the EIF to WS-Notification Converter (introduced in 3.5 "Installing and Configuring the EIF to WS-Notification Converter") can be received in a monitoring context, and that all event attributes can be extracted and written to metrics. Only events for the ITM_Services_Inventory_610 event class are processed by this model. An overview is shown in Figure 17. The complete model can be downloaded from [14]. Figure 17: Test Monitor Model for ITCAM for SOA Situation Events The single inbound event definition (Event) has no filter, and specifies extension name of ITM_Services_Inventory_610. Therefore, it subscribes to any CBE having this extension name, which includes the Open, Close, and Stop events that are presented in section 6 " Page 50 of 59

51 Situation Monitor" on page 27. Its correlation predicate should always evaluate to false, and the zero/one/multiple correlation settings are create new instance/treat as error/treat as error. As a result, any ITM_Services_Inventory_610 event received instantiates a new monitoring context, and copies its attributes to the 58 metrics. The maps are just copying event attributes to metrics and applying the quotation mark removal function to any string values. A dashboard view for this model is shown in Figure 18. Figure 18: Monitoring Contexts for ITCAM for SOA Test Monitor (instance view) Page 51 of 59