IBM Tivoli Software Configuring Netcool/Impact Event Correlation to resolve a Netcool/OMNIbus Events Flood Document version 1.0 Yasser Abduallah
Copyright International Business Machines Corporation 2014. US Government Users Restricted Rights Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
1 Netcool/Impact Overview IBM Tivoli Netcool/Impact is a highly scalable event processing engine that provides a common platform for ubiquitous data access that easily circumvents traditional organizational boundaries. Leveraging this data, IT operations staff can correlate, calculate, enrich, deliver, notify, escalate, visualize and perform a wide range of automated actions. The advanced policy-engine provided by Netcool/Impact allows IT operations staff to add business context to IT events, thereby making it easier to perform essential functions such as event prioritization, business impact analysis and measurement of key performance indicators. The Operator View allows user to create Web-based views consolidate multiple data sources and provide a single point from which to view distributed data. Netcool/Impact is comprised of a set of runnable server components that work together to provide event management and integration functionality for the Netcool suite of products. From an implementation perspective, you can understand Netcool/Impact as a development tool that you use to customize, enhance, and expand the functionality of an existing Netcool installation. It is a platform that you can use to build new functions into your current installation of the Netcool product suite. For more information about Netcool/Impact, see: http://www.ibm.com/developerworks/servicemanagement/bsm/tni/index.html
2 Scenario In this scenario, you learn how to use the Netcool/Impact policy and the OMNIbus ObjectServer event reader service to perform event correlation to solve events flood. In Netcool/Impact you create an OMNIbus event reader service based on a specific filter that will execute the correlation policy. The policy queries the OMNIbus ObjectServer again based on the same filter as the reader or a different one to check if there are older events within a threshold and how many they are. The scenario performs a simple X in Y correlation example. Where X is the number of events that occurred in a specified time window threshold Y, ie 50 events in the past 120 seconds. This specific scenario is focuses on an ITM TEMS Server that sends a flood of events tagged as MS_Offline. MS_Offline events are sent when the TEMS agents detect that servers are down or restarted. For example, if ITM TEMS sends 3 events per second per agent for 5 agents until the agents are responsive, it would result in: 3 events * 5 * (5*60 seconds) = 4500 events in 5 minutes. Because the 4500 events are coming from the same source, they should be correlated by either updating the new incoming event or deleting them. In this example, the events are updated. ITM TEMS sends events to the OMNIbus ObjectServer table with updated fields such as: Summary Like 'MS_Offline', ITMHostname='TEMS hostname', Agent = 'ITM' The fields are used to query the ObjectServer 2.1 Scenario Assumptions This particular scenario is using a standard Netcool/Impact policy and an OMNIbus ObjectServer Event Reader service for Version 5.x and up. NOTE: The screenshots and links are taken from the Impact 6.1.1 Server
3 Steps 1. Download the correlation policy file from the Scenarios and Examples page on the Netcool/Impact developerworks wiki at: https://www.ibm.com/developerworks/mydeveloperworks/wikis/home? lang=en#/wiki/tivoli Netcool Impact/page/Netcool Impact 6.1.1 2. The policy name is: EventCorrelationUsingXinYExample.ipl 3. Import the policy to the Impact Server: a. Login to Netcool/Impact Server: https://<host>:<port>/ibm/console, the default port is 16311. b. Navigate to System Configuration Event Automation Policies. c. You can optionally select an ITM project and load the policy to it. d. To select the file, click Upload a Policy file. e. Select EventCorrelationUsingXinYExample.ipl and click Upload. Figure 1 - Loading a Policy File to Impact Server
4. Update the policy to match a specific filter. Each section of the policy has a description. Also note that the policy uses @ITMHostname for ITM TEMS because this example is specifically for MS_Offline event floods. Make sure to update the filter accordingly. Policy Configuration: a. Setting the threshold (Y): /*Threadshold time window in seconds:*/ CorrelationThreshold =120; Log("LastOccurrence : " + @LastOccurrence ); /** *DiffTime can be calculated using DiffTime = GetDate() - Int(CorrelationThreshold) *Using GetDate() instead of @LastOccurrence makes sure that the policy checks period of time from "now time" - Threadshold *which keeps the time constant to check instead of using relative timestamp value from LastOccurrence */ //DiffTime=@LastOccurrence - Int(CorrelationThreshold); DiffTime=GetDate() - Int(CorrelationThreshold); Log("DiffTime: " + DiffTime); This is the threshold (time window) b. Filter: /*The following filter is used to correlate the events. It can be changed as needed *This specific example is to filter events to handle ITM MS_OFFLine events flood */ CorrelationFilter="ITMHostname='" + @ITMHostname + "' AND Summary Like 'MS_Offline' AND Severity = 5 AND Serial!= " + @Serial ; CorrelationFilter = CorrelationFilter + " AND LastOccurrence <= " + DiffTime ; /*ORDER BY can be used to rank the events and check which one came in first */ CorrelationOrderBy = "ORDER BY LastOccurrence ASC"; c. Number of Events (X) /*The following is to get COUNT(*) as EventCount from the same object server data source used by the event reader */ CorrelationFields="COUNT(*) AS EventCount"; /*form the correlation query including the threadshold filter*/ SQLQuery = "SELECT " + CorrelationFields + " FROM status WHERE " + CorrelationFilter ;
Log("Reader Policy Query: " + SQLQuery); Log("Check older events..."); Nodes=DirectSQL('defaultobjectserver',SQLQuery,NULL); Log("Number of Old Events: " + Num + " Nodes: " + Nodes); /*The following if condition checks if there is an X events occurred in the threshold * default is 5 events older than the incoming event that was picked up by the reader. * If there are older events, the incoming event will be correlated by updated the Severity * and SuppressEscl *in this example X is set to 5 */ if (Nodes[0].EventCount > 5) { Log("Found older events correlating this event: " + @Serial); @Severity=2; @SuppressEscl=6; //event can be deleted if the following is un-commented: //@DeleteEvent=true; ReturnEvent(EventContainer); } else { Log("No older events found..."); } 5. Create an Object Server Event Reader that executes the EventCorrelationUsingXinYExample policy. Refer to the Netcool/Impact documentation for more information on how to create an OMNIbus ObjectServer event reader service. http://publib.boulder.ibm.com/infocenter/tivihelp/v8r1/topic/com.ibm.netcoolimpact.doc _6.1.1.1/common/dita/event_readers_c.html For example, the following filter is used in the Event Mapping: Summary Like 'MS_Offline' AND Severity = 5 AND ITMHostname <> '' AND Agent ='ITM' 6. Run the Event Reader service and send some test events.
4 How the Example Works When the Event Reader finds a matching event, it executes the correlation policy. The policy queries the same ObjectServer using the same filter (or different, based on the configuration) and adds a threshold and time window (Y) as well as number of events found. If The number of events found in the threshold is greater than the count required (X), the incoming event will be correlated by updating the Severity and the SupressEscl fields. Another action can be done if the criteria is met such as deleting the event or update other fields.
REFERENCES IBM Tivoli Netcool/Impact 6.1.1 Infocenter: http://publib.boulder.ibm.com/infocenter/tivihelp/v8r1/topic/com.ibm.netcoolimpact.doc6.1.1/w elcome.html
Copyright IBM Corporation 2014 IBM United States of America Produced in the United States of America US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PAPER AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON- INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes may be made periodically to the information herein; these changes may be incorporated in subsequent versions of the paper. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this paper at any time without notice. Any references in this document to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation 4205 South Miami Boulevard Research Triangle Park, NC 27709 U.S.A. All statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. This information is for planning purposes only. The information herein is subject to change before the products described become available. If you are viewing this information softcopy, the photographs and color illustrations may not appear.
Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol ( or ), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the web at "Copyright and trademark information" at http://www.ibm.com/legal/copytrade.shtml.