Antonio Sgro, Agent Architect for SAPM John Alvord, L2 Support IBM Tivoli Monitoring

Transcription

1 ITM v6.x Situations November,2012 Antonio Sgro, Agent Architect for SAPM John Alvord, L2 Support

2 Introduction and Agenda Situation Architecture Situation Definition Situation Best Practices Q&A 2

3 Situation Architecture Situation Objects Situations are monitoring entities that are launched from the Situation Monitor/Automated Facilities task within the TEMS Server runtime environment. Situations are used to return results for monitoring (normal case) Situations are used to control flow in Policies Situations are used to drive collectors for History and Reports TEP workspace views are driven by Queries Situation Definitions are kept in the table called TSITDESC 3

4 Situation Architecture (continued) Situation Objects (continued) Important Column Information SITNAME Name of the situation object PDT Predicate formula of the situation SITINFO Special attributes for specific situation processing Display Item (Atomize) ATOM=attribute_name Event persistence counter COUNT=n Severity for TEC forwarding AUTOSTART Automatically start situation at startup if value is *YES or *SYN HUB Events for this situation are propagated to clients if this value is Y REEV_DAYS and REEV_TIME Combined, these columns form a monitoring evaluation interval for this situation AFFINITIES Application affinity to match situation record with eligible application distribution node list records 4

5 Situation Architecture (continued) Situation Objects (continued) AUTOSOPTS Options for Reflex Automation Indicator to run at Server versus Agent Indicator for forced reflex emulation Indicator to execute action each true event versus first true event only after state change. CMD Command to execute when this situation is true Abides rules as set by AUTOSOPTS 5

6 Situation Architecture Situation Objects (Continued) Situation Predicates Formula that uses attributes, functions, and other situations as operands along with threshold values to describe a filtering criteria. Similar to WHERE Clause SQL Syntax Example: *IF *VALUE Local_Time.Seconds *GE 30 *AND *VALUE Local_Time.DayOfWeek *EQ 03 Operand Type Identifiers *VALUE indicates following operand is an attribute name *SIT indicates following operand is an embedded situation *MISSING indicates following operand is an attribute name that will cause the condition to evaluate to TRUE if is not in the data list. *COUNT, *MIN, *MAX, *STR, *SCAN, *AVG, *TIME *IN: a list of possible values; equivalent to *VALUE *EQ xxx Operator Values *EQ,*NE,*GE, *GT, *LT, *LE Until Clause: *UNTIL indicates following operand is a situation name or TTL that resets the situation when true. TTL should be used only for pure events. *UNTIL only for sampled events. 6

7 Situation Architecture (continued) Situation Objects (Continued) Attributes Attributes used in predicates have characteristics that define how a situation is to be evaluated and filtered. Attributes are created by agents as part of their processing. Polled Attributes Attributes that return a value when queried (Metrics) Pure Event Attributes Attributes that return values upon the occurrence of an event Combination of predicate attribute event types assume behavior of polled versus pure event based on the encountered event type of the leftmost attribute in the situation formula. It is suggested to not be used since it could cause confusion on the situation logic Cost Attributes cost value determines which set of other attributes, within the same attribute table group, will be returned in response to the attributes contained within the situation formula. 7

8 Situations Definition Situations are conditions that you want to monitor on managed systems Each situation contains a name, a predicate formula, special attributes for specific situation processing, information on whether the situation is automatically started or not, and the sampling interval It also may contain a command to execute when the situation is true, and advice to give the client when an alert for the situation A situation predicate is a formula that uses attributes, functions, and other situations as operands along with threshold values to describe a filtering criterion. The situation formula is made up of one or more logical expressions. Each expression takes the form: [Attribute name] / [logical operator] / [value] 8

9 Situations Definition The situation predicates are similar to a WHERE clause in SQL. In, predicates are processed sequentially, and perform better if you put the most restrictive condition as the first predicate. You can use the situation editor to create/edit situations by choosing attributes, logical operator, value, and sampling interval, etc. Situations are assigned to run on behalf of managed systems or lists of managed systems 9

10 Situations Definition The Tivoli Enterprise Monitoring Server acts as a collection and control point for alerts received from the monitoring agents, and collects performance and availability data from them. The hub Tivoli Enterprise Monitoring Server correlates the monitoring data collected by monitoring agents and remote Tivoli Enterprise Monitoring Server and passes it to the Tivoli Enterprise Portal Server for presentation on Tivoli Enterprise Portal. Tivoli monitoring agents are installed on the systems or subsystems that the applications or resources you want to monitor. If a situation is running on that system, the agent collects the current values of the attributes specified in the formula, and tests the values against a threshold. When a condition is not met a zero row is returned and then nothing at each sampling interval When a condition is met, the result rows are returned every sampling interval forever. The TEMS decides when to create an event based on many aspects, like persistence- UNTIL sits etc. 10

11 Situations Definition However, some of the SQL features that cannot be tested at the agent level are: Only *IN, *MISSING and *STR group functions are evaluated at agent level whole others like MIN, MAX, AVG, SUM, COUNT, *CHANGE are not. Correlated sits only run at the hub TEMS and depend on result data in the TSITSTSH table If those are being used in the situation definition, the agent returns all rows back to the Tivoli Enterprise Monitoring Server that it is directly connected to, and the server performs the evaluation. The agent returns all the data rows to Tivoli Enterprise Monitoring Server for correlated and embedded situations as well. In large scale environments, especially if the sampling interval for the situation is short, that dramatically increases the workload on the Tivoli Enterprise Monitoring Server, and also increases the memory usage on the Tivoli Enterprise Monitoring Server that the agents are directly connected to. 11

12 Situations Guidelines Since the performance of the Tivoli Enterprise Monitoring Server is greatly affected by the volume and frequency of state changes, we do not run the situations and collect data unless we are interested in the results. Situation Guidelines Try to avoid using situations that have group functions. Since agents cannot process group functions themselves, all rows are returned to the Tivoli Enterprise Monitoring Server (or remote Tivoli Enterprise Monitoring Server) they report to, this can require large amounts of memory at the Tivoli Enterprise Monitoring Server. You may need to reduce the row count. Sometimes you cannot. Other choices increase sampling interval. Note: If group functions are required, you need to reduce the row count. 12

13 Situations Definition Put the most stringent condition at the beginning of the formula when creating a new situation because, in, the formula is evaluated sequentially. Put mission critical systems in a separated managed system list. If a heavy workload situation is required, you can limit the distribution of the situation to the managed system list containing the mission critical systems. Avoid distributing situations with short sampling intervals to a large scale environment. With a modern processor, it is difficult, but still possible to raise the CPU utilization to 100% through situation processing. The actual values will vary based on the particular processors, but given a long enough situation list and enough rows of data, the time required to process the situations can take longer than the interval between samples. Leading to a CPU pegged at 100%, and lost data. Avoid distributing too many situations since they could never complete in the needed sampling interval. Increase sampling interval in case you will need them. 13

14 Situations Definition Minimize the number of situations on attribute groups that typically return multiple rows like the process attribute group and disk attribute group. In database environments customers have reported problems when long lists of shared disk mounts have had to be processed. For example: 500 database machines each mounting 60 shared disks. Consider spreading business critical situations amongst several different remote Tivoli Enterprise Monitoring Servers. For instance a DB monitoring situation might be high load, but business critical. Instead of having all 500 DB machines report through the same remote Tivoli Enterprise Monitoring Server, consider configuring 100 DB across five remote Tivoli Enterprise Monitoring Servers (that include a mix of lower demand OS agents). Use the group function workaround where possible, if group functions are required. 14

15 Situations Definition If group functions are being used in the situation definition, the agent returns all rows back to the Tivoli Enterprise Monitoring Server that it is directly connected to, and the TEMS performs the testing. In large scale environments, this can dramatically increase the memory usage and CPU workload on the Tivoli Enterprise Monitoring Server. Group functions However, there are certain "special" attributes that the TEMS can pass to the agent if group functions are involved which will allow the agent to reduce the amount of data it returns to the TEMS. We will call these the indexed attributes. Although not technically indexed in the full SQL definition, it is the closest metaphor without creating new terminology. This section describes how to find indexed attributes and how to modify existing situations to take advantage of their properties. 15

16 Situations Definition As a last caveat, this is a limited workaround. There may not always be a suitable indexed attribute, but when there is, this can be one of the most effective methods for reducing the memory at the RTEMS. Analyze existing situations. Check: It will provide a list of situations that can benefit from the workaround and a list of indexed attributes to enable this workaround. 16

17 Situations Examples Add an indexed attribute from the list provided by the analyzesituations.pl script. Make it the first (leftmost) attribute evaluated. Move it to leftmost column if necessary. Example #1: Using an indexed attribute Linux_Too_Many_Http_Servers From *IF *VALUE Linux_Process.Proc_CMD_Line *EQ '/usr/sbin/httpd' *AND *COUNT Linux_Process.Process_ID *GT 10 to *IF *VALUE Linux_Process.Process_Command_Name *EQ '/usr/sbin/httpd' *AND *COUNT Linux_Process.Process_ID *GT 10 In this example the Process_Command_Name is synonymous with Proc_CMD_Line, however since Process_Command_Name is indexed, the situation can be processed at the agent level. 17

18 Situations Examples Example 1: Using an indexed attribute to create a subset Linux_System_MsgBus_Down From: *IF *VALUE Linux_Process.Proc_CMD_Line *EQ 'dbus-daemon-1 --system' *AND *COUNT Linux_Process.Proc_CMD_Line *NE 1 TO: *IF *VALUE Linux_Process.Process_Command_Name *EQ 'dbus-daemon-1' *AND *VALUE Linux_Process.Proc_CMD_Line *EQ 'dbus-daemon-1 --system' *AND *COUNT Linux_Process.Proc_CMD_Line *NE 1 Because the first version uses a group function, all the process rows are returned by the agent. By adding the indexed Process_Command_Name, only message bus processes are returned. However, using only Process_Command_Name is not a perfect filter, the argument list "--system" provided by Proc_CMD_Line is still needed. 18

19 Situations Best Practices Additional rules specific to the group function COUNT Rule 1: Note that when using COUNT in an AND expression, the COUNT must be the last function in the expression Rule 2: The attribute used for the count does matter. The attribute used in the COUNT cannot be the same attribute as used in the VALUE. Rule 3: Only evaluate one COUNT in an expression. Move complicated COUNT expressions to the warehouse. These might then be processed by a full SQL implementation. Example #2: Attribute used in the COUNT cannot be as used in the VALUE Linux_Too_Many_Http_Servers From *IF *VALUE Linux_Process.Process_Command_Name *EQ '/usr/sbin/httpd' *AND *COUNT Linux_Process.Process_Command_Name *GT 10 To *IF *VALUE Linux_Process.Process_Command_Name *EQ '/usr/sbin/httpd' *AND *COUNT Linux_Process.Process_ID *GT 10 Often there is a simpler and more efficient means to get the same answer. Consider looking at "summary" attribute groups instead of COUNT(), AVG(), and SUM(). 19

20 Situations Examples Example #3: Linux_Too_Many_Logins From *IF *COUNT Linux_User_Login.Login_PID *GT 32 To *IF *VALUE Linux_System_Statistics.Number_of_Users *GT 32 20

21 Situations Best Practices WildCards and Missing Using wildcard characters (*) and MISSING function in situations is one of the most common source of high impact on CPU usage related to situations evaluation. Situation with this simple pattern 'LIKE "*/process"' can be re-written as 'SCAN "/process"' Put the most stringent condition at the beginning of the formula when creating a new situation because, in, the formula is evaluated sequentially ( #UNIXPS.UCOMMAND == '/opt/ibm/itm/*' AND #UNIXPS.CPUPERCENT > 75.00) can be rewritten as: ( #UNIXPS.CPUPERCENT > AND SCAN(#UNIXPS.UCOMMAND) == '/opt/ibm/itm' ) ( MISSING(#UNIXPS.UCOMMAND) == ( '/opt/ibm/itm/*kux_vmstat' )) can be rewritten as: ( STR(#UNIXPS.UCOMMAND) == 1,'/opt/IBM/ITM/' AND MISSING(#UNIXPS.UCOMMAND) == ( '*kux_vmstat*' )) 21

22 Situations Best Practices WildCards and Missing ( MISSING (#UNIXPS.UCOMMAND) == ('/usr/*websphere/appserver/java/bin/java*was.status.socket*')) can be rewritten as: ( STR(#UNIXPS.UCOMMAND) == 1,'/usr/' AND SCAN(#UNIXPS.UCOMMAND) == 'WebSphere/AppServer/java/bin/java AND MISSING(#UNIXPS.UCOMMAND) == ( '*was.status.socket*' )) Try to use Process Filter instead of wildcards since it will provide 99% less CPU time: 22

23 Situations Best Practices WildCards and Missing A complex situation: ux_system_chm_proc_c1 *IF *MISSING Process.Process_Command *EQ ('ora_vmon_wmssldpr','ora_pmon_wmssldpr','ora_smon_wmssldpr','ora_arc0_wmssndp R','ora_pmon_WMSSNDPR','ora_smon_WMSSNDPR','*qmaster_ll/bin/qme','*qmaster_ll/bin/jme', '*qmaster_sl/bin/qme','*qmaster_sl/bin/jme','svrrun*4604','svrrun*9985','svrrun*4605','*webmeth ods65/sldc_prod/jvm*','*webmethods65/sndc_prod/jvm*' ) LIKE pattern matching. is a character by character NLS routine call, rather than a single NLS routine call for the entire pattern/string. Possibly add trailing '*' to these patterns, this will create single NLS SCAN call. May add blank(s) before trailing asterisk if "must end" e.g. '.../qme *' '*qmaster_ll/bin/qme' change to '*qmaster_ll/bin/qme*' '*qmaster_ll/bin/jme' change to '*qmaster_ll/bin/jme*' 23

24 Situations Best Practices WildCards and Missing A complex situation: ux_sys_chm_proc_c1 *IF *MISSING Process.Process_Command *EQ ('ora_arc0_wmssldpr','ora_pmon_wmssldpr','ora_smon_wmssldpr','ora_arc0_wmssndp R','ora_pmon_WMSSNDPR','ora_smon_WMSSNDPR','*qmaster_ll/bin/qme','*qmaster_ll/bin/jme', '*qmaster_sl/bin/qme','*qmaster_sl/bin/jme','svrrun*4604','svrrun*9985','svrrun*4605','*webmeth ods65/sldc_prod/jvm*','*webmethods65/sndc_prod/jvm*' ) Possibly create a separate situation. create a separate *SCAN or *SUBSTR for prefix part and *MISSING with the trailing part of the Missing Like. 'SVRrun*4604' add scan and change missing to (*SCAN EQ 'SVRrun' *AND *MISSING ('*4604*', '*4605*', '*9985*')) 'SVRrun*9985' move to above 'SVRrun*4605' move to above 24

25 Situations Best Practices Useful Tools for situations analysis: ITMSuper: KG=1TW10TM6L&lang=en_US&cp=UTF-8 25