Nimsoft Monitor. logmon Guide. v3.2 series

Transcription

1 Nimsoft Monitor logmon Guide v3.2 series

2 Legal Notices Copyright 2013, CA. All rights reserved. Warranty The material contained in this document is provided "as is," and is subject to being changed, without notice, in future editions. Further, to the maximum extent permitted by applicable law, Nimsoft LLC disclaims all warranties, either express or implied, with regard to this manual and any information contained herein, including but not limited to the implied warranties of merchantability and fitness for a particular purpose. Nimsoft LLC shall not be liable for errors or for incidental or consequential damages in connection with the furnishing, use, or performance of this document or of any information contained herein. Should Nimsoft LLC and the user have a separate written agreement with warranty terms covering the material in this document that conflict with these terms, the warranty terms in the separate agreement shall control. Technology Licenses The hardware and/or software described in this document are furnished under a license and may be used or copied only in accordance with the terms of such license. No part of this manual may be reproduced in any form or by any means (including electronic storage and retrieval or translation into a foreign language) without prior agreement and written consent from Nimsoft LLC as governed by United States and international copyright laws. Restricted Rights Legend If software is for use in the performance of a U.S. Government prime contract or subcontract, Software is delivered and licensed as "Commercial computer software" as defined in DFAR (June 1995), or as a "commercial item" as defined in FAR 2.101(a) or as "Restricted computer software" as defined in FAR (June 1987) or any equivalent agency regulation or contract clause. Use, duplication or disclosure of Software is subject to Nimsoft LLC s standard commercial license terms, and non-dod Departments and Agencies of the U.S. Government will receive no greater than Restricted Rights as defined in FAR (c)(1-2) (June 1987). U.S. Government users will receive no greater than Limited Rights as defined in FAR (June 1987) or DFAR (b)(2) (November 1995), as applicable in any technical data. Trademarks Nimsoft is a trademark of CA. Adobe, Acrobat, Acrobat Reader, and Acrobat Exchange are registered trademarks of Adobe Systems Incorporated. Intel and Pentium are U.S. registered trademarks of Intel Corporation. Java(TM) is a U.S. trademark of Sun Microsystems, Inc. Microsoft and Windows are U.S. registered trademarks of Microsoft Corporation. Netscape(TM) is a U.S. trademark of Netscape Communications Corporation. Oracle is a U.S. registered trademark of Oracle Corporation, Redwood City, California. UNIX is a registered trademark of the Open Group. ITIL is a Registered Trade Mark of the Office of Government Commerce in the United Kingdom and other countries. All other trademarks, trade names, service marks and logos referenced herein belong to their respective companies. For information on licensed and public domain software, see the Nimsoft Monitor Third-Party Licenses and Terms of Use document at:

3 Contact Nimsoft For your convenience, Nimsoft provides a single site where you can access information about Nimsoft products. At you can access the following: Online and telephone contact information for technical assistance and customer services Information about user communities and forums Product and documentation downloads Nimsoft Support policies and guidelines Other helpful resources appropriate for your product Provide Feedback If you have comments or questions about Nimsoft product documentation, you can send a message to support@nimsoft.com.

4

5 Contents Chapter 1: logmon logmon Overview... 7 Chapter 2: logmon Probe Deployment 11 Supported Platforms System Requirements Software Requirements Monitoring System Requirements Probe Deployment Information Chapter 3: logmon Configuration 13 Probe Configuration Interface Installation Probe Defaults Probe GUI Probe Configuration Edit the General Properties Create a New Profile Test Profile Copy Profile Delete Profile Chapter 4: Hints and Examples 43 Notes on Regexp Constructs Notes on Building Regexp Notes on QoS for Variables String Matching Methods Example 1: Logmon Configuration Example 2: HTML Page Example 3: XML File Chapter 5: QoS Threshold Metrics 55 logmon Metrics logmon Alert Metrics Default Settings Contents 5

6

7 Chapter 1: logmon 3.2 This description applies to logmon probe version 3.2x. This section contains the following topics: logmon Overview (see page 7) Documentation Changes (see page 10) logmon Overview The Logfile Monitoring probe will scan ASCII based logfiles and look for essential information in system and application logfiles automatically. Typically, this will be application log files but the probe is also able to monitor information presented via HTML web pages, command output and messages read from Nimsoft queues. The probe searches the specified targets and tries to find match with expressions defined in profiles/watchers. The alarms are generated when contents in the monitored target matches the expressions defined. The probe can also extract metric data from the matched log file entry and store this data in the Nimsoft Quality of Service (QoS) database. Note: Logmon also monitors files from remote shared folders. What logmon does? One important source of information for an IT operations staff is the wide variety of log-files on the systems they maintain. Unfortunately, checking these files manually is a very time-consuming job, and it may also be a challenge for all members of the staff to be able to interpret all types of messages in all types of logs. Logmon can simplify the job for the systems operation staff by: Automatically informing about error situations immediately after they have occurred. Filtering out the log-file entries that need manual action. Usually the majority of entries in a log-file are not of interest to the daily operations staff. By setting up watchers and filters in logmon, alarms are generated only for the important log-file messages. Specifying a more informative text in the alarm message by overwriting the text, which system has placed in the log-file, may help the operations staff locate and fix the problem more easily, without requiring assistance from the system specialist. Chapter 1: logmon 3.2 7

8 logmon Overview Logmon can be configured to monitor ASCII log-files in any format. Experience has shown that very few log-files have the same layout. Some files are line-oriented (single-line files like the UNIX system log-files /usr/adm/messages), while other log-files are record-oriented (multiple-line files, like the ones produced by Oracle). The logmon probe monitors just as effectively line-oriented and record-oriented log-files, using a powerful regular expression and/or pattern-matching scheme. The probe checks the log-file for new entries at user-configurable, timed intervals, keeping track of the position within the file between each run. This assures that only one alarm is sent per log-entry, even if the log-file is truncated in the meantime. One logmon probe can be configured to monitor many log-files. Within each log-file, logmon can be set up to look for occurrences of many different log-file entries. Different log-file entries can trigger different alarm messages, which may contain text from the log-file and/or user-defined text set up by the person who configured the probe. How logmon works? The Logmon probe runs through a set of profiles which essentially contains information about: Which file to monitor? How to monitor it? What to look for? How to report it? The probe monitors ASCII based log-files, and keeps information about the files being monitored, according to their operating mode. This information is used to determine whether the file has been modified or not. If the file has been modified, a set of formatting and watcher rules determine whether or not an alarm message should be generated. Profile A logmon profile contains all the necessary information needed to maintain full surveillance of an ASCII log-file. The profile is tagged with a name that should describe the purpose of the profile. One file may be monitored by one or more profiles without interfering with each other. All data within a profile (watcher and formatting data) are private to the profile, meaning it cannot be shared among profiles. 8 logmon Guide

9 logmon Overview The Watcher definition A watcher definition instructs logmon, which message footprints to look for, by using regular expressions or pattern matching rules. You may define multiple watcher definitions to extract various messages from the log-file. If you wish to extract only part of the message (such as a process name, Cluster number etc.), you can define a set of variables that are extracted from the line(s) that matches the watcher rules and the formatting rules, if defined. A variable may be extracted by defining positioning criteria (such as column or character position). There may be cases where you wish to look for messages within a block of lines, and therefore restrict the watchers to the active formatting definition. This is possible within the configuration of a watcher. The Format definition The default formatting is to match a line, however, some log-files are as already mentioned record-oriented. To be able to correctly extract information from a record-oriented log-file, logmon needs to determine when a block starts and when it ends. This may be specified using string matching rules and/or a line counter. A typical scenario when monitoring record-oriented log-files is to have a set of formatting blocks with watchers. QoS messages from logmon Be aware of the following QoS issue when configuring variables: One QoS message can be sent for the profile and another for the variable, both with the same QoS definition and target. You should therefore use different QoS definitions for the profile and the variable. Note: As a limitation of the logmon probe, the below encoding files are supported for Japanese environment: Linux: UTF-8 Windows: SHIFT_JIS Chapter 1: logmon 3.2 9

10 logmon Overview Documentation Changes This table describes the version history for this document. Version Date What's New? 3.2 February 2013 Added a check box for Exit Code Monitoring and a Settings button on the General tab with Mode selected as command. Added a field - Max. Alarm Level on the General tab. The limitation of 50 line text block has been increased to 200 lines. Added a check box - Text block on the Variable settings dialog of the Watcher Rules tab. Added Probe defaults. 3.1 March 2012 Added support for web-based SOC. Related Documentation Documentation for other versions of the logmon probe The Release Notes for the logmon probe Getting Started with CA Nimsoft Probes Monitor Metrics Reference Information for CA Nimsoft Probes 10 logmon Guide

11 Chapter 2: logmon Probe Deployment This section contains the prerequisites, system requirements and deployment information for the logmon probe. This section contains the following topics: Supported Platforms (see page 11) System Requirements (see page 11) Software Requirements (see page 11) Monitoring System Requirements (see page 12) Probe Deployment Information (see page 12) Supported Platforms The logmon probe supports the same set of operating systems and databases as supported by the Nimsoft Server solution. Please refer to the Nimsoft Compatibility Support Matrix for the latest information on supported platforms. System Requirements The logmon probe should be installed on systems with the following minimum resources: Memory: 2-4GB of RAM. Probe's OOB configuration requires 256MB of RAM' CPU: 3GHz dual-core processor, 32-bit or 64-bit Software Requirements The logmon probe requires the following software environment: Nimsoft Monitor Server or later Nimsoft Robot 5.23 or later Java Virtual Machine 1.6 or later (typically installed with NMS 5.0 and above) Note: For SOC functionality, NM Server 5.6 or later and UMP or later is required. Chapter 2: logmon Probe Deployment 11

12 Monitoring System Requirements Monitoring System Requirements The Log-file monitoring probe (logmon) monitors targets, such as: log-files web pages messages from queues output from commands Probe Deployment Information There are two ways to distribute archive packages. You can distribute the package within Infrastructure Manager or use the standalone Nimsoft Distribution application. See Probe Deployment for more information on deploying probes. 12 logmon Guide

13 Chapter 3: logmon Configuration This section contains the configuration concepts and procedures for setting up the logmon probe. This probe is configured to monitor log files, web pages, messages from queues, and output from commands. This section contains the following topics: Probe Configuration Interface Installation (see page 13) Probe Defaults (see page 13) Probe GUI (see page 13) Probe Configuration (see page 14) Probe Configuration Interface Installation The probe configuration interface is automatically downloaded and installed by the Nimsoft Infrastructure Manager when the probe is deployed on a robot. Probe Defaults At the time of deploying a probe for the first time on robot, some default configuration will get deployed automatically. These probe defaults could be Alarms, QoS, Profiles and so on which save time to configure the default settings. These probe defaults will be seen on a fresh install, that is no instance of that probe is already available on that robot in activated or deactivated state. Probe GUI This section contains the basic GUI information for the logmon probe. Left Pane: The left-side pane contains the list of configured log-file monitoring profiles.the checkmark, beside the profile name, indicates whether the profile is active or not, that is, whether the tests defined within the profile are performed. You can select a profile and view its details in the tree sub-tabs.. Right Pane: The right-side pane contains several fields and controls for adjusting the setup, organized under several tabs (General (see page 21), Format Rules (see page 29), Watcher Rules (see page 30), and Exclude Rules (see page 37)). These tabs are used to configure the selected profile. Chapter 3: logmon Configuration 13

14 Probe Configuration Probe Configuration Like all probes delivered by Nimsoft, the logmon package also contains a configuration tool. This small executable is installed on the workstation where you run Infrastructure Manager. This gives you an intuitive way to access and configure any logmon probe on any Robot within your Domain. Perform the following steps to access the logmon configuration: 1. Select the probe in Infrastructure Manager. 2. Right-click the probe record. 3. Select Configure from the shortcut menu. The logmon configuration application window opens. You can perform the below mentioned functionalities to configure the logmon probe: Edit the general properties (see page 15) Create a new profile (see page 19) Test a profile (see page 37) Copy a profile (see page 40) Delete a profile (see page 41) 14 logmon Guide

15 Probe Configuration Edit the General Properties Click the Edit General Properties button (top-left) on the logmon configuration application window to open the Setup dialog. The General Tab The Setup dialog contains two tabs - General and Quality of Service Definitions. By default, the General tab is opened (see General tab (see page 15) and Quality of Service Definitions (see page 16) tab). The options available on the General tab manipulate the logging facilities in the probe. You can set the detail level on the messages logged on the log-file. Chapter 3: logmon Configuration 15

16 Probe Configuration The fields in the above dialog are explained below: Active Activates the probe, if checked. Logfile size Sets the maximum size of the probe s log file to which probe-internal log messages are written. The default size of the log file is 100 KB. When this size is reached, a backup is made and the contents of the file are cleared. This field allows you to change the size of the log file according your needs. Log level Sets the level of details written to the log-file. Log as little as possible during normal operation to minimize disk consumption, and increase the amount of detail when debugging. Run as user The Quality of Service Definitions Tab Provides user name and password, required by directories and files, to allow access. Note that the user name also must include the Windows domain name (<domain name>/<user name>). The log-file for the logmon probe (not to be confused with the system log-files logmon monitors!) contains information about the activity in the logmon probe itself. The log-file resides on the machine where the probe is installed. You can view it locally using any text editor or from Infrastructure Manager centrally by right-clicking the logmon probe and selecting the View log menu entry. Selecting the Quality of Service Definitions tab displays the dialog where you can define your own QoS definitions. Here, you can add or delete QoS definitions. 16 logmon Guide

17 Probe Configuration Follow these steps to add a QoS Definition: 1. Click the button. The Quality of Service Properties dialog gets displayed. The fields in the above dialog are explained below: QoS Name Defines the new name of QoS. QoS Description Provides short description for the new QoS. QoS Unit and Unit abbreviation Select a unit from the drop-down list. Available units are: Bytes Mbytes Gbytes Seconds Milliseconds Bytes/Second Kilobytes/Second Megabytes/Second Percent An abbreviation will automatically be suggested. If you do not want to use any of the units listed, you can type your own. Chapter 3: logmon Configuration 17

18 Probe Configuration Has maximum value Select this check box to make the data type an absolute max. For example, disk size, memory usage. Type the value in the field. As expected If not selected, the QoS messages on variables that are QoS enabled are sent as regular numbers. If selected, QoS messages on variables are sent as true or false according to the expected value set for that variable. 2. Click OK to add the new QoS Definition. To delete a QoS Definition, do as follows: 1. Click the button. A confirmation pop-up gets displayed. 2. Click Yes to delete the selected definition. 18 logmon Guide

19 Probe Configuration Create a New Profile This functionality is used to create a new profile to configure the logmon probe. You can create a new profile either by: Clicking the Create a New Profile button (top-left of the logmon Configuration window), or Right-clicking in the available profile list pane and selecting the New option, as shown below: Note: The Copy, Delete, and Test Profile functionalities are explained in the respective sections. Follow these steps: 1. Click the Create a New Profile button or select the New option (as explained above). The Add a New Profile Definition dialog gets displayed. Chapter 3: logmon Configuration 19

20 Probe Configuration Note: The OK button gets enabled when you enter the profile name. 2. Enter the name of the new profile in the Name box and click OK. The new profile appears in the list of profiles. t 3. Click the profile check box to activate it and start configuring the profile through various tabs on the right-side pane - General, Format Rules, Watcher Rules, and Exclude Rules. Note: It is mandatory to configure, activate the new profile, and click Apply before it starts working. 20 logmon Guide

21 Probe Configuration The General Tab The General section configures the file to monitor as well as the operating mode of the logmon probe. As mentioned earlier, one file may have multiple profiles defined. Each profile may have its private operating mode and private subject. The subject field gives the administrator the option to make applications (for example, a Perl script) that subscribes to the given subject. The default subject is "alarm" which is subscribed to by for example, the Nimsoft Alarm Server. The fields in the above dialog are explained below: Mode The operating mode used by this profile. (see Operating Modes (see page 23)) You may choose one of: Cat Command Full Full_time Queue Updates URL Chapter 3: logmon Configuration 21

22 Probe Configuration File / Command / Queue / URL It is mandatory to specify the target to be monitored: If selecting mode cat, full, full_time or update a file must be specified. If selecting mode command, a command must be specified. If selecting mode queue, a queue on the hub must be specified. If selecting mode URL, a URL must be specified. See also Optional URL Settings (see page 24). Special time-formatting primitives are also available if the target is a file (see Time Formatting Primitives (see page 26)). Note: The title of this field depends on the operating mode selected. Check Interval Set the time interval at which the profile runs, for example, 5 minutes. Exit Code Monitoring Only visible if the Mode field is selected as command. This field serves as an option to provide multiple exit codes as threshold. If command is selected in the Mode field and thresholds for exit code monitoring are defined, then the probe will collect exit code of the command that has been configured in Command field and then perform threshold checking on it. Based on the results of threshold checking, alarm messages will be generated according to the configuration Settings Only visible when Mode is selected as command and is enabled only when the Exit Code Monitoring check box is selected. Allows you to add multiple thresholds. See Adding New Exit Code Thresholds (see page 26). Generate Quality of Service Enables sending of QoS messages for variables and number of matches. Generate Alarm Instructs logmon to generate an alarm message for each watcher match. Send message using a specific subject Instructs logmon to generate a Nimsoft message using the specified subject. Browse / Settings The caption of this button depends on the operating mode selected: Settings if URL is selected, otherwise Browse. Enables the users to browse the remote system for a specific log-file. When operating mode URL is selected, clicking the button opens the Optional URL Settings (see page 24) dialog. 22 logmon Guide

23 Probe Configuration View Brings up a simple viewer, which displays the contents of the monitored target. Max. alarm count Sets the limit to the number of alarms being sent. Max Alarm Level Sets the maximum alarm level. Max. alarm message Message that should be sent when maximum alarm count is reached. Operating Modes The different operating modes in logmon probe are explained below: Mode command full full_time updates queue cat URL Description Scans the output of a user-specified command. Scan the file from beginning to end, if the file has been modified. Note that the contents of the file must change. The file is not considered as modified unless its content is updated. As full (see above), except that a file not updated but only read operation is done is also considered modified. Scan the file from the last EOF mark when it s modified. A checksum taken from the end of the file and the file size + file modification time is used to determine whether the file is considered modified or not. Read messages of a permanent queue instead of from a named file. Logmon searches for the following string fields in the message: message, text and line. Used to test the configuration from the command-line. The file is scanned from beginning to end always. This mode makes logmon perform the scan of the URL specified, retrieved by the Nimsoft url_response probe. Note: When URL is selected, the tabs Format Rules and Exclude Rules are not available, since they do not apply in this case. Chapter 3: logmon Configuration 23

24 Probe Configuration Optional URL Settings Clicking the Settings button (appearing when the URL mode is selected) opens the Url Settings dialog, enabling you to specify additional parameters. The fields in the above dialog are explained below: URL Defines the full URL to the web page you want to scan. For example, Timeout (sec.) Specifies the wait time for the probe to load the complete web page. If the page takes longer to load, then the data is not received. An alarm message is sent. Retries before failure Specifies the number of attempts to access the URL before giving up, regarding it as a failure and sending an alarm. Failure alarm level Select the severity level of the alarm to be sent if an error occurs (timeout or too many retries). 24 logmon Guide

25 Probe Configuration Windows NT authentication Provides/enables authentication with web servers and proxy servers is normally of the 'Basic' type. If 'Windows Integrated Authentication' is required, the 'Windows NT Authentication' check box in the profile must be selected. Consequently, the Proxy settings are fetched from the registry, as saved by Internet Explorer. If the Windows NT authentication option is selected, it is mandatory to fill the Impersonate field. The Impersonate field appears when the Windows NT authentication is checked. Specify the user and password used by the probe when logging on the server. The specified user s proxy settings are then fetched from the registry, as saved by Internet Explorer. Proxy properties (optional) Provides details of the proxy properties when your access to the URL is via a proxy server. These fields appear when the Windows NT authentication is not checked. Host Port User Defines the name of the host, which forwards the URL requests. Defines the Proxy http-port. Specifies some proxy hosts requires a login, in which case it is mandatory to fill the User and Password fields. Password Defines the password corresponding to the proxy login user. SSL Settings Determine the degree of encryption of the data traffic. The options are: Automatic On Off Tunnel User authentication (optional) This is necessary when the web server hosting the URL requires you to log in. User The user account on the web server host to be used for accessing the web page. Password The password corresponding to the user account. Chapter 3: logmon Configuration 25

26 Probe Configuration Time Formatting Primitives The log-file name/path can be constructed by a combination of text and special primitives used by the system-call strftime. %a - Abbreviated weekday name %A - Full weekday name %b - Abbreviated month name %B - Full month name %c - Date and time representation appropriate for locale %d - Day of month as decimal number (01 31) %H - Hour in 24-hour format (00 23) %I - Hour in 12-hour format (01 12) %j - Day of year as decimal number ( ) %m - Month as decimal number (01 12) %M - Minute as decimal number (00 59) %p - Current locale s A.M./P.M. indicator for 12-hour clock %S - Second as decimal number (00 59) %U - Week of year as decimal number, with Sunday as first day of week (00 53) %w - Weekday as decimal number (0 6; Sunday is 0) %W - Week of year as decimal number, with Monday as first day of week (00 53) %x - Date representation for current locale %X - Time representation for current locale %y - Year without century, as decimal number (00 99) %Y - Year with century, as decimal number %z, %Z - Time-zone name or abbreviation; no characters if time zone is unknown %% - Percent sign Example: Adding New Exit Code Threshold Monitors a log-file that switches every day, on the form ddmmyy.log: C:\Program files\mylogs\%d%m%y.log This functionality allows you to add multiple thresholds. You can add a new threshold, copy an existing threshold details to a new one, modify an existing threshold, and delete a threshold. 26 logmon Guide

27 Probe Configuration Follow these steps: 1. Click the Settings button on the General tab when the Mode is selected as command. The Thresholds dialog displays. 2. Right-click and select the New option. The Threshold [New] dialog displays. Chapter 3: logmon Configuration 27

28 Probe Configuration The fields in the above dialog are explained below: Threshold Value Specifies an exit code. Condition Specifies relational operator. Severity Indicates severity of the alarm message. ID Displays the numeric sequence of the thresholds. Message Specifies message string to be used in alarm message. 3. Click OK. The threshold gets added. Similarly, you can add other thresholds also. In a similar manner you can use the copy, edit, and delete functionalities. 28 logmon Guide

29 Probe Configuration The Format Rules Tab As briefly mentioned earlier, the formatting definition is primarily used to recognize record-oriented (multiple-lines) log-files. To identify a block, you need a starting definition and an ending definition. The formatting definition may be used to 'tie' a set of watchers to a formatting block by setting the Restrict to field in the Advanced tab of the watcher definition. When operating mode URL is selected, the Format Rules tab is not available, since the tab is not applicable in this case. Having two or more format rules, and if all of them are applicable, then only the first match in the target file of the pattern specified results in an alarm. The other ones are ignored. The fields in the above dialog are explained below: Format properties Start expression A regular expression specifying the start of all the log-file entries you are looking for. Chapter 3: logmon Configuration 29

30 Probe Configuration End rule Select one of the two options from the following: End expression: A regular expression specifying the end of the log-file entry. or End after: A fixed number of lines (200), which makes up the log-file entry. The Watcher Rules Tab The watcher section of the logmon configuration tool enables management of all the watchers within a profile. A watcher is dedicated to 'watch' for specific patterns in the target file. You may define specific formatting definitions that enable you to monitor record-oriented log-files, e.g., Oracle log-files, or you can simply ignore this feature when monitoring line-oriented logs. Anyhow, the Match field determines in most cases what you are looking for in the log-file. This string may be a pattern string or it may be a regular expression. When you select the Watchers Rules tab in the configuration tool, you see a list of all the watchers defined for the selected profile. The checkmark indicates whether the tests specified in the watcher are actually performed. Select a watcher to see its detailed setup. Note: Watchers are executed in the order they appear in the list. Right-clicking a watcher in the list, you can move the watcher up or down in the list. 30 logmon Guide

31 Probe Configuration The Watcher Rules tab contains the following sub tabs - Standard, Variables, Advanced, QoS, and Alarm. The Standard tab: Match expression A regular expression specifying what to look for in the log messages. Using parentheses inside the match expression, the contents inside the parentheses can be made available as variables (see the Variables tab in the following section). Message to send on match If the expression specified above match an entry in the log-file, an alarm with the text specified here is generated. This message may contain information from the log entry if the information is stored in a variable and you specify the name of the variable preceded by a dollar sign ($). If running in Send QoS mode, then the content of the Message to send on match field is converted to a number (double) unless the keyword "NULL" is used. Severity level of message Defines the severity level of the generated alarm message. Run command on match Select this option if you want a command to be run if expression specified above match an entry in the monitored target. Specify the command to be run and optional parameters. Note: The Command executable and Command arguments fields get enabled only when you select this check box. Chapter 3: logmon Configuration 31

32 Probe Configuration The Variables tab: To simplify the definition of match expressions and to allow the alarm messages to contain information extracted from the log entry, you can define variables. The variables are assigned values according to the extract definition. In its simplest form, an extract definition is simply a number indicating which word on each line to use as the value for the variable. The Field separator field allows you to specify which character is used to separate the fields in each record in the log-file. Create a new variable definition by right-clicking the variables window in the Watcher Rules > Variables tab. The Variable settings dialog opens. The fields in the above dialog are explained below: Name Specify a variable a name of your own choice. Source Line This field is not used when the monitored target is a URL. Variables can be extracted from source files or from the match expression defined for the watcher. While extracting the variable from a source file, define from which line in the source file to extract the variable (1 means to extract the variable from line one). This is useful when a Format is used. Text block Allows you to refer to entire text block as variable. When you select this check box, the content is used as it is without doing any truncation of control characters. 32 logmon Guide

33 Probe Configuration Source FROM position Extracting from a source file: Define from which position in the source line defined to extract the variable, either a column or a character. Extracting from a match expression: If extracting from the match expression defined for the watcher, click the match expression radio button and specify the number referring to the correct position in the match expression. You may extract variables from the contents inside parentheses in the match expression. Using number 1, refers to the first parenthesis in the expression, using number 2, refers to the second parenthesis in the expression and so on. Source TO position Extracting from a source file: Define to which position in the source line defined to extract the variable, either a column or a character. Extracting from a match expression: Ignore 'to' is automatically selected (as the only valid option). Threshold alarm definition Define the operator and threshold value for the variable. Normally alarms are sent on match expression for the watcher. However, if the threshold value is not breached for any of the variables for watchers defined for the profile, no alarm is sent even though the match expression matches the line. Examples: Space (the space character) means white space (any space, e.g. <return>, <tab>, <space>). This is the default behavior. Ctrl + I indicates that <ctrl>+i (ASCII 9) is used as field separator. Ctrl +A indicates that <ctrl>+a (ASCII 1) is used as field separator. Ctrl + I + : indicates that both <tab> and colon(: ) will be used as field separators. First the record is divided into fields, based on <tabs> found. Then, if a colon character (: )is found in a field (e.g. time format hh:mm), this field is again divided. \\ indicates that the <backslash> character (\) is used as a field separator. Note: Please note that all variables are stripped of trailing whitespace after they are extracted, no matter how they are extracted. Leading whitespace is handled differently based on which mechanism is used, and which character(s) make up the separator list. See also section Hints and examples (see page 43). Chapter 3: logmon Configuration 33

34 Probe Configuration The Advanced tab: Some rarely used fields are assembled under the Advanced tab: Restrict to format definition If one format definition is specified here, the watcher is active only when the probe has found a log-file entry, which matches that format definition. Abort on match Aborts on match, does not attempt to run remaining watchers on the current line. Match on every run Logmon expects a 'match' for the specified watcher every time the log-file is parsed. Some applications write 'heartbeat' messages to a log. Notes: When this check box is selected, the Message to send on match and Severity level of message fields are enabled. Here, you can specify the alarm message for match occurrence and severity level of the alarm. If after the specified time period, given in Check Interval field (on General tab), the match occurs, then the message in the Message to send on match field is published. Else, the message in the Message to send on match field, on the Standard tab, is published. Send clear alarm Selecting this option triggers a clear alarm for a checkpoint if all variables are as expected. This option is only applicable when testing if a variable is as expected, and a suppression key is set in the Alarm section. It does not have any effect on a Watcher that is only checking for log entries, or where no checking of variables is done. Note: Special care should be taken when setting up the suppression key for alarms you want to clear automatically, since a suppression key that is not unique may clear an alarm you did not intend to clear. The easiest way to ensure that the correct alarm is cleared is to use a variable from the log itself as a part of the suppression key. 34 logmon Guide

35 Probe Configuration Example: Given the file: Service aaa has status OK Service bbb has status OK Service ccc has status ERROR Service ddd has status OK The watcher might look something like this: Match string = /Service (\w+) has status (\w+)/ The Variables would be: Service = $1 (the first regular expression evaluation). Status = $2 (the second regular expression evaluation with the added Expect rule: = OK). The Alarm section could have the suppression key: $PROFILE. $WATCHER.$Service. The QoS tab: This would be unique for each service. If the suppression key consisted of only the first two elements then the clear alarm for service ddd would clear the alarm for service ccc immediately after it was generated. The setup outlined above would generate four alarms on every run. Three of them would be clear alarms, and the fourth would be set to the severity in the Watcher. If the status of service ccc changed to OK it would generate four clear alarms. This added alarm traffic should be taken into consideration when activating this functionality. The QoS tab lets you set the QoS properties for the selected watcher. Note: The QoS tab is not available unless the Generate Quality of Service option is selected in the General tab. The fields in this tab are described below: Count matches When selected, the count of matches are sent as QoS messages. QoS Name Select the default QoS or one you have created yourself, or you may select <Add new QoS definition> to define a new one. QoS Description Short description of the QoS. Chapter 3: logmon Configuration 35

36 Probe Configuration QoS Target The QoS target (where the QoS value is measured). Default target is profilename.watchername. QoS on variables You can specify a QoS to be sent for each of the variables defined. All variables defined under the Variables tab are listed here. None of them are activated or configured for QoS. All entries in the list can be activated or deactivated. Clicking a variable opens the properties dialog for the variable, enabling you to set the QoS properties for the variable. Note that QoS on variables can only be sent on numeric value or state (true or false), provided that the option As expected is selected on the QoS definition dialog (the dialog launched when creating a new QoS). The Alarm tab: The Alarm tab lets you set the alarm properties for the selected watcher. Message subsystem id The subsystem ID of alarms generated by this watcher. A string or an id managed by the nas probe. Suppression key Source Suppression key is a messages suppression feature, used to avoid multiple instances of the same alarm-event (variables may be used). Avoid using underscore in the suppression key. Rather than using e.g. $PROFILE_$WATCHER, you are advised to use $PROFILE-$WATCHER or $PROFILE/$WATCHER. This field signifies the "sender" of the alarm. It can be used to impersonate a system/host. For example, many networking devices record/log information to syslogd, hence the need to impersonate the target/originator device. Count matches This field is enabled only when the Count matches check box is checked in the QoS tab. Generates alarms when the conditions specified in the Operator and Threshold fields are breached. Pattern match threshold Generate pattern match alarms when the threshold value specified in the Threshold field is breached. 36 logmon Guide

37 Probe Configuration The Exclude Rules Tab The Exclude Rules tab lets you can set the exclude rules for the selected watcher. You can define one or more Exclude Rules. Each block or line of input is checked to see if the expression entered in the Exclude expression field is found. If a match occurs, the line/block is ignored and does not trigger any action (even if one or more of the patterns the watcher is dedicated to 'watch' for is found). Test Profile Test Profile functionality enables you to test a profile to ensure valid regular expressions, before executing the profile in a production environment. Chapter 3: logmon Configuration 37

38 Probe Configuration Before using the Test Profile functionality, you need to configure one or more watcher rules. Make sure that the profile to be tested is properly configured and saved. Follow these steps: 1. From the list of configured profiles, right-click the profile you wish to test and select Test Profile option from the context menu. The Test Profile dialog appears. 38 logmon Guide

39 Probe Configuration 2. Select the Test File option if you wish to execute the profile on an existing log file and click Browse to navigate to and select the required log file. Chapter 3: logmon Configuration 39

40 Probe Configuration Note: Select the Test String option if you wish to test the profile on a dummy log data rather than an actual log file. In the text area provided, enter the string on which you wish to run the profile. 3. From the Watcher Name section, choose one or more watchers that you wish to run. 4. Click Test. The Matching results section displays the results, if any. Note: Similarly, by selecting the required option (Copy or Delete), by right-clicking in the available profile list pane, you can copy or delete a profile. Copy Profile This functionality allows you to add a new profile using the copy functionality. Follow these steps: 1. Select the profile that you want to copy, right-click, and select the Copy option. The Copy of profile [profile name] displays. 2. Provide the required name and click OK. The new profile gets added and then you can configure the profile as mentioned in Create a New Profile section. 40 logmon Guide

41 Probe Configuration Delete Profile This functionality allows you to delete a selected profile. Follow these steps: 1. Select the profile that you want to delete, right-click, and select the Delete option. The delete confirmation pop-up displays. 2. Click Yes to delete the selected profile. Chapter 3: logmon Configuration 41

42

43 Chapter 4: Hints and Examples This section contains the following topics: Notes on Regexp Constructs (see page 44) Notes on Building Regexp (see page 45) Notes on QoS for Variables (see page 45) String Matching Methods (see page 46) Chapter 4: Hints and Examples 43

44 Notes on Regexp Constructs Notes on Regexp Constructs General: [^<]* Anything (or nothing) not a '>'. Useful for getting the rest of a tag without knowing what it might be! \"([^\"]*?)\" Capture anything (or nothing) inside quotes. Note that the quotes themselves are required here, also that the quote needs to be escaped with the backslash \ character. The data you capture inside parentheses is available for use in variables in the logmon probe. You can use these variables in alarm messages, check values and send QoS messages with the contents, if they are numeric, or if the results are as expected <A[^>]*?> Matches and anchor tag up until the end of the tag. Generalizes for other tags as required. <\/A[^<]*? Matches an end anchor tag and anything up until the start of the next tag. Note that a slash / must be escaped with a backslash since it's a reserved character in regular expressions. If the monitoring target is an URL: Avoid using.* since regexps tries to match anything. The page is stripped of newlines when we get it, so everything is on one long line, and '.*' matches more than you probably expect it to. Be restrictive, and try to limit yourself to what is either inside the tag or outside the tag (don't go for a combination unless you really know what you are doing). Carriage Return (\r) and Newline (\n) characters are stripped out and replaced by spaces in the file when we get it. This means that you have to be ready to deal with spaces in places you might not expect them! Either add \s* (zero or more spaces) to your regexp or handle anything up to the next start or end of a tag. 44 logmon Guide

45 Notes on Building Regexp Notes on Building Regexp Break what you are looking for down into manageable component parts. The following example is a table entry in an html file. 1. Start small - figure out where what you are looking for starts. In this example, start on a table row (<TR bgcolor=#efeee9>). 2. Figure out a regexp to find that in the file and add it to your configuration. 3. Print out a dummy alarm for every match and count number of alarms you received. If it matches the number you were expecting then go on to the next part of the regexp. 4. Add fields one at a time. As long as you take it one step at a time you can easily see which regexp is causing you to get unexpected data. 5. Open an editor and keep track of the regexps component parts. Add notes. Judge the complexity of the regexp is up to after the fact can be difficult. Also keep track of the different parentheses groupings, so that you know which ones to use in the variable list. 6. When you match everything you are looking for, you can start adding variable definitions. Do not add thresholds yet - make sure you are matching everything you are looking for first. Print them in the alarms to see that you get what you expect. 7. Add threshold values so that matching entries that are within acceptable limits are not generating alarms. Note: The threshold is an expected value. So, if the value is 1, the operator is < and the threshold is 5 then an alarm is NOT generated (1<5 = true). However if the value is 7 then an alarm is sent (7<5 = false!). Numeric checks (<, <=, >, >=) require that the value be numeric as well. Checking strings can be done with the =,! = and re operators, where re requires that you specify a regexp. Don't forget to start and end your regexp with slashes (/regexp/). Notes on QoS for Variables QoS requires numeric values. If what you are looking for is a string then you can only send if it matches your expected value or not. To signify that a QoS is the result of an expected value there is a flag you set when defining the QoS, 'As Expected', and if the QoS is used then the result of the check against a threshold is used even if you have a numeric value. If you are looking for a set of variables as a single entity, like we are below, only set one of the variables to send QoS. If you are expecting more than one match on a page try to use a QoS Target that is unique for each match. Chapter 4: Hints and Examples 45

46 String Matching Methods String Matching Methods Logmon supports two methods of string-matching. The simple method is called pattern matching, and is very similar to the one found in many popular shells (UNIX), and the more powerful one, called regular expressions. The logmon probe implements pattern matching by default, while regular expressions are denoted by a pair of / / encapsulating the expression. The capabilities are best highlighted by considering a couple of logmon configuration examples. 46 logmon Guide

47 String Matching Methods Example 1: Logmon Configuration This extract is produced by Microsoft Scandisk and placed in C:\SCANDISC.LOG whenever the user or windows automatically starts scandisk and it detects problems with the disk Clusters. Scandisk could not properly read from or write to Cluster 66293, which contains some or all of C:\WINDOWS\SYSTEM\MSJET35.DLL. Resolution: Repair the error Results: Error was partially corrected. Scandisk could not properly read from or write to Cluster , which contains some or all of C:\Program Files\DevStudio\VC\lib\MSVCRTD.PDB. Resolution: Repair the error Results: Error was partially corrected. Restarted 10 times because another program wrote to this disk. In order to write a configuration that reacts on the changes in the scandisk.log, and reports this by issuing an alarm, it is required to instruct the logmon probe that search for a block of text that: 1. Starts with "properly read from or write to Cluster". 2. Ends on a blank line. The cookbook below demonstrates how we define a profile, a formatting definition and a watcher definition by using the file extract from SCANDISK.LOG. It reports when it detects data from the file matching the formatting criteria above. If a match is detected data is extracted using the active watcher(s) defined. The purpose is to extract the actual Cluster number and the files involved with the problem, and report it using a user-defined message. Define Profile Follow these steps: 1. Define which file to monitor and how to monitor the file. 2. Click the New Profile button. The Add a New Profile Definition dialog is displayed. 3. Create a profile named ScanDisk and activate the profile by selecting the checkbox against it. 4. Enter the required File parameter and set the Mode parameter to updates. Chapter 4: Hints and Examples 47

48 String Matching Methods Define a Formatting Definition Follow these steps: 1. Select the Format Rules tab. 2. Create a new formatting definition by right-clicking the formats window and selecting the New option. The Add a New Format Definition dialog is displayed. 3. Name the definition, say, ClusterBlock and click OK. 48 logmon Guide

49 String Matching Methods The new definition gets added in the list. 4. Activate the Format Rule by selecting the checkbox once the new definition is displayed in the list. 5. Replace the Start expression field with *read from or write to Cluster*, select the End expression option, and select blankline from the End expression combo-box. Define the Watcher Definitions Follow these steps: 1. Select the Watcher Rules tab and create a new watcher definition by right-clicking the watchers window. Name the definition ClusterError. Activate the watcher rule. Chapter 4: Hints and Examples 49

50 String Matching Methods 2. Select Watcher Rules > Advanced tab and select Restrict to format definition as ClusterBlock. 3. Create a new variable definition by right-clicking the variables window in the Watcher Rules > Variables tab and name this variable as Cluster. 4. Check the Source Line and set it to 1 (means extract the variable from line one). 5. Select the Column option in the FROM frame and set it to 11 (this means extract column 11). 6. Select the Ignore 'To' option. All this instructs logmon to take whatever's in column 11 on line 1 (one) and store it into a variable that may be referred to as 'Cluster' (or rather $Cluster) in the message string. 50 logmon Guide

51 String Matching Methods 7. Create another variable from the data found in line 2, column 7 to the end of the line and call this file. 8. Activate both the variables. 9. Select the Watcher Rules > Standard tab. 10. Clear the Match expression field (you've restricted this watcher to the formatting block ClusterBlock, so a match is implicit). Set the Message to send on match field to ScanDisk reports error on Cluster $Cluster containing $file. 11. Save the definition by clicking the Apply button. Chapter 4: Hints and Examples 51