Best Practices Guide for Managing Statistics Poller Error Messages

Transcription

1 F Best Practices Guide for Managing Statistics Poller Error Messages Understanding and Resolving ehealth Polling Errors The ehealth statistics polling cycle is critical to the integrity of the data within the ehealth database. This document will provide the ehealth Administrator with the tools and knowledge to understand and resolve the possible errors which can be encountered by the ehealth Statistics poller. Prepared by: Jason Normandin/Sheldon Chapman Concord Technical Support Copyright 2004 Concord Communications, Inc. ehealth, the Concord Logo, Live Health, Live Status, SystemEDGE, AdvantEDGE and/or other Concord marks or products referenced herein are either registered trademarks or trademarks of Concord Communications, Inc. Other trademarks are the property of their respective owners.

2 I. Table of Contents I. TABLE OF CONTENTS... 2 II. EXECUTIVE SUMMARY... 3 DOCUMENT FOCUS... 3 III. UNDERSTANDING THE EHEALTH STATISTICS POLLING CYCLE... 3 BACKGROUND INFORMATION... 3 OVERVIEW... 3 DETAILS OF THE STATISTICS POLLING CYCLE... 3 IV. STATISTICS POLLER LOGGING MECHANISMS... 6 STATISTICS POLLING STATUS WINDOWS... 6 STATISTICS POLLER MESSAGES FILE... 6 V. REVIEWING THE STATISTICS POLLER ERROR MESSAGES... 7 HOW OFTEN SHOULD THE LOGS BE CHECKED... 7 PARSING THE ERROR MESSAGES LOG... 7 OTHER TOOLS AVAILABLE... 8 VI. RESOLVING NO RESPONSE TO PING ERROR MESSAGES... 8 CAUSE... 8 RESOLUTION... 9 VII. RESOLVING NO RESPONSE TO SNMP ERROR MESSAGES... 9 CAUSE... 9 RESOLUTION VIII. RESOLVING DEVICE EXCEEDED ALLOWABLE TIMEOUTS ERROR MESSAGES CAUSE RESOLUTION IX. RESOLVING LARGE DELTA' ERROR MESSAGES CAUSE RESOLUTION X. RESOLVING GENERIC SNMP ERROR MESSAGES CAUSE RESOLUTION XI. OPENING A TICKET WITH CONCORD TECHNICAL SUPPORT XII. OTHER RESOURCES

3 II. Executive Summary Document Focus This document is designed to introduce the new or established ehealth Administrator to the ehealth Statistics Poller process and the types of Statistics Poller errors that the user may encounter. Through the use of this document the ehealth Administrator should be able to better manage the ehealth Statistics Polling cycle and minimize potential data loss that may occur due to polling errors. III. Understanding the ehealth Statistics Polling Cycle Background Information Overview In order to understand the ehealth polling process it is necessary to understand the components involved in providing and collecting the data. The first component is the device s Management Information Base (MIB). The MIB is a formal description of the set of object identifiers (OID s) that can be managed using the SNMP protocol. The actual information that ehealth collects and stores in the database is contained within the OID s of the device s MIB. The second component is the MIB translation file (MTF). Each MTF file identifies the objects and variables associated with a given element. These variables are listed as a set of statements that map MIB variables to the appropriate database columns. This proprietary technology allows ehealth to normalize data collection from standard and proprietary agents available from multiple vendors. By normalizing data collection, ehealth can reliably analyze data collected from different agents and vendors and display it in the same report using a standard set of labels, such as Bandwidth Utilization. The final component is the precompiled MIB file (PCM). A PCM file contains the formal object names and corresponding object identifiers (OID s) for the variables that will be polled by ehealth. ehealth utilizes this file as a guide to determine which object identifiers will be requested from the device s MIB during the poll cycle. ehealth utilizes the statistics polling cycle to obtain SNMP statistics for the devices within the ehealth element configuration. The poller process performs SNMPGET operations for each MIB object associated with the ehealth variables for that element. The device responds to our SNMP requests and the data is then stored within the ehealth database for that element and variable. Details of the Statistics Polling Cycle Poller Initialization The first action taken by the ehealth statistics poller is the poller initialization process. This occurs only when either ehealth or the poller itself is stopped and restarted. 3

4 In ehealth versions 5.6.x and earlier, the following actions are taken by the initialization process: 1. The element configuration information from the ehealth $NH_HOME/poller/poller.cfg file is read into memory 2. The element configuration information from the ehealth database is read into memory 3. The database and poller.cfg configurations are then compared to determine if any differences exist between the two configurations a. If elements exist in the poller.cfg file that do not exist in the database configuration, these elements are added to the database as specified within the poller.cfg file b. If elements exist in the database that do not exist in the poller configuration, stub elements are created within the poller configuration to mark the place of each missing element i. These stub elements will not be polled and the element configuration will be marked by the assignment of unknown.mtf within the mtffile parameter ii. This is due to the fact that there is not enough information within the database s element configuration tables to accurately rebuild the element within the poller configuration 4. The MTF file associated with each element within the configuration is then read into the poller processes, nhipoller, memory. 5. The PCM file associated with each MTF file is then read into poller memory 6. The poll records are then created within the universal poll list for each element documenting the following information obtained from the above initialization: a. Agent IP Address b. Community String c. SNMP Port (default is 161) d. Element index values e. The MIB objects to be polled for each element from the PCM and MTF files f. The MIB object type for each object is noted (Eg. Gauge, Counter, or Octet String object) In ehealth version 5.7 and later, there is no longer a poller.cfg file. All element configuration information is stored within the ehealth database. This improves the performance of the poller initialization process as steps 1 and 3 are no longer required. Poll Cycle The ehealth statistics poller will then begin to poll each element within the universal poll list starting from the top and continuing until the end of the list is reached. The steps performed during this process are as follows: First Poll 4

5 1. A ping request is sent to the Agent IP Address or configured interface IP Address a. A ping will not be sent if the following conditions have been met i. Ping request previously sent and response received for element having the same Agent IP Address ii. The NH_POLL_PING_DISABLED variable is set to yes 2. SNMP requests are sent to each Agent IP Address for each element for the associated OID s observing the limits defined by the following ehealth environmental variables: a. NH_STATS_POLLS_PER_SECOND b. NH_POLL_AGENT_THROTTLE c. NH_SNMP_TIMEOUT d. NH_SNMP_RETRIES e. NH_SNMP_DEVICE_TIMEOUTS 3. If the MIB object polled is a counter, store this first value as the baseline for delta calculations 4. Once all requests are received or all SNMP requests have exceeded the allowable timeouts and retries defined above, the polltime (the actual time it took to complete the poll) value is compared to the configured basepolltime (the configured normal stats poll cycle value, 300 seconds by default) Second Poll a. If polltime > basepolltime, start a new cycle immediately i. For example, if the configured poll cycle is 5 minutes (300 seconds) and the last poll took 309 seconds, a new poll would begin immediately b. Else wait until next poll cycle is due 1. Repeat steps above i. For example, if the configured poll cycle is 5 minutes (300 seconds) and the last poll took 209 seconds, the poller process would sleep for 91 seconds and then begin the next poll 2. Perform any necessary availability backfill operations if the ehealth application was down prior to the first poll 3. If the MIB object polled is a counter, subtract the previous counter value from the current counter value to determine the delta value. This is the value that is stored within the ehealth database Subsequent Polls 1. Repeat steps above calculating the delta values for all counter values 5

6 IV. Statistics Poller Logging Mechanisms The ehealth statistics poller will log all polling related error messages or informational messages to the Statistics Poller Status Window and the Statistics Poller messages file ($NH_HOME/log/pollerStatus/messages.stats.log). Statistics Polling Status Windows The Statistics Polling Status Window provides the ehealth Administrator with a quick view of the overall ehealth polling cycle. The information contained within this window includes: 1. Time of the next poll a. The time of the next poll is the next time that any poll (Fast, Normal, or Slow) will occur 2. Number of successful polls (Good Polls) a. This window displays separate bars for elements polled at each of the different poll rates. 3. Number of unsuccessful polls (Bad Polls) a. See above 4. Polling Status Indicator a. The circle changes color to indicate the polling status i. The circle changes from blue to yellow to indicate that the poll time is approaching ii. An orange circle indicates that the poller is initializing iii. A green circle indicates that a poll is in progress iv. A red circle indicates that polling has stopped 5. Additionally, the Statistics Poller Status Window provides the option to view the poller error messages that are being logged by the Statistics Poller. The messages are available by selecting the Show Messages option. Statistics Poller Messages File The ehealth Statistics Poller will log all error messages or informational messages displayed within the Statistics Polling Status Window to Statistics Poller Messages File (messages.stats.log). This file is stored within the $NH_HOME/log/pollerStatus directory. This directory also contains message files for the other ehealth polling processes (not covered within this document). Message File messages.stats.log Description Statistics Polling Status window messages 6

7 messages.stats.log.bak messages.traffic.log messages.traffic.log.bak messages.import.log messages.import.log.bak Previous messages.stats.log Conversations Polling Status window messages Previous messages.traffic.log Import Polling Status window messages Previous messages.import.log Once the messages.stats.log file reaches the default maximum size of 1 MB, ehealth moves it to a backup log file named messages.stats.log.bak and overwrites the existing backup log file if one exists. ehealth then starts a new log file using the default file name. The 1 MB maximum size value can be increased or decreased using the NH_POLL_LOG_SIZE environmental variable if more or less information is needed by the ehealth Administrator. NH_POLL_LOG_SIZE specifies the maximum size of the log files and backup files in bytes. The default value is and it defines the maximum value for the log file size in bytes with valid values being 1 to any integer. V. Reviewing the Statistics Poller Error Messages The messages within either the Statistics Poller Status Window or the messages.stats.log file should be reviewed by the ehealth Administrator on a regular basis to minimize the number of bad polls which can result. How Often Should the Logs Be Checked The poller messages should be reviewed at minimum on a daily basis to ensure that the elements in the configuration are being polled correctly. The polling errors must be resolved to prevent continued data loss that occurs when elements are unable to be polled successfully. If left unresolved, errors can build up to a point where the actual poll cycle is extended due to timeout and retries attempts for unresponsive devices. Parsing the Statistics Poller Messages Log Through the use of command line utilities and scripts provide by Concord Communications Technical Support the messages can be reformatted in a way that allows for quick review and analysis of the messages contained within the log file. The UNIX grep command can be utilized to prioritize the messages that are frequently occuring versus those that may have been a one time occurrence. The following command effectively counts the number of times an error appears in the messages.stats.log file and sorts them in order of highest occurring to lowest. grep "'" $NH_HOME/log/pollerStatus/messages.stats* cut -f2- -d"'" sed 's/delta.*//g' sort uniq -c sort -nr Output of this command will provide information similar to the following: Count Element Name Error Message 7

8 132 JNORMANDIN-C400-SH-disk-Harddisk0': no response. 131 JNORMANDIN-C400-SH-enet-port-4': no response. 130 JNORMANDIN-C400-SH-enet-port-65542': no response. 130 JNORMANDIN-C400-SH-Cpu-1': no response. 130 JNORMANDIN-C400-SH-C:': no response. 99 JNORMANDIN-C400-SH-D:': no response. 99 JNORMANDIN-C400-SH': no response. 33 missouri-1691-sh-ehealth-nhirmtout': no response (Empire process polling received snmp timeout while in state EpSentPMonPoll) Additionally, the nhssortpollingerrorslite utility is available from the Concord Communications Technical Knowledgebase ( using solution ID TS This utility creates comma separated value (CSV) files which can be viewed and sorted easily using Microsoft Excel or another equivalent application. Please see solution ID TS16405 for additional information regarding the nhssortpollingerrorslite utility. Other Tools Available Additional tools are also available to assist the ehealth Administrator in managing the Statistics Polling Cycle. The nhlistelements utility accepts the nodbdatafor flag. This flag will display all elements that have not had data inserted in the DB for the number of hours specified by this flag. The data loss is usually the result of polling errors for the element in question. For example, to obtain a list of all elements that have not had data inserted within the past 12 hours, the following command could be used: nhlistelements nodbdatafor 12 This list can then be compared to the messages seen in the messages.stats.log to determine the cause of the data gap. VI. Resolving No Response to Ping Error Messages Cause The ehealth poller process first pings each IP address before sending out an SNMP request during the poll cycle. If the device does not respond to ehealth s ping request, the poll for this element is considered a "Missed Poll" and no SNMP data is collected. Given this information it is crucial that the network path between the ehealth server and the device to be polled allow for a ping request/response. If the device responds to a manual ping via the command line, then it may be possible that ping packet size limitations may exist on the network or destination device. The default ping packet size utilized by ehealth is 100 bytes which differs from the default ping packet size used by the command line ping command (Windows=32 bytes, UNIX=64 bytes). 8

9 Resolution Ensure that the ping test performed via the command line utilizes the correct 100 byte ping packet size as utilized by the ehealth poller process. Additionally, the ping packet size utilized by ehealth can be modified via the NH_POLL_PING_PKTSIZE variable. If the command line ping using the correct ping packet size is successful, then network restrictions are likely not the cause for the failure and network latency or other factors may be contributing to a timeout issue. Routers for example may give lower priority to ping requests which can result in delayed responses when the device is busy. Increasing the value of the NH_POLL_PING_TIMEOUT environment variable will increase the wait time of the poller before timing out the request. The default value for the NH_POLL_PING_TIMEOUT variable is 1 second. Alternately the value of the NH_POLL_PING_RETRIES environment variable could be increased. This variable controls how many times ehealth will attempt to ping a device. The default value for the NH_POLL_PING_RETRIES variable is 4. NOTE: Each time ehealth retries a failed poll attempt the timeout period is, by default, doubled. When the device is queried ehealth will wait 1 second before sending another query. On the second query ehealth will wait 2 seconds. The third query will wait 4 seconds and the final 8 seconds. This means if this device fails to respond the total time elapsed from the first to last attempts would be 16 seconds ( ). Based on this modifying the timeout and retries variables can have a dramatic affect on the performance of the ehealth statistics poller. If the device does not respond to manual pings from the command line, then network restrictions may be prohibiting the ping request/response. If that is the case and changes to the network configuration cannot be made to allow for this communication, it is possible to disable the ping portion of each poll. To disable pings set the NH_POLL_PING_DISABLE environment variable to yes. It must be noted that this change will affect all elements polled and cannot be made on a device by device basis. Disabling the poll-ping will cause ehealth to calculate the reachability values based on SNMP responses rather than true device reachability via ping. This change should only be made as a last resort. Please see the ehealth Administration Reference for additional details regarding the available poller ping variables. VII. Resolving No Response to SNMP Error Messages Cause The most common cause of "No response to SNMP" errors during a polling cycle is due to an ehealth timeout waiting for the polled device to respond to the SNMP request. 9

10 Resolution Increasing the value of the NH_SNMP_TIMEOUT environment variable will allow ehealth to wait longer for device responses. The default value for the NH_SNMP_TIMEOUT variable is equal to microseconds (4 seconds). Additionally the value of the NH_SNMP_RETRIES environment variable could be increased. This variable controls how many times ehealth will send an SNMP request to a device. The default value for the NH_SNMP_RETRIES variable is 3. NOTE: By default each time ehealth resends an SNMP request to a device the timeout period is by default doubled. This setting is controlled by the NH_POLL_BS variable. Example: NH_SNMP_TIMEOUT set to NH_POLL_PING_RETRIES set to 5 In the above example an SNMP request is sent to a network device that does not respond. The first time the device is queried ehealth will wait 4 seconds before sending another query. On the second query ehealth will wait 8 seconds. The third query will wait 16 seconds. The fourth 32 seconds, and the fifth will wait 64 seconds. This means if this device fails to respond the total time elapsed form the first attempt through all of the retries would be = 124 seconds. Increasing this variable can have a dramatic affect on the overall poll cycle time and should only be modified incrementally to limit this affect. Additionally, an alternative retry/timeout algorithm can be specified using the NH_POLL_BS variable. The valid values for that variable are: Constant sets a progression of equal timeout periods with the default timeout value of 4 seconds. Linear sets a progression of timeout periods where each subsequent period is incremented by the timeout value itself. For example, three retries yields timeout periods of 4, 8, and 12 seconds. Exponential sets a progression of timeout values where subsequent timeout values are increased by a factor of 2. For example, for three retries, the timeouts are 4, 8, and 16 seconds. Please see the ehealth Administration Reference for additional information regarding these variables. VIII. Resolving Device Exceeded Allowable Timeouts Error Messages 10

11 Cause Resolution A device exceeded allowable timeouts message is caused by multiple elements on a device not responding to ehealth SNMP requests. ehealth will stop sending SNMP requests to a device after a given number of elements associated with that device have not responded to any SNMP requests. Any elements associated with that device that have not yet been polled will not be polled. The device timeout value can be adjusted via the NH_SNMP_DEVICE_TIMEOUTS environmental variable. NH_SNMP_DEVICE_ TIMEOUTS specifies the maximum number of SNMP request timeouts for each device on a given poll before the poller considers it a missed poll and proceeds to the next device. The poller attempts the requests again on all subsequent polls. Device timeouts help reduce the time spent polling devices that are not responding to polls on a consistent basis. Reducing the value may cause missed polls for agents that are slow to respond. Increasing the value will increase poll times in configurations where devices are not responding. Setting this variable to 15, for example, indicates that if any 15 elements on the same agent fail to respond after all retries and timeouts, the poller will give up trying to poll that agent for this poll period. Please see the ehealth Administration Reference for additional information regarding this variable. IX. Resolving Large Delta Error Messages Cause Resolution A large delta error is reported when the delta between the counter value for the last poll and the counter value for the current poll counter reaches or exceeds 50% of its maximum value. For a 32 bit counter the error is reported when this value is 2^31 ( ) or greater. For a 64 bit counter the error is reported when this value is 2^63 ( ) or greater. The four possible causes for this type of message are: 1. Index Shift The index has shifted on the device and ehealth is now polling a port that is incorrect for that element type. Re-discover the device and review the discover log to see if this element is updated. 2. Fast Network Link The speed of the device is so fast (> 10MB) that the counters are wrapping within the time interval of the polling cycle. Switch the polling speed to Fast Polling or change the element s associated agent type to a 64 bit counter if available. 3. The device may not be certified or certified for these specific statistics 11

12 Check to make sure the device is certified on the Concord Web Site. If the device is not certified, you may request certification through the Concord Web Site. 4. MIB Agent Failure The MIB agent on the device in question is decrementing the counter. No counters should ever decrement and therefore this would be considered a bug initiated by the vendor s firmware. To verify a broken (decrementing) counter, run the following commands from $NH_HOME as $NH_USER and output the results to a file: Command syntax version P06 and prior the -s switch is in milliseconds: nhsnmptool -n -c <comm_string> -POLLsa -s o <Object_IDentifier> <ip_address> > <OutPutFileName> & Command syntax version P07 and after the -s switch is in seconds: nhsnmptool -n -c <comm_string> -POLLsa -s 60 -o <Object_IDentifier> <ip_address> > <OutPutFileName> & Note: To determine the application object identifiers utilize the ehealth Element Variable Report From the WEB GUI, Run Reports Tab > AT-A-Glance >Drill Down > Trend > Drill Down > Element Variable Report > MIB Variable From the WEB GUI, Organization Tab > Expand Elements > Select Element > Element Variable Report > MIB Variable Note: If an index number is required, it can be obtained from the ehealth Console > Setup > Poller Configuration. Select the element in question and determine the applicable element index value(s). Note: The index values should be substituted into the object identifier value as follows: $1 Index 1 $2 Index 2 $3 Index 3 $4 Index 4 These commands will output the counter values polled directly from the device s MIB agent documenting the decrementing counters. This information should then be utilized to contact the device manufacturer for a firmware upgrade to correct the issue. Please see the ehealth Administration Guide for more information regarding the nhsnmptool command. 12

13 X. Resolving Generic SNMP Error Messages Cause Resolution Error messages reporting that a particular element received an SNMP error indicate that the device was unable to provide an OID requested in the ehealth SNMP request. The most common causes of this issue are: 1. The device has not been certified by Concord 2. The element in question has had an index shift and is no longer available using the index polled by ehealth To resolve these issues: 1. Verify the device is supported using the Concord Communications certified devices web page ( and re-discover the device after applying the latest device certification patch 2. Re-discover the device to resolve any index shifts that may have occurred XI. Opening a Ticket with Concord Technical Support If the above information does not resolve the specific error messages you are receiving or if you are receiving an error message which is not detailed within this document, the following information is often necessary for Concord Technical Support to sufficiently diagnose the issue. 1. Output of the nhgetsupportinfo utility using the statspoller flag: nhgetsupportinfo type statspoller 2. The CSV file created by the nhssortpollingerrorslite utility 3. The poller messages files 4. An ehealth ASCII Trend report for Good Polls, Bad Polls, Missed Polls, Reachability, and Availability for the timeframe and element in question 5. A full description of the problem and circumstances surrounding the problem such as network changes etc This information should be provided to Concord Technical Support when opening the issue. XII. Other Resources In addition to this document there are many other resources available to the ehealth Administrator to assist in the resolution of statistics polling errors. These include, but are not limited to: 13

14 Standard ehealth Documentation: o o ehealth Administration Guide o ehealth Administration Reference Concord Communications Best Practices Guides, Whitepapers, and Focus Topics: o Discover Best Practices o Data Validation Best Practices Guide o Tuning the Statistics Poller Guide Concord Communications Knowledgebase o 14