Slide 1. Best Practices. Copyright 2003 Concord Communications, Inc. All Rights Reserved. EDU Module Name Here Slide 1

Transcription

1 Slide 1 Best Practices Copyright 2003 Concord Communications, Inc. All Rights Reserved. EDU Slide 1

2 Slide 2 Slide 2 Proprietary Notice The information and descriptions contained herein are the property of Concord Communications, Inc. ( Concord ). Such information and descriptions may not be copied, modified, distributed or used in any manner without the express written consent of Concord. Concord assumes no responsibility for any inaccuracies that may appear in this document. Concord reserves the right to change its products, documentation, literature or training materials at any time without notice. U. S. Government Restricted Rights Use, reproduction, and disclosure by the U.S. Government are subject to the restrictions set forth in FAR (c) (1) and (2) and DFARS (c) (1) (ii). Copyright by Concord Communications, Inc. All Rights Reserved.

3 Slide 3 Table of Contents Slide 3 Table of Contents Introduction Session Objectives Main Topics Daily Status Check Daily Activities: Poller Status Daily Activities: Change Control Daily Activities: Error Message Daily Activities: Manual Discovery Daily Activities: Data Collection Weekly Activities Monthly Activities Database: Backups Nightly File Save Recovery From Catastrophic Failure Recovery From Catastrophic Failure (continued) Discovery Practices: The Order Discovery Practices: Device Changes Discovery Practices: Use Logs Discovery Problems Discovery Problems (continued) Poller Errors: Large Delta Large Delta Errors Large Delta Errors: Getting the OID Large Delta Errors (continued) Large Delta Errors Assistance ehealth Poller Ping ehealth Poller: No Response to Ping

4 ehealth Poller: Ping Variables ehealth Poller Errors: Ping Blocked ehealth Poller Errors: No Response to SNMP ehealth Poller: Timeout Variable ehealth Element Variables ehealth Element Variables: Validating Data

5 Slide 4 Purpose To provide information that will guide you in implementing the best ehealth practices and operating procedures for minimizing interruptions and maintaining quality data for reporting. Slide 4

6 Slide 5 Workshop Objectives Upon successful completion you will be able to: Implement the Daily recommended practices Implement the Weekly recommended practices Describe the backup strategy List the recommended file backups Outline Catastrophic recovery procedures Resolve Large Delta errors Describe the recommended Discovery techniques Diagnose and resolve Poller ping timeouts Diagnose and resolve SNMP timeouts Validate ehealth element variables Slide 5

7 Slide 6 Main Topics Daily Status Checks Daily Activities Weekly Activities Monthly Activities Database: Backups Nightly File Saves Catastrophic Recovery Discovery Practices Poller Errors ehealth Element Variables Slide 6

8 Slide 7 Daily Status Check Check all ehealth system log files for errors. Files are located in $NH_HOME/log/ Review the following Files: Statistics_Rollup log. Data_Analysis log. Report logs Database_Save log file. HINT: You can easily search multiple logs the $NH_HOME/log directory using grep i failed * **Please physically check to make certain that the DB save completed successfully.** Slide 7 By checking your log files daily, you can resolve many problems before they seriously impact ehealth daily operations. Check the files for errors, warnings, and failed. Check the report logs for errors and no data. SystemEdge can be configured for Log file monitoring, and automatically alert you via e- mail or SNMP trap when a regular expression is matched.

9 Slide 8 Daily Activities: Poller Status Check the Statistics Poller Status Determine the causes of Bad Polls Review Scheduled Report Only Discovery files Note: You should never schedule ehealth to automatically save any discoveries. You should always review any changes that you save to the ehealth system to insure you are getting exactly the information you want. Slide 8 Review all scheduled report-only scheduled discovery files. If none are currently scheduled, then schedule auto-discovery using seed files. These files are best when focused on a set of common elements, routers/switches or WAN elements. These reportonly scheduled discoveries can be used to locate and discover new devices that can manually be added to the poller configuration. These report-only scheduled discoveries can be used to identify elements that have changed based on change management, or replaced hardware or new system identity.

10 Slide 9 Daily Activities: Change Control Change Control Review Change Management Documentation. The best way to be successful with change control is to have strong ties to change management in your organization. You need to be notified when anyone is going to make changes to a system monitored by ehealth. This will allow you to be proactive in managing changes as they come along. Slide 9 You should also schedule to rediscover your network every week to maximize change control for issues such as index shifting and to keep ehealth running at top performance. To do this schedule a section of your network to be discovered every night. For example you can discover all Routers on Monday then all WAN elements on Tuesday. If you need assistance on how to set up your scheduled discoveries you should speak to either your Concord Sales Engineer or Concord Technical support. Note: Do not automatically save scheduled discoveries without first reviewing them for possible duplicates. You risk losing the historical thread of data to report on any elements that have unresolved changes.

11 Slide 10 Daily Activities: Error Messages Review and reconcile the error messages in the poller log file: $NH_HOME/log/pollerStatus/messages.stats.log Slide 10 Parse the log using grep -i separating the elements into groups of poller failures. This makes it easy to take the appropriate actions to resolve poller issues. Common poller issues are: Invalid community strings Incorrect SNMP access rights No response to ping No response Received and SNMP error (the request referenced an incorrect index or an unsupported MIB) Large packets

12 Slide 11 Daily Activity: Manual Discovery Implement Manual Discoveries based on: change management new device groups. reconciled elements (In the $NH_HOME/pollerStatus/messages.stats.log file) Implement occur after normal or high system traffic hours. Slide 11 The traffic associated with the discovery process includes ARPs, Pings, SNMP requests and SNMP responses. Discovering some elements (System) can generate more than 100 small ethernet frames

13 Slide 12 Daily Activity: Data Collection Identify elements that have: no collected or archived data in the last 24 hours by using the following commands: nhlistelements -nodbdatafor 24 nhelementstatus nodbdatafor 1440 errors with the following command: nhelementstatus -showerrors Slide 12

14 Slide 13 Weekly Activity Check the logs for weekly generated reports Implement any new groups or add new elements reported to ehealth administration via the Change Management process. Slide 13

15 Slide 14 Monthly Activities Review the Concord Communications web site for: New ehealth Device Certification Software release patches. (Notification is sent via to all registered users for any Patch or Certification release) Review all scheduled reports: Are they needed? Are they being used? Maintain the ehealth web users profiles and accounts. Slide 14

16 Slide 15 Database: Backups Backup Database Regularly: Schedule full Database backups Determine Backup interval Have latest backup on disk NOTE: The database procedures will run faster if the ehealth poller has been stopped. Slide 15 The preferred method to preserve the integrity of the ehealth database, regular Database Saves should be scheduled via the ehealth Scheduler. The interval can be determined by how much tolerance there is for lost data, or a gap in the historical data. Identify a disk and partition that has ample available space to store the database backup, and follow desired method the instructions are outlined in the user guide.

17 Slide 16 Nightly File Saves The following UNIX directories and files should be saved nightly: $NH_HOME = /export/home/ehealth $II_DATABSE/idb $NH_HOME/alias.tcl $NH_HOME/idb $NH_HOME/log $NH_HOME/lmgr/license.dat file $NH_HOME/poller/poller.cfg file $NH_HOME/output directory $NH_HOME/reports $NH_HOME/web directory /export/home1/file_save/ directory Slide 16 There are many working files that are frequently referenced and updated. Should one of these files become corrupted it could have a catastrophic effect on (hose) the system. If this should happen, replacing the corrupted file with a fresh backed-up copy is preferable than a full restore.

18 Slide 17 Catastrophic Failure Recovery Procedures On rare occasions there is a catastrophic failure due to hardware issues. The recommended recovery procedure is: 1) Reinstall the ehealth software from distribution media 2) Load the latest version of Software and Device Certification patches. 3) Stop all ehealth server processes; use $NH_HOME/bin/nhServer stop. 4) Stop the ehealth license manager (FlexLM); use $NH_HOME/bin/nhLmgr stop. 5) Move the following directories and files to.orig-dist a. $NH_HOME/custom b. $NH_HOME/output c. $NH_HOME/web Slide 17

19 Slide 18 Catastrophic Failure Recovery Procedures 6) Restore the custom, web, and output directories from the backup media onto the newly installed location. 7) Restore the license.dat file from system backup media into $NH_HOME. 8) Start the ehealth license manager (FlexLM); use $NH_HOME/bin/nhLmgr start. 9) Start the ehealth server processes; use $NH_HOME/bin/nhServer start. 10) Stop the ehealth poller, Setup > Poller Controls, click off. 11) Restore the archived database save. 12) Restart the ehealth poller, Setup > Poller Controls, click on Slide 18

20 Slide 19 Discovery Practices: The Order Discover devices in the following order: Router, Systems, and LAN/WAN elements simultaneously (or) Router or Systems first, and LAN/WAN last. Example:! Routers! Systems! LAN/WAN! Process Sets Slide 19 It is always preferable that if you can not discover Routers and LAN/WAN simultaneously, to discover your routers first and your LAN/WAN second. Your router discoveries should create the parent element with all the child elements (polling will be turned off for the child elements). The subsequent LAN/WAN discovery will do one of two things. 1) It will turn polling on for the child elements associated with the parent device. 2) It will give you the option of creating a new LAN/WAN specific device. The reason that ehealth gives you that option is because occasionally we can collect more specific data on a LAN/WAN specific element than we can with an element that needs to be aggregated. Aggregated elements are always defined as MIB2 for aggregation purposes. Stand alone elements can take into account the nature of the element itself. The same applies when discovering

21 Slide 20 Discovery Practices: Device Changes When a Polled device is going to be changed due to upgrade or configuration: Discovery device before the change Resolve all unresolved changes Discover device after the change Resolve one change at a time per device Rediscover after change is saved Slide 20 Discover the device before the upgrade and resolve all unresolved changes; then discover the device again after the upgrade to resolve the element changes that occurred as a result of the upgrade. Whenever possible, make only one change at a time to each device in your network and run discover after each change. If you plan to make several changes to an element (such as changing sysname, IP addresses, or ifdescr values), make one change at a time and discover the device after each change. If you change two or more attributes at the same time, discover might interpret the changes to be an entirely new device rather than updates to an existing device.

22 Slide 21 Discovery Practices: Use Logs After a discovery review the discovery logs before saving the results. If the changed device wasn t automatically updated there is more than one difference Edit old element in poller configuration using attributes of new discovered element and rerun discovery. Slide 21 **After a discovery, always review your discovery logs prior to saving the results! If ehealth is intending to make a new or duplicate element determine what fields are different in the discovery log of the new device in comparison to the old device in the poller configuration. If the device isn t automatically updated there will be more than one difference. You will need to edit the old element in the poller configuration with the attributes of the new element and then rerun the discovery to link the device correctly.

23 Slide 22 Discovery problems Unable to Discovery something Is the device certified for the technology being discovered? Are you licensed for the module in question? Is the element already in the poller.cfg? Does the device respond to ping? Can this particular device/element type be discovered elsewhere in the network? Slide 22 1) Is the device certified for the technology being discovered? Verify device is certified on our web page. < 2) Are you licensed for the module in question? 3) Is the element already in the poller.cfg? You need to check the poller.cfg Determine if element exists in the poller.cfg. If yes, then the element has already been discovered. If no, continue with troubleshooting. 4) Does the device respond to ping? Ping the device from the ehealth server command line. If the device does not respond, the problem lies with the device. If it responds, continue. 5) Can this particular device/element type be discovered elsewhere in the network? If not, proceed to step 7. If yes, continue.

24 Slide 23 Discovery Problems (continued) Unable to Discovery something Do you have the correct community string? Does the device respond to the following; nhsnmptool -c (community string) <-Server when applicable> -t 60 <IP> If none of this works send all files listed below to requesting a new ticket be open. $NH_HOME/log/discoverResults.log NH_HOME/log/discoverInteractive.date.time.log $NH_HOME/poller/poller.cfg Output from nhsnmptool command. nhsnmptool -c <community string> -t60 <IP> Slide 23 6) Do you have the correct community string? If not, correct community string and attempt discover. If yes, continue. Is comm. string read write? If no, please change the permissions and try again. If it is fine, continue. 7) Open a command prompt. Does the device respond to the following; nhsnmptool -c (community string) <-Server when applicable> -t 60 <IP> If the problem item is an interface, run an nhsnmptool -i -n -c<community string> -t 60 <IP>. Is the index present in the output? Is the element administratively up? Are there MIB2 statistics associated with the index in the MIB (such as ifinoctets, ifinucastpkts)? ehealth discovery looks at the interface table and will only discover those interfaces that have ifadminstatus = up and MIB2 statistics associated with them. 9) If none of this works send all files listed below to support@concord.com requesting a new ticket be open. $NH_HOME/log/discoverResults.log NH_HOME/log/discoverInteractive.date.time.log $NH_HOME/poller/poller.cfg Output from nhsnmptool command. nhsnmptool -c <community string> -t60 <IP>

25 Slide 24 Poller Errors: Large Delta Common causes for Large Delta errors during ehealth Polling: Speed: 32bit counters are wrapping causing a Large Delta (large change) errors. Slide 24 There are four major reasons why the ehealth poller reports Large Delta errors in the the Statistics Poller window. 1) SPEED: The speed of the device is so fast that the 32-bit counters are wrapping within the time interval of the polling cycle. To fix this, you would need to open the console and select Setup -> Poller Configuration. Double click on the device in question. This will open the modify window. On the right hand side of the screen change the polling interval to fast. To verify the poller speed, go to Setup -> Poller Controls. The statistics poll rate on the right will reflect the polling speed. Default is one minute. Note: Regardless of the poll rate, statistics data will only be inserted into the database every 5 minutes.

26 Slide 25 Large Delta Errors Common causes for Large Delta Errors during ehealth Polling: Index Shift: Resulting in polling a port that is incorrect for the element type. Decrementing Counter: The MIB agent is decrementing the counter. Slide 25 2) Index Shift The index has shifted on the device and we are now polling a port that is incorrect for that element type. To solve this, rediscover the device. Make sure you edit before saving. If the device has had an index shift the discover log will point it out. Save the discover results and continue polling. 3) The counter is decrementing The MIB agent on the device in question is decrementing the counter. No counters should ever decrement and therefore this would be considered a software bug in the vendor s MIB. To check this, please do the following: a) Open a command line and Log in as the $NH_USER. b) UNIX only: Source the appropriate nethealthrc file in the $NH_HOME directory, depending on the shell you are in: % source nethealthrc.csh (if in C shell) or $../nethealthrc.sh (if in Bourne shell) or $../nethealthrc.ksh (if in Korn shell) Run the following; nhsnmptool c <community string> -o <oid> -s <seconds in between polls> -POLLsa This will poll the device at the specified OID outside of ehealth. It will wait the specified amount of seconds in between polls, and continue to run until killed.

27 Slide 26 Large Delta Errors: Getting the OID Two ways of determining the OID of the variable generating Large Delta Errors: From the WEB GUI, Run Reports Tab > AT-A-Glance >Drill Down >Trend > Drill Down > Element Variable Report > Select > MIB Variable. From the WEB GUI, Organization Tab > Expand Elements > Click on > <ehealth element> Click > Element Variable Report > Select > MIB Variable. Note: If an Index is required it can be obtained from the ehealth console > Setup > Poller Configuration. Slide 26 An example of finding the OID without using ehealth WEB GUI follows: In the following example we will assume that the $NH_HOME/log/pollerStatus/messages.stats.log file indicates an error on ifinoctets and we want to poll it to see why the errors occur. The ** below represents fictitious examples. Go to the console and select Setup -> Poller Configuration. Find the device in question (**Cisco-RH-ethernet0/1) and write down the agent type that is assigned to the element (**MIB2 ) Go into the $NH_HOME/poller directory and open the agent.types file. In this file, you will see the agent type associate with an MTF file (in this case, **MIB2.mtf). Note: The.mtf stands for MIBs Translation File. These files are also located in the $NH_HOME/poller directory. Open the MTF file and you will see a line that says something like file **mib2.mib. This is the name of the.pcm file associated with the MTF. Add a.pcm to this name, and you will see this file in the $NH_HOME/poller directory as well. In the.pcm file is the actual list of OIDs we use in various MTF files. In this case, it is ifinoctets. From the.pcm file: ifinoctets

28 Slide 27 Large Delta Errors: Using the OID Once you have the OID of the MIB variable Use the nhsnmptool Utility to retrieve the data values. If variable is a counter it should not decrement If counter is decrementing contact Manufacturer Slide 27 Using this OID, you can invoke the nhsnmptool utility as shown below, and see the raw values of the variable. If this variable is a counter (like ifinoctets) and not a gauge, then it should never decrement. If it is a counter and the value is decrementing, please contact the device manufacturer to resolve the issue with the MIB nhsnmptool c <community string> -o <oid> -s <seconds in between polls> -POLLsa

29 Slide 28 Large Delta Errors (Continued) The device may not be certified, or it may not be certified for these specific statistics. This can be verified by your support rep or by viewing: Your support rep can also verify the certification status of this vendor s device. Slide 28 The 4 th major cause for Large Delta errors: Device is not properly certified.

30 Slide 29 Large Delta Errors: Assistance Please contact support if you are not able to locate the cause of the problem. It would be helpful to send the following to support: - $NH_HOME/poller/poller.cfg - $NH_HOME/log/discoverResults.log after rediscovering the device (you don t have to save the changes - The discoverinteractive log from $NH_HOME/log for discovery of this device - $NH_HOME/log/system.log - $NH_HOME/log/pollerStatus/messages.stats.log - The output of typing nhshowrev - The output of the nhsnmptool command shown above (which one?) (Specify) nhsnmptool c <community string> -t 30 f mibdump.txt Slide 29

31 Slide 30 ehealth Poller: Ping Poller pings IP address before SNMP requests Used to calculate Latency. Determines Reachability of a device or interface. When no response to a ping is received: It is considered a Missed Poll No SNMP requests are sent Slide 30 The ehealth poller process first pings each IP address before sending out an SNMP request during each poll cycle. We send 4 ping packets to the device, and if there is no response to ping from the device the poll for this element is considered a "Missed Poll", and no SNMP requests are sent. Given this information, it is crucial that the network path between the ehealth server and the device to be polled allow for a ping request/response.

32 Slide 31 ehealth Poller: No Response to Ping When a device response to ping via the command line, but not a ping from ehealth: There may be a restriction on ICMP packet sizes Decrease the value of NH-POLL_PING_PKTSIZE When a device response to ping via the command line from a different machine, but not a ping from ehealth: Delay in the ehealth path to the device Increase the value of NH_POLL_PING_TIMEOUT Slide 31 If the device responds to a manual ping via the command line, but gets no response to ping from ehealth, find out if there are network restrictions on ICMP packet sizes. The default sizes are as follows: Windows: 32 bytes Unix: 64 bytes ehealth: 100 bytes The ehealth size can be changed using NH_POLL_PING_PKTSIZE. Please set this variable to the size (in bytes) for ehealth to use. Also, increasing the value of the NH_POLL_PING_TIMEOUT environment variable will increase the wait time of the poller before timing out. Also some devices such as routers give lower priority to ping requests which can results in delayed responses when the device is busy. Make sure that a router in the path of the ehealth ping has not been configured to block echo request / echo reply (ping).

33 Slide 32 NH_POLL_PING_TIMEOUT Default value = 1 second NH_POLL_PING_RETRIES Default value = 4 ehealth Poller: Ping variables NH_POLL_PING_NUM_PINGS Default value = 3 (ehealth adds 1) total = 4 NOTE: Each time ehealth retries to poll a device the timeout period is doubled. Slide 32 The default value for the NH_POLL_PING_TIMEOUT variable is equal to 1 second. Alternately, the value of the NH_POLL_PING_RETRIES environment variable could be increased. This variable controls how many times Network Health will attempt to ping a device. The default value for the NH_POLL_PING_RETRIES variable is 4. NOTE: Each time ehealth retries to poll a device the timeout period is, by default, doubled. For example if: NH_POLL_PING_TIMEOUT set to 1 NH_POLL_PING_RETRIES set to 5 Then: 1 st poll timeout = (times out after 1 second, no response) 2 nd poll timeout = (times out after 2 seconds, no response) 3 rd poll timeout = (times out after 4 seconds, no response) 4 th poll timeout = (times out after 8 seconds, no response) 5 th poll timeout= (times out after 16 seconds stops polling the device, polls next device.)

34 Slide 33 ehealth poller errors- Ping blocked IF ping (echo request / echo response) is blocked ehealth pinging can be disabled. Disable pinging during a poll cycle by setting the: NH_POLL_PING_DISABLE variable to yes Will apply to all elements being polled Latency will not be calculated Reachability will be based on SNMP responses Slide 33 If a device does not respond to an ehealth ping, and doesn t respond to a ping generated via the command line of the ehealth server, then network restrictions may be in place that prohibit the ping (echo request / echo response). If this is the case and changes to the network configuration cannot be made to allow for pinging, it is possible to disable the ping portion of each poll. To disable pinging, set the NH_POLL_PING_DISABLE environment variable to yes. Keep in mind that this change will affect all elements polled and cannot be made on a device by device basis. Disabling the poll-ping will cause ehealth to calculate the Reachability values based on SNMP responses rather than ping replies. Also ehealth will no longer be able to calculate latency. This change should only be made as a last resort.

35 Slide 34 ehealth Poller errors: No Response to SNMP No response to SNMP error messages caused by: ehealth timing out while waiting for an SNMP response Can be resolved by: Increasing the value of NH_SNMP_TIMEOUT Default value is microseconds (4 seconds) Increasing the value of NH_SNMP_RETRIES Default value is 3 Slide 34 The most common cause of "No response to SNMP" errors during a polling cycle is due to an ehealth timeout waiting for the polled device to respond to the SNMP request. Increasing the value of the NH_SNMP_TIMEOUT environment variable will allow ehealth to wait longer for device responses. The default value for the NH_SNMP_TIMEOUT variable is microseconds (4 seconds). In addition, the value of the NH_SNMP_RETRIES environment variable could be increased. This variable controls how many times ehealth will send an SNMP request to a device. The default value for the NH_SNMP_RETRIES variable is 3. You should be very careful about increasing the value of any of these variables. These variables affects the performance of the ehealth statistics poller. Do not modify this variable unless you have experience with and a very clear understanding of tuning poller performance.

36 Slide 35 ehealth Poller: Timeout Variable Each time ehealth resends an SNMP request to a device the timeout period is doubled by default. The value defined in the NH_POLL_BS environment variable determines the timeout period for SNMP requests The valid values are: Constant Linear (default value) Exponential Slide 35 NH_POLL_BS specifies the back off strategy used to determine timeout intervals for SNMP requests that time out and need to be resent. The valid values are: constant sets a progression of equal timeout periods with the default timeout value of 4 seconds. linear sets a progression of timeout periods where each subsequent period is incremented by the timeout value itself. For example, three retries yields timeout periods of 4, 8, and 12 seconds. exponential sets a progression of timeout values where subsequent timeout values are increased by a factor of 2. For example, for three retries, the timeouts are 4, 8, and 16 seconds. If the NH_SNMP_TIMEOUT value needs to be increased, use constant progression to prevent subsequent retries from having excessive timeout values. The exponential value is not recommended because the timeout periods increase so rapidly.

37 Slide 36 ehealth Element Variables To determine how ehealth calculates the values presented in reports run an Element Variable report From the WEB GUI, Run Reports Tab > Trend > Drill Down > Element Variable Report. (or) From the WEB GUI, Organization Tab > Expand Elements > Click on > <ehealth element> Click > Element Variable Report Slide 36 By default, ehealth does not give you access to this drilldown report. To be able to generate the Element Variable report, the ehealth Web administrator must modify your Web user account configuration through the Modify Account page. The first page of the Element Variable report is typically quite long. It lists all Trend variables that ehealth collects for an element and their fully expanded associated expressions. The second page lists the MIB variables and associated OIDs for each variable.

38 Slide 37 ehealth Element Variables: Validating Data Using this information, you may then query the OID's directly using the SNMP tool utility. To run the SNMP tool, simply do the following: -Log in as the NH_USER -CD to the NH_HOME directory -If on UNIX, source the nethealthrc.csh file. -The command syntax is as follows: nhsnmptool -f <output file> -POLLsa -o <OID in question> -s Slide 37 Let the command run for 15 minutes, then ^c to cancel. Then, run a trend report in ASCII for the same time period, and compare the results of the SNMP tool output to those in the reports. If there are multiple OID's in the calculation, then multiple SNMP tool queries will need to be run simultaneously. Each individual query must be run in a different shell window. Try to start each as close as possible to one another, to ensure the data will be valid.