NSC , Cisco Systems, Inc. All rights reserved. 1

Transcription

1

2

3 Understanding Fault Management Session

4 Do You Know What Went Wrong? Did a router fail? Did several routers fail? Was there a fiber cut? Did the path to the network management station fail?

5 A Few Things to Remember Faults happen Redundancy only works if you notice the first failure and fix it before the second failure It s easier to resolve problems before they affect your customers You won t know what went wrong unless you have appropriate instrumentation

6 We re Trying to Avoid this

7 What Is a Fault? A fault is an unplanned failure of software, hardware Not every outage is due to a fault Not every fault results in an outage

8 Sources Routers, switches, wires, fiber, PDUs, DNS servers, power companies, floods, soft drinks, clumsy people, squirrels, nuclear accelerators,.

9 Fault Tolerance DNS Internet WWW WWW DNS

10 Agenda Fault Tuning and Process Cycles Fault Management Tuning Fault Management Process

11 Fault Management Tuning Cycle Fault Management Tuning

12 Baselining What s Nominal? Take a baseline of your network when times are good Include: Network and wiring diagrams Device configurations Performance metrics Service level agreements (SLAs)

13 NMS Placement WAN Avoid Flying Blind

14 NMS Placement WAN Cadillac Solution

15 The Cadillac Solution Monitoring doesn t impact your live network bandwidth Same CPU cost Costs more money Screws up auto-discovery Keep Management Net Routing Simple!

16 Use HA Principles for Network Management! Have redundant connectivity all the way to the NMS Have multiple DNS servers near by Ditto for NTP servers Have sufficient bandwidth to poll (when needed) Have reliable AAA; if you can t get in, you can t fix the problem

17 Fault Management Tuning Cycle Fault Management Tuning

18 Setup Alarms and Thresholds Alarms may be SNMP notifications, Syslog events, or events triggered from polling Many applications include default thresholds Defaults may not be the best for your network Each network is unique Check which alarms are covered by built-in notifications or events Some polling may be necessary to check other thresholds

19 Management Domain Service Level and QoS Management Users Performance Management Fault Management Devices and Software

20 Architectural View of a Router CLI SNMP SYSLOG Higher Level Services Serial Fast Ethernet ATM OC-48 DS3

21 Basic SNMP Objects Defined in Management Information Base (MIB) Extensible tree-like structure Data content is self describing ASN.1 subset Semantics of each object is predefined

22 MIBs Object Identifiers SNMP AGENT ISO (1) Organization (3) DOD (6) Internet (1) Hierarchically structured Each object uniquely identified Directory (1) Management (2) Experimental (3) Private (4) ID for System System (1) MIB-2 (1) TCP (6) Proteon (1) Enterprise (1) Sun (42) Interfaces (2) UDP (7) IBM (2) Apple (63) Address Translation (3) IP (4) ICMP (5) EGP (8) CMOT (9) Transmission (10) SNMP (11) Cisco (9) HP (11) Wellfleet (18) Microsoft (311). Unassigned (9118) ternet Activities Board (IAB) Administered Vendor Administered

23 What s in a MIB? Mnemonic Type How to Encode and Interpret This Variable sysuptime OBJECT-TYPE SYNTAX TimeTicks MAX-ACCESS read-only STATUS current DESCRIPTION "The time (in hundredths of a second) since the network management portion of the system was last re-initialized." ::= { system 3 } Parent OID

24 What s in a MIB? (Cont.) Mnemonic Type VarBinds Included in Notification linkdown NOTIFICATION-TYPE OBJECTS { ifindex, ifadminstatus, ifoperstatus } STATUS current DESCRIPTION "A linkdown trap signifies that the SNMPv2 entity, acting in an agent role, has detected that the ifoperstatus object for one of its communication links is about to transition into the down state." ::= { snmptraps 3 } Parent OID

25 Different Types in MIBs Counter32, Counter64 TimeTicks (time) Gauge32 DisplayString (usually with a size limit) Tables with indices

26 Which Objects Does Cisco Support? Mandatory: MIB-II and IF-MIB For IOS For all other OSes Describes which products support which MIBs Also has the MIBs themselves

27 Looking for Notifications in MIBs Three ways: Check the documentation Spot it in your NMS Configure a device for all notifications and trigger an event

28 Downloading MIBs and Related Files in Bulk ftp://ftp.cisco.com/pub/mibs Consider downloading the whole OID directory for quick searches

29 Polling vs. Notifications Polling puts more load on: Network devices Network links Network management stations

30 Polling vs. Notifications Polling needed for: Availability monitoring Utilization and forecasting Polling not needed for: Operational monitoring Fault monitoring (in many cases)

31 Polling vs. Notifications Notifications puts more load on: Network engineer To do it well, requires careful setting of thresholds Requires configuration on all devices

32 Difference between Ping and SNMP Ping Session/ Application Transport Internet RMON2, ART, RTTMON TCP, Multicast MIBs MIB-2, BGP4 SNMP MIBs Entity, Event, Expression L2 IF, EtherLike, RMON

33 Cost of Polling Assume: Query and response per object X each device Every N seconds (Yes, some commercial systems do this)

34 More Math Checking interface status of 20 interfaces = 20 queries X 100 devices = 2000 queries Basic query for ifinoctets = ~87 byte query and 90 byte response

35 Cost of Queries IfInOctets IfOutOctets Both in one packet Sum in two packets Querying 4 Variables in one packet Querying 4 in Separate packets One Ten Hundred Bytes to Bits: 277,600 And Those Are Interfaces

36 Cost of Notifications No queries But you may need to poll for other reasons (performance metrics) SMART polling engines can really make the difference!

37 SNMP Trap Reliability SNMP Traps Are Unreliable True Use v2c Informs!

38 Traps and Informs Trap Inform Acknowledgement

39 Configuring Notifications Basic command snmp-server enable traps [notification type [options]] Types include: bgp, config, entity, frame-relay, isdn, rtr, snmp snmp-server host host-addr [traps informs] [version {1 2c 3 [auth noauth priv]}] community-string [udp-port port] [notification-type] snmp-server informs [retries retries] [timeout seconds] [pending pending]

40 Configuring Informs To enable bgp informs to host using SNMPv2 and community string public configure: snmp-server enable traps bgp snmp-server host informs version 2c public bgp snmp-server informs command not needed unless tuning is required Informs Require ENTERPRISE-PLUS Image

41 Checking Your Informs Configuration Check your notification receiver log Show snmp pending Show snmp sessions

42 SNMP Version Differences Version 1 Version 2c Version 3 Informs No Yes Yes RMON/Event Yes Yes Yes Authentication Community Community Users Privacy No No Yes IOS/CATOS Supported Supported Supported NMS Support Ubiquitous Pretty Good Limited

43 How Useful Is Notification-Based Polling? There Are So Few SNMP Notifications! There Are as Many Notifications as There Are MIB Variables!

44 Notifications How to setup Notifications RMON Events/Alarms Available in IOS since 11.1 Available in CatOS always Event MIB Available in IOS since 12.1(3)T Customize objects for use in Notifications Expression MIB Available in IOS since 12.0(5)T

45 Set a Threshold RMON alarms Command line accessible EVENT-MIB Has wildcards, better variety of tests EXPRESSION-MIB Create new objects that can be tested System Polls Itself!

46 Example MIB to create Notification from CISCO-PROCESS-MIB SNMP version of show process Provides total CPU usage as well as perprocess usage Now, let s create a new notification based upon total CPU utilization

47 Checking CPU Using RMON RMON alarm enterprises absolute rising-threshold 90 1 falling-threshold 70 Tests the OID (CPU utilization found in CISCO-PROCESS-MIB) every 60 seconds If the absolute value exceeds 90% fire event 1 Reset the alarm when the value falls below 70% Equivalent of HPOV Threshold Events but no polling!

48 Event MIB More flexible than RMON events/alarms RMON is tailored for use with Counter objects Can be used to test objects on other devices No CLI yet need to use SNMP or tool

49 Event MIB Based on IETF draft and numbered in Cisco s namespace from 12.1(3)T through 12.2T RFC 2981 implemented in 12.2(4)T Allows you to create custom notifications and log them and/or send them as SNMP traps or informs Persistence added in 12.2(4)T (same time as RFC)

50 Event MIB Show commands show management event Debug commands debug management event mib No CLI configuration support yet

51 Expression MIB Based on IETF draft and numbered in Cisco s namespace, not RFC 2982 Work is being done to implement RFC Allows you to create new SNMP objects based upon formulas Available in IOS since 12.0(5)T, added delta and wildcard support in 12.1(3)T Persistence added in 12.2(4)T

52 Expression MIB Delta and wildcard support allows, for example: Calculating utilization for all interfaces with one expression This will allow you to recreate the locifinoctets and locifoutoctets objects in a standards-based manner With more flexibility Calculating errors as a percentage of traffic

53 Expression MIB Show commands show management expression Debug commands debug management expression? Evaluator Expression MIB evaluator Mib Expression MIB SNMP operations Parser Expression MIB parsing No CLI configuration support yet

54 SNMP Security SNMP Is Insecure True (with v1 and v2c Communities) And False (with v3 Users) V3 Users and Passwords NOT Stored in Normal Config

55 In Cisco IOS The simple way: snmp-server community PASSWORD [ro rw] [ACL] Allows access to all objects on the router for a given ACL No ACL is probably not wise unless interface level ACLs (much more expensive) or other access-limiting measures are taken

56 Security SNMP should only be accessible to NMS Use ACLs where appropriate Use SNMPv3 where available

57 Silliness The entire MIB tree can be retrieved from a device Doing so WILL cause problems This can be prevented with views

58 Silliness snmpwalk of iproutetable Snmp-server View Enabled Cisco 2621 w/64mb RAM and 4000 Routes snmpwalk Would Have Run for 25 ½ Minutes Unrestricted

59 SNMP Views enterprises rttmon interfaces bgp iproutetable

60 SNMP Views enterprises rttmon interfaces bgp iproutetable

61 SNMP Views Cisco IOS command: snmp-server view VIEWNAME OID-tree [include exclude] Followed by snmp-server community PASSWORD view VIEWNAME [ro rw] This prevents a runaway tree walk, and also limits access to necessary objects Consider different community strings with different views for different applications

62 SAA and CISCO-RTTMON-MIB Service assurance agent Session level probe Generates availability and threshold traps Also collects statistics

63 SA Agent Today Increasing Service Value HTTP FTP Connect DLSw QoS Support (ToS) TCP Jitter DNS/ DHCP UDP Echo SNA Path Echo ICMP Cisco IOS-Based Service Assurance* Agent Echo Path Jitter APM ATM* Frame Relay MPLS VPN Aware *With Cisco IOS 12.2(9)T

64 IOS 12.0(5)T and 12.0(8)S HTTP Operation Operation Mode for HTTP Transaction time: Get-Mode Minimal parameters required; Only the base html page will be retrieved, not the links contained in the page Raw-Mode User specifies entire http request (optional payload) It allows for security options, proxy servers, web caching Each Mode Measures DNS Lookup Time TCP Connect Time HTTP Transaction time Uniform Resource Locator:

65 Configuring SAA (config)# rtr 1 (config-rtr)# type http operation get url (config-rtr)# exit (config)# rtr schedule 1 start-time now (config)#

66 SAA Traps Defined in the CISCO-RTTMON-MIB rttmonconnectionchangenotification rttmontimeoutnotification rttmonthresholdnotification

67 Know Thy Router! DNS WAN SAA get /

68 SAA Shadow Router Shadow router: Simulated CE used to measure data travel time between two devices; reasons: Memory consuming IOS independent MPLS/VPN: one shadow router for several VPN s The loopback IP address is unique and globally known Access to the management IP address via the grey management VPN or via the PE global routing table X parallel links to the PE Note: BGP doesn t need to run in the shadow router, static route with redistribution on the PE side is enough

69 VPNSC SA Agent Management Green VPN Site 1 SA Agent Green Probe Traffic SA Agent Blue Probe Traffic Green VPN Connectivity Blue VPN Connectivity Green VPN Site 2 CE Shadow Router PE SA Agent VPN Connectivity to Green and Red VRF s Provider Core PE Shadow Router CE Red VPN Site 1 SA Agent SA Agent Green Probe Traffic SA Agent Blue Probe Traffic SA Agent CE PE PE/P PE Netflow Collector SA Agent CE Non-MPLS/VPN MCE Router Management VPN Blue VPN Site 2 CE Blue VPN Site 1 VPN SC Management Network SA Agent

70 What Happens if a MIB Doesn t Exist? Information may be found using Syslog Syslog produces (mostly) structured logs of information Syslog receivers/processors include: Unix syslogd Cisco Info Center CNS Notification Engine CW2K RME Syslog Analyzer

71 Syslog Very basic reporting mechanism Very basic standard Text messages on UDP port 514

72 Syslog Problems It s not reliable It s not secure Not much worse than SNMP traps One way: No query capability Priority isn t consistently used

73 Syslog Format <selector> NETASCII \0 <facility>.<level> user.info nms-host /usr 90% full <selector> is not retained in the syslog message file Timestamp is added by logging host Logged message looks like: Apr 18 12:02:34 nms-host /usr 90% full

74 There Is a Cisco IOS Message Standard for Syslog %FACILITY-SUBFACILITY-SEVERITY- MNEMONIC: Message-text Documentation for each release explains the meaning of many of these events Facility is not the same as the syslog facility syslog facility is set by configuring: logging facility<facility> syslog facility defaults to local7 Severity maps to syslog level

75 Configuring Syslog Logging command Sets host, facility, rate limit, and more

76 What If There Is No Syslog Event? Screen Scraping Telnet/expect scripts sometimes can be the only way to get information

77 Fault Management Tuning Cycle Fault Management Tuning

78 Does Your Fault Console Look Like This?

79 Monitor Excessive or Missed Faults Adjust thresholds as necessary to catch all faults (no false-negatives) but still have a tolerable false-positive rate Iterative process The network administrator s job is never done

80 The Nice Picture Fault Correlation Workflow Management Fault Tracking DB Trouble Ticket

81 Fault Tuning and Process Cycles Fault Management Tuning Fault Management Process

82 Fault Process Cycle Fault Management Process

83 The Fault

84 Fault Process Cycle Fault Management Process

85 Knowing There s a Problem Event reporting mechanisms SNMP Traps/Informs SYSLOG Polling Help Desk Collect to central repository

86 SAA Example DNS WAN SAA Trap get / ( )

87 Going Up the Stack and on to Resolution Session/ Application Transport RTTMON SNMP Internet L2 Onward to Fault Correlation

88 What to Do When You Get a Trap or Inform? Now s the time to poll! Both the device itself and perhaps neighboring devices! This is known as event-based polling

89 When Did an Event Happen? Having the right time is very important Watch multiple time zones Use NTP

90 Fault Process Cycle Fault Management Process

91 Definition Fault correlation: Tying of two or more events to a single fault or group of faults Fault correlation! = Diagnosis

92 Element-Based Correlation Router B Router A WAN CLOUD Router C Router D linkdown(1) pimneighborloss(b) Router E Router F

93 Network-Based Correlation Router B Router A ATM CLOUD Router C Router D LinkDown(1) pimneighborloss(b) pimneighborloss(c) pimneighborloss(d) pimneighborloss(e) pimneighborloss(f) Router E Router F

94 Correlating Faults and Our SAA Example DNS WAN LinkDown SAA Timeout get/ ( )

95 Bunches of Events WAN Device Duplicates Overload!

96 Remove Duplicates and Correlate WAN Fault Correlation

97 Hierarchical Mechanisms Fault Correlation Fault Correlation Fault Correlation

98 Fault Correlation Tools Rule-based systems that determine that two or more faults are related Good ones classify and prioritize as well Ciscoworks Device Fault Manager Device level only Cisco Information Center CNS Notification Engine

99 Fault Process Cycle Fault Management Process

100 Two Ways to Diagnose Problems Smart computers Smart humans Sometimes both See troubleshooting course

101 Root Cause Analysis WAN

102 Root Cause Analysis WAN

103 Fault Process Cycle Fault Management Process

104 Root Cause Analysis Network-based mechanisms require some knowledge of topology Deals mostly with hard failures Usually limited to some subset of the network

105 Who Fixes Those Problems? Network managers don t know everything Workflow management is the process of delegating to the right person to fix the right problem in a timely fashion

106 Workflow Management Tools to track the state of network Used for escalation, division of labor, provisioning, and planning Examples include remedy and clarify

107 How Important Is That Fault? Importance High Moderate None Time to Repair

108 Work Flow Management Importance High Moderate Low IP Management Web Voice DBA Function

109 Helping the NOC Importance NOW! High Moderate Low IP Management Web Voice DBA Fault Correlation Maintenance Window Function

110 Multiple Sources and Prioritizing warmstart High Management Fault Correlation authfailure Trap Moderate IP Web Voice DBA Low Function Disk Space 90%

111 Fault Process Cycle Fault Management Process Nominal at Last!

112 Summary Move away from the fishbowl approach Aim to tackle faults before they are visible to your users Know how to monitor each service Notification-based polling scales! Fault Management should significantly improve network availability

113 Recommended Reading Performance and Fault Management ISBN: Available on-site at the Cisco Company Store

114 Please Complete Your Evaluation Form Session Understanding Fault Management

115