Lecture 20: Automated Fault Management

Transcription

1 Lecture 20: Automated Fault Management Prof. Shervin Shirmohammadi SITE, University of Ottawa Prof. Shervin Shirmohammadi CEG Fault Management Fault Management consists of 5 steps: 1. Fault detection 2. Fault location 3. Service restoration 4. Identification of the problem s root 5. Problem resolution Service restoration has a higher priority than 4 and 5, but this is not always possible Steps 4 and 5 can be complex, and is the subject of this lecture After the successful completion of step 4, a trouble ticket will be generated. These steps can be made automatic for many cases, but not always. Prof. Shervin Shirmohammadi CEG

2 Correlation and Filtering Before a failure is registered by an agent, typically some correlation and filtering are done. Correlation: to ensure that if a fault is the result of one or more other faults, the former is not reported. Otherwise too many faults would be unnecessarily reported E.g., a Loss of Connection, would also generate a Loss of Signal, a Degradation in Signal, and perhaps an Excessive Loss fault Filtering: For most faults, they should only be reported if the persist. In other words, if something is very temporary, we do not need to bother the manager. Very temporary must of course be defined for the given context. Filtering is used to catch and suppress these temporary conditions Prof. Shervin Shirmohammadi CEG Example (1/2) ITU-T G.784: Synchronous Digital Hierarchy (SDH) management Related to management of high-speed fiber optics systems The diagram shows the inside of the agent (not all of it though): G.783 G.784 Another document, G.783, specifies the correlation rules, in addition to specifying many other things. Prof. Shervin Shirmohammadi CEG

3 Example correlation rule from G.783: clof dlof and (not dlos) ctim dtim and MON etc Example (2/2) Filter rule from G.784: A transmission failure (fxxx) shall be declared if the fault cause persists continuously for 2.5 ± 0.5 s. The failure shall be cleared if the fault cause is absent continuously for 10 ± 0.5 s. timer expired Wait 2.5 sec fault Fault no fault no fault No Fault fault Wait 10 sec timer expired Prof. Shervin Shirmohammadi CEG Management Correlation Techniques The correlations we saw in the previous slides happen within the agent and are simple. Management correlation, which happens at the manager, is used for steps 4 and 5 of Fault Management to identify a problem and take appropriate action can be more complicated. Typically, one of the following techniques are used for that purpose: Rule-Based Reasoning Model-Based Reasoning Case-Based Reasoning Codebook Correlation Model State Transition Graph Model Finite State Machine Model Prof. Shervin Shirmohammadi CEG

4 Rule Based Reasoning (1/2) The earliest correlation technique, a.k.a. expert system. Used in HP OpenView. Knowledge base consists of 1) problem definition and 2) action to be taken In the form of if-then or conditionaction Working memory contains the actual topology and state information of the network. When something happens, the Inference engine compares it against the knowledge base and picks the closest match and takes appropriate action on the working memory, as pre-defined for that rule. Prof. Shervin Shirmohammadi CEG Rule Based Reasoning (2/2) Rules are established from the expertise of people in the field. If a rule does not exist, it can be added. But this increases the size of the knowledge base to an unscalable level, causing a knowledge acquisition bottleneck. Also, actions are too specific, causing unwanted behaviour. An adaptation of HP s OpenView is shown in Figure Similar to G.783 correlation Prof. Shervin Shirmohammadi CEG

5 Model Based Reasoning The event correlator is built on an object-oriented model associated with each managed object. Used by Cabletron s Spectrum The correlator in NMS has the model shown in b, which is an objectoriented model of the actual network in a. Each object in the model corresponds to its own managed object. For example, the hub 1 model periodically pings Hub 1 to see if it s alive. If not, it checks with the Router s model to see if the router is alive. If yes, it generates an alarm for Hub 1; otherwise, it doesn t do anything. Prof. Shervin Shirmohammadi CEG Case-Based Reasoning (CBR) Unlike Rule Based Reasoning, where the until of Knowledge is a rule, CBR s unit of knowledge is a case. It is based on the intuition that situations repeat themselves, and that what was done in one case is applicable to another similar case, although not exactly identical. Hence, solutions are adapted to the case. Used in SpectroRx. Adaption techniques: Parameterized adaptation Abstraction/respecialization adaptation Critic-based adaptation Prof. Shervin Shirmohammadi CEG

6 Codebook Correlation Model (CCM) Previous methods are all based on algorithms, and hence do not expertly associate events with problems. In CCM, problems are messages generated by the system and are encoded in sets of alarms that they cause. The correlator than decodes those messages to identify the problems. This is done in two phases: 1. Produce a problem-symptom matrix 2. Compare the stream of alarms with the codebook to identify the problem Prof. Shervin Shirmohammadi CEG CCM Example We can now generate a problem-symptom matrix codebook from the labeled causality graph from figure Usually the number of symptoms exceeds the number of problems, so the matrix is reduced to a correlation matrix which uniquely identifies a problem given specific symptoms. This matrix gives us the correlation graph. Used in System Management ARTS (SMARTS) InCharge. Prof. Shervin Shirmohammadi CEG

7 State Transition Graph Model Can be used instead of or as a complement to other techniques. A state transition graph, similar to a sequence diagram, is followed to detect problems and make decisions. Used in Seagate s NerveCenter Prof. Shervin Shirmohammadi CEG Finite State Machine Models the behaviour of the system as s set of finite number of states, and makes decisions based on this model. Prof. Shervin Shirmohammadi CEG

8 Monitoring for Event Notification Event: something that occurs in the network that is noteworthy. Most of the time this is a problem or a failure in a network element. Threshold may be set on end-to-end or element characteristics for notification of events. This is know as real time analysis. Real time analysis usually involves short polling intervals: capacity, CPU, memory, storage needed. Traffic not insignificant! Rule of thumb: This should not be more than 10% of the capacity. For Ethernet keep this at 2-5%. Prof. Shervin Shirmohammadi CEG Traffic Example A network has 100 routers, each with 4 interfaces, each with 8 characteristics. Polling is every 5 seconds. How much is the monitoring overhead traffic? 100 network elements x 4 interfaces / network device x 8 characteristics = 3200 characteristics. Assume Each characteristic = 8 bytes of data + 60 bytes of overhead. (why so much overhead?) Total traffic = 3200 x (8 + 60) = KB = 1.74 Mb If we poll every 5 seconds ~ 1.74Mb / 5s = 384 Kbps. (not likely!) More likely it is a bursty rush of 1.74 Mbps every 5 seconds. Over a period of a day we have: 1.74Mbps * 720 polls per hour* 24 hours per day =30 Gb of traffic and we have 3200 * 8 * 720 * 24 = 442 MB of data are stored per day. Prof. Shervin Shirmohammadi CEG

9 In-Band and Out-of-Band (1/2) In-band: having the NM data flow over the same network that the user network traffic uses Simple network management architecture In case of network problems monitoring and troubleshooting may be difficult Out-of-band monitoring: providing different paths for NM traffic and user network traffic ISDN D-channel Separate Frame Relay/ATM virtual circuit Telephone lines Prof. Shervin Shirmohammadi CEG In-Band and Out-of-Band (2/2) In-band cons: troubleshooting is adversely affect if data flows are delayed or blocked, which can happen during trouble times. Event monitoring when the network is under stress, such as during congestion, can also be impacted negatively. Out-of-band cons: Extra equipment and networking resources are needed. Speed of monitoring might not be the same as the speed of the actual network (specially if costs were reduced in the installation of the management network) A separate method to check availability of the management network is needed Compromise: Hybrid approach. Prof. Shervin Shirmohammadi CEG