Monitoring and Alerting in Cisco MDS Fabric

Transcription

1 Monitoring and Alerting in Cisco MDS Fabric 2016 Cisco and/or its affiliates. All rights reserved. 1

2 Contents About This Document 3 Supported Hardware and Software 3 Introduction 4 Cisco Prime Data Center Network Manager 4 Remote Monitoring 4 Overview 4 RMON Alarms 4 RMON Events 5 Configuring RMON 5 Guidelines and Limitations 5 Enabling RMON 5 Setting RMON Alarms 6 Verifying RMON 6 Using RMON to Monitor MIBs 6 Port Monitor 7 Overview 7 Comparison of PMON and RMON 9 Architectural Components 9 Configuring PMON 10 Verifying PMON 11 Configuring PMON Alerting Mechanisms 11 Configuring Alerting Based on Poll Interval 11 Configuring Alerting Based on Check Interval 12 PMON s 14 Cisco Embedded Event Manager 24 Overview 24 Defining Cisco Embedded Event Manager Policy 25 Defining Events and Actions 25 Supervisor Upgrade and Downgrade Support 26 For More Information 26 Appendix A: Remote Monitor Configuration 27 Appendix B: Port Monitor Configuration 27 Normal Template PMON Configuration 28 Aggressive Template PMON Configuration 29 Most Aggressive Template PMON Configuration 30 Appendix C: Cisco Embedded Event Manager Configuration Cisco and/or its affiliates. All rights reserved. 2

3 About This Document This document describes the mechanisms available to monitor Cisco MDS fabric to effectively check SAN fabric health. It discusses important hardware and software counters present in Cisco NX-OS Software and the values that are built into 8- and 16-Gbps Cisco MDS 9000 Family switches. It describes the counters, the techniques employed to monitor them, the monitoring capabilities available, and the configurable threshold limits for the counters. It also provides in-depth discussions of the three main tools available to monitor Cisco MDS fabric: Remote Monitor (RMON), Port Monitor (PMON), and Cisco Embedded Event Manager (EEM). The document discusses how to configure each tool and when to use each one. SAN administrators can use the policies and threshold recommendations provided in this document as guidelines for monitoring their SAN fabrics for overall health and for setting alerts so that they are informed if a particular counter or setting exceeds its preset threshold value. The appendixes at the end of this document provide sample configurations and command-line interface (CLI) commands that can be used to deploy the various monitoring techniques on Cisco MDS 9000 Family switches. Supported Hardware and Software This document supports the hardware platforms and NX-OS software releases listed in Table 1. The features discussed in this document may be available on other platforms or NX-OS releases as well, but no attempt has been made to cover them in this document. This document is based on NX-OS Release 6.2(13) and Cisco Prime Data Center Network Manager (DCNM) Release 7.2(1). Table 1. Supported Hardware and Cisco NX-OS Software Releases Cisco MDS 9000 Family Platforms Cisco MDS 9000 Family 16-Gbps platforms Cisco MDS 9000 Family advanced 8-Gbps platforms Cisco MDS 9000 Family 8-Gbps platforms Model Cisco MDS 9700 Series Multilayer Directors with DS-X K9 line card Cisco MDS 9396S 16G Multilayer Fabric Switch Cisco MDS 9148S 16G Multilayer Fabric Switch Cisco MDS 9250i Multilayer Fabric Switch Cisco MDS 9500 Series Multilayer Directors with DS-X92xx-256K9 line cards Cisco MDS 9500 Series Multilayer Directors with MDS 9148 Multilayer Fabric Switch and DS-X K9 and DS-X92xx-96K9 line cards 2016 Cisco and/or its affiliates. All rights reserved. 3

4 Introduction In today s SAN deployments, finding the correlation between a network event and its root cause can be difficult. Without proactively monitoring the SAN fabric and being informed about events as they occur, organizations have difficulty preventing incidents in the network. Incidents may be simple, such as the flapping of one member link of a port channel interface on a Cisco MDS 9000 Family switch, or complex, such as application performance degradation due to a slow-performing device or host bus adapter (HBA) somewhere in the network. Monitoring fabricwide events, various port parameters, Small Form-Factor Pluggable (SFP) transceiver settings, system CPU memory values, and other environmental parameters permits early fault detection and isolation of such network events. It enables the SAN administrator to effectively measure performance on the SAN fabric network. Benefits include: Improved high availability of the network through reduced downtime Effective root-cause analyses of a network event Alerts that are delivered proactively, rather than reactively, from counters and events that are monitored Reduced troubleshooting time through notifications from alerts and traps Cisco MDS 9000 Family switches track a variety of SAN fabric metrics, counters, and events in real time. These counters are made available by the MDS platform for use by native NX-OS applications such as PMON, RMON, and embedded event managers. A few of these counters are exposed in their raw Simple Network Management Protocol (SNMP) object identifier (OID) format for monitoring or are made available through various system policies. SAN administrators can use existing templates and policy scripts (for example, the default PMON policy) or write custom policy scripts that can be applied to their SAN fabric switches to help improve monitoring, alerting, and troubleshooting for their SAN infrastructure. These system policies, scripts, counters, and tools can be extended to form a monitoring and alerting framework that can be developed and deployed by SAN administrators on their fabric network from day one. Continuous monitoring of fabricwide events, ports, system resources, and environmental parameters enables early fault detection and helps keep the network healthy. Cisco Prime Data Center Network Manager Cisco Prime DCNM is a data center network monitoring tool that monitors events occurring across the network on multiple switches. It can manage both SAN and LAN networks, and it can correlate events across multiple switches and proactively alert users of these fabricwide events. SAN administrators should deploy and use the network manager to gain fabricwide visibility, perform diagnosis and troubleshooting at the fabric level, analyze trending statistics, generate reports, and perform forecasting for the network, among other tasks. Administrators can also use the network manager to poll a specific management information base (MIB) or OID and plot the values in a graph over an extended period of time. With Cisco Prime DCNM Release and later, administrator can choose to deliver only network events of interest and suppress unwanted events and alerts. Remote Monitoring Overview Remote Monitoring, or RMON, is an SNMP IETF standard monitoring specification that allows various network agents and console systems to exchange network monitoring data. NX-OS supports RMON alarms, events, and logs to monitor NX-OS devices. You can use the RMON alarms and events to monitor Cisco MDS 9000 Family switches running NX-OS. All switches in the Cisco MDS 9000 Family support the RMON functions discussed in this section (and defined in RFC 2819). RMON Alarms Each alarm monitors a specific MIB object for a specified interval. When the MIB object value exceeds a specified value (rising threshold), the alarm condition is set and only one event is triggered regardless of how long the condition exists. When the MIB object value falls below a certain value (falling threshold), the alarm condition is cleared. Clearing allows the alarm to be triggered again the next time the rising threshold is crossed Cisco and/or its affiliates. All rights reserved. 4

5 When you create an alarm, you specify the following parameters: MIB object to monitor Sampling interval: The interval that NX-OS uses to collect a sample value of the MIB object Sample type: Absolute or delta -- Absolute samples take the current snapshot of the MIB object value. -- Delta samples take two consecutive samples and calculate the difference between them. Rising threshold: The value at which NX-OS triggers a rising alarm or resets a falling alarm Falling threshold: The value at which NX-OS triggers a falling alarm or resets a rising alarm Event: The action that NX-OS takes when an alarm (rising or falling) is triggered RMON Events You can associate a particular event with each RMON alarm. RMON supports the following event types SNMP notification: Sends an SNMP risingalarm or fallingalarm notification when the associated alarm is triggered Log: Adds an entry in the RMON log table when the associated alarm is triggered Both: Sends an SNMP notification and adds an entry in the RMON log table when the associated alarm is triggered You can specify different events for falling alarms and rising alarms. Configuring RMON RMON is disabled by default, and no events or alarms are configured in the switch. You can configure your RMON alarms and events by using the CLI or an SNMP-compatible network management station. Guidelines and Limitations The Cisco MDS 9000 Family MIB files can be obtained through FTP from sw-center/netmgmt/cmtk/mibs.shtml, under Cisco Storage Networking. Cisco and IETF MIBs are updated frequently. You should download the latest MIBs from cmtk/mibs.shtml whenever you upgrade your Cisco MDS 9000 NX-OS Software. RMON has the following limitations: You must configure an SNMP user and a notification receiver to use the SNMP notification event type. You can configure an RMON alarm only on a MIB object that resolves to an integer. When you configure an RMON alarm, the object identifier must be complete, with its index, so that it refers to only one object. For example, corresponds to cpmcputotal5minrev, and.1 corresponds to index cpmcputotalindex, which creates object identifier Enabling RMON RMON is disabled by default. To enable it, use the following commands: MDS9000# config t MDS9000(conf t)# snmp-server enable traps rmon 2016 Cisco and/or its affiliates. All rights reserved. 5

6 Setting RMON Alarms RMON alarms help configure rising- and falling-threshold values for a specific MIB object using OIDs. If the OID that is configured is related to port counters, then the monitoring will apply to the entire port set available in the system. The following example shows configuration of RMON alarm number 20 to monitor the SNMP MIB object , where represents the ifinoctets value and represents the ifindex value of the physical port. Monitoring occurs once every 60 seconds until the alarm is disabled and checks the change in the variable s rise or fall. If the value shows a MIB counter increase of 15 or more, the software triggers an alarm. The alarm in turn triggers event number 1, which is configured with the RMON event command. Possible events include a log entry or an SNMP trap. If the MIB value changes by 0, the alarm is reset and can be triggered again. switch(config)# rmon alarm delta risingthreshold 15 1 falling-threshold 0 owner test RMON is a software process that runs on a supervisor module. There is no limit on the number of OIDs that can be configured for monitoring purpose. The lowest frequency for polling an SNMP counter is 1 second. For example, if you want to monitor the CPU use percentage rise with a threshold of 80 percent, you can set the polling frequency to once every 30 seconds. Verifying RMON Use the show rmon and show snmp commands to display the configured RMON and SNMP information. Using RMON to Monitor MIBs RMON can monitor all the MIBs that the platform supports. The MIB reference guide for Cisco MDS 9000 Family switches lists all the available MIBs that can be used for managing SAN fabric. Table 2 lists common MIBs that the network administrator can use to monitor the SAN fabric. Table 3 lists MIBs for monitoring CPU and memory utilization and their recommended thresholds. Table 2. Common MIB OIDs for Monitoring on Cisco MDS 9000 Family Switches Serial Number MIB OID 1 CISCO-CFS-MIB Provides global-level control over the features in the system that support Cisco Fabric Services x 2 CISCO-ENTITY-FRU- CONTROL-MIB Manages field-replaceable units (FRUs), such as power supplies, fans, and modules x 3 CISCO-ENTITY-SENSOR-MIB Provides sensor data for environmental monitors such as temperature gauges 4 CISCO-FSPF-MIB Configures and monitors the Fabric Shortest Path First (FSPF) parameters on all VSANs configured on the local switch 5 CISCO-LICENSE-MGR-MIB Manages license files on the switch x 2016 Cisco and/or its affiliates. All rights reserved. 6

7 Serial Number MIB OID 6 CISCO-SYSLOG-MIB Collects system messages generated by switch (These messages are typically sent to a syslog server. Use this MIB to retrieve SYSLOG-MIB system messages.) 7 CISCO-VSAN-MIB Enables you to create and monitor VSANs 8 CISCO-PROCESS-MIB Displays memory and process utilization 9 CISCO-ZS-MIB Provides notifications related to zone changes in Cisco MDS 9000 Family switches x x Table 3. CPU and Memory Utilization MIBs Category OID Poll (seconds) Rising Falling CPU CPU utilization Normal Aggressive Memory Memory utilization Port Monitor Overview Normal Aggressive Port Monitor, or PMON, is a monitoring feature available on Cisco MDS 9000 Family switches that can monitor physical port related critical counters and alert external monitoring software of any anomalies. The configured entities are monitored in hardware, thereby saving CPU cycles on the supervisor. The software component of PMON relies on RMON capabilities, which in turn rely on the SNMP component to generate threshold messages whenever a monitored entity rises or falls beyond the set threshold value. Administrators can optionally use Cisco Prime DCNM software to view these alerts and gain fabricwide control by identifying nodes that are consuming network resources and hampering the flow of data traffic. The PMON feature hides the complexity of SNMP OIDs and provides an easy-to-use CLI. Cisco MDS 9000 Family switches have two predefined PMON policies: default and slowdrain. Administrators can also create other custom policies for monitoring and alerting purposes. The slowdrain policy is automatically active when you launch a Cisco MDS 9000 Family switch. Note that at any given time, only one policy can be active in the system Cisco and/or its affiliates. All rights reserved. 7

8 The default policy monitors the objects as shown in Table 4 for each physical port interface on the Cisco MDS 9000 Family switch. This policy is not active and cannot be modified. Table 4. Default PMON Policy Serial Number Hardware s Interval (seconds) Rising Threshold Falling Threshold 1 Link loss Sync loss Signal loss Invalid words Invalid CRCs LR Rx LR Tx Timeout Discards Credit Loss Reco Tx Credit Not Available 1 10% 0% 11 RX Datarate 60 80% 20% 12 TX Datarate 60 80% 20% 13 TX Discards ASIC Error Pkt from port ASIC Error Pkt from xbar ASIC Error Pkt to xbar Tx Slowport Count Tx Slowport Oper Delay 1 50 ms 0 ms 19 TxWait 1 40% 0% The slowdrain PMON policy monitors two counters: Credit Loss Reco and Tx Credit Not Available, as shown in Table 5. This policy is active by default. The Tx Credit Not Available counter is sampled at 100-ms intervals. A 10 percent rising threshold indicates 10 percent of a second (100 milliseconds [ms]). If there are zero remaining Tx B2B credits for 100 ms, then the rising-threshold criterion will be met. Table 5. Slowdrain PMON Policy Serial Number Hardware s Interval (seconds) Rising Threshold Falling Threshold 1 Credit Loss Reco Tx Credit Not Available 1 10% 0% 2016 Cisco and/or its affiliates. All rights reserved. 8

9 PMON allows end users to further define and modify new custom policies for monitoring access ports or trunk ports, or both, on a Cisco MDS 9000 Family switch. You can also optionally use PMON to errordisable or flap the port if the set threshold is met for a configured counter. Note: Different counters are available depending on the types of Cisco MDS 9000 Family modules in the system. The unit for threshold values (Rising and Falling) differs across different counters. For all platforms if the default value for tx-slowportcount is modified then ISSD will be restricted. To proceed with ISSD, use the no form of the command to roll back to the default value. For all platforms if the default value for tx-slowportoper-delay is modified then ISSD will be restricted. To proceed with ISSD, use the no form of the command to roll back to the default value. For all platforms if the default value for txwait is modified then ISSD will be restricted. To proceed with ISSD, use the no form of the command to roll back to the default value. Comparison of PMON and RMON PMON and RMON can be configured to work in tandem to monitor various parameters on the Cisco MDS 9000 Family switches. For example, you can configure RMON to monitor CPU and memory statistics and use PMON to monitor hardwarespecific counters related to a slow-drain condition. PMON differs from RMON in monitoring port counters in the way that PMON processes events and alarms on the ports. RMON is dependent on SNMP and runs only on supervisor CPUs. It listens to SNMP-based events and traps from various components and relays this information to the external management application. PMON, in contrast, performs most computations on the port application-specific integrated circuit (ASIC) present on the individual modules, and counters exposed by PMON features can be monitored at frequencies of less than 1 second. If a counter exceeds its configured threshold value, an alarm event will be generated by the line-card port ASIC and sent to the supervisor module. This architectural design of PMON frees CPU time on the supervisor, relieving it from having to continuously monitor the PMON counters. The supervisor CPU can instead be used to service other important functions: to service control traffic, run online Cisco Generic Online Diagnostics (GOLD), etc. Architectural Components Figure 1 shows the interaction of PMON components with other software components. Figure 1. PMON Interaction with Various Software Modules Port Monitor Configuration: Type, Interval, Threshold, Event, and Object PMON CLI Configuration (on Supervisor) Configuration List and Database RMON Component (on Supervisor) SNMP Component (on Supervisor) Persistent Storage Broadcast to All Line Cards Cisco Prime DCNM (External Monitoring Application) Port Monitor Agent in LC1 (Line-Card Component) Port Monitor Agent in LC2 (Line-Card Component) 2016 Cisco and/or its affiliates. All rights reserved. 9

10 The PMON component on the supervisor interacts with SNMP and RMON components to alert and send traps to external monitoring software. It also interacts with the persistent storage system (PSS) component, which stores the configuration list and policy details in a database. The PSS helps enable you to retrieve the PMON policy if a switch reloads or fails. When PMON policy is configured on the supervisor, the policy is internally broadcast to all line cards using internal NX-OS messaging. The PMON agent process on the line card is notified and then applies the policy according to the settings configured on all the active ports on the module. Relevant hardware-assisted timers are started to monitor the port counters according to the polling interval specified. If a threshold exceeds its configured value, the PMON process on the line card will inform the SNMP agent on the supervisor of this violation. RMON will also send traps to the external monitoring application, such as Cisco Prime DCNM, that is registered to receive such notifications from the switch. PMON-triggered notifications can also be viewed in RMON by using the show rmon logs and show rmon alarm commands. Configuring PMON Here is a sample configuration to specify the PMON policy, monitor the link-loss counter, and activate the policy. MDS9000-A(config)# port-monitor name port-health-check MDS9000-A(config-port-monitor)# monitor counter link-loss MDS9000-A(config-port-monitor)# counter link-loss poll-interval 60 delta risingthreshold 5 event 4 falling-threshold 1 event 4 portguard errordisable MDS9000-A(config-port-monitor)# port-monitor activate port-health-check The preceding policy configures a PMON policy named port-health-check to continuously monitor the link-loss counter every 60 seconds. The delta value specified in the configuration represents the difference between the value from the current polling and the value from previous polling performed by PMON 60 seconds earlier. If this value is greater than or equal to 5 between two successive polls, then a port-guard action to error-disable the affected port is performed, and an event with the value 4 is generated to RMON. Note: The port types all and access port include all F-ports and all F-port-channel interfaces. Also, the port-guard action to flap or error-disable the port should not be used when monitoring an F-port-channel interface connected to a network port virtualization (NPV) switch. If another PMON policy is already active for the port type in question, then you will first have to deactivate that policy before you can activate a new policy. PMON can generate five levels of events. The levels reflect the importance of this event occurrence. Table 6 shows the event levels. Table 6. PMON Events Event 1 Fatal 2 Critical 3 Error 4 Warning 5 Informational 2016 Cisco and/or its affiliates. All rights reserved. 10

11 When configuring PMON, administrators should be sure to choose the right event ID levels so that meaningful actions can be taken by the administrator or the third-party tool that receives these alerts. Generally, events are classified as follows: An event level 1 alert signifies a general network failure or a situation in which a critical part of your network is unreachable. An event level 2 alert signifies a critical situation in the network that needs immediate attention. For example, such an alert might indicate that one of the modules of the Cisco MDS 9000 Family switch is down, or that the flow of traffic has changed so that traffic that was earlier traversing the SAN A path has been switched to a SAN B path. Event level 3, 4, and 5 alerts signify less critical situations. For example, these alerts may be used to indicate that one of the member links in a port-channel interface has flapped, that the module temperature is rising, or that the count of cyclic redundancy check (CRC) errors on an interface has increased. After the PMON policy is configured, enabled, and activated for monitoring, you can use the show portmonitor port-health-check and show port-monitor active CLI commands to view the various PMON counters and their configured thresholds. To deactivate monitoring for the configured policy, use this command: MDS9000(config)# no port-monitor activate port-health-check Verifying PMON You can use the following commands to check whether any Cisco MDS 9000 Family module has triggered an alarm and whether PMON was able to inform RMON about generating an SNMP event. MDS9000# show port-monitor active MDS9000# show port-monitor < policy name > MDS9000# show port-monitor status MDS9000# show rmon logs Configuring PMON Alerting Mechanisms PMON provides two ways to configure port counters for monitoring and alerting. You can configure monitoring and alerting based on the poll interval or check interval. Configuring Alerting Based on Poll Interval Consider a PMON policy named pmon-test with the counter link-loss configured as shown here: port-monitor name pmon-test port-type all counter link-loss poll-interval 30 delta rising-threshold 20 event 4 fallingthreshold 0 event 4 This policy configures the PMON policy named pmon-test to continuously monitor the link-loss counter every 30 seconds. Rising and falling thresholds are configured as 20 and 0, respectively. If a counter value exceeds the threshold settings, the rising or falling alert is generated and logged in RMON. If an external monitoring tool such as Cisco Prime DCNM is configured to listen for SNMP traps, RMON will inform this tool though an SNMP trap of PMON alerts. Figure 2 and the following section show how poll-interval-based PMON alerting works internally and generates rising or falling PMON alerts Cisco and/or its affiliates. All rights reserved. 11

12 Figure 2. Poll-Interval-Based Alerting Rising Threshold: 20 Falling Threshold: 0 Polling Interval: 30 Seconds Rising Alarm Value Rising Alarm Falling Alarm Time (seconds) When you configure a PMON policy, the system begins polling the hardware counters internally. The PMON agent process on the Cisco MDS 9000 Family module runs according to the configured poll-interval duration (such as 30 seconds), reads the PMON counters, and performs validation checks against the configured rising and falling alarm thresholds. In the sample pmon-test policy configured in the example here, during the first polling interval (that is, at the thirtieth second), PMON reads the counter s absolute value from the module s port ASIC and stores this value internally as the base value. During next polling cycle (that is, at the sixtieth second), PMON starts polling again, reads the counter value, and stores this value internally as the current value. A validation check is then performed between the counter s current value and the base value, which determines whether an alarm needs to be raised. If the delta between the current counter value and the previous base counter value is greater than the risingthreshold value configured, then a rising alarm is generated. [(Current_value Base_value) > Rising threshold] = Generate rising alarm Similarly, if the delta between the current and previous counters is less than or equal to the falling-threshold value configured, then a falling alarm is generated. [(Current_value Base_value) <= Falling threshold] = Generate falling alarm Note: The base value is overwritten with the counter s current value during each polling cycle by PMON. The validation always is performed against the base value during each polling cycle. Two successive rising alarms or falling alarms will not be generated unless an alarm of the other type occurs between them. Configuring Alerting Based on Check Interval Beginning from the Cisco MDS NX-OS Release , along with the rising and falling threshold, the user can configure a new threshold, called warning threshold, to generate syslogs. In poll-interval-based alerting, hardware counters are polled only at the configured polling intervals, triggering alerts at the polling intervals. However, if you need a more fine-tuned alerting mechanism, you should consider the check-interval-based Alerting mechanism. In this approach, a new global configuration command-line interface (CLI) is introduced which checks for counters value in between the configured polling interval and will send rising or falling alerts if the thresholds conditions are met Cisco and/or its affiliates. All rights reserved. 12

13 port-monitor check-interval <interval> where <interval> is the duration specified in seconds. The minimum interval value that can be set is 5, and it can be in increments of 1 second. To illustrate the alerting mechanism using this approach, let us consider a similar process monitor (PMON) policy named pmon-test for monitoring the counter link-loss as configured below. The checkinterval is configured with value 15 seconds and the warning-threshold with a value of 10. The warning-threshold is an optional parameter. It can be configured with a value in between rising and falling threshold values. It can also be used to inform the administrator through syslog message generation that the monitored counter is nearing its rising or falling threshold values. port-monitor check-interval 15 port-monitor name pmon-test port-type all counter link-loss poll-interval 30 delta rising-threshold 20 event 4 warningthreshold 10 falling-threshold 0 event 4 Figure 3 illustrates check-interval-based alerting. Figure 3. Check-Interval-Based Alerting Raising and Falling Alerts Rising Threshold Falling Threshold Rising Threshold Falling Threshold Warning Threshold (Downward) Warning Threshold (Upward) Warning Threshold (Downward) Warning Threshold (Upward) Warning Threshold (Upward) Value Rising Threshold: 20 Falling Threshold: 0 Warning Threshold : 10 Polling Interval: 30 Seconds Check Interval: 15 Seconds Time (Seconds) As shown in Figure 3, in this approach PMON performs validation checks at every configured check-interval period and at every poll-interval period. Calculations to generate alarms in this method are similar to those shown for poll-interval-based alerting, as shown here. If the delta between the current counter value and the previous base counter value is greater than the risingthreshold value configured, then a rising alarm is generated. [(Current_value Base_value) > Rising threshold] = Generate rising alarm 2016 Cisco and/or its affiliates. All rights reserved. 13

14 Similarly, if the delta between the current counter value and the previous counter value is less than or equal to the falling-threshold value configured, then a falling alarm is generated. [(Current_value Base_value) <= Falling threshold] = Generate falling alarm In addition: [(Current_value Base_value) >= Alert threshold] = Generate a syslog message Sample syslog messages generated by PMON for rising, falling and warning thresholds are as below: MDS9000(config)# 2016 Jan 6 06:51:35 sw-apex-40 %PMON-SLOT2-4-WARNING_THRESHOLD_REACHED_UPWARD: Invalid Words has reached warning threshold in the upward direction (port fc2/18 [0x ], value = 30) Jan 6 06:51:35 sw-apex-40 %PMON-SLOT2-3-RISING_THRESHOLD_REACHED: Invalid Words has reached the rising threshold (port=fc2/18 [0x ], value=30) Jan 6 06:51:36 sw-apex-40 %SNMPD-3-ERROR: PMON: Rising Alarm Req for Invalid Words counter for port fc2/18( ), value is 30 [event id 1 threshold 30 sample 2 object 4 fcifinvalidtxwords] 2016 Jan 6 06:52:21 sw-apex-40 %PMON-SLOT2-5-WARNING_THRESHOLD_REACHED_DOWNWARD: Invalid Words has reached warning threshold in the downward direction (port fc2/18 [0x ], value = 0) Jan 6 06:52:21 sw-apex-40 %PMON-SLOT2-5-FALLING_THRESHOLD_REACHED: Invalid Words has reached the falling threshold (port=fc2/18 [0x ], value=0) Jan 6 06:52:21 sw-apex-40 %SNMPD-3-ERROR: PMON: Falling Alarm Req for Invalid Words counter for port fc2/18( ), value is 0 [event id 2 threshold 0 sample 2 object 4 fcifinvalidtxwords] PMON s The Cisco MDS 9000 Family switches include a set of hardware counters. Advanced 8- and 16-Gbps and higher Cisco MDS 9000 Family modules include additional counters that are specific to the line-card type being used. To view the list of available PMON counters on the software release you are running, use the following command: MDS9000(config)# port-monitor name <pmon_policy_name> MDS9000(config-port-monitor)# monitor counter? credit-loss-reco err-pkt-from-port err-pkt-from-xbar err-pkt-to-xbar invalid-crc invalid-words link-loss Configure credit loss recovery counter Configure err-pkt-from-port counter Configure err-pkt-from-xbar counter Configure err-pkt-to-xbar counter Configure invalid-crc counter Configure invalid-words counter Configure link-failure counter 2016 Cisco and/or its affiliates. All rights reserved. 14

15 lr-rx lr-tx rx-datarate signal-loss sync-loss timeout-discards tx-credit-not-available tx-datarate tx-discards tx-slowport-count tx-slowport-oper-delay txwait Configure the number of link resets received by the fc-port Configure the number of link resets transmitted by the fc-port Configure rx performance counter Configure signal-loss counter Configure sync-loss counter Configure timeout discards counter Configure credit not available counter Configure tx performance counter Configure tx discards counter Configure slow port sub-100ms counter Configure slow port operation delay Configure tx total wait counter This section describes the PMON counters available, the CLI show commands for the counters, and the recommended threshold values configured in three templates: Normal, Aggressive, and Most Aggressive. The CLI configuration commands for these templates are also shown in Appendix B at the end of this document. PMON offers various counters to monitor Cisco MDS fabric and provide notifications if events exceed the configured threshold values. These counters range from generic interface counters such as receive (Rx) and transmission (Tx) rate counters to more platform-specific counters such as TxWait and Tx Slowport Count, for slow-drain detection. If you are experiencing slowness in your network, applications that are not performing up to their potential, or congestion on network ports, a prudent approach is to deploy PMON on all Cisco SAN switches and monitor the counters for potential slow-drain conditions. By integrating PMON with Cisco Prime DCNM, system administrators can be alerted proactively and notified whenever a configured counter exceeds its set threshold value. PMON is easy to deploy. System administrators can quickly configure a PMON policy using any of the three templates (Normal, Aggressive, and Most Aggressive) and start monitoring their SAN fabric from day one. A general guideline is to first deploy a Normal PMON policy on the SAN fabric and monitor for alerts or events for a few weeks. If the network is stable and does not result in alerts or event notifications and yet you continue to see application degradation, then you can move to the next level of monitoring and deploy a PMON policy using the Aggressive template and then, if necessary, the Most Aggressive template. PMON also provides facility to adjust the individual counter threshold settings when you deploy the policy. It also provides mechanisms for configuring alert and notification actions such as error-disabling a port if a configured counter exceeds its threshold settings. In Appendix B, the port-guard action is set to the default value for all the templates: that is, neither flapping nor error-disabling the physical port. The policy will alert RMON, which can then trigger alerts to the management application and notify the administrator if a threshold breach occurs on the Cisco MDS 9000 Family switch. Table 7 lists the 19 available hardware counters and the recommended threshold settings for each counter value. These values are further divided into three templates: Normal, Aggressive, and Most Aggressive. The polling interval specified is per second unless specified alongside its value Cisco and/or its affiliates. All rights reserved. 15

16 Table 7. Hardware s Name Normal Aggressive Most Aggressive Rise Fall Poll Rise Fall Poll Rise Fall Poll Signal Loss Sync Loss Link Loss Invalid CRC Invalid Words Tx Discards LR Rx LR Tx Timeout Discard Credit Loss Reco Tx Credit Not Available 10% % % 0 1 Rx Datarate 80% 70% 60 90% 60% 60 90% 50% 30 Tx Datarate 80% 70% 60 90% 60% 60 90% 50% 30 ASIC Error from Port ASIC Error Pkt to Xbar ASIC Error Pkt From Xbar Tx Slowport Count Tx Slowport Oper Delay 2 50 ms ms ms 0 1 TxWait 40% % % 0 1 Table 8 provides a support matrix for slow-drain-specific counters for PMON. Table 8. PMON Slow-Drain s Name Supported Platforms Minimum Cisco NX-OS Release credit-loss-reco All Release or lr-rx All Release or lr-tx All Release or timeout-discards All Release or tx-credit-not-available All Release or tx-discards All Release or Cisco and/or its affiliates. All rights reserved. 16

17 Name Supported Platforms Minimum Cisco NX-OS Release tx-slowport-count 1 Cisco MDS 9500 Series only with DS-X K9, DS-X K9, or DS-X K9 line card tx-slowport-oper-delay 2 Cisco MDS 9700 Series, 9396S, 9148S, 9250i, and 9500 Series only with DS-X K9 or DS-X K9 line card txwait Cisco MDS 9700 Series, 9396S, 9148S, 9250i, and 9500 Series only with DS-X K9 or DS-X K9 line card Release 6.2(13) Release 6.2(13) Release 6.2(13) 1 The slow-port-monitoring feature must be enabled for this counter to work. This counter will increment only if the amount of time that an interface is at 0 Tx buffer-to-buffer (B2B) credits is equal to or greater than the slow-port-monitoring value configured (administrative) delay. It would increment at most 1 time per 100 ms, or 10 times per second. Use the following configuration command to enable slow-port monitoring : MDS9700(config)# system timeout slowport-monitor? <1-500> Configure number of milliseconds default Default timeout value for HW slowport monitoring MDS9700(config)# system timeout slowport-monitor default? mode Enter the port mode MDS9700(config)# system timeout slowport-monitor default mode? E E mode 2 The slow-port-monitoring feature must be enabled for this counter to work. The threshold is exceeded only if the slow-port-monitoring administrative delay (the configured value) is less than the configured rising threshold and the reported operation delay (oper-delay) is more than the configured rising threshold. 3 These counters work on 4-and 8-Gbps Cisco MDS 9000 Family switches. On 16-Gbps Cisco MDS 9000 Family switches, these counters are not operable. The remainder of this section provides detailed descriptions of the PMON counters, their associated switch CLI commands, and their threshold values. Show command Signal Loss (signal-loss) Signal loss typically occurs at the physical transport medium. If Laser is off or the optical medium is down, then the counter is incremented. The optical ASIC on Cisco MDS 9000 Family switches detects this incrementation and raises a firmware interrupt to flag such conditions in the underlying Layer 1 signaling. The usual result of this event is that the link must be reset and the port must be reinitialized. The loss of signal usually indicates a hardware problem. show interface x/y counters detail include ignore signal Normal Aggressive Most Aggressive Cisco and/or its affiliates. All rights reserved. 17

18 Show command Sync Loss (sync-loss) This counter tracks the loss of the synchronization (sync) bit pattern when a frame is received from the peer Fibre Channel switch. In Fibre Channel, if the frame that is transmitted is encoded in 66-bit format, the first 2 bits form sync bits and remaining 64 bits form the data. Upon receiving a frame, the receiver locks itself to a sync-bit pattern. During data transfer, if any packet is received in which the sync pattern is different from the one to which the receiver is locked, then the counter goes high, indicating noise in the transmitted data. Generally a sync loss is followed by a logical reset of link reset (LR), link reset response (LRR), and IDLE. Sync-loss errors frequently are the result of a faulty transceiver or cable. show interface x/y counters detail include ignore sync Normal Aggressive Most Aggressive Show command Link Loss (link-loss) Link loss occurs whenever a port flaps. For example, if a peer sends an OLS or NOS signal to the switch, the counter is incremented. Link loss can trigger a series of other hardware counter updates as well. Both physical and hardware problems can cause link failures. Link failures frequently occur due to a sync loss or signal loss. A port or link flap causes other associated PMON counters to increment as well (sync loss, signal loss, link reset, etc.). None Normal Aggressive Most Aggressive Show command Invalid CRC (invalid-crc) In Fibre Channel, words are packaged inside a frame. If bit-level corruption occurs (that is, if cyclic redundancy-check [CRC] failure, checksum mismatch, etc. occurs) at the frame level, then the invalid-crc counter is incremented. The CRC field is 32 bits long and inserted before the end-of-frame (EOF) delimiter in the Fibre Channel frame format. show interface x/y include ignore crc Normal Aggressive Most Aggressive Cisco and/or its affiliates. All rights reserved. 18

19 Show command Invalid Words (invalid-words) A checksum and signal integrity check occurs for every word received on a Cisco MDS 9000 Family switch. The invalid-words counter is incremented each time a Fibre Channel word is detected with checksum or bit-level errors. show interface x/y counter detail include ignore words Normal Aggressive Most Aggressive Show command Tx Discards (tx-discards) This counter increments when one of the following conditions occurs: an egress policy map (access control list [ACL]) forcibly drops the packets, traffic is received on the port ASIC when port is physically down, or packets are discarded as a result of a timeout (500-ms threshold). show interface x/y counter detail include ignore discard Normal Aggressive Most Aggressive Show command LR RX (lr-rx) IIf a peer port is continuously running out of transmit B2B credits for a long time (1 second for F-ports and 1.5 seconds for E-ports), it can invoke the credit-lossrecovery mechanism by transmitting a Fibre Channel link reset (LR) primitive. The node (the Fibre Channel switch) connected to this port will receive a link reset in the receive (Rx) direction and increment the lr-rx counter. The switch that sent this link reset would have its lr-tx (transmit) counter incremented. This example is just one of the situations in which this counter incremented. A link flap can also trigger the counter to increment. Usually, no traffic flows are received on this port when this condition occurs. An increasing number of received link resets can indicate congestion at a port, or the resets can be a cascading effect of congestion on other ports to which this port is transmitting data. It can also indicate that the receiver-ready (R_RDY) B2B credits are being lost due to corruption on the link. show interface fc2/11 count details include ignore reset responses received Normal Aggressive Most Aggressive Cisco and/or its affiliates. All rights reserved. 19

20 Show command LR TX (lr-tx) If a port is continuously running out of transmit B2B credits for a long time (1 second for F-ports and 1.5 seconds for E-ports), it will invoke the credit-loss-recovery mechanism by transmitting a Fibre Channel link reset (LR) primitive. A link reset is only a credit reset and does not actually flap the port if it is successful. Usually, no traffic flows out of this port when this condition occurs. This example is just one of the situations in which this counter is incremented. A link flap can also trigger the counter to increment. incrementation can also indicate that the R_RDY B2B credits are being lost due to corruption on the link. show interface x/y count details include ignore reset responses transmitted Normal Aggressive Most Aggressive Show command Timeout Discards (timeout-discards) All frames are time-stamped when they enter a Cisco MDS 9000 Family switch. The frame is put into the receive buffer queue when it enters the switch. The frame will be dropped if it is not delivered to the egress port within the time specified by the congestion-drop timeout value (the default is 500 ms). These dropped frames are accounted for at the next-hop (egress) port and signify congestion at this egress port. This example is one of the conditions that causes the timeout-discards counter to increment. show interface x/y count details include ignore timeout Normal Aggressive Most Aggressive Show command Credit Loss Reco (credit-loss-reco) IThis counter is incremented when B2B credits are unavailable for more than 1 second for F-ports or more than 1.5 seconds for E-ports. Because this is an important counter for slow-drain detection, you should keep the value the same for all three templates while monitoring. show interface x/y count details include ignore credit loss Normal Aggressive Most Aggressive Cisco and/or its affiliates. All rights reserved. 20

21 Name TX Credit Not Available (tx-credit-not-available) This counter is incremented by 1 when Tx B2B credits are not available for 100 ms (default). Both thresholds are configured as a percentage of the polling internal time. For example, assume that you set the poll interval to 1 second, the rising threshold to 10% (which translates to 100 ms: that is, 10% of a second), the falling threshold to 0% (which translates to 0 ms), and the port-guard action to error-disable the port. If there are zero remaining Tx B2B credits for 100 ms, then the rising-threshold criterion is met. If a credit is indeed returned, then the falling-threshold criterion is met. In addition, when the rising-threshold criterion is met, the port-guard action is initiated, which puts the port in the error-disable state. The port will remain in that state until someone manually issues a shut or no shut command on that port. Because this is an important counter for slow-drain detection, you should keep the value the same for all three templates while monitoring. Show command Note: Because this counter is sampled at 100-ms intervals, you should configure both threshold percentages only in multiples of 10 (10, 20, 30, etc.). show interface x/y count details include ignore Transmit B2B Normal 1 10% 0 Aggressive 1 10% 0 Most Aggressive 1 10% 0 Show command Rx Datarate (rx-datarate) This counter indicates the rate at which frames are received on the interface. Both thresholds are configured as a percentage of the operational speed of the interface. show interface x/y include ignore input rate Normal 60 80% 70% Aggressive 60 90% 60% Most Aggressive 30 90% 50% Show command TX Datarate (tx-datarate) This counter indicates the rate at which frames are transmitted out of a particular interface. Both thresholds are configured as a percentage of the operational speed of the interface. show interface x/y include ignore output rate Normal 60 80% 70% Aggressive 60 90% 60% Most Aggressive 30 90% 50% 2016 Cisco and/or its affiliates. All rights reserved. 21

22 ASIC Error Pkt from Port (err-pkt-from-port) This counter is incremented when the internal path between the port ASIC to the forwarding ASIC is causing frames to be corrupted. If the forwarding ASIC detects corrupted frames from the port ASIC due to any internal problems, this counter is incremented. Show command - The counter is deprecated on Cisco MDS 9000 Family fourth-generation and later line cards due to the nature of the hardware architecture on these modules. Normal Aggressive Most Aggressive ASIC Error Pkt to XBAR (err-pkt-to-xbar) Show command - This counter indicates the rate at which frames are received on the interface. Both thresholds are configured as a percentage of the operational speed of the interface. Normal Aggressive Most Aggressive ASIC Error Pkt from XBAR (err-pkt-from-xbar) Show command - This counter is incremented when the forwarding ASIC on the ingress module (line card) is sending corrupted frames to the crossbar (fabric module or XBAR) ASIC. Normal Aggressive Most Aggressive Cisco and/or its affiliates. All rights reserved. 22

23 Show command tx-slowport-count The tx-slowport-count counter is applicable only for 8-Gbps modules in Cisco MDS 9500 Series Multilayer Directors. In the default configuration, PMON sends an alert when a slow-port condition is detected 10 times in 1 second for the configured administrative delay. Because this is an important counter for slow-drain detection, you should keep the value the same for all three templates while monitoring. Note: Because this counter is sampled at 100-ms intervals and there can be, at most, one event per 100-ms interval, this counter can increment, at most, 10 times per second. Therefore, configuring a rising-threshold value greater than 10 (with a polling interval of 1 second) will cause this alert never to be generated. In addition, system timeout slowport-monitor default mode must be configured for alerting to work. show process creditmon slowport-monitor-events show logging onboard slowport-monitor-events Normal Aggressive Most Aggressive Show command tx-slowport-oper-delay The tx-slowport-oper-delay counter is applicable to advanced 8- and 16-Gbps modules for Cisco MDS 9000 Family switches. There are two defaults based on the module type. For advanced 8-Gbps modules, the default rising-threshold value is 80 ms with a 1-second polling interval. For 16-Gbps modules and switches, the default rising-threshold value is 50 ms with a 1-second polling interval. show process creditmon slowport-monitor-events show logging onboard slowport-monitor-events Normal 1 50 ms 0 Aggressive 1 40 ms 0 Most Aggressive 1 30 ms 0 TxWait TxWait counts the time whenever the port Tx B2B credits available are zero and frames are waiting in the egress queue. The counter increments if a port has zero remaining Tx B2B credits for 2.5 microseconds. This counter is used to report the credit unavailability on a port in multiple intuitive ways. TxWait is reported only on 16-Gbps and advanced 8-Gbps platforms. The other platforms in the Cisco MDS 9000 Family will report zero. Both rising and falling thresholds are configured as a percentage of the polling-interval time. Therefore, for example, a 30% threshold with a 1-second polling interval is a setting of 300 ms Cisco and/or its affiliates. All rights reserved. 23