Any ASR 5000 including PDSN, epdg, Home Agent, GGSN, SGW, MME, etc. with LAG ports would be affected.

Transcription

1 Contents Overview Products Affected CDETS Symptoms Fix Workaround / Recovery Method Root Cause Analysis Related Cisco Support Community Discussions Overview This article is to provide brief information and notification on an issue related to Single Point of Failure (SPOF) alarms on ASR 5000 for Link Aggregation (LAG) ports after a port bounce. The false alarms may cause unnecessary tickets to be opened when in fact there is no issue to be concerned about. Products Affected Any ASR 5000 including PDSN, epdg, Home Agent, GGSN, SGW, MME, etc. with LAG ports would be affected. CDETS CSCun74136: ASR 5000 Alarm The 10 Gig Ethernet Line Card is a SPOF Symptoms There is an issue related to the unnecessary triggering of single point of failure (SPOF) alarms for LAG-based 10 GB Line Cards (XGLC) on ASR 5000 platform. Whenever a LAG port goes down (trap PortDown), the CardSPOFClear trap will trigger, and whenever the port comes up (trap PortUp), the CardSPOFAlarm trap will trigger. Port bounces can be for any number of reasons including PSC migrations, npumgr restarts, hardware failures, chassis reload, or externally caused link issues. The following snippet shows the respective SPOF traps for a port 19/1 bounce, meanwhile a LAG switchover often results in traps for all the ports that may bounce in the process. Tue Jan 21 07:35: Internal trap notification 1024 (PortDown) card 19 port 1 port type 10G Ethernet Tue Jan 21 07:35: Internal trap notification 1503 (EntStateOperDisabled) Port(19/1) Admin state:"locked", Alarm severity:"major" Tue Jan 21 07:35: Internal trap notification 93 (CardStandby)

2 Tue Jan 21 07:35: Internal trap notification 140 (CardSPOFClear) Tue Jan 21 07:40: Internal trap notification 1025 (PortUp) card 19 port 1 port type 10G Ethernet Tue Jan 21 07:40: Internal trap notification 139 (CardSPOFAlarm) Starting in v15.0 deployed in January 2015, in addition to the SNMP traps, the alarming mechanism also started to be notified. Here is the matching alarm from the example: ******** show alarm outstanding verbose ******* Severity Object Timestamp Alarm ID Alarm Details Minor Card 19 Tuesday January 21 07:40: The 10 Gig Ethernet Line Card in slot 19 is a single point of failure. A 10 Gig Ethernet Line Card is needed in slot 20. Fix Future versions may allow for the ability to suppress the SPOF trap and alarm for LAG or other configurations if desired. Workaround / Recovery Method SPOF alarms for LAG-configured cards can be simply ignored and cleared per the reason explained in the Root Cause Analysis. The clear alarm command can be used to clear all outstanding alarms (including non-spof ones if that is desired), or to clear just specific SPOFs by specifying the alarm ID reported by show alarm outstanding [verbose]. For the example above: clear alarm id or clear alarm all Note: Alarms will remain in place indefinitely unless another port bounce occurs, in which case a fresh alarm (as evidenced by the timestamp) replaces the existing one.

3 Root Cause Analysis Due to the design of LAG, card redundancy is done by LAG and is NOT done at the card level, such that all LAG-configured cards always are operationally state active - none of them are standby. Therefore the config for LAG-configured cards does NOT specify any redundancy. show port info... Card 23: card 26: Card Type: 10 Gig Ethernet Line Card Card Card Type: 10 Gig Ethernet Line Operational State : Active Operational State : Active Redundant With : None Redundant With : None ******** show card table all ******** Slot Card Type Oper State SPOF Attach : LC 10 Gig Ethernet Line Card Active Yes 3 20: LC 10 Gig Ethernet Line Card Active Yes 4 21: LC 1000 Ethernet Line Card Active No 5 22: LC 1000 Ethernet Line Card Active No 6 23: LC 10 Gig Ethernet Line Card Active Yes 7 24: SPIO Switch Processor I/O Card Active 25: SPIO Switch Processor I/O Card Active 26: LC 10 Gig Ethernet Line Card Active Yes 10 27: LC 10 Gig Ethernet Line Card Active

4 Yes 11 28: LC 10 Gig Ethernet Line Card Active Yes 12 29: LC 10 Gig Ethernet Line Card Active Yes 13 30: LC 10 Gig Ethernet Line Card Active Yes 14 Meanwhile, the config for non-lag cards does specify redundancy. For example, here is the config that does NOT have any LAG ports, in which case SPOF alarms have significance and should be investigated. Following is the card table showing the respective pairs of active/standby XGLCs. card 19 redundant with 20 card 23 redundant with 26 card 27 redundant with 28 card 29 redundant with 30 [local]asr5000> show card table all Slot Card Type Oper State SPOF Attach : LC 10 Gig Ethernet Line Card Active No 3

5 20: LC 10 Gig Ethernet Line Card Standby : LC 1000 Ethernet Line Card Active No 5 22: LC 1000 Ethernet Line Card Active No 6 23: LC 10 Gig Ethernet Line Card Active No 7 24: SPIO Switch Processor I/O Card Active 25: SPIO Switch Processor I/O Card Active 26: LC 10 Gig Ethernet Line Card Standby : LC 10 Gig Ethernet Line Card Active No 11 28: LC 10 Gig Ethernet Line Card Standby : LC 10 Gig Ethernet Line Card Active No 13 30: LC 10 Gig Ethernet Line Card Standby - 14