Prostředky návrhu a zajištění dostupné LAN sítě

Transcription

1

2 Prostředky návrhu a zajištění dostupné LAN sítě TECH-LANWAN Radek Boch Cisco Systems Engineer CCIE#7095 rboch@cisco.com

3 Todays Agenda Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Foundations of the Structured Network Design High Availability Distribution Block Architectures HA System Recovery Analysis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Designing High Availability Switching Networks for the Enterprise

5 Seconds of Data Loss Evolving Campus Design High Availability Requirements Availability Requirements for voice and video are more than just five 9 s Consider the subjective impact to real time communications No Impact to Voice or Video Minimal Impact to Voice User Hangs Up Phone Resets* * The Time for a Phone to Reset Is Variable and Depends on the gnaling Protocol, SCCP or SIP, and the State of the Call, Active, Ringing, TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

6 High Availability Switching Design Key Principals Enterprise network design architectures continue to evolve to meet business and technology needs, but the key principals of high availably network design still apply; Add redundancy and resiliency components as needed to meet the business requirements. mplify network designs and configurations through virtualization techniques. Implement network-monitoring tools with automation where appropriate, and analyze all aspects of network outages for indications of where improvement is needed. TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

7 The Redundancy Effect Availability = % Downtime = ~ min/yr Availability = % Downtime = ~10 min/yr Unit % ~5 min/yr Linecard % ~5 min/yr Unit 1 Unit 2 Blocks in Series Supervisor % ~5 min/yr Supervisor Unit % ~5 min/yr Supervisor Blocks in Parallel Cisco Public 8

8 Systems Hardware and Software Resiliency

9 Todays Agenda Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Hardware Resiliency Physical and Environmental Device Operational Resiliency In Service Software Upgrades Embedded Management Foundations of the Structured Network Design High Availability Distribution Block Architectures HA System Recovery Analysis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

10 Catalyst 6500-E and 4500-E Switches Integrated Resiliency Hot swap capability (Online Insertion and Removal OIR) Redundant Supervisors (1:1) NSF/SSO switchover results in sub-second recovery Redundant Supervisors are supported on all Cisco Catalyst 6500 chassis Redundant Supervisors are supported on the Cisco Catalyst 4507R-E and 4510R-E chassis Redundant fans (1:N) Secondary fans provide sufficient cooling to keep the system running at full capacity Catalyst 6513-E Redundant power supplies (1+1) Secondary power supply kicks in instantly to provide full uninterrupted power to the system Separation of Data Plane and Control Plane Redundant clocks Catalyst 4507R-E Cisco Public 11

11 Utilize Hardware Redundancy Where It Makes Sense Redundant Power Supplies gnificant value for minimal cost Redundant Supervisor modules ngle attached devices, such as the access layer Faster software upgrades with In-Service Software Upgrade. Hitless In-Service Software Upgrade with the Nexus Consistent Switchover times with NSF/SSO. Hitless Switchover with the Nexus 7000 Cost of Hardware Cost of Downtime Stacking Improves availability Improves performance mplifies device management Cisco Public 12

12 Todays Agenda Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Hardware Resiliency Device Operational Resiliency Stateful Switchover, Non-Stop Forwarding, StackWise In Service Software Upgrades Embedded Management Foundations of the Structured Network Design High Availability Distribution Block Architectures HA System Recovery Analysis Cisco Public 13

13 Redundant Supervisors IOS Active Standby Model Active Supervisor Control Plane Data Plane Active Supervisor Control Plane Console access Manages Configurations Manages Chassis Environmentals L2 L3 Protocols Data Plane Hardware-based switching CF RF Standby Supervisor Not part of the active forwarding path COLD Standby WARM Standby HOT Standby Control Plane Data Plane Multiple Redundancy modes COLD Standby WARM Standby HOT Standby Synchronization CF Checkpoint Facility RF Redundancy Facility Cisco Public 14

14 Stateful Switchover Mode IOS SSO-Aware and SSO-Compliant IOS Applications SSO-Compliant Applications Routing Protocols NetFlow Cisco Discovery Protocol and more Cisco IOS Redundancy Facility Checkpointing Facility SSO-Aware Applications Forwarding Information Base IEEE 802.1x PAgP / LACP and more Active Supervisor Standby Hot Supervisor SSO-Compliant Applications TECH-LANWAN Routing Protocols NetFlow Cisco Discovery Protocol and more Redundancy Facility Checkpointing Facility Cisco IOS 2014 Cisco and/or its affiliates. All rights reserved. SSO-Aware Applications Forwarding Information Base IEEE 802.1x PAgP / LACP and more Cisco Public 15

15 NSF/SSO in Campus Networks NSF Router Roles Non-Stop Forwarding, NSF, allows a router to continue forwarding data along routes that are already known, while the routing protocol information is being restored NSF Aware router or NSF Helper router* A router running NSFcompatible software, capable of assisting a neighbor router perform an NSF restart NSF-Aware NSF Capable router A router configured to perform an NSF restart, therefore able to rebuild routing information from neighbour NSF-aware or NSF capable router NSF-Capable * NSF Helper - This term is used in IETF terminology Cisco Public 16

16 Non-Stop Forwarding OSPF Implementations Cisco NSF NSF Capable NSF Aware NSF Capable IETF NSF NSF Aware Restart Event Fast Hello (2 Sec Interval RS Bit Set) Fast Hello (2 Sec Interval RS Bit Set) Fast Hello (2 Sec Interval RS Bit Clear) Fast Hello (2 Sec Interval RS Bit Clear) Fast Hello Restart Event LS Update (Grace LSA) Hello LS ACK (Grace LSA) Hello Announce Graceful- Restart OSPF Discovery Database Description LSA Requests/ Update Hello (RS Bit Clear) Database Description LSA Requests/ Update Hello (RS Bit Clear) Out-of-Band Sync Database Description LSA Request s/update Hello Database Description LSA Requests /Update Hello Database Exchange Cisco Public 17

17 Data Plane Control Plane NSF/SSO Switchover Operation IOS NSF Aware Router Active Supervisor Fails 3 1 Active Supervisor Newly Active Supervisor RP CPU CPU 5 Control Path 2 Cisco IOS CEF Tables FIB Table Hardware Forwarding Path OSPF EIGRP IS-IS BGP Process Process Process Process FIB Table Adjacency Table ARP Table Adjacency Table Cisco Public Routing Information Base Global Epoch = 1 Prefix Next Hop Interface Epoch Next Hop MAC Epoch Vlan AA-BB Vlan EE-DD.. 10

18 NSF with Stateful Switchover IOS NSF/SSO Switchover Details For Your Reference 1. Switchover is triggered. Standby Supervisor becomes active 2. Control plane and data plane separation: the FIB is detached from the RIB 3. Packet forwarding continues based on last-known FIB and adjacency entries while the standby takes over 4. The global epoch number is incremented 5. The Supervisor brings its interfaces and control plane online 6. The software adjacency table is populated with the pre-switchover ARP table contents. Updated entries receive the new global epoch number. New adjacency entries are downloaded in hardware 7. The routing protocol specific neighbour and adjacency reacquisition occurs 8. The routing protocol specific database synchronization occurs 9. The RIB is repopulated with new routing entries. The corresponding CEF entries are updated 10. Updated entries receive the global epoch number to indicate that they have been refreshed. Corresponding FIB entries and hardware entries are updated 11. Each routing protocol notifies CEF that it has converged. Once all of them have converged, the last one flushes the stale route and adjacency information 12. The IOS CEF tables on the RP and the hardware forwarding tables are now synchronized. Cisco Public 19

19 NSF Configuration - IOS Configuration is required to enable NSF Capable No configuration required to enable NSF Helper with default settings The Nexus 7000 is NSF Capable by default for all the routing protocols in all NX-OS software releases. router eigrp 1 nsf! router ospf 1 nsf ietf! router isis 1 nsf cisco core1# show ip ospf nsf Routing Process "ospf 1" IETF Non-Stop Forwarding enabled restart-interval limit: 120 sec IETF NSF helper support enabled IETF NSF helper strict-lsa-checking enabled Cisco NSF helper support enabled OSPF restart state is NO_RESTART Handle , Router ID , checkpoint Router ID Config wait timer interval 10, timer not running Dbase wait timer interval 120, timer not running Cisco Public 20

20 Design Considerations for NSF/SSO Distribution Layer FHRP Timer Tuning By contrast, fast hello timers intend to provide availability through fast convergence, SSO attempts to avoid convergence No FHRP Active Gateway Change with Default Timers HSRP / GLBP hold timers should be greater than SSO restart time + time to send first hello Recommendation: Use HSRP / GLBP default timers with redundant Supervisors Default hold timers are: HSRP default 10 seconds GLBP default (10 sec for the Nexus 7000, 18 sec for the Catalyst 6500) Aggressive sub-second timers can be used if desired, but will cause an active gateway flap in the event of a Supervisor switchover On the Catalyst 6500 HSRP and GLBP Are SSO Aware Beginning with 12.2(33)SXH and Newer Cisco Public 21

21 Design Considerations for NSF/SSO Routing Protocol Timer Tuning NSF is intended to provide availability by avoiding a route flap By contrast, fast IGP hello and hold timers are intended to provide availability through fast route convergence Recommendation is to use default IGP hello and hold timers Allows NSF/SSO recovery Provides consistency and simplicity NSF-AWARE Hello If tighter hello / hold timers are desired in an NSF configuration the routing protocol dead timer must be greater than: SSO recovery + Routing Protocol restart + time to send first hello (typically 7 seconds) NSF Restart RP Restart OSPF First Hello NSF-CAPABLE IGP hold timers lower than 8 seconds are not recommended Cisco Public 22

22 Standalone Chassis Redundant Core Redundant Supervisors Yes or No? Catalyst 6500 Seconds of Lost Voice Redundant topologies with equal cost paths provide sub-second convergence NSF/SSO provides superior availability in environments with non-redundant paths? RP Convergence Is Dependent on IGP and Tuning Link Failure Node Failure NSF/SSO OSPF Convergence Cisco Public 24

23 Standalone Chassis Redundant Core Redundant Supervisors Yes or No? Catalyst 6500 Seconds of Lost Voice Redundant topologies with equal cost paths provide sub-second convergence NSF/SSO provides superior availability in environments with non-redundant paths? RP Convergence Is Dependent on IGP and Tuning Link Failure Node Failure NSF/SSO OSPF Convergence Cisco Public 25

24 Seconds of Lost Voice Distribution - Redundant Supervisors Yes or No? HSRP doesn t flap on Supervisor SSO switchover Reduces the need for sub-second HSRP timers SSO Aware HSRP 6500-E (33)SXH (31)SG? Cisco Public 26

25 Design Considerations for NSF/SSO Where Does It Make Sense? Seconds of Lost Voice Access switch is the single point of failure in best practices HA design Supervisor failure is most common cause of access switch service outages Recommended design with NSF/SSO provides for sub 600 msec recovery of voice and data traffic? Cisco Public 27

26 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Hardware Resiliency Device Operational Resiliency Stateful Switchover, Non-Stop Forwarding, Cisco StackWise- 480 and Cisco StackWise In Service Software Upgrades Embedded Management Foundations of the Structured Network Design High Availability Distribution Block Architectures HA System Recovery Analysis Cisco Public 28

27 Catalyst 3750 StackWise Stack Master and Stack Member Refresh 1:N redundancy where any member can become stack master if the master fails Stack master provides centralized functionality Controls the console and Propagates the configuration to the entire stack Represents the active Layer 3 control plane and management plane (telnet, SSH, SNMP, HTTP) Builds and propagates the hardware information (L3 FIB, ACL, QoS) TCAM TCAM TCAM Switching Switching Switching CPU 1 Master TCAM TCAM TCAM Switching Switching Switching CPU 2 Slave Stack member provides distributed forwarding of data and local control plane functionalities Local instance of STP BPDU processing MAC address management TCAM TCAM TCAM Switching 3 Slave Switching CPU Switching Cisco Public 29

28 Comparing HA Between StackWise Plus to StackWise X StackWise Plus - Hybrid control-plane processing - N:1 stateless control-plane redundancy - Distributed L2/L3 Forwarding Redundancy - Stateless L3 protocol Redundancy 3850 StackWise Centralized control-plane processing NSF/SSO based stateful redundancy - Distributed L2/L3 Forwarding Redundancy - IOS HA Framework alignment for L3 protocol - ISSU Ready* TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

29 Catalyst 3750 StackWise Multi-Layer Access Stack Master with uplink failure introduces two failures Master control plane Uplink interface When the stack master fails, the master election process starts Upstream, HSRP / GLBP will detect link down, and D2 will start answering to the virtual MAC c07.ac00 Distribution Access D1 HSRP ACTIVE vip: vmac: c07.ac00 Summar y Subnets Master S1 S2 S3 ngle Logical Switch D2 HSRP STANDBY L2 Downstream traffic is re-routed to D2 via L3 link TECH-LANWAN IP: MAC: aaaa.aaaa.aa01 GW: ARP: c07.ac Cisco and/or its affiliates. All rights reserved. IP: MAC: aaaa.aaaa.aa03 GW: ARP: c07.ac00 Cisco Public

30 Catalyst 3750 StackWise Multi-Layer Access Stack Master Failure (without uplink) When the master fails, the master election process starts Distribution D1 HSRP ACTIVE vip: vmac: c07.ac00 Summar y Subnets D2 HSRP STANDBY No HSRP/GLBP failover, while the new master being elected, MAC address of HSRP/GLPB still used by the rest of the stack for data forwarding Access Maste S1 S2 r S3 ngle Logical Switch L2 No downstream re-route convergence IP: MAC: aaaa.aaaa.aa01 GW: ARP: c07.ac00 IP: MAC: aaaa.aaaa.aa03 GW: ARP: c07.ac00 Cisco Public 34

31 Catalyst 3750 StackWise Multi-Layer Access Convergence msec of lost Voice Stack Master with Uplink Downstream traffic reroute Upstream traffic has HSRP convergence due to link down Stack Master without Uplink Downstream & Upstream follows the master recovery Catalyst 3750 StackWIse Convergence Downstream Upstream Recommendation Use stack master without uplink for the best convergence Master With Uplink Master Without Uplink Downstream Upstream Cisco Public 35

32 Catalyst 3750 StackWise Routed Access For Your Reference Stack master (without uplink) failure Upstream traffic is not interrupted by master failure Downstream traffic is interrupted due to routing protocol restart and adjacency reset Distribution Summar y Subnets Stack master provides the MAC address for all the control plane activities such as Routed & SVI Interfaces CDP Spanning Tree When master fails new MAC address are provisioned from newly elected master. Both Routed and SVI interfaces announce this change via Gratuitous ARP Forces the MAC change in end host and network devices for next hop connectivity CLI show switch before and after the failure shows different switch/stack MAC address Access S1 Maste S2 r S3 ngle Logical Switch IP: MAC: aaaa.aaaa.aa01 GW: ARP: 000c.cece.7c80 Cisco Public 36 L3 IP: MAC: aaaa.aaaa.aa03 GW: ARP: 0011.bbb9.9480

33 Catalyst 3750 StackWise Routed Access For Your Reference CLI stack-mac persistent timer 0 enables MAC consistency New master inherits the MAC address of the previous master No MAC changes for end hosts and adjacent routers, significantly improves upstream recovery Distribution Summar y Subnets L3 Caution Do not re-introduce the 3750 in order to avoid duplicate MAC introduction Access S1 Maste S2 r S3 ngle Logical Switch Best practice Use no stack-mac persistent timer 0 during planned downtime to inherit the MAC provisioned from existing master IP: MAC: aaaa.aaaa.aa01 GW: ARP: 000c.cece.7c80 NO MAC Changes IP: MAC: aaaa.aaaa.aa03 GW: ARP: 000c.cece.7c80 Cisco Public 37

34 Catalyst 3750 StackWise Routed Access Convergence For Your Reference Upstream traffic is not interrupted by master failure Downstream traffic is interrupted due to routing protocol restart and adjacency reset * Recommended Design Configure priority for master and its backup for deterministic failures Use same IOS feature set in all stack members Avoid using master as uplink to reduce uplink related losses Use stack-mac persistent timer 0 to avoid the gratuitous ARP changes for Best convergence Where GARP processing is disabled in the network, e.g. Security Where network devices / host do not support GARP, e.g. Phones msec Catalyst 3750 OSPF NSF Aware Convergence 6000 Down_Min 5000 Down_Max Up_Min 4000 Up_Max Master Fail WITH MAC Change Master Fails WITHOUT MAC Change Down_Min Down_Max Up_Min Up_Max Cisco Public 38

35 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Hardware Resiliency Device Operational Resiliency In Service Software Upgrades Embedded Management Foundations of the Structured Network Design High Availability Distribution Block Architectures HA System Recovery Analysis Cisco Public 39

36 In-Service Software Upgrade IOS ISSU Stages Initial State Abortversion Final State 12.2(31)SGA 12.2(31)SGA 12.2(31)SGA 12.2(31)SGA 12.2(31)SGA1 12.2(31)SGA 12.2(31)SGA1 12.2(31)SGA1 12.2(31)SGA1 12.2(31)SGA1 Loadversion ISSU upgrade is a 4 step process Runversion Possible to rollback (abort) up until you complete the 4 th step (commit to final state) Acceptversion (Optional) Leverages NSF/SSO to implement Supervisor transition Requires that the two images are compatible for upgrade / downgrade processing Commitversion Cisco Public 40

37 ISSU Client and Infrastructure Interactions IOS Active Supervisor ISSU Endpoint V1 Application XYZ ISSU Client V1 Register Client Info Versioning Infrastructure Store Client Info Register ClientID,, Msg Capabilities, MSG Versions, Card Type Endpoints Agree on a Common Set of Capabilities Hot Standby Supervisor ISSU Endpoint V3 Versioning Infrastructure Store Client Info Application XYZ ISSU Client V3 Register Client Info Propose Capabilities Propose Message Version Capabilities Negotiation Message Version Negotiation Endpoints Agree on a Common Message Version Capabilities Negotiation Message Version Negotiation Propose Capabilities Propose Message Version Compatible Y N V1 Agree V1 If Compatible, then Message Exchange Can Proceed V1, V2,V3 Compatible Y N Message Exchange Message Transformation Message Transformation Message Exchange MSG V1 MSG V3 Cisco Public 41

38 What About Upgrades Between Incompatible Versions? Core1# issu image-version compatibility disable ********************************************************** * WARNING WARNING WARNING!!!!!!!! * * * * The ISSU compatibility matrix check has been disabled. * * No image version compatibility checking will be done. * * Please be sure this is your intention. * ********************************************************** Core1#3w0d: %COMP_MATRIX-4-DISABLED: Compatibility Matrix has been disabled. No image compatibility checking will be done. Core1# issu image-version compatibility enable Cisco is committing to provide ISSU compatible software releases, where compatibility is possible, for an 18 month sliding window When two versions are not compatible, you may will still need to upgrade Enhanced Fast Software Upgrade can still be done using the ISSU process Disable the version compatibly checking service (shown above) ISSU process will leverage RPR mode rather than SSO Only for ISSU aware software versions If downgrading to a pre-issu version, must use the manual Fast Software Upgrade process Cisco Public 42

39 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Hardware Resiliency Device Operational Resiliency In Service Software Upgrades Embedded Management Generic Online Diagnostics, EEM, Call Home Foundations of the Structured Network Design High Availability Distribution Block Architectures HA System Recovery Analysis Cisco Public 43

40 Proactive Fault Detection and Reaction Distributed Management Intelligence Challenge In today s highly available networks improved physical redundancy is not enough, intelligent system failure detection and recovery are key Memory Corruption Software Inconsistency System Faults Link Faults Detect and Isolate Enhanced System Stability Enhanced Network Stability Generic Online Diagnostics (GOLD) Provides Proactive, Scheduled and Manual System Diagnostics Enhanced Object Tracking (EOT), Embedded Event Manager (EEM), and Smart Call Home (SCH) Provide Intelligent Response to System Events Cisco Public 44

41 Diagnostics, Analysis, GOLD vs. Other Forms of Diagnostics GOLD performs functional tests typically using diagnostic packets switching through the system, as well as ASIC memory testing Can be performed during runtime Typically use the same hardware path and IOS software drivers and user traffic Power On Self Test (POST) occurs early on in the IOS initialization Focused on the CPU subsystem and memory components Improved Analysis Tools System Event Archive (SEA) maintains a secure log of system events and critical messages across reboots On Board Failure Logging (OBFL) is like a black box recorder on each line card that records diagnostic and environmental information Cisco Public 45

42 Generic Online Diagnostics Diagnostic Operation For Your Reference Boot-Up Diagnostics Switch(config)#diagnostic bootup level complete Runtime Diagnostics Health-Monitoring Switch(config)#diagnostic monitor module 5 test 2 Switch(config)#diagnostic monitor interval module 5 test 2 00:00:15 Run During System Bootup, Line Card OIR or Supervisor Switchover Makes Sure Faulty Hardware Is Taken Out of Service Non-Disruptive Tests Run in the Background Serves as HA Trigger On-Demand Switch#diagnostic start module 4 test 8 Module 4: Running test(s) 8 may disrupt normal system operation Do you want to continue? [no]: y Switch#diagnostic stop module 4 Scheduled Switch(config)#diagnostic schedule module 4 test 1 port 3 on Jan :32 Switch(config)#diagnostic schedule module 4 test 2 daily 14:45 All Diagnostics Tests Can Be Run on Demand, for Troubleshooting Purposes. It Can Also Be Used as a Pre-Deployment Tool Schedule Diagnostics Tests, for Verification and Troubleshooting Purposes Cisco Public 46

43 Generic Online Diagnostics Recommendations For Your Reference Bootup diagnostics Configure level to complete On demand diagnostics Use as a pre-deployment tool run complete diagnostics before putting hardware into production environment Use as a troubleshooting tool when suspecting hardware failure Scheduled diagnostics Schedule Supervisor Switchovers periodically, run active and standby specific tests Schedule all non-disruptive tests periodically Cisco Public 47

44 Generic Online Diagnostics Recommendations (cont.) For Your Reference Health-monitoring diagnostics Key tests are already running by default Configure additional non-disruptive tests for specific functionalities enabled in your network IPv6, MPLS, NAT Review Corrective Actions and adjust actions if needed Configure a notification method for test failures that meet corrective action threshold SNMP Trap, Call Home, Syslog, EEM Keep in mind that GOLD tests are functional tests, meaning test failures do not automatically mean faulty hardware or software, failures must be put in context with the overall system Cisco Public 48

45 Embedded Event Manager What is EEM? EEM is a Cisco IOS technology that runs on the control plane It is a combination of processes designed to monitor key system event parameters such as CPU utilization, interface errors, counters, SNMP and SYSLOG events, and take action based upon the user or system defined policies Events Application CLI Counter GOLD Interface OIR Resource Thresholding SNMP SYSLOG Timer IOS Watchdog Timer IOS Modularity WatchDog Timer EEM Policy Director Actions Execute IOS CLI Command Increment/Decrement an EEM Counter Force an SSO Switchover Request System Information Send an Run Another EEM Policy Re-Load the Switch Generate an SNMP Trap Generate a SYSLOG Message CLI Based (Applet) Script Based (TCL) Default Cisco Policies Cisco Public 49

46 Embedded Event Manager EEM Application Example Upon Matching the Provided SYSLOG Message LINK-3-UPDOWN, the Switch Performs the Following Actions Display error statistics for the link that has gone down Start a Time Domain Reflectometry (TDR) test Start a GOLD Loopback test Send the results using a provided template to a user-configurable address Interface Down EEM Interface Error Counters P O R T Cable TDR Test Fault P O R T GOLD Loopback Test Send Results in Alert Cisco Public 50

47 Embedded Event Manager Configuration Example For Your Reference event manager applet TEST EEM Applet Example event syslog pattern "%LINK-3-UPDOWN: Interface GigabitEthernet7/1" maxrun 20 action 1.0 cli command en action 2.0 cli command "test cable-diagnostics tdr interface G7/1 action 3.0 cli command diagnostic start module 7 test 2 port 1" action 4.0 mail server x.x.x.x" to _id@x.com" from "Switch-1" subject "Urgent! Interface went down" body "G7/1 went down" EEM TCL Script Example event manager environment _ _server <IP_address> event manager environment _ _to _id@x.com event manager environment _syslog_pattern.*updown.*state to down.* event manager environment _ _from Switch1@mylab.com event manager environment intchk_template disk1:/interfacecheck.template event manager directory user policy disk1:/ event manager policy interfacecheck.tcl Cisco Public 51

48 Smart Call Home Proactive Problem Identification Call Home 37) TestErrorCounterMonitor > F Error code > 1 (DIAG_FAILURE) Total run count > 2484 Last test execution time ----> Feb :55:52 First test failure time -----> Jan :55:17 Last test failure time > Feb :55:52 Last test pass time > Jan :54:45 Total failure count > 2474 Consecutive failure count ---> 2474 Error Records as following. ID -- Asic Identification IN -- Asic Instance PO -- Asic Port Number RE -- Register Identification RM -- Register Identification More EG -- Error Group DV -- Delta Value CF -- Consecutive Failure TF -- Total Failure ID IN PO RE RM DV EG CF TF Smart Call Home Automated Diagnosis Capability ASIC # 49, Register #240 Failed 2483 Consecutive Times Indicates ngle Bit ECC Errors Detected and Recovered Customer Indicated Developing Unrecoverable Failure. This Is Usually a Problem Related to Improper Grounding or Excessive Radiation Emitted into the Device. Make Sure that the Device Is Properly Grounded and that Neighbouring Devices Are Not Emitting Excessive Radiation Levels. Cisco Public 52

49 Foundations of the Structured Network Design The Building Blocks for High Availability

50 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Foundations of the Structured Network Design Modularity, Hierarchy, and Structure Leveraging Hardware-Based Path Restoration High Availability Distribution Block Architectures HA System Recovery Analysis Cisco Public 54

51 High Availability Design Optimization of the Elements Data Center WA N Optimize the interaction of the physical redundancy with the network protocols Provide the necessary amount of redundancy Pick the right protocol for the requirement Optimize the tuning of the protocol The network looks like this so that we can map the protocols onto the physical topology We want to build networks that look like this Redundant Switches Redundant Links Redundant Supervisor Services Block Layer 3 Equal Cost Link s Layer 2 or Layer 3 Distribution Blocks Cisco Public 55

52 High Availability Design Optimization of the Elements Not This!! Server Farm WAN Internet PSTN Cisco Public 56

53 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Foundations of the Structured Network Design Modularity, Hierarchy, and Structure Leveraging Hardware-Based Path Restoration High Availability Distribution Block Architectures HA System Recovery Analysis Cisco Public 57

54 Optimizing Network Convergence Failure Detection and Recovery Optimal high availability network design attempts to leverage local switch fault detection and recovery Design should leverage the hardware capabilities of the switches to detect and recover traffic flows based on these local events Design Principle Hardware failure detection and recovery is both faster and more deterministic Design Principle Software failure detection mechanisms provide a secondary, not primary, fault detection and recovery mechanism in the optimal design L3 HW Initiated Recovery Hello s SW Initiated Recovery Cisco Public 58

55 Optimizing Network Convergence Layer 1 Link Redundancy and Failure Detection Direct point to point fiber provides for fast failure detection IEEE 802.3z and 802.3ae link negotiation define the use of Remote Fault Indicator & Link Fault gnaling mechanisms Do not disable auto-negotiation on GigE and 10GigE interfaces IOS debounce GigE and 10GigE fiber ports is 10 msec Minimum for copper is 300 msec NX-OS debounce Currently 100 msec by default All 1G and 10G SFP / SFP+ based interfaces (MM, SM, CX-1) changing to a default of 10 msec RJ45 based Copper interfaces on NX-OS will remain at 100 msec Design Principle Understand how hardware choices and tuning impact fault detection and response to link failures Cisco IOS Throttling: Carrier Delay Timer Linecard Throttling: Debounce Timer 1 Remote IEEE Fault Detection Mechanism Cisco Public 59

56 Optimizing Network Convergence Layer 2 Software Fault Detection (e.g. UDLD, LACP) While 802.3z and 802.3ae link negotiation provide for L1 fault detection, hardware ASIC failures can still occur UDLD provides an L2 based keep-alive mechanism that confirms bi-directional L2 connectivity Tx Rx Each switch port configured for UDLD will send UDLD protocol packets (at L2) containing the port s own device / port ID, and the neighbor s device / port IDs seen by UDLD on that port Rx Tx If the port does not see its own device / port ID echoed in the incoming UDLD packets, the link is considered unidirectional and is shutdown LACP with fast timers configured can provide a similar function for dual NIC servers (detect failures of the local adapters,.) Design Principle Redundant fault detection mechanisms are required (leverage SW as a backup to HW whenever possible) UDLD Keepalive LACP Hellos Cisco Public 60

57 Optimizing Network Convergence Layer 2 and 3 Why Use Routed Interfaces? Configuring L3 routed interfaces provides for faster convergence than an L2 switchport with an associated L3 SVI ~ 8 msec Loss 1. Link Down 2. Interface Down 3. Routing Update 21:38: UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/1, changed state to down 21:38: UTC: %LINK-3-UPDOWN: Interface GigabitEthernet3/1, changed state to down 21:38: UTC: IP-EIGRP(Default-IP-Routing-Table:100): Callback: route_adjust GigabitEthernet3/1 L3 1. Link Down 2. Interface Down L2 ~ msec Loss 3. Autostate 4. SVI Down 5. Routing Update 21:32: UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1, changed state to down 21:32: UTC: %LINK-3-UPDOWN: Interface GigabitEthernet2/1, changed state to down 21:32: UTC: %LINK-3-UPDOWN: Interface Vlan301, changed state to down 21:32: UTC: IP-EIGRP(Default-IP-Routing-Table:100): Callback: route, adjust Vlan301 Cisco Public 61

58 EtherChannel Design Considerations EtherChannel and L2 Links In the STP topology the port cost for port-channel is calculated based on Catalyst / IOS Sum of operational port-channel member ports Nexus / NX-OS Sum of configured port-channel member ports On single link failure in a bundle Catalyst / IOS Potential change to the STP topology Nexus / NX-OS No change to the STP topology Cisco Public 62

59 Optimizing Network Convergence Equal Cost Multi-Path (ECMP) Recovery Equal Cost Multi-Path (ECMP)-based routing provide the second key building block for switching HA designs Time to restore traffic flows is based on Time to detect link failure Process the removal of the lost routes from the SW FIB Update the HW FIB No dependence on external events (no routing protocol convergence required) Behavior is deterministic Design Principle Designs that have pre-calculated installed paths (e.g. you already have two routes) take less time to recover traffic Equal Cost Links: Link / Box Failure Does Not Require Multi-Box Interaction Cisco Public 63

60 High Availability Distribution Block Architectures Multi-Layer Campus Distribution

61 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Foundations of the Structured Network Design High Availability Distribution Block Architectures Multi-Layer Campus Distribution Virtual Switching System (VSS) HA System Recovery Analysis Cisco Public 65

62 Optimizing the Layer 2 Design Spanning Tree L3 L2 Vlan 10 Vlan 20 Vlan 30 Vlan 30 Vlan 30 Vlan 30 Each access switch has unique VLANs No Layer 2 loops Layer 3 link between distribution No blocked links More typical of a Campus Design At least some VLANs span multiple access switches Layer 2 loops Layer 2 and 3 running over link between distribution Blocked links Typical Data Center Design Deterministic recovery (L2 CAM, no STP dependant recovery) Cisco Public 66

63 Optimizing the Layer 2 Design STP Toolkit PortFast and BPDU Guard PortFast is configured on edge ports to allow them to quickly move to forwarding bypassing listening and learning and avoids TCN (Topology Change Notification) messages BPDU Guard can prevent loops by moving PortFast configured interfaces that receive BPDUs to errdisable state BPDU Guard prevents ports configured with PortFast from being incorrectly connected to another switch When enabled globally, BPDU Guard applies to all interfaces that are in an operational PortFast state Switch(config-if)#spanning-tree portfast Switch(config-if)#spanning-tree bpduguard enable VLAN 30 PortFast + BPDU Guard X BPDU Receive 1w2d: %SPANTREE-2-BLOCK_BPDUGUARD: Received BPDU on port FastEthernet3/1 with BPDU Guard enabled. Disabling port. 1w2d: %PM-4-ERR_DISABLE: bpduguard error detected on Fa3/1, putting Fa3/1 in err-disable state Cisco Public 67

64 Optimizing the Layer 2 Design STP Best Practices for Campus The root bridge should stay where you put it Loopguard and Rootguard UDLD STP Root Loopguard Only end station traffic should be seen on an edge port BPDU Guard & PortFast Port-Security Rootguard Loopguard There is a reasonable limit to Broadcast and Multicast traffic volumes Configure storm control on backup links to aggressively rate limit B-Cast and M-Cast Utilize Sup720 rate limiters, or Sup IV or higher with HW queuing structure Storm Control BPDU Guard PortFast Port Security Cisco Public 68

65 First Hop Redundancy Sub-Second Timers Improve Convergence HSRP Config interface Vlan4 ip address standby 1 ip standby 1 timers msec 250 msec 750 standby 1 priority 150 standby 1 preempt standby 1 preempt delay minimum 180 GLBP Config interface Vlan4 ip address glbp 1 ip glbp 1 timers msec 250 msec 750 glbp 1 priority 150 glbp 1 preempt glbp 1 preempt delay minimum 180 FHRP Active R1 Access-a FHRP Standby R2 VRRP Config interface Vlan4 ip address vrrp 1 description Master VRRP vrrp 1 ip vrrp 1 timers advertise msec 250 vrrp 1 preempt delay minimum 180 HSRP Is Widely Used with Its Rich Feature Set GLBP Facilitates Uplink Load Balancing Not Optimal for L2 Looped Topology VRRP If You Need Multi-Vendor Interoperability HSRP, GLBP and VRRP Provide Millisecond Timers and Excellent Convergence Performance Critical for VoIP and Video Recovery in < 1 Sec (Compared to: Default Recovery is ~10 Seconds) Cisco Public 69

66 HSRP Preemption Why It s Desirable Core Spanning Tree root and HSRP primary aligned When Spanning Tree root is re-introduced, traffic will take a two-hop path to HSRP active HSRP preemption will allow HSRP to follow the Spanning Tree topology Spanning-Tree Root HSRP Active Distribution Access Cisco Public 71

67 HSRP Preemption Why It s Desirable Core Spanning Tree root and HSRP primary aligned When Spanning Tree root is re-introduced, traffic will take a two-hop path to HSRP active HSRP preemption will allow HSRP to follow the Spanning Tree topology HSRP Active Spanning-Tree Root Distribution Access Cisco Public 72

68 HSRP Preemption Why It s Desirable Core Spanning Tree root and HSRP primary aligned When Spanning Tree root is re-introduced, traffic will take a two-hop path to HSRP active HSRP preemption will allow HSRP to follow the Spanning Tree topology Spanning-Tree Root HSRP Active Distribution Access Cisco Public 73

69 HSRP Preemption Why It s Desirable Core Spanning Tree root and HSRP primary aligned When Spanning Tree root is re-introduced, traffic will take a two-hop path to HSRP active HSRP preemption will allow HSRP to follow the Spanning Tree topology Spanning-Tree Root HSRP Preempt HSRP Active Distribution Access Without Preempt Delay, HSRP Can Go Active Before the Switch Is Completely Ready to Forward Traffic L1 (Linecards), L2 (STP), L3 (IGP Convergence) Cisco Public 74

70 HSRP Preemption Why It s Desirable Core Spanning Tree root and HSRP primary aligned When Spanning Tree root is re-introduced, traffic will take a two-hop path to HSRP active HSRP preemption will allow HSRP to follow the Spanning Tree topology Spanning-Tree Root HSRP Active Distribution Access Without Preempt Delay, HSRP Can Go Active Before the Switch Is Completely Ready to Forward Traffic L1 (Linecards), L2 (STP), L3 (IGP Convergence) Cisco Public 75

71 FHRP Design Considerations Preempt Delay Needs To Be Longer Than Box Boot Time HSRP is not always aware of the status of the entire switch and network Test Tool Timeout 30 Seconds Ensure that you provide enough time for the entire system to be up Diagnostics (full or partial), L1 (Line cards), L2 (STP), L3 (IGP convergence) Tune delay and preempt delay conservatively as the network is already forwarding data interface Vlan standby delay minimum 60 reload 600 standby 1 ip standby 1 timers msec 250 msec 750 standby 1 priority 110 standby 1 preempt delay minimum 60 reload 600 standby 1 authentication ese standby 1 name HSRP-Voice hold-queue 2048 in standby delay Controls How Long Before the Interface Needs to Be Up Before HSRP Starts and preempt delay Controls How Long to Wait After HSRP Establishes a Neighbour Relationship. You Should Configure Both. Cisco Public 76

72 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing This can cause a problem with unicast flooding Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists B B B B In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 78

73 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 This can cause a problem with unicast flooding ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists B B B B CAM Table Empty for VLAN 3 In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 79

74 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 This can cause a problem with unicast flooding ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists B B B B CAM Table Empty for VLAN 3 In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 80

75 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 This can cause a problem with unicast flooding ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists CAM Table Empty for VLAN 2 B B B B CAM Table Empty for VLAN 3 In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 81

76 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 This can cause a problem with unicast flooding ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists CAM Table Empty for VLAN 2 B B B B CAM Table Empty for VLAN 3 In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 82

77 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 This can cause a problem with unicast flooding ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists CAM Table Empty for VLAN 2 B B B B CAM Table Empty for VLAN 3 In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty B VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 83

78 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) For Your Reference Alternating HSRP Active between distribution switches can be used for upstream load balancing Switch 1: Active HSRP and Root Bridge VLAN 3 Switch 2: Active HSRP and Root Bridge VLAN 2 This can cause a problem with unicast flooding ARP timer defaults to four hours and CAM timer defaults to five minutes ARP entry is valid, but no matching L2 CAM table exists CAM Table Empty for VLAN 2 B B B B CAM Table Empty for VLAN 3 In many cases when the HSRP standby needs to forward a frame, it will have to unicast flood the frame since it s CAM table is empty B VLAN 3 VLAN 2 VLAN 3 VLAN 2 Cisco Public 84

79 FHRP Design Considerations Asymmetric Routing (Unicast Flooding) Multiple Solutions Deploy Virtual Switching System in the distribution block Using V based design with unique voice and data VLANs per access switch, this problem has no user impact Don t deploy stacking switches that depend on Spanning Tree for managing interconnects in the stack Tune ARP timer to 270 seconds and leave CAM timer to default, unless ARP > 10,000, change CAM timers MultiChassis EtherChannel (VSS or vpc) VLAN 3 VLAN 2 *Note: CAM Timers Traditionally Default to 5 Minutes to Allow for MAC Addresses (Devices) to Move in the Network. It Is Safe to Increase the CAM Timers If the Client Devices Will Generate Unicast or Multicast Traffic to Refresh the CAM Table. Cisco Public 85

80 Multi-Layer Network Design Good Solid Design, But Seconds of VOIP packet loss Utilizes multiple Control Protocols Spanning Tree (802.1w), HSRP / GLBP, EIGRP, OSPF Convergence is dependent on multiple factors FHRP 900msec to 9 seconds Spanning Tree Up to 50 seconds Multi-Layer Convergence 50 Load balancing Asymmetric forwarding HSRP / VRRP per subnet GLBP per host Unicast flooding in looped design STP, if it breaks badly, has no inherent mechanism to stop the loop Looped PVST+ (No RPVST+) 9.1 Non-looped Default FHRP 0.91 Non-looped Sub- Second FHRP DST MAC /2 3/2 3/1 3/1 Switch 1 Switch 2 DST MAC Cisco Public 86

81 High Availability Distribution Block Architectures Virtual Switching System (VSS)

82 Today s Agenda Getting to Non-Stop Communications Designing High Availability Switching Networks for the Enterprise System Hardware and Software Resiliency Foundations of the Structured Network Design High Availability Distribution Block Architectures Multi-Layer Campus Distribution Virtual Switching System (VSS) HA System Recovery Analysis Cisco Public 90

83 VSS Enabled Campus Design End-to-End VSS Design Option TECH-LANWAN WAN Data Center 2014 Cisco and/or its affiliates. All rights reserved. Internet Cisco Public 1

84 VSS Enabled Campus Design End-to-End VSS Design Option WAN Data Center Internet TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

85 Cisco VSS Architecture Overview Catalyst 6500E/6800/4500E Catalyst 6500E/6800/4500E Line Card Inter-Chassis SSO Redundancy SF PFC RP Internal EOBC Active Sup Line Card Standalone VSS-SW1 VSS-SW2 Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 1

86 Cisco VSS Architecture Overview Catalyst 6500E/6800/4500E Catalyst 6500E/6800/4500E Line Card Inter-Chassis SSO Redundancy Intra-Chassis SSO Redundancy SF PFC RP Internal EOBC Active Sup SF PFC RP Standby Sup Line Card Standalone VSS-SW1 VSS-SW2 Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 1

87 Cisco VSS Architecture Overview Catalyst 6500E/6800/4500E Catalyst 6500E/6800/4500E Line Card Inter-Chassis SSO Redundancy SF PFC RP Active Sup SF PFC RP Standby Sup Internal EOBC External EOBC (VSL) Line Card VSS-SW1 VSS-SW2 Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 1

88 Cisco VSS Architecture Overview Catalyst 6500E/6800/4500E Catalyst 6500E/6800/4500E Line Card SF PFC RP Internal EOBC Active Sup Inter-Chassis SSO Redundancy External EOBC (VSL) Line Card SF PFC RP Standby Sup Internal EOBC Line Card Line Card VSS-SW1 VSS-SW2 Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 1

89 Cisco VSS Architecture Overview Catalyst 6500E/6800/4500E Catalyst 6500E/6800/4500E Line Card SF PFC RP Internal EOBC Active Sup Inter-Chassis SSO Redundancy External EOBC (VSL) Line Card SF PFC RP Standby Sup Internal EOBC Line Card Line Card VSS-SW1 VSS-SW2 Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 1

90 Cisco Instant Access Architecture Overview Catalyst 6500E/6800 Catalyst 6500E/6800E Line Card SF PFC RP Active Sup Interna l EOBC Line IA Client Card External EOBC (VSL) Line Card SF PFC RP Standby Sup Line IA Client Card Interna l EOBC VSS-SW1 VSS-SW2 FEX FEX Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis and Instant Access Client switches TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 2

91 Cisco Instant Access Architecture Overview Catalyst 6500E/6800 Catalyst 6500E/6800E Line Card SF PFC RP Active Sup Interna l EOBC External EOBC (VSL) Line Card SF PFC RP Standby Sup Interna l EOBC VSS-SW1 VSS-SW2 FEX FEX Line IA Client Card Line IA Client Card Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis and Instant Access Client switches TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 2

92 Cisco Instant Access Architecture Overview Catalyst 6500E/6800 Catalyst 6500E/6800E Line Card SF PFC RP Active Sup Interna l EOBC External EOBC (VSL) Line Card SF PFC RP Standby Sup Interna l EOBC VSS-SW1 VSS-SW2 FEX FEX IA Client IA Client Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis and Instant Access Client switches TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 2

93 Cisco Instant Access Architecture Overview Catalyst 6500E/6800 Catalyst 6500E/6800E Line Card SF PFC RP Active Sup Interna l EOBC External EOBC (VSL) Line Card SF PFC RP Standby Sup Interna l EOBC VSS-SW1 VSS-SW2 FEX FEX External EOBC (FEX) IA Client IA Client Internal EOBC : Internal communication control channel between supervisor and linecards within single-chassis External EOBC : External communication control channel between supervisors between two-chassis and Instant Access Client switches TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public SF : Switch Fabric PFC : Policy Feature Card RP : Route Processor EOBC : Ethernet Out-of-Band Channel 2

94 VSS Supervisor Redundancy Summary Catalyst 6500E Sup2T Catalyst 6500E Sup720-10GE Catalyst 4500E/4500X/6500E TECH-LANWAN Quad-Sup (SSO) 2014 Cisco and/or its affiliates. All rights reserved. Quad-Sup (RPR-WARM) Cisco Public Dual-Sup Supported Platforms Catalyst 6500E/6800 Sup2T Catalyst 6500E Sup720-10GE Catalyst 6500E, 4500E and 4500X Switch Fabric Inter-Chassis(ICA) Active Intra-Chassis (ICS) Ready Inter-Chassis (ICA) Active Intra-Chassis (ICS) Inactive Inter-Chassis Active Switching Capacity 4 Tbps 1.4 Tbps 4500E / 4500X 1.6 Tbps 6500E Sup720-10GE 1.4 Tbps 6500E Sup2T 4 Tbps Policy Feature BOOT, VLAN Dbase and Startup config Sync Inter-Chassis(ICA) Active Intra-Chassis (ICS) Inactive Inter-Chassis (ICA) Active Intra-Chassis (ICS) Inactive Inter-Chassis Active Inter-Chassis (ICS) + Intra-Chassis (ICA) Inter-Chassis (ICA) + Intra-Chassis (ICS) Inter-Chassis Running configuration Inter-Chassis (ICA) Inter-Chassis (ICA) Inter-Chassis SSO State Synchronization Inter-Chassis (ICA) Inter-Chassis (ICA) Inter-Chassis efsu Software Upgrade Inter-Chassis (ICA) + Intra-Chassis (ICS) Inter-Chassis (ICA) + Intra-Chassis (ICS) Inter-Chassis 105

95 Understanding Virtual Switch Link Inter-Chassis System Link No network protocol operations Invisible in network topology Transparent to network level troubleshooting VSL Control Link Carries all system internal control traffic ngle member-link and dynamic election during bootup Shared interface for network/data traffic < 50 msec switchover to pre-determined VSL path Payload Overhead Every single packet encapsulated with Virtual Switch Header (VSH) Non-bridgeable and Non-routeable. VSL must be directly connected between two virtual switch systems Control Link V S H L 2 VSL 4500E-VSS#show switch virtual link L 3 Control Link Payload Executing the command on VSS member switch role = VSS Active, id = 1 VSL Status : UP VSL Uptime : 1 day, 1 hour, 16 minutes VSL Control Link : Te1/3/1 Executing the command on VSS member switch role = VSS Standby, id = 2 C R C VSL Status : UP VSL Uptime : 1 day, 1 hour, 17 minutes VSL Control Link : Te2/3/1 TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 106

96 Virtual Switching System VSLP Framework Building Virtual System Link Management Protocol (LMP) LMP protocol operates on each VSL member-link for peer-switch detection, link integrity and bi-directionality health check Default hello and dead timers are non-tunable and are optimal for various purpose. LMP hello timers (aka VSLP timers) : Catalyst 6500E LMP Hello / Dead Timer = 0.5 sec / 60 sec Catalyst 4500E/4500X LMP Hello / Dead Timer = 1 sec / 30 sec For older 6500E VSS deployments, it is strongly recommended not to modify default LMP(VSLP) timer Role Resolution Protocol (RRP) RRP runs on control link of the VSL bundle Determines whether software versions allow a virtual switch to form Determines which chassis will become Active or Hot Standby from a control plane perspective by checking configuration of switch priority or pre-emption RRP roles are negotiated when either of the switch member initializes or when VSL link is restored 4500-VSS#show vslp lmp timer LMP hello timer 6500-VSS#show vslp lmp timer LMP hello timer LMP RRP VSL LMP RRP Hello Tx (T4) ms Hello Rx (T5*) ms Interface State Cfg Cur Rem Cfg Cur Rem Te1/3/1 operational Te1/4/1 operational Hello Tx (T4) Hello Rx (T5*) ms Interface State Cfg Cur Rem Cfg Cur Rem Te2/5/4 operational Te2/2/8 operational VSS#show switch virtual role Switch Switch Status Preempt Priority Role Session ID Number Oper(Conf) Oper(Conf) Local Remote LOCAL 1 UP FALSE(N) 110(110) ACTIVE 0 0 REMOTE 2 UP FALSE(N) 100(100) STANDBY TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 107

97 6500E/6800 VSS Dual Sup VSL Design Sup2T and Sup720-10GE Design Two Cisco recommended designs Profile 1 VSL on Supervisor (Sup2T/Sup720-10GE) Profile 2 Diversified VSL between Supervisor (Sup2T/Sup720-10GE) and VSL capable Linecard Sup Sup Sup Sup VSL VSL Cost-effective solution to leverage both uplinks. Continue to use non-vsl capable linecard for 10G core connection. Redundant fibers connects thru common fabric and ASICs, this could result vulnerability in system stability. Optimal and preset VSL parameters Load-Balancing, QoS, HA, Traffic-engg, Dual-Active etc. Restricted to bundle 2 x VSL ports or 20G switching capacity on per virtual-switch node basis. Redundant and diversified fibers between supervisor and nextgen VSL capable linecards. Same design as Profile 1 but increases system reliability as each VSL port are diversified across different fabric/asics. Optimal and preset VSL parameters Load-Balancing, QoS, HA, Traffic-engg, Dual-Active etc. Flexible to scale up to 8 x VSL for high-dense system to aggregate uplink, service modules, single-home etc. TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 108

98 6500E/6800 VSS VSL Design Quad-Sup (SSO / RPR-WARM) Sup2T Quad-Sup NSF/SSO VSL Redundancy Recommended Full-Mesh VSL on Quad-Sup Sup-1 Sup-2 Sup-1 Sup-2 Sup-3 VSL Sup-4 Sup-3 VSL Sup-4 SW1 SW2 SW1 SW2 Same Design Profile 1 Dual Sup Flexible to increase VSL Capacity Continue to leverage existing non-vsl 10G linecard for uplink connection Retains all original VSL benefits Vulnerable design during any supervisor selfrecovery fault incident Highly Redundant and cost-effective VSL Design. Increases overall VSL Capacity Maintains 20G VSL Capacity during supervisor failure. Increases network reliability by minimizing the dual-active probability TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

99 6500E/6800 VSS VSL Design Quad-Sup (SSO / RPR-WARM) Sup2T Quad-Sup NSF/SSO VSL Redundancy Recommended Full-Mesh VSL on Quad-Sup Sup-2 Sup-1 Sup-2 Sup-3 VSL Sup-4 Sup-3 VSL Sup-4 SW1 SW2 SW1 SW2 Same Design Profile 1 Dual Sup Flexible to increase VSL Capacity Continue to leverage existing non-vsl 10G linecard for uplink connection Retains all original VSL benefits Vulnerable design during any supervisor selfrecovery fault incident Highly Redundant and cost-effective VSL Design. Increases overall VSL Capacity Maintains 20G VSL Capacity during supervisor failure. Increases network reliability by minimizing the dual-active probability TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

100 6500E/6800 VSS VSL Design Quad-Sup (SSO / RPR-WARM) Sup2T Quad-Sup NSF/SSO VSL Redundancy Recommended Full-Mesh VSL on Quad-Sup Sup-2 Sup-1 Sup-2 Sup-3 VSL Sup-4 Sup-3 VSL Sup-4 SW1 SW2 SW1 SW2 Same Design Profile 1 Dual Sup Flexible to increase VSL Capacity Continue to leverage existing non-vsl 10G linecard for uplink connection Retains all original VSL benefits Vulnerable design during any supervisor selfrecovery fault incident Highly Redundant and cost-effective VSL Design. Increases overall VSL Capacity Maintains 20G VSL Capacity during supervisor failure. Increases network reliability by minimizing the dual-active probability TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

101 6500E/6800 VSS VSL Design Quad-Sup (SSO / RPR-WARM) Sup2T Quad-Sup NSF/SSO VSL Redundancy Recommended Full-Mesh VSL on Quad-Sup Sup-2 Sup-2 Sup-3 VSL Sup-4 Sup-3 VSL Sup-4 SW1 SW2 SW2 Same Design Profile 1 Dual Sup Flexible to increase VSL Capacity Continue to leverage existing non-vsl 10G linecard for uplink connection Retains all original VSL benefits Vulnerable design during any supervisor selfrecovery fault incident Highly Redundant and cost-effective VSL Design. Increases overall VSL Capacity Maintains 20G VSL Capacity during supervisor failure. Increases network reliability by minimizing the dual-active probability TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

102 4500E VSS Dual-Sup VSL Network Design Sup7E and Sup7-LE Design Two Cisco recommended designs Profile 1 VSL on Sup7-E Profile 2 Diversified VSL between Supervisor (Sup7-E/Sup7-LE) and VSL capable Linecard For Your Reference Sup Sup Sup Sup VSL VSL Cost-effective solution to leverage Quad uplinks for VSL and Core connections For reliable internal connection diversify fibers between Uplink ports groups thru different fabric and ASICs connection Optimal and preset VSL parameters Load-Balancing, QoS, HA, Traffic-engg, Dual-Active etc. Restricted to bundle 2 x VSL ports or 20G switching capacity on per virtual-switch node basis. Redundant and diversified fibers between supervisor and VSL capable linecards. Same design as Profile 1 but increases system reliability as each VSL port are diversified across different ASICs. Optimal and preset VSL parameters Load-Balancing, QoS, HA, Traffic-engg, Dual-Active etc. Flexible to scale up to 8 x VSL for high-dense system to aggregate uplink, service modules, single-home etc. TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 114

103 4500X VSS VSL Network Design For Your Reference Fixed switch hardware architecture 24 or 48 10G/1G Front Panel Ports 8 port 1G/10G Pluggable Uplink Module Any ports can be bundled into VSL EtherChannel. Recommended to use front-panel ports to build VSL connections. Minimizes system instability during accidental uplink module OIR/reset Splits VSL member-link interfaces to different internal ASICs groups : ASIC Group 4500X 16 Port ASIC to Port Mapping 4500X 32 Port ASIC to Port Mapping 4500-X Ten1/1/1 Ten2/1/1 Ten1/1/9 Ten2/1/9 VSL Front Panel Ports Front / Uplink Ports 4500-X Internal Stub ASIC Internal Stub ASIC SW-1 SW-2 Internal Stub ASIC 3 N/A Internal Stub ASIC 4 N/A Consistent software design and VSL function as 4500E TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 115

104 Virtual Switch Link Capacity Planning Plan VSL capacity to reduce congestion point, handle failures and specific configurations Supported VSL interfaces types : Catalyst 6500E/6800 : 10G and 40G Catalyst 4500E/4500X : 1G and 10G Four major factors : Total Uplink BW Per Chassis. Ability to handle data re-route during uplink failures without network congestion Handling egress data to single-homed devices (Nonrecommended design) Catalyst 6500E/6800 services module integration may require centralized forwarding on remote chassis Remote network services such as SPAN VSL Up to 8 member-links supported in VSL EtherChannel. Recommended to implement in power of 2 for optimal forwarding decision TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

105 Virtual Switch Link Capacity Planning Plan VSL capacity to reduce congestion point, handle failures and specific configurations Supported VSL interfaces types : Catalyst 6500E/6800 : 10G and 40G Catalyst 4500E/4500X : 1G and 10G Four major factors : Total Uplink BW Per Chassis. Ability to handle data re-route during uplink failures without network congestion Handling egress data to single-homed devices (Nonrecommended design) Catalyst 6500E/6800 services module integration may require centralized forwarding on remote chassis Remote network services such as SPAN VSL Up to 8 member-links supported in VSL EtherChannel. Recommended to implement in power of 2 for optimal forwarding decision TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

106 Virtual Switch Link Capacity Planning Plan VSL capacity to reduce congestion point, handle failures and specific configurations Supported VSL interfaces types : Catalyst 6500E/6800 : 10G and 40G Catalyst 4500E/4500X : 1G and 10G Four major factors : Total Uplink BW Per Chassis. Ability to handle data re-route during uplink failures without network congestion Handling egress data to single-homed devices (Nonrecommended design) Catalyst 6500E/6800 services module integration may require centralized forwarding on remote chassis Remote network services such as SPAN VSL Up to 8 member-links supported in VSL EtherChannel. Recommended to implement in power of 2 for optimal forwarding decision TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

107 Virtual Switch Link Capacity Planning Plan VSL capacity to reduce congestion point, handle failures and specific configurations Supported VSL interfaces types : Catalyst 6500E/6800 : 10G and 40G Catalyst 4500E/4500X : 1G and 10G Four major factors : Total Uplink BW Per Chassis. Ability to handle data re-route during uplink failures without network congestion Handling egress data to single-homed devices (Nonrecommended design) Catalyst 6500E/6800 services module integration may require centralized forwarding on remote chassis Remote network services such as SPAN VSL Up to 8 member-links supported in VSL EtherChannel. Recommended to implement in power of 2 for optimal forwarding decision TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

108 Virtual Switch Link Capacity Planning Plan VSL capacity to reduce congestion point, handle failures and specific configurations Supported VSL interfaces types : Catalyst 6500E/6800 : 10G and 40G Catalyst 4500E/4500X : 1G and 10G Four major factors : Total Uplink BW Per Chassis. Ability to handle data re-route during uplink failures without network congestion Handling egress data to single-homed devices (Nonrecommended design) Catalyst 6500E/6800 services module integration may require centralized forwarding on remote chassis Remote network services such as SPAN Up to 8 member-links supported in VSL EtherChannel. Recommended to implement in power of 2 for optimal forwarding decision VSL Analyzer TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

109 VSS Multi-Chassis EtherChannel Multi-Chassis EtherChannel (MEC) in VSS enables distributed link bundling into single logical L2/L3 Interface Combining VSS with MEC builds simplified, scalable and highly resilient campus network MEC is an imperative network design component to enable mplified STP loop-free network topology Consistent L3 control-plane and network design as traditional Standalone mode system Deterministic sub-second network recovery MECs can be deployed in two modes Layer 2 = Supported on 6500E/6800, 4500E and 4500X Layer 3 = Supported on 6500E/6800, 4500E * and 4500X * SW-1 (ACTIVE) VSL SW-2 (HOT-STANDBY) MEC scalability support varies on system basis Catalyst 6500E supports 512 L2/L3 MEC Catalyst 4500E and 4500X supports 256 L2 MEC A1 A2 * = Starting from IOS software release TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 122

110 Optimize EtherChannel Load Balancing Load share egress data traffic based on input hash Optimal load sharing results with : Bucket-based load-sharing Bundle member-links in power-of-2 (2/4/8) Multiple variation of input for hash (L2 to L4) Recommended algorithm * : Access Src/Dst IP 6500E/6800 Dist/Core Src/Dst IP + Src/Dst L4 Ports 4500E / 4500X Dist Src/Dst IP Default : src-dst-ip vlan Recommended : src-dst-mixed-ip-port Default : src-dst-ip vlan Recommended : src-dst-mixed-ip-port vlan Core Dist Default : src-mac Recommended : src-dst-ip Access * May vary based on your network traffic pattern TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 123

111 6500E/6800 VSS MEC EtherChannel Hash Algorithm For Your Reference Catalyst 6500E in VSS or in non-vss configuration mode has common support of EtherChannel Hash algorithms. 6500E EtherChannel Hash result computation mode: Fixed Recomputes hash results and programs each time when member-link flaps. This is default mode and recommended if each virtual-switch has single local physical port bundled in L2/L3 MEC. Adaptive Pre-computes hash results and programs member-link ports. Do not recompute when member-link flaps and improves network convergence. Best practice to modify to adaptive hash method only if each virtual-switch has >=2 local physical ports in L2/L3 MEC. Unlike EtherChannel load sharing, the EtherChannel Hash can be globally enabled for entire system or it can be on per MEC basis. Modifying EtherChannel Hash algorithm requires manually EtherChannel reset to make effective vss#show etherchannel 10 detail inc Hash Last applied Hash Distribution Algorithm: Fixed 6500-vss#show interface po10 etherchannel inc Load Gi Index Load Port EC state No of bits 0 FF Gi1/4/1 Desirable-Sl 8 2 FF Gi2/4/1 Desirable-Sl vss#show etherchannel 10 detail inc Hash Last applied Hash Distribution Algorithm: Fixed 6500-vss#conf t 6500-vss(config)#port-channel hash-distribution adaptive 6500-vss(config)#do show etherchannel 10 detail inc Hash Last applied Hash Distribution Algorithm: Fixed 6500-vss(config)#interface port-channel <id> 6500-vss(config-if)#shutdown 6500-vss(config-if)#no shutdown 6500-vss#show etherchannel 10 detail inc Hash Last applied Hash Distribution Algorithm: Adaptive TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 124

112 Layer 3 Load Balancing Can Be Randomized with a Unique ID Associated with Switch For Your Reference Universal ID concept (also called Unique ID) is used to prevent CEF polarization Universal ID generated at bootup (32-bit pseudo-random value seeded by router s base IP address) Universal ID used as input to ECMP hash, introduces variability of hash result at each network layer Universal ID supported on Catalyst 6500 Sup GE and Sup2T Universal ID supported on Catalyst 4500E Sup7E, Sup7LE and Catalyst 4500X Catalyst 4500E/X Load-Sharing Options Catalyst 6500E Load-Sharing Options Original Src IP + Dst IP Default * Src IP + Dst IP + Unique ID Universal * Src IP + Dst IP + Unique ID Full Src IP + Dst IP + Src Port + Dst Port Include Port Src IP + Dst IP + (Src OR Dst Port) Unique ID (Recommended) Full Exclude Port mple Full mple Src IP + Dst IP + (Src OR Dst Port) (Recommended) Src IP + Dst IP Src IP + Dst IP + Src Port + Dst Port * = default load-sharing mode TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 125

113 Cisco PAgP and IETF LACP Best Practices Link bundling protocols builds reliable logical network connections between two systems Cisco PAgP and IETF LACP protocol provides consistent solution Ensure link aggregation parameters consistency and compatibility between the VSS and neighbor switch. Ensure interface compliance with various aggregation requirements. Dynamically react to runtime changes and failures on local and remote Etherchannel systems Detect and remove unidirectional links and multidrop connections from the Etherchannel bundle Cisco PAgP MEC can be use for in-direct dual-active detection LACP Layer 3 Port-Channel SW1 PAgP Layer 2 Port-Channel VSL SW2 interface TenG1/2/1, TenG2/2/1 channel-protocol lacp channel-group <id> mode active interface TenG1/1/1, TenG2/1/1 channel-protocol pagp channel-group <id> mode desirable Recommended to implement in following modes for Layer 2 or Layer 3 EtherChannel : Cisco PAgP = Desirable / Desirable on both MEC end IETF LACP = Active / Active on both MEC end Keep PAgP and LACP timers to default settings Implement non-negotiable EtherChannel mode (ON) only when remote device do not support PAgP or LACP protocols, i.e. multi-home PC Catalyst 2K/3K/4K 4500E-VSS#show pagp neighbor Flags: S - Device is sending Slow hello. C - Device is in Consistent state. A - Device is in Auto mode. P - Device learns on physical port. Channel group 101 neighbors Partner Partner Partner PartnerGroup Port Name Device ID Port Age Flags Cap. Gi1/2/4 M c8c.a780 Gi1/1/1 17s SC Gi2/2/4 M c8c.a780 Gi1/1/2 4s SC TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 126

114 LACP Secondary Aggregator Interface For Your Reference During EtherChannel bundling process, LACP performs configuration check between physical bundle ports and port-channel and takes 2 following sequential actions : If configuration check pass, both end system establishes control and forwarding-plane information on user-defined port-channel group and both system function normally. If configuration check fails than it automatically generate an EtherChannel interface with unique alphabetical ID on each end device of an EtherChannel. System generated LACP MEC will bundle all the physical ports into an MEC that failed configuration check. All control, forwarding and management-plane will be independently operated over system generated LACP MEC. Such type of EtherChannel configuration mis-match condition will trigger dual individual layer 2 EtherChannel paths between access and virtualswitch nodes. STP topology will consider such network as a loop and block high STP port priority. Recommendation keep member-link configuration consistent to minimize network impact Active SW-1 Po20 Gi2/1 Switch#show etherchannel 20 summary inc Gi 20Po20(SU) LACP Gi2/1(P) Gi2/2(P) Switch#show spanning-tree inc Po20 Po20 Root FWD P2p VSL Gi2/2 Standby Switch(config)#int gi2/2 Switch(config-if)#switchport nonegotiate Switch(config-if)#shut Switch(config-if)#no shut %EC-SPSTBY-5-CANNOT_BUNDLE_LACP: Gi2/2 is not compatible with aggregators in channel 20 and cannot attach to them (trunk mode of Gi2/2 is trunk, Gi2/1 is dynamic) %EC-SP-5-BUNDLE: Interface Gi2/2 joined port-channel Po20B Switch#show etherchannel 20 summary inc Gi 20Po20(SU) LACP Gi2/1(P) 21Po20B(SU) LACP Gi2/2(P) 6500-access#show spanning-tree inc Po20 Po20 Root FWD P2p Po20B Altn BLK P2p Po20B Po20A SW-2 STP Block port MEC config check fail TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

115 mplify STP Network Topology with VSS VSS simplifies STP. VSS does not eliminate STP. Never disable STP Multiple parallel Layer 2 network path builds STP loop network VSS with MEC builds single loop-free network to utilize all available links. Distributed EtherChannel minimizes STP complexities compared to standalone distribution design STP toolkit should be deployed to safe-guard multilayer network STP BLK Port TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

116 mplify STP Network Topology with VSS VSS simplifies STP. VSS does not eliminate STP. Never disable STP Multiple parallel Layer 2 network path builds STP loop network VSS with MEC builds single loop-free network to utilize all available links. Distributed EtherChannel minimizes STP complexities compared to standalone distribution design STP toolkit should be deployed to safe-guard multilayer network STP BLK Port Loop-free L2 EtherChannel TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

117 mplify STP Network Topology with VSS VSS simplifies STP. VSS does not eliminate STP. Never disable STP Multiple parallel Layer 2 network path builds STP loop network VSS with MEC builds single loop-free network to utilize all available links. Distributed EtherChannel minimizes STP complexities compared to standalone distribution design STP toolkit should be deployed to safe-guard multilayer network STP BLK Port Loop-free L2 EtherChannel TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

118 mplify STP Network Topology with VSS VSS simplifies STP. VSS does not eliminate STP. Never disable STP Multiple parallel Layer 2 network path builds STP loop network VSS with MEC builds single loop-free network to utilize all available links. Rootguar d STP Root Distributed EtherChannel minimizes STP complexities compared to standalone distribution design STP toolkit should be deployed to safe-guard multilayer network STP BLK Port BPDU Guard or PortFast Port Security Loop-free L2 EtherChannel TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

119 FHRP Tuned Convergence vs. VSS Multi-Chassis EtherChannel based forwarding topologies Per-Flow Load Balancing based on Layer 2 to Layer 4 + VLANs Hardware-Based Fault Detection and Recovery Deterministic network convergence with simplistic approach Increases Network Scale for system reliability FHRP Active VSS-SW1 FHRP Standby No reliability compromise to enable path and system-level Quad-Sup redundancy Multilayer Standalone Network Scale And Convergence 6500-Sup2T TECH-LANWAN 4500-Sup7E SVI - Aggressive Timer Convergence (msec) Cisco and/or its affiliates. All rights reserved. Multilayer VSS Network Scale And Convergence HSRP Config 6500-Sup2T interface Vlan2 ip address standby 1 ip standby 1 timers msec 250 msec 750 standby 1 priority 150 SVI (Validated Limit) standby 1 preempt standby 1 preempt delay minimum Convergence 180 (msec) Cisco Public 4500-Sup7E 1

120 FHRP Tuned Convergence vs. VSS Multi-Chassis EtherChannel based forwarding topologies Per-Flow Load Balancing based on Layer 2 to Layer 4 + VLANs Hardware-Based Fault Detection and Recovery Deterministic network convergence with simplistic approach VSS-SW1 Increases Network Scale for system reliability No reliability compromise to enable path and system-level Quad-Sup redundancy Multilayer Standalone Network Scale And Convergence 6500-Sup2T TECH-LANWAN 4500-Sup7E SVI - Aggressive Timer Convergence (msec) Cisco and/or its affiliates. All rights reserved Sup2T Multilayer VSS Network Scale And Convergence Cisco Public 4500-Sup7E SVI (Validated Limit) Convergence (msec) 1

121 HSRP and VRRP Design Consideration Asymmetric Routing (Unicast Flooding) Alternating HSRP Active between distribution switches can be used for upstream load balancing, however downstream traffic hits both distribution block switches ARP (4 hours) and CAM (5 min) table timer mismatch may build inconsistent tables and cause unicast flooding VSS eliminates unicast flooding problem by automatically synchronizing ARP and CAM tables in local and remote switch hardware SW1: Active HSRP and Root Bridge VLAN 3 CAM Table Empty for VLAN 2 SW1 B SW1: ngle Root Bridge and Gateway for VLAN 2 and VLAN 3 B B SW2: Active HSRP and Root Bridge VLAN 2 B ngle auto synchronized ARP and CAM Table SW1 B SW2 CAM Table Empty for VLAN 3 VLAN 3 VLAN 2 VLAN 3 VLAN 2 VLAN 3 VLAN 2 VLAN 3 VLAN 2 TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

122 HSRP and VRRP Design Consideration Asymmetric Routing (Unicast Flooding) Alternating HSRP Active between distribution switches can be used for upstream load balancing, however downstream traffic hits both distribution block switches ARP (4 hours) and CAM (5 min) table timer mismatch may build inconsistent tables and cause unicast flooding VSS eliminates unicast flooding problem by automatically synchronizing ARP and CAM tables in local and remote switch hardware SW1: ngle Root Bridge and Gateway for VLAN 2 and VLAN 3 ngle auto synchronized ARP and CAM Table SW1 VLAN 3 VLAN 2 VLAN 3 VLAN 2 TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 1

123 Multi-Chassis EtherChannel Performs Better In Any Network Design Convergence (sec) Network Recovery mechanic varies in different distribution design Standalone Protocol and Timer dependent VSS Hardware dependent 1 VSS logical distribution system ngle P2P STP Topology ngle Layer 3 gateway ngle PIM DR system Distributed and synchronized forwarding table MAC address, ARP cache, IGMP L2-FHRP L2-MEC All links are fully utilized based on Ether-channel load balancing Upstream Downstream Multicast TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 142

124 Convergence (sec) Best Practice for Module OIR Module OIR is supported on all modular systems. Network recovery have higher impact with Module OIR due to OIR detection Hardware Synchronization Protocol Dependencies Forwarding Updates Minimize network impact with following techniques : Admin Power Down 0.5 Admin Reset 0 OIR Power Down Soft Reset 6500/6800 Standalone 6500E(config)# no power enable module <slot-id> Upstream Multicast Downstream 6500 /6800VSS 6500-VSS(config)# no power enable switch <1 2> module <slot-id> TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 143

125 Summary VSS vs Standalone For Your Reference STP Loop FHRP FHRP Tunings PIM DR Priority PIM Tunings Protocol Dependent Scale Unicast Flooding Asymmetric Forwarding L2 Hardening Network/System Redundancy Tradeoff Protocol Dependent Recovery CAM/ARP Tunings OSPF LSA/SPF Tuning Control/Mgmt/Forwarding Complexities Scale-independent Recovery Network/System Level Redundancy Hardware Driven Recovery Increase Unicast Capacity Increase Multicast Capacity mplified Network Topologies Control-plane mplicity Operational mplicity L2-L4 Load Sharing Flat L2 Network Cisco Public 144

126 VSS Core Network Design Alternatives ngle Link Network Design Full-Mesh Network Design VSL VSL SW1 SW2 SW1 SW2 Physical Design VSL VSL SW1 SW2 SW1 SW2 ECMP MEC ECMP Dual MEC ngle MEC Routing Design Recommended Design : Full-Mesh Physical Network with ngle MEC TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 145

127 VSS Core Network Design Analysis For Your Reference ngle Link ECMP ngle Link MEC Full-Mesh ECMP Full-Mesh Dual-MEC Full-Mesh ngle MEC Total physical links Total logical links Total layer 3 links ECMP routing path Per switch local forwarding path Routing Peers Double ngle Quadrupled Double ngle ngle link failure recovery mechanic ECMP via VSL ECMP MEC MEC NSF/SSO benefits No Yes Yes Yes Yes MEC Load-sharing benefits No No No Yes Yes Dual-Active Trust Support No Yes No Yes Yes Fast-Link Notification capability No Yes No Yes Yes ngle Link Failure Upstream Network Convergence (ave) ngle Link Failure Downstream Network Convergence (ave) Variable ~600 msec ~200-msec <=100 msec <=100 msec Variable ~600 msec ~200-msec <=100 msec <=100 msec Recommended Best Practice Core routing Design No No No No Yes TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 146

128 Convergence (sec) Implementing Non-Stop Forwarding VSS software design is built on NSF/SSO architecture. Catalyst 4500E/X and 6500E/6800 deployed in VSS mode must enabled NSF. No configuration required on NSF Helper system NSF capability must be manually enabled for all Layer 3 routing protocols : EIGRP, OSPF, ISIS, BGP, MPLS etc. In VRF environment the NSF must be manually enabled on per-vrf IGP instance Multicast NSF capability is default ON Inter-Chassis NSF/SSO Recovery Analysis Without NSF TECH-LANWAN With NSF 2014 Cisco and/or its affiliates. All rights reserved. EIGRP NSF Configuration 4500E(config)#router eigrp <AS#> 4500E(config-router)#nsf! 4500E#show ip protocols inc Routing EIGRP NSF *** IP Routing is NSF aware *** Routing Protocol is "eigrp 100" EIGRP NSF enabled <snip> OSPF NSF Configuration 6500E(config)#router ospf <PID#> 6500E(config-router)#nsf (cisco ietf)! 6500E#show ip ospf inc Routing Non-Stop NSF Routing Process "ospf 100" with ID Non-Stop Forwarding enabled IETF NSF helper support enabled Cisco NSF helper support enabled Multicast Redundancy Configuration 4500E#show ip multicast redundancy state Multicast IPv4 Redundancy Mode: SSO <snip> Cisco Public 147

129 VSS Dual-Active Detection Redundancy Dual-Sup or Quad-Sup VSL Redundancy Two Detection and Recovery Mechanic : In-Direct Detection = Enhanced PAgP (epagp) Direct Detection = Dual-Active Fast Hello Recommended to use epagp and Fast-Hello mechanic for redundancy. Catalyst 4500E/X supports Dual-Active Fast Hello from IOS-XE software release 1 epagp Trusted L3 Port-Channel Core 6500E VSS BFD detection mechanic is deprecated from 15.0(SY1) IOS software release 2 VSL Fast Hello Dist SW1 ACTIVE SW2 HOT-STANDBY Platform Enhanced PAgP Dual Active Fast Hello BFD Catalyst 6500E/6800 (Deprecated) Catalyst 4500E 1 epagp Trusted L2 Port-Channel Catalyst 2K/3K/4K Access Catalyst 4500X TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 148

130 Convergence (sec) Convergence (sec) 6500E Dual-Active Recovery Analysis Dual-Sup or Quad-Sup VSL Redundancy Dual-Active Network Recovery depends on Uplink Network Design ECMP vs MEC Routing Protocols EIGRP vs OSPF Detection Mechanic Fast-Hello vs epagp OSPF ECMP faster in failure detection then epagp. Slow network convergence Starting 12.2(33)SXI3 Dual-Active Fast-Hello performs rapid failure detection and delivers deterministic recovery independent of network design and protocol E VSS Dual-Active Recovery Analysis epagp EIGRP - ECMP EIGRP - ECMP Upstream EIGRP - MEC 6500E VSS Dual-Active Recovery Analysis Fast-Hello EIGRP - MEC OSPF - ECMP Downstream OSPF - ECMP OSPF - MEC OSPF - MEC Upstream Downstream TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 149

131 VSS Best Practices Summary For Your Reference Design each VSS domain with unique ID Configure mac-address use-virtual under virtual switch configuration mode Select appropriate VSS capable system that fits in network and solution requirements Deploy 6500E Quad-sup NSF/SSO for mission-critical networks to protect network availability and capacity Do not compromise network foundation baselines. Deploy full-mesh physical connections for redundancy and load sharing across the network MEC enables network benefits with VSS. Bundle all physical connections into single logical connection for simplified and resilient network topologies Layer 3 MEC highly recommended for 4500E/X VSS enabled Campus network Always use link bundling protocols Cisco PAgP or IETF LACP Plan and design VSL with appropriate capacity, diversification and redundancy TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 150

132 VSS Best Practices Summary For Your Reference Configure nsf under L3 routing protocols Keep Layer 2 and Layer 3 protocol timers at factory default. Do not enable protocols with aggressive timers Configure redundant dual active trusted epagp neighbors (L2/L3) Configure redundant dual active mechanics epagp and Fast Hello Exclude dual active management interface for connectivity and troubleshooting reload command on 6500E/6800 resets both virtual-switch chassis, whereas 4500E/X resets ACTIVE switch. Issue redundancy reload shelf on 4500E/X to reload ACTIVE and STANDBY system TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public 151

133 High Availability System Recovery Analysis

134 High Availability Recovery Analysis Let s look at the application of everything we have talked about today Analyze recovery times after major component failures in the recommended hierarchical design Understand better why we made the design choices High Availability Recovery Analysis Cisco Public 153

135 HA Analysis ECMP Node / Link Failure in the Core Upstream Recovery Downstream Recovery Recovery Mechanisms Node Failure 200 msec 200 msec L3 Equal Cost Path Core-to-Distribution Link Failure 200 msec 200 msec L3 Equal Cost Path Core-to-Core Link Failure 0 msec 0 msec No Loss of Forwarding Path Three failure / recovery cases 1 Node failure and restoration 2 Core-to-Distribution link failure and restoration 3 Core-to-Core link failure and restoration Other core failures covered by these three cases The network is symmetrical upstream and downstream and as a result the upstream and downstream convergence results are the same 1 Cisco Public Distribution Core Distribution

136 Distribution Block Recovery HSRP, EIGRP with Rapid PVST+ Uplink Fiber Fail to Active HSRP Uplink Fiber Fail to Standby HSRP Active HSRP Distribution Switch Failure Standby HSRP Distribution Switch Failure Inter-Switch Distribution Fiber Fail Upstream Recovery 900 msec 0 msec Downstream Recovery msec msec 800 msec 200 msec 0 msec 200 msec Recovery Mechanisms Upstream HSRP Downstream EIGRP Upstream No Loss Downstream EIGRP Upstream HSRP Downstream L3 Equal Cost Path Upstream No Loss Downstream L3 Equal Cost Path 0 msec 0 msec No Loss of Active Data Path Cisco Public 155

137 Distribution Block Recovery VSS EtherChannel EIGRP and Rapid PVST+ Access to Distribution Uplink Failure Active VSS Switch Failure Upstream Recovery Downstream Recovery < 200 msec 600 msec msec < 200 msec Recovery Mechanisms Upstream EtherChannel Downstream Multichassis EtherChannel Upstream SSO & EtherChannel Downstream SSO & ECMP Standby VSS Switch Restoration msec < 200msec Upstream SSO & EtherChannel Downstream ECMP VSL Link Failure (Dual Active Recovery) msec msec Upstream EtherChannel Downstream ECMP Note Please See VSS Design Guide for More Complete Analysis of All Potential Failure Cases Cisco Public 156

138 sec of lost voice VSS-Enabled Campus Design VSS Switchover Convergence For Your Reference Several factors to consider in planning a large campus with many VLANs spanning multiple switches include Max number of VLAN supported on lower-end switch platforms MAC address learning and TCAM capacity of other switches in the STP domain e.g. access-switch such as 2960 or 37xx Rate of MAC address change offered to VSS-enabled network, which may increases the control plane activity Exposure domain for virus and other infection control Existing subnet structure and VLAN sizing Validated with ESE Campus network environment 70 access-layer switches aka 70 MEC 8 VLANs Spanning multiple switches, total 150 VLANs Total 4K MAC 720 MAC / VLANs NSF aware adjacent node Default EIGRP and OSPF hello / hold Timers Native IOS 12.2(33) SXH2a Switchover from ACTIVE to HOT_STANDBY chassis is sub-second, without the complexity of existing design options NSF-aware core improves downstream convergence significantly L2 MEC Access Layer Average convergence for 37xx and 45xx is 200 msec Active VSS ACTIVE to HOT_STANDBY switchover convergence (Unicast) VSS Max Min HSRP Sub-second Timers HSRP Default Max Min Hot_ Standby Cisco Public 157

139 High Availability Switching Design Key Principals Enterprise network design architectures continue to evolve to meet business and technology needs, but the key principals of high availably network design still apply; Add redundancy and resiliency components as needed to meet the business requirements. mplify network designs and configurations through virtualization techniques. Implement network-monitoring tools with automation where appropriate, and analyze all aspects of network outages for indications of where improvement is needed. TECH-LANWAN 2014 Cisco and/or its affiliates. All rights reserved. Cisco Public

140 High Availability Network Design Stick to Your Principles Develop an architecture and stick to it Ease operational support Consistent deployment Balance OpEx and CapEx Remember you will have to live with this for a long time Requirements will change vs. Cisco Validated Designs: Plan for evolution The one thing that doesn t change is that there will be change Data Center Cisco Public 160

141 Prosíme, ohodnoťte tuto přednášku Děkujeme