Enterprise Campus Design: Routed Access

Transcription

1 Enterprise Campus Design: Routed Access 2

2 Some Loops are Fun... 5

3 But Not All... Sounds Familiar...? The whole network is down %IP-4-DUPADDR: Duplicate address on Vlan100, sourced by 00d0.04e0.63fc %IP-4-DUPADDR: Duplicate address on Vlan100, sourced by 00d0.04e0.63fc %IP-4-DUPADDR: Duplicate address on Vlan100, sourced by 00d0.04e0.63fc... I can t access anything Nothing seems to work All systems are unreachable %C4K_EBM-4-HOSTFLAPPING: Host 00:02:A5:8A:8B:5E in vlan 60 is flapping between port Gi3/6 and port Po9 %C4K_EBM-4-HOSTFLAPPING: Host 00:02:A5:8A:8B:5E in vlan 60 is flapping between port Gi3/6 and port Po9 %C4K_EBM-4-HOSTFLAPPING: Host 00:02:A5:8A:8B:5E in vlan 60 is flapping between port Gi3/6 and port Po9... Number of topology changes last change occurred 00:00:02 ago %PM-SP-4-LIMITS: Virtual port count for module 5 exceeded the recommended limit of 1800 %PM-SP-4-LIMITS: Virtual port count for switch exceeded the recommended limit of Many of us have suffered the consequences of a L2 loop 6

4 The Problem? One Solution... L2 Fails Open i.e. Broadcast and Unknowns flooded L3 Fails Closed i.e. neighbour lost L2 Control Plane Failure L3 Control Plane Failure... a loop and a network down... some subnets down 7

5 This Is Not About... L2 = BAD L3 = GOOD This is about... A design alternative that leverages L3 routing all the way down to the access layer, to see where it brings an advantage while we analyze the trade offs of using it. 8

6 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 9

7 Borderless Campus 21st Century Business Realities One Time Zone Real Time Workers, Customers, and Partners Operate Anywhere Rapid Collaborative Decisions Strict Governance for Compliance and Risk Reduction Resources Must be Leveraged to Their Maximum 10

8 Borderless Campus New Users, Applications, Services Badge Readers Unknown or Guest Partners Employees Subcontractor Consultant Campus Data Center 11

9 Borderless Campus Collaboration and Video Evolution IP Telephony (IPT) is now a mainstream technology Ongoing evolution to the full spectrum of Unified Communications High Definition Video Communications requires stringent Service-Level Agreement (SLA) Reliable Service High Availability Infrastructure Application Service Management End-to-End QoS 12

10 Traffic (Kbps) Medianet Application Requirements The Effect of Convergence Times on Media Flows 0.8 sec loss 0.4 sec loss Stresses and demands of video on the network expose shortcomings of good enough convergence Effect of 0.8 sec of Interruption on Diverse Multimedia Traffic 0.8 sec > 1 min 13

11 Fast Convergence and Reliability Are Essential... 14

12 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 15

13 Hierarchical Network Design Without a Rock Solid Foundation the Rest Doesn t Matter Access Distribution Core Distribution Offers hierarchy each layer has specific role Modular topology building blocks Easy to grow, understand, and troubleshoot Creates small fault domains clear demarcations and isolation Promotes load balancing and redundancy Promotes deterministic traffic patterns Incorporates balance of both Layer 2 and Layer 3 technology, leveraging the strength of both Can be applied to both the multilayer and routed campus designs Access WAN Building Block Internet 16

14 Multilayer Campus Network Design Layer 2 Access with Layer 3 Distribution L3 L2 Vlan 10 Vlan 20 Vlan 30 Vlan 30 Vlan 30 Vlan 30 Each access switch has unique VLAN s No layer 2 loops Layer 3 link between distribution No blocked links At least some VLAN s span multiple access switches Layer 2 loops Layer 2 and 3 running over link between distribution Blocked links

15 Multilayer Campus Network Design Well Understood Best Practices Mature, 10+ year old design Evolved due to historical pressures Cost of routing vs. switching Speed of routing vs. switching Non-routable protocols Well understood optimization of interaction between the various control protocols and the topology STP Root and HSRP primary tuning to load balance on uplinks HSRP Standby Root Bridge & HSRP Active RootGuard LoopGuard Spanning Tree Toolkit (RootGuard, LoopGuard, ) etc., CISF, BPDU Guard BRKCRS-2031 Multilayer Campus Architectures and Design Principals 18

16 Time to restore VoIP data flows (seconds) Multilayer Campus Network Design Good Solid Design Option Utilizes multiple Control Protocols FHRP Convergence Spanning Tree (802.1w, ) 10 FHRP (HSRP, VRRP, GLBP ) Routing Protocol (EIGRP, ) 8 Convergence is dependent on multiple factors 6 FHRP - 900msec to 9 seconds Spanning Tree - 400msec to 50 seconds 4 FHRP Load Balancing HSRP/VRRP Per Subnet 2 GLBP Per Host msec 3 secs HSRP Hello Timers 19

17 Multilayer Campus Network Design Layer 2 Loops and Spanning Tree Campus Layer 2 topology has sometimes proven a operational or design challenge Spanning tree protocol itself is not usually the problem, it s the external events that triggers the loop or flooding L2 has no native mechanism to dampen down a problem: L2 fails Open, as opposed to L3 which fails closed Implement Spanning Tree loops only when you have to DST MAC /2 3/2 3/1 3/1 Switch 1 Switch 2 DST MAC

18 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 21

19 Best Practices Campus Routing Leverage Equal Cost Multiple Paths Use routed pt2pt links and do not peer over client VLANs, SVIs. ECMP to quickly re-route around failed node/links with load balancing over redundant paths Tune CEF L3/L4 load balancing hash to achieve maximum utilization of equal cost paths (CEF polarization) Build triangles not squares for deterministic convergence Layer 3 Equal Cost Link s Layer 3 Equal Cost Link s Insure redundant L3 paths to avoid black holes Summarize distribution to core to limit event propagation Utilized on both Multi-Layer and Routed Access designs WAN Data Center Internet 22

20 Routed Interfaces Offer Best Convergence Properties Configuring L3 routed interfaces provides for faster convergence than a L2 switchport with an associated L3 SVI ~ 8 msec loss 1. Link Down 2. Interface Down 3. Routing Update L3 21:38: UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/1, changed state to down 21:38: UTC: %LINK-3-UPDOWN: Interface GigabitEthernet3/1, changed state to down 21:38: UTC: IP-EIGRP(Default-IP-Routing-Table:100): Callback: route_adjust GigabitEthernet3/1 1. Link Down ~ msec loss 2. Interface Down 3. Autostate 4. SVI Down 5. Routing Update L2 21:32: UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1, changed state to down 21:32: UTC: %LINK-3-UPDOWN: Interface GigabitEthernet2/1, changed state to down 21:32: UTC: %LINK-3-UPDOWN: Interface Vlan301, changed state to down 21:32: UTC: IP-EIGRP(Default-IP-Routing-Table:100): Callback: route, adjust Vlan301 23

21 Best Practice Build Triangles Not Squares Deterministic vs. Non-Deterministic Triangles: Link/Box Failure Does Not Require Routing Protocol Convergence Squares: Link/Box Failure Requires Routing Protocol Convergence Model A Model B Layer 3 redundant equal cost links provide fast convergence Hardware based fast recovery to remaining path Convergence is extremely fast (dual equal-cost paths: no need for OSPF or EIGRP to recalculate a new path) 24

22 Convergence (sec) CEF ECMP Optimize Convergence ECMP Convergence Is Dependent on Number of Routes Until recently, time to update switch HW FIB was linearly dependent on the number of entries (routes) to be updated Summarization and Filtering will decrease RP load as well as speed up convergence Time for ECMP/MEC Unicast Recovery Time for ECMP Recovery ECMP ECMP (SXI2) MEC Number or Routes in Area Sup720 25

23 CEF Load Balancing Underutilized Redundant Layer 3 Paths The default CEF hash input is L3 source and destination IP addresses Access Default L3 Hash Redundant Paths Ignored Imbalance/overload could occur CEF polarization: in a multihop design, CEF could select the same left/left or right/right path Redundant paths are ignored/underutilized Two solutions: 1. CEF Hash Tuning 2. CEF Universal ID Distribution Default L3 Hash Core Default L3 Hash Distribution Default L3 Hash Access Default L3 Hash L L 70% load R 30% load R 26

24 CEF Load Balancing 1. Avoid Polarization with CEF Hash Tuning With defaults, CEF could select the same left/left or right/right paths and ignore some redundant paths Access Default L3 Hash All Paths Used Alternating L3/L4 hash and default L3 hash will give us the better load balancing results The default is L3 hash no modification required in core or access Distribution L3/L4 Hash Core Default L3 Hash L R L R In the distribution switches use: mls ip cef load-sharing full to achieve better redundant path utilization Distribution L3/L4 Hash Access Default L3 Hash L R L Left de Shown 27

25 CEF Load Balancing 2. Avoid Polarization with Universal ID Cisco IOS uses Universal ID concept (also called Unique ID) to prevent CEF polarization Universal ID generated at bootup (32-bit pseudorandom value seeded by router s base IP address) Universal ID used as input to ECMP hash, introduces variability of hash result at each network layer Universal ID supported on Catalyst 6500 Sup-32, Sup-720, Sup-2T Universal ID supported on Catalyst 4500 SupII+10GE, SupV-10GE and Sup6E Hash using Source IP (SIP) + Destination IP (DIP) + Universal ID Catalyst 4500 Load-Sharing Options Original Src IP + Dst IP Universal* Src IP + Dst IP + Unique ID Include Port Src IP + Dst IP + (Src or Dst Port) + Unique ID Catalyst 6500 PFC3** Load-Sharing Options Default* Src IP + Dst IP + Unique ID Full Src IP + Dst IP + Src Port + Dst Port Full Exclude Port Src IP + Dst IP + (Src or Dst Port) * = Default Load-Sharing Mode mple Full mple Src IP + Dst IP Src IP + Dst IP + Src Port + Dst Port 28

26 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 29

27 Routed Access Design Layer 3 Distribution with Layer 3 Access: no L2 Loop EIGRP/OSPF EIGRP/OSPF Layer 3 Layer 3 Layer 2 EIGRP/OSPF GLBP Model EIGRP/OSPF Layer 2 Data / :DB8:CAFE:20::/64 Voice / :DB8:CAFE:120::/64 Data / :DB8:CAFE:40::/64 Voice / :DB8:CAFE:140::/64 Move the Layer 2/3 demarcation to the network edge Leverages L2 only on the access ports, but builds a L2 loop-free network Design Motivations: simplified control plane, ease of troubleshooting, high availability 30

28 Routed Access Advantages mplified Control Plane mplified Control Plane No STP feature placement (root bridge, loopguard, ) No default gateway redundancy setup/tuning (HSRP, VRRP, GLBP...) No matching of STP/HSRP priority No asymmetric flooding No L2/L3 multicast topology inconsistencies No Trunking Configuration Required L3 L2 Port Edge features still apply: Spanning Tree Portfast Spanning Tree BPDU Guard L3 L3 L3 L3 Port Security, DHCP Snooping, DAI, IPSG Storm Control 802.1x QoS Settings... 31

29 Routed Access Advantages mplified Network Recovery Routed Access network recovery is dependent on L3 re-route Time to restore upstream traffic flows is based on ECMP re-route Time to detect link failure Process the removal of the lost routes from the SW RIB Update the HW FIB Time to restore downstream flows is based on a routing protocol re-route Time to detect link failure Time to determine new route Process the update for the SW RIB Update the HW FIB Upstream Recovery: ECMP Downstream Recovery: Routing Protocol 32

30 Routed Access Advantages Faster Convergence Times RPVST+ convergence times dependent on FHRP tuning Proper design and tuning can achieve sub-second times EIGRP converges <200 msec OSPF converges <200 msec with LSA and SPF tuning Both L2 and L3 Can Provide Sub- Second Convergence RPVST+ FHRP OSPF Upstream Downstream EIGRP 33

31 Routed Access Advantages A ngle Router per Subnet: mplified Multicast Layer 2 access has two multicast routers per access subnet, RPF checks and split roles between routers Routed Access has a single multicast router which simplifies multicast topology and avoids RPF check altogether IGMP Querier (Low IP address) Non-DR has to drop all non-rpf Traffic Designated Router (High IP Address) Designated Router & IGMP Querier 34

32 Routed Access Advantages Ease of Troubleshooting Routing troubleshooting tools Consistent troubleshooting: access, dist, core show ip route / show ip cef Traceroute Ping and extended pings Extensive protocol debugs IP SLA from the Access Layer Failure differences Routed topologies fail closed i.e. neighbor loss Layer 2 topologies fail open i.e. broadcast and unknowns flooded L3 L3 switch#sh ip cef /24 nexthop TenGigabitEthernet9/4 L3 L3 L3 35

33 Routed Access Design Considerations Design Constrains Can t span VLANs across multiple wiring closet switches + Contained Broadcast Domains + But can have the same VLAN ID on all closets RSPAN no longer possible Can use ER-SPAN on Catalyst 6500 L3 IP addressing do you have enough address space and the allocation plan to support a routed access design? L3 L3 L3 L3 36

34 Routed Access Design Considerations Platform Requirements Catalyst Requirements Cisco Catalyst 3560 and 3750 Cisco Catalyst 4500 Cisco Catalyst 6500 Catalyst IOS IP Base minimum feature set EIGRP-Stub Edge Router L3 PIM Stub Edge Router OSPF for Routed Access 200 Dynamically Learned Routes L3 L3 L3 L3 Catalyst 3x00 Series IOS 12.2(55)SE Catalyst 4500 Series IOS 12.2(53)SG Catalyst 6500 Series IOS 12.2(33)SXI4 37

35 Routed Access Design Migrating from a L2 Access Model interface Vlan20 ip address ip helper-address standby 1 ip standby 1 timers msec 200 msec 750 standby 1 priority 150 standby 1 preempt standby 1 preempt delay minimum 180 VLAN 20 VLAN VLAN / / /24 EIGRP/OSPF 20, DHCP DNS interface GigabitEthernet1/1 switchport switchport trunk encapsulation dot1q switchport trunk allowed vlan switchport mode trunk switchport nonegotiate VLAN 20 VLAN VLAN 120 VLAN 20 GLBP Model VLAN VLAN 120 User Groups User Groups Typical deployment uses Vlan/Subnet for different user groups To facilitate user mobility, vlans extend to multiple closets 38

36 Routed Access Design Migrating from a L2 Access Model interface Vlan20 ip address ip helper-address standby 1 ip standby 1 timers msec 200 msec 750 standby 1 priority 150 standby 1 preempt standby 1 preempt delay minimum 180 VLAN 20 VLAN VLAN / / /24 EIGRP/OSPF L3 20, DHCP DNS interface GigabitEthernet1/1 description switchport Distribution Downlink ip switchport address trunk encapsulation dot1q switchport trunk allowed vlan switchport mode trunk switchport nonegotiate VLAN 20 VLAN VLAN 120 L3 L3 L3 L3 VLAN 20 GLBP Model VLAN VLAN 120 User Groups User Groups As the routing is moved to the access layer, trunking is no longer required /31 addressing can be used on p2p links to optimize ip space utilization 39

37 Routed Access Design Migrating from a L2 Access Model interface Vlan20 ip address ip helper-address standby 1 ip standby 1 timers msec 200 msec 750 standby 1 priority 150 standby 1 preempt standby 1 preempt delay minimum / / / / / / / / /25 EIGRP/OSPF L3 DHCP DNS interface Vlan20 ip address ip helper-address VLAN 20 VLAN VLAN 120 L3 L3 L3 L3 VLAN 20 GLBP Model VLAN VLAN 120 User Groups User Groups SVI configuration at the access layer is simplified Larger subnets are split into smaller ones and assigned to new DHCP scopes 40

38 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design EIGRP Design to Route to the Access Layer OSPF Design to Route to the Access Layer Other Design Considerations Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies 41

39 Deploying a Stable and Fast Converging EIGRP Campus Network The key aspects to consider are: 1. Using EIGRP Stub at the access layer 2. Route Summarization at the distribution layer 3. Leverage Route filters 4. Consider Hello and Hold Timer tuning 42

40 EIGRP Neighbors Event Detection EIGRP neighbor relationships are created when a link comes up and routing adjacency is established When physical interface changes state, the routing process is notified Carrier-delay should be set as a rule because it varies based upon the platform Some events are detected by the routing protocol Neighbor is lost, but interface is UP/UP To improve failure detection Use routed interfaces and not SVIs Decrease interface carrier-delay to 0 Decrease EIGRP hello and hold-down timers* Hello = 1 Hold-down = 3 * Not recommended with NSF/SSO interface GigabitEthernet3/2 ip address ip hello-interval eigrp ip hold-time eigrp carrier-delay msec 0 Hellos L2 Switch or VLAN Interface Routed Interface 43

41 EIGRP in the Campus Conversion to an EIGRP Routed Edge The greatest advantages of EIGRP are gained from the use of summarization and stub routers /16 EIGRP allows for multiple tiers of hierarchy, summarization and route filtering Relatively painless to migrate to a L3 access with EIGRP Deterministic convergence time in very large L3 topology / /17 EIGRP maps easily to campus topology 44

42 EIGRP Design Rules for HA Campus Limit Query Range to Maximize Performance EIGRP convergence is dependent on query response times Minimize the number of queries to speed up convergence Summarize distribution block routes to limit how far queries propagate across the campus Upstream queries are returned immediately with infinite cost Configure access switches as EIGRP stub routers No downstream queries are ever sent interface TenGigabitEthernet 4/1 ip summary-address eigrp router eigrp 100 network distribute-list Default out <mod/port> ip access-list standard Default permit router eigrp 100 network eigrp stub connected 45

43 EIGRP Query Process Queries Propagate the Event EIGRP relies on neighbors to provide routing information If a route is lost and no feasible successor is available, EIGRP actively queries its neighbors for the lost route(s) The router waits for replies from all queried neighbors before the calculating a new path If any neighbor fails to reply, the queried route is stuck in active and the router resets neighbor adjacency Query Reply Query Reply Query Reply Query Reply Query Reply Query Reply Access Distribution Core The fewer routers and routes queried, the faster EIGRP converges; solution is to limit query propagation Query Query Reply Distribution Query Reply Reply Access 46

44 Limiting the EIGRP Query Range With Summarization Summarization from distribution to core for the subnets in the access limits the upstream query/reply process Queries will now stop at the core; no additional distribution blocks will be involved in the convergence event The access layer is still queried Summary Route Reply Query No Queries to Rest of Network from Core Reply Query Reply Summary Route interface gigabitethernet 3/1 ip address ip summary-address eigrp Query Reply Reply 47

45 Limiting the EIGRP Query Range With Stub Routers A stub router signals (through hellos) that it is a stub and not a transit path Queries are not sent towards the stub routers but marked as if a No path this direction reply had been received D1 knows that stubs cannot be transit paths, so they will not have any path to /24 D1 will not query the stubs, reducing the total number of queries in this example to one Stubs will not pass D1 s advertisement of /24 to D2 D2 will only have one path to /24 D1 STUB /24 Distribution Query Access Reply D2 I m Not Going to Send You Any Queries nce You Said That Hello, I m a Stub Stub Stub Stub 48

46 EIGRP Query Process With Summarization and Stub Routers When we summarize from distribution into core we can limit the upstream query/reply process Queries will now stop at the core; no additional routers will be involved in the convergence event With EIGRP stubs we can further reduce the query diameter Non-stub routers do not query stub routers so no queries will be sent to the access nodes Only three nodes involved in convergence event No secondary queries Summary Route Reply Query No Queries to Rest of Network from Core Stub Reply Reply Stub Summary Route 49

47 EIGRP Route Filtering in the Campus Control Route Advertisements Campus bandwidth not a constraining factor but it is recommended to limit the number of routes advertised Remove/filter routes from the core to the access and inject a default route with distribute-lists Smaller routing table in access is simpler to troubleshoot Deterministic topology Default Default & other Routes ip access-list standard Default permit router eigrp 100 network distribute-list Default out <mod/port> 50

48 EIGRP Routed Access Campus Design Summary Detect the event: Set hello-interval = 1 second and hold-time = 3 seconds to detect soft neighbor failures * Set carrier-delay = 0 Propagate the event: Configure all access layer switches as stub routers to limit queries from the distribution layer Summarize the routes from the distribution to the core to limit queries across the campus Process the event: Summarize and filter routes to minimize calculating new successors for the RIB and FIB Default Default & other Routes Summary Route * Not recommended with NSF/SSO Stub Stub Stub 51

49 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design EIGRP Design to Route to the Access Layer OSPF Design to Route to the Access Layer Other Design Considerations Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies 52

50 Deploying a Stable and Fast Converging OSPF Campus Network Key Objectives of the OSPF Campus Design: 1. Map area boundaries to the hierarchical design 2. Enforce hierarchical traffic patterns 3. Minimize convergence times 4. Maximize stability of the network 53

51 OSPF Design Rules for HA Campus Where Are the Areas? Area size/border is bounded by the same concerns in the campus as the WAN In campus the lower number of nodes and stability of local links could allow you to build larger areas however- Area design also based on address summarization Area boundaries should define buffers between fault domains Keep area 0 for core infrastructure do not extend to the access routers Area 100 Area 110 Area 120 Area 0 WAN Data Center Internet 54

52 Hierarchical Campus Design OSPF Areas with Router Types Area 10 Area 20 Area 30 Access Internal Internal Distribution ABR ABR ABR Core Area 0 Backbone Backbone Area 0 Distribution ABR ABR ASBR Area 300 Access Area 200 WAN Data Center Area 100 Internet BGP 55

53 OSPF in the Campus Conversion to an OSPF Routed Edge OSPF designs that utilize an area for each campus distribution building block allow for straight forward migration to Layer 3 access Converting L2 switches to L3 within a contiguous area is reasonable to consider as long as new area size is reasonable How big can the area be? It depends Switch type(s) Number of links Area 20 Dist 2 Stability of fiber plant Area 200 Branches Area 0 Core Area 10 Dist 1 56

54 When a Link Changes State Router 1, Area 1 LSA Router 2, Area 1 ACK Link State Table Every router in area hears a specific link LSA Each router computes shortest path routing table Old Routing Table Dijkstra Algorithm New Routing Table 57

55 OSPF LSA Process LSAs Propagate the Event Area 0 OSPF is a Link State protocol; it relies on all routers within an area having the same topology view of the network. LSA SPF 2 LSA SPF 2 Access If a route is lost, OSPF sends out an LSA to inform it s peers of the lost route. LSA SPF 2 LSA SPF 2 Distribution All routers with knowledge of this route in the OSPF network will receive an LSA and run SPF to remove the lost route. The fewer the number of routers with knowledge of the route, the faster OSPF converges; Solution is to limit LSA propagation range LSA SPF 2 LSA SPF 2 LSA SPF 2 LSA SPF 2 Core Distribution LSA SPF 2 Area 0 SPF Access 58

56 OSPF Regular Area ABRs Forward All LSAs from Backbone External Routes/LSA Present in Area 120 ABR Forwards the Following into an Area Summary LSAs (Type 3) ASBR Summary (Type 4) Specific Externals (Type 5) Backbone Area 0 Distribution Config router ospf 100 area 120 range cost 10 network area 120 network area 0 Area 120 Access Config: router ospf 100 network area

57 OSPF Stub Area Consolidates Specific External Links Default Eliminates External Routes/LSA Present in Area (Type 5) Backbone Area 0 Stub Area ABR Forwards Summary LSAs Summary Default Distribution Config router ospf 100 area 120 stub area 120 range cost 10 network area 120 network area 0 Area 120 Access Config: router ospf 100 network area

58 OSPF Totally Stubby Area Use This for Stable Scalable Internetworks Minimize the Number of LSAs and the Need for Any External Area SPF Calculations Backbone Area 0 A Totally Stubby Area ABR Forwards Summary Default Distribution Config router ospf 100 area 120 stub no-summary area 120 range cost 10 network area 120 network area 0 Area 120 Access Config: router ospf 100 network area

59 Summarization Distribution to Core Reduce SPF and LSA Load in Area 0 Minimize the Number of LSAs and the Need for Any SPF Recalculations at the Core Backbone Area 0 ABRs Forward Summary /16 Area Border Router Distribution Config router ospf 100 area 120 stub no-summary area 120 range cost 10 network area 120 network area 0 Area 120 Access Config: router ospf 100 network area

60 OSPF Design Considerations What Area Should the Distribution Link Be In? Two aspects of OSPF behavior can impact convergence OSPF ABRs ignore LSAs generated by other ABRs learned through non-backbone areas when calculating least-cost paths In a stub area environment the ABR will generate a default route when any type of connectivity to the backbone exists Ensure loopbacks are not in area 0 Configure dist to dist link as a trunk using 2 subnets one in area 0 and one in stub area when possible 63

61 OSPF Timer Tuning High-Speed Campus Convergence OSPF by design has a number of throttling mechanisms to prevent the network from thrashing during periods of instability Campus environments are candidates to utilize OSPF timer enhancements Sub-second hellos* Generic IP (interface) dampening mechanism Back-off algorithm for LSA generation Exponential SPF backoff Configurable packet pacing Reduce Hello Interval Reduce LSA and SPF Interval * Not recommended with NSF/SSO 64

62 Subsecond Hellos Neighbor Loss Detection Physical Link Up OSPF hello/dead timers detect neighbor loss in the absence of physical link loss Useful where an L2 device separates L3 devices (Layer 2 core designs) Fast timers quickly detect neighbor failure Not recommended with NSF/SSO Interface dampening is recommended with sub-second hello timers OSPF Processing Failure (Link Up) OSPF point-to-point network type to avoid designated router (DR) negotiation. Access Config: interface GigabitEthernet1/1 dampening ip ospf dead-interval minimal hello-multiplier 4 ip ospf network point-to-point router ospf 100 timers throttle spf timers throttle lsa all timers lsa arrival 80 A B 65

63 Time to Restore Voice Flows (sec) OSPF Requires Sub-Second Throttling of LSA Timers to Speed Convergence OSPF has an SPF throttling timer designed to dampen route recalculation After a failure, the router waits for the SPF timer to expire before recalculating a new route By default, there is a 500ms delay before generating router and network LSAs; the wait is used to collect changes during a convergence event and minimize the number of LSAs sent Propagation of a new instance of the LSA is limited at the originator Acceptance of a new LSAs is limited by the receiver Make sure lsa-arrival < lsa-hold Default Convergence msec. SPF 10 msec. SPF and LSA timers throttle spf timers throttle lsa all timers lsa arrival 80 66

64 OSPF Design Rules for HA Campus LSA/SPF Exponential Back-off Throttle Mechanism Topology Change Events msec 1600 msec SPF Calculations Time [ms] timers throttle spf timers throttle lsa all timers throttle spf <spf-start> <spf-hold> <spf-max-wait> timers throttle lsa all <lsa-start> <lsa-hold> <lsa-max-wait> Sub-second timers without risk 1. spf-start (initial hold timer) controls how long to wait prior to starting the SPF calculation 2. If a new topology change event is received during the spf-hold interval, the SPF calculation is delayed until the hold interval expires and the hold interval is temporarily doubled 3. The spf-hold interval can grow until the maximum period spf-max-wait is reached 4. After the expiration of any hold interval, the spf-hold timer is reset 67

65 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design EIGRP Design to Route to the Access Layer OSPF Design to Route to the Access Layer Other Design Considerations Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies 68

66 IP Event Dampening to Reduce Routing Churn Prevents routing protocol churn caused by constant interface state changes Dampening is applied on a system: nothing is exchanged between routing protocols Supports all IP routing protocols Static routing, RIP, EIGRP, OSPF, IS-IS, BGP In addition, it supports HSRP and CLNS routing Up Down Up Down Up Down Up Applies on physical interfaces and can t be applied on subinterfaces individually interface GigabitEthernet1/1 description Uplink to Distribution 1 dampening ip address Up Down Up Down Interface State Interface State Perceived by EIGRP or OSPF 69

67 Redundant Supervisors with L3 Non-Stop-Forwarding with Stateful Switchover (NSF/SSO) Active Supervisor Synchronization Standby Supervisor RP CPU Control Path Routing Protocol process Configuration RP CPU Routing Information Base ARP Table Cisco IOS CEF Tables Synchronization ARP Table IOS CEF Tables FIB Table Adjacency Table IOS CEF Tables FIB Table Adjacency Table Hardware FIB Table Adjacency Table Hardware Tables Synchronization Hardware FIB Table Adjacency Table Forwarding Path 70

68 Access Layer Redundant Supervisors with SSO 1. Supervisor switchover event occurs 2. SSO maintains SSO-aware applications, including L2 tables, L2/L3 forwarding is maintained 3. Routing protocols will restart on the newly active Supervisor L3 routes are purged stopping L3 forwarding 4. Routing neighbors lose adjacency with the restarting router Routes to the lost neighbor are purged 5. Routing neighbors reestablish adjacencies, forwarding to and from nondirectly connected L3 networks resumes SSO alone is not enough with a Routed Access do not run SSO w/o NSF in the RA design 71

69 NSF Configuration and Monitoring EIGRP Switch(config)#router eigrp 100 Switch(config-router)#nsf OSPF Switch(config)#router ospf 100 Switch(config-router)#nsf Router#sh ip protocol *** IP Routing is NSF aware *** Routing Protocol is "eigrp " <snip EIGRP NSF-aware route hold timer is 240s EIGRP NSF enabled NSF-Aware NSF-Capable Router#sh ip ospf Routing Process "ospf 100" with ID Start time: 00:01:37.484, Time elapsed: 3w2d Supports Link-local gnaling (LLS) <snip> Non-Stop Forwarding enabled, last NSF restart 3w2d ago (took 31 secs) Recommendation Is to Not Tune IGP Hello Timers. Use Default Hello and Dead Timers for EIGRP/OSPF When Peering to a Device Configured for NSF/SSO 72

70 Access Layer Redundant Supervisors, Now with NSF/SSO 1. Supervisor switchover event occurs 2. SSO maintains SSO-aware applications, including L2 tables, L2/L3 forwarding is maintained 3. NSF-capable router signals NSF-aware routing peers of a routing protocol restart 4. NSF-aware routers detect the restarting router 4 Assist in re-establishing full adjacency Maintain forwarding to and from the restarting router 5. NSF restart complete, traditional L3 convergence event is avoided

71 Design with Redundant for NSF/SSO Status of Uplinks of the Supervisor Catalyst 4500 Supervisor II+, Supervisor IV: 2 x GigE ports are active 1/1 1/2 Cisco Catalyst 4500: supervisor uplink ports are active and forward traffic as long as the supervisor is fully inserted Uplink ports do not go down when a supervisor is reset. There are restrictions on which ports can be active simultaneously in redundant systems Cisco Catalyst 6500: both the active supervisor and the standby supervisor uplink ports are active as long as the supervisors are up and running Uplink ports go down when the supervisor is reset 2/1 2/2 Catalyst 4500 Supervisor II+10GE: 2 x 10GE and 4 x GigE ports are active 1/1 1/2 1/3 1/4 1/5 1/6 2/1 2/2 2/3 2/4 2/5 2/6 Catalyst 6500 Supervisors: all ports are active An NSF/SSO switchover also modifies topology 74

72 StackWise at the Access Layer Recommended Design: Configure priority for master and its backup for deterministic failures Avoid using master as uplink to reduce uplink related losses Use stack-mac persistent timer 0 to avoid the gratuitous ARP changes for Best convergence Where GARP processing is disabled in the network, e.g. Security Where network devices/host do not support GARP, e.g. Phones Upstream traffic is not interrupted by master failure Downstream traffic is interrupted due to routing protocol restart and adjacency reset Run 12.2(37)SE or higher for NSF support Access Master S1 S2 S3 ngle logical Switch 75

73 Routed Access Does Not Require Switch Management Vlan In the L2 design it was considered a best practice to define a unique Vlan for network management In the routed access model, the best way is to configure a loopback interface The /32 address should belong to the summarized routed advertised from the distribution block The loopback interface should be configured as passive for the IGP ACLs should be used as required to ensure secure network management SNMP Server interface Loopback0 description Dedicated Switch Management ip address

74 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 77

75 Virtual Switch Catalyst 6500 Virtual Switching System (VSS) Virtual Switching System consists of two Catalyst 6500 s defined as members of the same virtual switch domain running a VSL (Virtual Switch Link) between them ngle Control Plane with Dual Active Forwarding Planes Extends NSF/SSO infrastructure to Two Switches Virtual Switch Domain Virtual Switch Link (VSL) Switch 1 + Switch 2 = VSS 78

76 Virtual Switch System Multi-Chassis Etherchannel Multi-chassis Etherchannel (MEC) replaces spanning tree to provide link redundancy MEC allows the physical members of the Etherchannel bundle to be connected to two separate physical switches MEC links on both switches are managed by PAgP or LACP running on the Master Switch via internal control messages PAgP or LACP packets for all links in the MEC bundle are processed by the active supervisor Multi-Chassis Etherchannel 79

77 Virtual Switch System Impact to the Campus Topology Physical network topology does not change Still have redundant chassis Still Still have have redundant redundant links links Logical topology is simplified as we now have a single control plane Allows the design to replace traditional topology control plane with Multi-chassis Etherchannel (MEC) No No reliance on on IGP spanning Protocol tree to to provide link redundancy Convergence and and load load balancing balancing are are based based on on Etherchannel Etherchannel BRKCRS-3035 Advance Enterprise Campus Design: Virtual Switching System (VSS) 80

78 Leveraging EtherChannel Time to Recovery Link failure detection Removal of the Portchannel entry in the software Update of the hardware Portchannel indices Notify the spanning tree and/or routing protocol processes of path cost change 1 2 Link Failure Detection Catalyst Switch Layer 2 Forwarding Table PortChannel 1 G3/1, G3/2, G4/1, G4/2 3 4 Routing Protocol Process Spanning Tree Process VLAN MAC Destination Index 10 AA Portchannel 1 11 BB G5/1 Load-Balancing Hash Destination Port G3/1 G3/2 G4/1 G4/2 81

79 VSS and Routed Access Design Link Down Convergence Without VSS Downstream traffic recovery is dependent upon the Interior Gateway Protocol reroute to the peer distribution switch Use Stub on the access devices, and proper summarization from distribution Tune IGP... etc. Upstream traffic recovery is dependent upon updates to the Access Switch s Forwarding Information Base removing the adjacency for the lost link (ECMP) Downstream IGP reroute Upstream CEF ECMP L3 ECMP 82

80 VSS and Routed Access Design Link Down Convergence with VSS MEC Access layer switch has one neighbor Distribution switch has neighbor count reduced by half Upstream and Downstream traffic convergence now is an Etherchannel link event No IGP reconvergence event No Impact of number of routes/vlans Fast IGP Timers not needed nor recommended (only 1 IGP peer) Summarization rules still recommended Achieves sub-second failure and no L2 loop on the topology Downstream IGP reroute Upstream CEF ECMP L3 MEC ECMP 83

81 VSS and Routed Access Design Enable MEC Links in L3 Core Best Multicast Use MEC uplinks from the access in routed access environments with multicast traffic VSS MEC local switch link preference avoids egress replication across the VSL link during normal conditions MEC Uplinks PIM Join In the event of link failure multicast traffic will pass across VSL link and will experience local switch replication SW2 HOT_STANBY SW1 ACTIVE Large scale mroute and s,g topology the convergence may vary, however much better then ECMP based topology PIM Joins L3 MEC Uplinks 84

82 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 85

83 Routed Access Layer and IPv6 Support for Dual Stack Deployment IPv4 and IPv6 Dual Stack is the recommended deployment model In RA model, the first hop switch must be capable of routing IPv6 EIGRP-Stub and OSPFv3 Routed Access Catalyst IPv6 Routing Cisco Catalyst 6500 Series Switches SUP32, SUP720, SUP2T Cisco Catalyst 4500 Series Switches SUP6-E and higher Cisco Catalyst 3750 Series, E Series, and X Series Switches Cisco Catalyst 3560 Series, E Series, and X Series Switches 86

84 Routed Access Layer and IPv6 Dual Stack Deployment Sample ipv6 unicast-routing ipv6 cef! [...] interface Vlan2 description Data VLAN for Access ipv6 address 2001:DB8:CAFE:2::CAC1:3750/64 ipv6 nd prefix 2001:DB8:CAFE:2::/64 no-advertise ipv6 nd managed-config-flag ipv6 nd other-config-flag ipv6 dhcp relay destination 2001:DB8:CAFE:10::2 ipv6 ospf 1 area 2 ipv6 cef! [...] ipv6 router ospf 1 router-id log-adjacency-changes auto-cost reference-bandwidth area 2 stub no-summary passive-interface Vlan2 timers spf 1 5 IPv6/IPv4 Dual Stack Hosts v6- Enabled v6- Enabled v6-enabled Dual-stack Server Dual Stack Dual Stack L3 v6- Enabled v6- Enabled v6-enabled For Your Reference Access Layer Distribution Layer Core Layer Aggregation Layer (DC) Access Layer (DC) 87

85 Routed Access Layer and IPv6 Dual Stack Deployment Sample! interface GigabitEthernet1/0/25 description To 6k-dist-1 ipv6 address 2001:DB8:CAFE:1100::CAC1:3750/64 no ipv6 redirects ipv6 nd suppress-ra ipv6 ospf network point-to-point ipv6 ospf 1 area 2 ipv6 ospf hello-interval 1 ipv6 ospf dead-interval 3 ipv6 cef! interface GigabitEthernet1/0/26 description To 6k-dist-2 ipv6 address 2001:DB8:CAFE:1101::CAC1:3750/64 no ipv6 redirects ipv6 nd suppress-ra ipv6 ospf network point-to-point ipv6 ospf 1 area 2 ipv6 ospf hello-interval 1 ipv6 ospf dead-interval 3 ipv6 cef IPv6/IPv4 Dual Stack Hosts v6- Enabled v6- Enabled v6-enabled Dual-stack Server Dual Stack Dual Stack L3 v6- Enabled v6- Enabled v6-enabled For Your Reference Access Layer Distribution Layer Core Layer Aggregation Layer (DC) Access Layer (DC) 88

86 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 89

87 Analyzing the Impact on Advanced Technologies Unified Communications Deployments work the same way. You still need to provision a voice vlan/subnet per wiring closet switch TrustSec (802.1x) solutions work the same: user vlan assignment still possible, as well as per user dacl (checkout BRKSEC-2005) Wireless LAN works seamlessly as well, since LWAPP works with UDP hence at L3. We will take a closer look at; Network Virtualization 90

88 Network Virtualization Functional Architecture Access Control Path Isolation Services Edge Branch Campus WAN MAN Campus Data Center Internet Edge Campus GRE VRFs MPLS VPNs Ethernet VRFs Access control techniques remain the same with a Routed Access Model Path Isolation techniques remain the same, but there are provisioning implications by running routing at the access layer BRKCRS-2033 Deploying a Virtualized Campus Network Infrastructure 91

89 Path Isolation Functional Components Device virtualization Control plane virtualization Data plane virtualization Services virtualization Per VRF Virtual Routing Table Virtual Forwarding Table VRF VRF Global Data path virtualization Hop-by-Hop: VRF-Lite End-to-End Multi-Hop: VRF-Lite+GRE, MPLS-VPN 802.1q IP VRF: Virtual Routing and Forwarding 92

90 Network Virtualization and Routed Access Path Isolation Issues VRFs to the Edge Define VRFs on the access layer switches Campus Core Layer 3 Links One VRF dedicated to each virtual network (Red, Green, etc.) Map device VLANs to corresponding VRF Provisioning is more challenging, because multiple routing processes and logical interfaces are required. The chosen path isolation technique must be deployed from the access layer devices EVNs VRF-lite Ethernet VRF-Lite GRE MPLS L3 VPNs VRF Red VLAN 21 Red VLAN 22 Green VLAN 23 Blue VLAN 21 Red VLAN 22 Green VLAN 23 Blue VRF Green VRF Blue 93

91 Virtualizing at the Access Layer VLANs to VRF Mapping Configuration ip vrf Red! ip vrf Green! vlan 21 name Red_access_switch_1! vlan 22 name Green_access_switch_1! interface Vlan21 description Red on Access Switch 1 ip vrf forwarding Red ip address ! interface Vlan22 description Green on Access Switch 1 ip vrf forwarding Green ip address Defining the VRFs Defining the VLANs (L2 and SVI) and Mapping Them to the VRFs 94

92 OSPF Example EIGRP Example VRF-Lite Routing Protocol Example router eigrp 100 network passive-interface default no passive-interface vlan 2000 no auto-summary! address-family ipv4 vrf green autonomous-system 100 network no auto-summary exit-address-family! address-family ipv4 vrf red autonomous-system 100 network no auto-summary exit-address-family router ospf 1 network area 0 passive-interface default no passive-interface vlan 2000! router ospf 100 vrf green network area 0 no passive-interface vlan 2001! router ospf 200 vrf red network area 0 no passive-interface vlan 2002 Defining the Routing Protocol within the VRFs 95

93 Network Virtualization and Routed Access Path Isolation Issues VRFs to the Edge (Cont.) Catalyst 6500 supports all three path isolation techniques: Campus Core 802.1Q Ethernet VRF-Lite GRE with VRF-Lite MPLS VPN Catalyst 3000s and 4500s only support 802.1Q Ethernet VRF-Lite Layer 3 Links Convergence times increase ~800ms for 9 VRFs + Global Increased load from multiple routing processes and logical interfaces VLAN 21 Red VLAN 22 Green VLAN 23 Blue VLAN 21 Red VLAN 22 Green VLAN 23 Blue Operational impact of managing multiple logical networks VRF Red VRF Green VRF Blue Network Virtualization--Path Isolation Design Guide 96

94 Easy Virtual Networks (EVN) Summary A simple, IP-based L3 VPN network virtualization solution Makes VRF-lite easier to deploy, operate and scale Based on existing network designs Interoperable with VRF-lite and MPLS-VPN LAN VNET Trunks VLAN-ID reuse Sub-interface inheritance Route Replication IGP based Shared Services Enhanced Troubleshooting and Usability routing-context, traceroute, debug condition, cisco-vrf-mib New BRKVIR Present and Future Services in Network Virtualization 97

95 EVN - Easy Virtual Network Roadmap Platform Phase 1 ASR1K IOS XE 3.2S Cat6K Sup2T Cat4K Cat6K Sup720* Cat3K-X ISR-G2 Nexus 7K 15.0(1)SY1 IOS XE SG, 15.1(1)SG Future Future Future Future EVN is planned to be available on all platforms and versions in months Many of the products and features described herein remain in varying stages of development and will be offered on a when-and-if-available basis. This roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document. 98

96 Enterprise Campus Design: Routed Access Agenda Introduction Cisco Campus Architecture Review Campus Routing Foundation and Best Practices Building a Routed Access Campus Design Routed Access Design and VSS Routed Access Design for IPv6 Impact of Routed Access Design for Advanced Technologies Summary 99

97 Routed Access Campus Design End to End Routing: Fast Convergence and Maximum Reliability B B B B B = STP Blocked Link STP-Based Redundant Topology Routed Access Redundant Topology 100

98 Summary Layer 2 designs remain valid Routed Access Design: mplified Control Plane: no dependence on STP, HSRP, etc. Increased Capacity: flow-based load balancing High Availability: 200 msec or better recovery mplified Multicast No L2 Loops fails closed, no flooding Easy Troubleshooting Flexibility to provide the right implementation for each requirement 101

99