vpc Best Practices and Design on NX-OS

Transcription

1

2 vpc Best Practices and Design on NX-OS Nemanja Kamenica Engineer, Technical Marketing BRKDCN-2378

3 Cisco Spark How Questions? Use Cisco Spark to chat with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space Cisco Spark spaces will be available until July 3, cs.co/ciscolivebot#brkdcn Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Session Goal To provide a thorough understanding of the Virtual Port Channel, design and best practices of vpc configuration. This session will examine best practice of vpc in environments with: Nexus 2000, firewalls, load-balancers, and vpc with Dynamic routing. Examine best practice of vpc with ACI, VxLAN, FCoE, FabricPath networks. This session will not examine in depth Nexus switch architecture, FCoE, Fabric Path, VxLAN, ACI, and Nexus BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 4

5 Agenda Introduction to vpc Feature Overview Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

6 vpc Feature Overview vpc Concept & Benefits STP S3 S3 vpc Physical Topology S3 vpc Logical Topology No Blocked Ports More Usable Bandwidth BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 6

7 Virtual Port Channel - vpc Benefits Eliminates Spanning Tree blocked ports by providing loop-free topology Better bandwidth utilization Provides device level redundancy with faster convergence MC-LAG on Cisco Nexus Devices Deployed by almost 95% of Nexus customers Unified Fabric BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Data Center Technology Evolution MPLS, OTV, LISP MPLS, OTV, LISP FabricPath with vpc+ FEX with vpc VPC STP 2010 VXLAN ACI Used to redundantly connect network entities at the edge of the Fabric BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Agenda Introduction to vpc Feature Overview Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

10 Feature Overview vpc Terminology L3 vpc Peer Layer 3 Cloud P vpc Domain S vpc Peer Keepalive Link Peer-Link Orphan Port CFS vpc vpc Member Port Orphan Device S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 vpc Peer-keepalive link L3 L3 link, connects vpc peers Carries periodic hart beet between vpc peers Uses UDP port 3200 Sends keep-alive heart beets every second vpc Domain S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 vpc Peer-link vpc peer link is a port channel that carries: vpc VLANs CFS messages Flooded traffic from the other peer device STP BPDUs, HSRP hello messages and IGMP updates vpc imposes the rule that peer-link should never be blocking L3 vpc Domain S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 vpc L3 Consists of port-channel member of vpc L2 port channel Ports in vpc can be in access or trunk mode VLANs allowed on vpc need to be allowed on peer-link vpc Domain LACP and Static port channel configuration S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 Cisco Fabric Services Protocol Synchronization and consistency checking mechanism Runs on VPC Peer-link CFS protocols mechanism: Validation and comparison for consistency check Synchronization of MAC addresses for member ports Status of member ports advertisement STP management Synchronization of HSRP and IGMP snooping Enabled by default L3 vpc Domain CFS S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 vpc Consistency check System configuration must be in sync Type 1 Consistency Check Graceful Consistency check suspends: Per-interface inconsistent parameters vpc member ports on secondary peer set to down state Globally inconsistent parameters misconfigured member ports on secondary peer suspended Parameters: STP mode, STP VLAN state, STP global settings, LACP mode, MTU Type 2 Consistency Check Forwards traffic in case of inconsistency Possible undesirable traffic forwarding behavior Parameters: VLAN interface (SVI), ACL, QOS, IGMP snooping, HSRP BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 15

16 Agenda Introduction to vpc Feature Overview Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

17 vpc Configuration Best Practices Building a vpc domain Configuration Steps L3 1. Define domains 2. Establish Peer Keepalive connectivity 3. Create a Peer link 4. Create vpcs 5. Make Sure Configurations are Consistent CFS (Order does Matter!) S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 vpc Configuration Best Practices vpc Domain-ID vpc Domain 10 The vpc peer devices use the vpc domain ID to automatically assign a unique vpc system MAC address You MUST use unique Domain id s for all vpc pairs defined in a contiguous layer 2 domain! Configure the vpc Domain ID It should be unique within the layer 2 domain NX-1(config)# vpc domain 20 S3 S4 vpc Domain 20! Check the vpc system MAC address NX-1# show vpc role <snip> vpc system-mac : 00:23:04:ee:be:14 S5 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 vpc Configuration Best Practices vpc Peer-Keepalive link Recommendations (in order of preference): Preference Nexus 9500 / 7X00 series 1 Dedicated link(s) (1GE/10GE Links) Nexus 9X00 / 6000 / 5X00 / 3X00 series mgmt0 interface 2 mgmt0 interface Dedicated link(s) (1GE/10GE Links) 3 L3 infrastructure L3 infrastructure BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 vpc Configuration Best Practices vpc Peer-Keepalive link Dual Supervisors When using dual supervisors and mgmt0 interfaces to carry the vpc peer-keepalive, DO NOT connect them back to back between the two switches vpc_pkl Management Switch Management Network vpc_pkl Only one management port will be active a given point in time and a supervisor switchover may break keepalive connectivity vpc1 vpc_pl vpc2 Use the management interface when you have an outof-band management network (management switch in between) Standby Management Interface Active Management Interface BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 vpc Configuration Best Practices vpc Peer-Link S3 S3 vpc Peer-link should be a point-to-point connection Peer-Link member ports can be 10/40/100GE interfaces Peer-Link bandwidth should be designed as per the vpc vpc imposes the rule that peer-link should never be blocking BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Design Best Practices Mixed Hardware across vpc Peers : Line Cards Always use identical switches and line cards on either sides of the peer link and vpc member ports! Examples: vpc Primary vpc Secondary vpc Primary vpc Secondary vpc Peer-link vpc Peer-link N EX 9732EX N EX 9536PQ 97160EX vpc 97160EX 9732EX vpc 9536PQ BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 22

23 Design Best Practices Mixed Hardware across vpc Peers : Line Card N9500 vpc Primary X X vpc Peer-link vpc vpc Secondary Y Y N9500 N9K-X9636PQ N9K-X9564PX N9K-X9564TX N9K-X9536PQ X Y vpc N9K-X9432PQ N9K-X9464PX N9K-X9464TX N9K-X9736PQ BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 Design Best Practices Mixed Hardware across vpc Peers : Line Cards Always use identical switches and line cards on either sides of the peer link and vpc member ports! VDC type should match on both peer device Examples: vpc Primary vpc Secondary vpc Primary vpc Secondary vpc Peer-link vpc Peer-link N7700 F2E F2E N7700 M1 M2 F3 vpc F3 F3 vpc F3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 24

25 Design Best Practices Mixed Hardware across vpc Peers : Chassis & Supervisors N7000 and N7700 in same vpc Domain - Supported vpc peers can have mixed SUP version* N9K: SUP-A, SUP-B N7K: SUP1, SUP2, SUP2E N5500 and N5600 in same vpc Domain Not Supported N7000 vpc Primary vpc Peer-link vpc Secondary N7700 N5500 vpc Primary vpc Peer-link vpc Secondary N5600 vpc vpc *Recommended only for short period BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 25

26 vpc Configuration Best Practices vpc Loop Avoidance Data plane loop prevention vpc peer forwards traffic locally when possible Traffic coming from vpc member port, crossing peer-link is NOT allowed to egress any vpc member port Exception of the rule, when member port goes down vpc 1 S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 26

27 vpc Configuration Best Practices Spanning Tree (STP) STP is running to manage loops outside of vpc domain, or before initial vpc configuration! S3 S4 S5 All switches in Layer 2 domain should run either Rapid-PVST+ or MST Do not disable spanning-tree protocol for any VLAN Always define the vpc domain as STP root for all VLANs in that domain BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 vpc Configuration Best Practices vpc Peer-switch Without Peer-switch: STP for vpcs controlled by vpc primary vpc primary send BPDU s on STP designated ports vpc secondary device proxies BPDU s to primary P BPDUs S Nexus(config-vpc-domain)# peer-switch With Peer-switch: Peer-Switch makes the vpc peer devices to appear as single STP root BPDUs processed by the logical STP root formed by the 2 vpc peer devices P S BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 Hybrid topology (vpc and non-vpc) Bridge Priority VLAN 1 4K VLAN 2 8K STP Root VLAN 1 vpc Primary STP Root VLAN 1 VLAN 2 vpc Secondary STP Root VLAN 2 Bridge Priority VLAN 1 8K VLAN 2 4K S3 vpc1 peer-switch Hybrid topology where vpc and non-vpc devices coexist in a vpc domain Need additional configuration parameters : spanning-tree pseudo-information STP pseudo configuration takes precedence over global STP configuration Not supported on Nexus 9200 and Nexus 9x00-EX S4 VLAN 1 (blocked) VLAN 2 (blocked) BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 vpc Configuration Best Practices vpc Peer-Gateway Allows a vpc switch to act as the active gateway for packets addressed to the peer router MAC Keeps forwarding of traffic local to the vpc node and avoids use of the peer-link Allows Interoperability with features of some NAS or load-balancer devices S3 S4 Nexus(config-vpc-domain)# peer-gateway BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 vpc Configuration Best Practices PVLAN on vpc PVLAN configuration across both vpc switches should be identical PVLAN configuration not supported on Peer-Link vpc Primary vpc Secondary Type-1 Consistency Check Port mode is a type-1 check vpc member port brought down if PVLAN port mode differs between vpc peers Type-2 Consistency Check PVLAN will bring down mismatched couples PVLAN- PROMISC (3500, 3501) P C P PVLAN- PROMISC (3500, 3501) Community VLAN BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 vpc Configuration Best Practices PVLAN vpc Type 1 Consistency Check vpc Primary vpc Secondary vpc Primary vpc Secondary P P I I PVLAN Promiscuous Trunk S3 PVLAN Isolated Trunk S3 vpc Primary vpc Secondary I T Type 1 Consistency Failure S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 32

33 vpc Configuration Best Practices PVLAN vpc Type 2 Consistency Check vpc Primary vpc Secondary vpc Primary vpc Secondary P P I I PVLAN- PROMISC (10, 201) S3 PVLAN- PROMISC (10, 201) Secondary Trunk (2,31) (3,30), (4,100) S3 Secondary Trunk (2,31) (3,30), (4,100) vpc Primary vpc Secondary Type 2 Consistency Failure Secondary Trunk (3,31) (2,30), (4,100) I S3 I Secondary Trunk (2,31) (3,30), (4,100) BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Failure Scenarios

35 vpc Failure Scenario vpc member port down On of the vpc member ports fails (optics failure or cable failure) P vpc Peer-keepalive S vpc primary and secondary peer remain primary and secondary, no change in roles vpc_plink Result in change in path, and traffic with destination to other peer, will cross peer-link to get to destination SW3 vpc1 vpc2 SW4 This is not desirable behavior, and peer-link can be oversubscribed P S Primary vpc peer Secondary vpc peer BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 vpc Failure Scenario vpc peer-keepalive link down vpc peer-keepalive link failure (link loss): vpc peer-link up P vpc Peer-keepalive S No role change Status of other vpc peer known vpc_plink Both peers forwarding No down time in the network vpc1 vpc2 SW3 SW4 Keepalive Heartbeat P S Primary vpc peer Secondary vpc peer BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 vpc Failure Scenario vpc Peer-Link down vpc peer-link failure (link loss): P vpc Peer-keepalive S vpc peer-keepalive up Status of other vpc peer known Both peers are active Secondary vpc peer disables all vpc s Traffic flows over vpc primary vpc1 vpc Peer-Link vpc2 Suspend secondary vpc Member Ports Traffic from orphan devices connected to secondary peer will be black-holed SW3 SW4 Keepalive Heartbeat P S Primary vpc peer Secondary vpc peer BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 vpc Failure Scenario Dual Active vpc Peer-Keepalive down followed by vpc Peer-Link down 1. vpc peer-keepalive DOWN P vpc Peer-keepalive P 2. vpc peer-link DOWN 3. DUAL-ACTIVE or SPLIT BRAIN scenario vpc_plink vpc primary peer remains primary and secondary peer becomes operational primary role Result in traffic loss / uncertain traffic behavior When links are restored, the operational primary (former secondary) keeps the primary role & former primary becomes operational secondary SW3 Traffic Loss / Uncertain Traffic Behavior vpc1 P S vpc2 SW4 Primary vpc peer Secondary vpc peer BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 Additional Features

40 vpc Configuration Best Practices vpc Orphan ports suspend Single attached devices to vpc domain, will black-hole traffic if peer-link fails P S With Orphan Port Suspend feature, will suspend orphan ports on vpc secondary peer Active or Standby Active or Standby When peer-link is restored, vpc secondary restores orphan ports S3 Nexus(config-if)# vpc orphan-ports suspend P S Primary vpc peer Secondary vpc peer BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 40

41 vpc Configuration Best Practices vpc ARP sync When peer device goes down or peer link goes down, SVIs are suspended ARP TABLE IP1 MAC1 VLAN 100 ARP TABLE IP1 MAC1 - VLAN After restore of the peer device, or peer link, ARP table is empty - traffic blac-kholed Before bringing up SVI, peer devices synchronize ARP table over CFS IP2 MAC2 VLAN 200 L3 L2 SVI 100 SVI 200 CFS IP2 MAC2 - VLAN SVI 100 SVI 200 Reduces convergence time Nexus(config-vpc-domain)# ip arp synchronize BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 vpc Configuration Best Practices vpc Delay Restore After vpc peer reload, traffic might be black-holed, before L3 connectivity is reestablished vpc link bring up can be delayed to allow L3 routing protocol convergence Default time 30 seconds L3 L2 OSPF Nexus(config-vpc-domain)# delay restore < sec> BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 42

43 vpc Configuration Best Practices vpc auto-recovery P S P S Operational Primary P S3 S3 S3 1. vpc peer-link down : - secondary shuts all its vpc member ports 2. down : vpc peer-keepalive link down : receives no keepalives 3. After 3 keepalive timeouts, changes role and brings up its vpc Nexus(config-vpc-domain)# auto-recovery P S Primary vpc Secondary vpc BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 vpc Configuration Best Practices vpc auto-recovery reload delay Until peer adjacency is reestablished between vpc devices, vpc member ports are suspended vpc auto-recovery reload delay allows alive vpc peer to assume primary role after delay time is expired Delay timer can be tuned S3 Nexus(config-vpc-domain)# auto-recovery reload-delay < seconds> BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 vpc Configuration Best Practices vpc auto-recovery Auto-recovery addresses two cases of single switch behavior Peer-link fails and after a while primary switch (or keepalive link) fails Both VPC peers are reloaded and only one comes back up How it works If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives will trigger auto-recovery After reload (role is none established ) auto-recovery timer (240 sec) expires while peer-link and peer-keepalive still down, autorecovery kicks in Switch assumes primary role VPCs are brought up bypassing consistency checks BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 vpc Configuration Best Practices vpc Self-Isolation P Error Triggered S P Self- Isolate S ISOLATED Operational Primary P S3 S3 S3 1. Error Triggered : All Line cards Fail or All Vlans s down on peer-link 2. sends self-isolation message through the peer-keepalive 3. takes over as operational Primary and is isolated from the vpc domain P S Primary vpc Secondary vpc BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 vpc Configuration Best Practices Example Configuration and Verification on Nexus 7x00 vpc domain 100 peer-keepalive destination peer-gateway self-isolation Switch# show vpc brief <snip> vpc domain id : 100 <snip> vpc role : primary <snip> Self-isolation : Enabled 2015 Sep 29 22:33:03 %$ VDC-1 %$ %vpc-2-enter_self_isolation: Local switch goes into self isolation state due to all linecards failure. Please resume failed linecards and do shut/no shut on peer-link to exit self-isolation state vpc domain 100 peer-keepalive destination peer-gateway self-isolation Switch# show vpc brief <snip> vpc domain id : 100 <snip> vpc role : secondary <snip> Self-isolation : Enabled 2015 Sep 30 10:33:14 %$ VDC-1 %$ %vpc-2-enter_self_isolation: Remote switch goes into self isolation state due to all linecards failure. Please resume failed linecards and do shut/no shut on peer-link to exit self-isolation state BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 vpc Configuration Best Practices vpc Self-Isolation vpc self-isolation is turned OFF by default No Impact on vpc operation if sellf-isolation enabled Functional only when enabled on both vpc peers. Not part of vpc type-1 and type-2 consistency checks BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 vpc Configuration Best Practices Why Object-Tracking? S4 S5 Modules hosting peer-link and uplink fail on the vpc primary Peer-Link is down and vpc Secondary shut all its vpc Auto-Recovery does not kick in as peerkeepalive link is active Primary L3 L2 Secondary Traffic is black holed S3 BRKDCT Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 vpc Configuration Best Practices Object-tracking vpc object tracking, tracks both peer-link and uplinks in a list of Boolean OR Object Tracking triggered when the track object goes down Suspends the vpcs on the impaired device Traffic forwarded over the remaining vpc peer! Track the vpc peer link track 1 interface port-channel11 line-protocol! Track the uplinks track 2 interface Ethernet1/1 line-protocol track 3 interface Ethernet1/2 line-protocol! Combine all tracked objects into one.! OR means if ALL objects are down, this object will go down track 10 list boolean OR object 1 object 2 object 3 S4 S5 L3 L2! If object 10 goes down on the primary vpc peer,! system will switch over to other vpc peer and disable all local vpcs vpc domain 1 track 10 S3 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 50

51 vpc Configuration Best Practices Spanning Tree Bridge Assurance Stopped receiving BPDUS! Root Network Network BPDUs BA Inconsistent BPDUs Network Network Malfunctioning switch BPDUs Network Network Blocked BA Inconsistent Edge Edge Stopped receiving BPDUS! %STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48 VLAN0700 switch# show spanning vl 700 in -i bkn Eth2/48 Altn BKN* Network P2p *BA_Inc BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Spanning Tree Bridge Assurance Almost like a routing protocol Turns STP into a bidirectional protocol Ensures spanning tree fails closed rather than open All ports with network port type send BPDUs regardless of state If network port stops receiving BPDUs, port is placed in BA-Inconsistent state (blocked) %STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48 VLAN0700. switch# sh spanning vl 700 in -i bkn Eth2/48 Desg BKN* Network P2p *BA_Inc BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 vpc Configuration Best Practices vpc & Bridge Assurance (BA) STP Bridge Assurance is enabled by default on vpc Peer-Link DON T disable Bridge Assurance on vpc Peer-link NO Bridge Assurance on vpc member ports (even with peer-switch) BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 vpc Configuration Best Practices Unidirectional Link Detection (UDLD) Light-weight Layer 2 failure detection protocol Designed for detecting: One-way connections due to physical or soft failure Miss-wiring detection (loopback or triangle) Cisco proprietary, but listed in informational RFC 5171 Runs on any single Ethernet link, even inside bundle Centralized implementation in switching platforms Message interval: 7-90 sec (default: 15 seconds) Detection: 2.5 x interval + timeout value (4 sec) ~ 41 sec Rx Rx Tx Tx BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 vpc Configuration Best Practices UDLD with vpc UDLD NOT recommended on vpc peer-link UDLD NOT recommended on vpc member ports if LACP is used UDLD only in normal mode on vpc member ports if required BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Agenda Introduction to vpc Feature Overview Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

57 Design Best Practices FHRP with vpc FHRP Active : Active for shared L3 MAC FHRP Standby : Active for shared L3 MAC L3 L2 S3 S4 FHRP in Active/Active mode with vpc Primary peer should be the HSRP/VRRP active device Best Practice : Use default FHRP timers BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 Design Best Practices ASA Cluster Cluster Control Link Cluster Data Link ASA Cluster Mode Use unique vpc for ASA Cluster Data Links to vpc domain Use vpc per ASA device for Cluster Control Link (CCL) to vpc domain Leverage peer-switch configuration BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 Nexus 2000 (FEX) Straight-Through Deployment with vpc Port-channel connectivity from the server Two Nexus switches bundled into a vpc pair Suited for servers with Dual NIC and capable of running Port-Channel HIF FEX 101 Fabric Links FEX 102 HIF vpc BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 59

60 Nexus 2000 (FEX) Active-Active Deployment with vpc Fabric Extender connected to two Nexus 5X00 / 6000 / 7x00 / 9300 Suited for servers with Single NIC or Dual NIC not having port-channel capability Scale implications of less FEX per system and less vpc Fabric Links Note : This design is currently not supported on Nexus 9500 Fex 101 HIF HIF Fex 102 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 Nexus 2000 (FEX) Active-Active Scale & Limitations (N7X00) N7X00 can support up to 64 FEXs N7X00 supports only 15 Active-Active FEX in 7.2(0)D1(1) N7X00 supports only 32 Active-Active FEX in 7.3(0)D1(1) Straight-Through FEX and Active-Active FEX cannot exist on the same ASIC instance Layer 3 HIF ports are not supported with Active-Active FEX Active-Active FEX is not supported with vpc+ BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 Nexus 2000 (FEX) - Enhanced vpc Port-channel connectivity to dual-homed FEXs From the server perspective a single access switch with port-channel support each line card supported by redundant supervisors Ideal design for a combination of single NIC and Dual NIC servers with portchannel capability Fabric Links Note : This design is currently not supported on N7000 / N7700 and N9X00 Fex 100 Fex 101 HIF HIF BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 Physical Port vpc Nexus 7x00 vpc domain vpc domain FEX101 e101/1/1 Po1 VPC1 VPC1 Po1 FEX102 e102/1/1 FEX101 e101/1/1 VPC1 VPC1 FEX102 e102/1/1 Port-channel vpc interface e101/1/1 switchport vpc 1 lacp mode active Physical port vpc vpc configuration on a physical Layer 2 port as opposed to a port-channel Front panel ports and FEX ports connected to F2/F2e/F3 only Improves scaling as separate port-channel interface not created for single-link vpc member port Key benefit: more than 1000 host facing vpcs with FEX BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Data Center Interconnect - vpc

65 Data Center Interconnect - DCI DCI provides connection of distant date centers Extend VLANs between data centers Technologies for DCI: Overlay Transportation Virtualization OTV (Multiple DC Interconnect) Virtual Port Channel vpc (Two DC Interconnect) vpc DCI: STP Isolation Between DC Easy to Configure Resilient Solution BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 65

66 ACCESS AGGR CORE vpc - Data Center Interconnect (DCI) DC 1 DC 2 vpc domain 11 Long Distance Dark Fiber vpc domain 21 N E Network port Edge or portfast - Normal port type - N R N - R - - N N E E F F F F E E N N - - R - N R - N CORE AGGR B F R BPDUguard BPDUfilter Rootguard 802.1AE (Optional) - - R R vpc domain 10 vpc domain R R - E B E B - ACCESS Server Cluster Server Cluster BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 Design Best Practices vpc - Data Center Interconnect (DCI) vpc Domain id for vpc layers should be UNIQUE BPDU Filter on the edge devices to avoid BPDU propagation STP Edge Mode to provide fast Failover times No Loop must exist outside the vpc domain No L3 peering between Nexus devices (i.e. pure layer 2) BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 Layer 3 over vpc

69 Dynamic Routing over vpc Problem? 1) Packet arrives at R1 2) R1 does lookup in routing table and sees 2 equal paths going north (to & ) 3) Assume it chooses (ECMP decision) 4) R1 now has rewrite information to which router it needs to go (router MAC or ) 5) L2 lookup happens and outgoing interface is port-channel 1 6) Hashing determines which port-channel member is chosen (say to ) 7) Packet is sent to 8) sees that it needs to send it over the peer-link to based on MAC address S3 Po2 Po1 R1 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 Dynamic Routing over vpc Problem? Note: 9) performs lookup and sees that it needs to send to S3 10) performs check if the frame came over peer link & is going out on a vpc. 11) Frame will ONLY be forwarded if: Outgoing interface is NOT a vpc or Outgoing vpc doesn t have active interface on other vpc peer (in the example ) Use of Peer-Gateway allows routing/forwarding traffic for the peer-router MAC locally, but does NOT Enable Dynamic Routing on vpc VLANs S3 Po2 Po1 Even with Peer-Gateway Routing protocols (e.g. OSPF) TTL expiry when traversing in transit the peer vpc Router device. R1 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 Dynamic Routing over vpc Devices without L3 over vpc support Not recommended to attach L3 devices to vpc domain via L2 port-channel Common workarounds: Individual L3 links for routed traffic Static route to FHRP VIP A B SVI 1 IP Y VIP A SVI 1 IP Z VIP A SVI 1 IP Y VIP A SVI 1 IP Z VIP A SVI 1 IP Y VIP A SVI 1 IP Z VIP A L3 ECMP SVI 2 IP X Router SVI 2 IP X Router SVI 2 IP X Router Static Route to VIP A BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 Design Best Practices Backup Routing Path Use one transit VLAN to establish L3 routing backup path over the vpc peerlink in case L3 uplinks were to fail, all other SVIs can use passive-interfaces Point-to-point dynamic routing protocol adjacency between the vpc peers to establish a L3 backup path to the core through PL in case of uplinks failure P S3 OSPF/EIGRP S4 P Define SVIs associated with FHRP as routing passive-interfaces in order to avoid routing adjacencies over vpc peer-link A single point-to-point VLAN/SVI (aka transit VLAN) will suffice to establish a L3 neighbor Alternatively, use an L3 point-to-point link between the vpc peers to establish a L3 backup path P L3 L2 Primary vpc VLAN 99 OSPF/EIGRP Secondary vpc S5 P P Routing Protocol Peer BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Dynamic Routing over vpc

74 Dynamic routing over vpc Configuration Nexus(config-vpc-domain)# layer3 peer-router Dynamic peering between Layer 3 device and vpc peers over vpc VLAN Dynamic unicast routing for IPv4 Traffic does not get decremented TTL if travers peer-link Peer-Gateway should be enabled. vpc domain 200 peer-keepalive destination source peer-gateway layer3 peer-router P P vpc domain 200 peer-keepalive destination source peer-gateway layer3 peer-router P BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 Dynamic routing over vpc Example Configuration and Verification on Nexus 7x00 vpc domain 200 peer-keepalive destination source peer-gateway layer3 peer-router P P vpc domain 200 peer-keepalive destination source peer-gateway layer3 peer-router Switch# show vpc brief <snip> vpc domain id : 100 <snip> Peer Gateway : Enabled <snip> Operational Layer3 Peer : Enabled P Switch# show vpc brief <snip> vpc domain id : 100 <snip> Peer Gateway : Enabled <snip> Operational Layer3 Peer : Enabled BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Benefits of Dynamic Routing over vpc No Static routes No Parallel links No design changes Route peering across vpc s over existing infrastructure Routing between vpc DCI BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Dynamic routing over vpc Use Case 1 : Firewall at Aggregation layer Peering Firewalls in routed mode over vpc L3 Cloud Firewalls may be in active-standby mode Static routing / L3 P2P links NOT required External and internal traffic traverse same port channel to firewall. FW-A FW-B Dynamic Peering Relationship BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 77

78 Dynamic routing over vpc Use Case 2 : Remote Orphan Site Peering in DCI Deployment vpc as Data Center Interconnect (DCI) Remote Site 1 Remote Site 2 Each Switch has routing adjacency with both vpc device in other DC Each DC connected to a remote site by orphan port Remote sites forms routing adjacency with both peers of its directly connected DC S3 S4 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 Dynamic Routing over vpc Supported Topologies

80 Dynamic routing over vpc Supported Designs Layer 3 services devices with vpc Layer 3 over DCI - vpc P P P P P P P Note : Supported on Nexus 9X00 in ACI and NX-OS mode Supported only in Nexus 7X00 on M3, F3, and F2E Line Cards, and Nexus 5x00 Currently not supported on Nexus 3X00, Nexus 7000 M1, M2, and F2 series Line card BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 Dynamic routing over vpc Supported Designs STP inter-connection using a vpc VLAN Orphan device with vpc peers over vpc VLAN P P P P P P Note : Supported on Nexus 9X00 in ACI and NX-OS mode Supported only in Nexus 7X00 on M3, F3, and F2E Line Cards, and Nexus 5x00 Currently not supported on Nexus 3X00, Nexus 7000 M1, M2, and F2 series Line card BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 Dynamic routing over vpc Supported Designs Peering with vpc peers over FEX vpc host interfaces P P P Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2. Supported on Nexus 9X00 in ACI mode Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (NX-OS mode), Nexus 7000 M-series Line card BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 Dynamic Routing over vpc Unsupported Topologies

84 Dynamic routing over vpc Unsupported Design Peering across vpc interfaces with unequal L3 metrics The routing metric on is less than the routing metric on (preferred path using ). B Router2 Int VLAN 20 Po2 SVI Traffic from A to B may hash to. This traffic will need to traverse to peer-link to get to B through. Due to the vpc loop avoidance rule will not allow traffic to flow to B. Int VLAN 20 Int VLAN 10 Metric 20 Po100 Po1 Router1 Int VLAN 10 Int VLAN 20 Int VLAN 10 Metric 10 SVI A BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 Multicast over vpc

86 Design Best Practices vpc and Multicast Source Source Receivers vpc supports PIM-ASM (on all platforms) vpc supports PIM-SSM (on N9000 and N5600) vpc uses CFS to sync IGMP state Sources in vpc domain both vpc peers are forwarders Duplicates avoided via vpc loop-avoidance logic Sources in Layer 3 cloud Active forwarder elected on unicast metric vpc Primary elected active forwarder in case metric are equal Active forwarder concept is per multicast group/source BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 Agenda Feature Overview vpc Terminology and Roles Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

88 vpc Configuration Best Practices vpc Shutdown Isolates a switch from the vpc complex to: Debug Troubleshoot Physically isolate Primary vpc Secondary Minimal disruption of traffic flows no shutdown brings switch up Part of configuration, persistent after reload Recommended to have peer-switch enabled S3 switch# configure terminal switch(config)# vpc domain 100 switch(config-vpc)# shutdown BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89 Graceful Insertion and Removal - GIR Change window begins vpc system mode maintenance vpc One command! Pre-change System Snapshot BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 89

90 Graceful Insertion and Removal - GIR Change window complete vpc vpc system mode normal One command! Pre/Post-change Snapshot Comparison BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 90

91 Graceful Insertion and Removal Flexible framework providing a comprehensive, systemic method to isolate a node. Configuration profile foundation in NX-OS Initial support for: vpc/vpc+ ISIS OSPF EIGRP BGP Interface Per VDC on Nexus 7x00 Platform Release Nexus 5x00/6000 NX-OS 7.1 Nexus 7x00 NX-OS 7.2 Nexus 9000 NX-OS 7.X BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 91

92 ISSU with vpc ISSU (In Service Software Upgrade) recommended way system upgrade in a vpc environment vpc system can be independently upgraded Upgrade must be run one peer at a time Start with vpc primary switch Configuration is locked on other vpc peer during ISSU vpc run seamlessly with two different version of software Aggressive timers not supported 5.2(x) / 6.2(x) BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 92

93 Agenda Feature Overview vpc Terminology and Roles Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

94 vpc with VXLAN

95 Virtual Extensible LAN - VXLAN Benefits VXLAN is a network overlay technology VXLAN builds Layer-2 & Layer-3 overlay network on top of an IP routed network VXLAN uses MAC in IP-UDP encapsulation (UDP dest. port 4789) VLAN scale VXLAN extends the L2 segment ID field to 24-bits MAN/WAN BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 95

96 VXLAN Packet Format Outer Mac Header Outer IP Header UDP Header VXLAN Header Original L2 Frame FCS FCS MAC Addr. Dst. Src. MAC Addr. VLAN Type 0x8100 VLAN ID Tag Ether Type 0x0800 IP Header Misc Data Protocol 0x11 Header Checksum Outer Src. IP Outer Dst. IP UDP Src. Port VXLAN Port UDP Length Checksum 0x0000 VXLAN RRRR1RRR Reserved VNI Reserved 14 Bytes (4 bytes optional) 20 Bytes 8 Bytes 8 Bytes VXLAN is a Layer 2 overlay scheme over a Layer 3 network. VXLAN uses Ethernet in UDP encapsulation VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 96

97 VXLAN Terminology VTEP Virtual Tunnel End Point Transport IP Network VTEP IP Interface VTEP IP Interface Local LAN Segment Local LAN Segment End System End System End System End System VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point) VTEP has two interfaces : Bridging functionality for local hosts IP identification in the core network for VXLAN encapsulation / de-encapsulation BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 97

98 vpc and VXLAN vpc VTEP When vpc is enabled an anycast VTEP address is programmed on both vpc peers Multicast topology prevents BUM traffic being sent to the same IP address across the L3 network (prevents duplication of flooded packets) vpc peer-gateway feature must be enabled on both peers Backup SVI, configured with PIM vpc VTEP vpc VTEP VLAN VXLAN VXLAN header is not carried on the vpc Peer link BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 98

99 VXLAN & vpc vpc Configuration Map VNI to VLAN Source Interface individual IP is used for single attached Hosts anycast IP is used for VPC attached Hosts VXLAN Tunnel Interface Vlan for VXLAN vpc peer-link SVI for the VXLAN vpc peer-link Enable the VLAN on the VXLAN vpc peer-link vtep 1 H VLAN 10 (vpc) vtep 2 VTEP1 vlan 10 vn-segment interface loopback 0 ip address <VTEP individual IP orphan) ip address <VTEP anycast IP per VPC domain> secondary! interface nve1 source-interface loopback0 member vni mcast-group ! Vlan 99! Interface vlan 99 ip address /24 ip ospf cost 10 ip router ospf 1 area ip pim sparse-mode! vpc nve peer-link-vlan 99 VTEP2 vtep vlan 10 4 vn-segment vtep 3 interface loopback 0 ip address <VTEP individual IP - orphan> ip address <VTEP anycast IP per VPC domain> secondary! interface nve1 source-interface loopback0 member vni mcast-group ! Vlan 99! Interface vlan 99 ip H2 address / ip ospf cost 10 VLAN ip 10 router ospf 1 area (vpc) ip pim sparse-mode! vpc nve peer-link-vlan 99 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 99

100 VXLAN & vpc Dual attached Host to dual attached Host (Layer-2) Host 1 (H1) and Host 2 (H2) are dual connected to a vpc domain As H1 is behind a VPC interface, the anycast VTEP IP is the source for the VXLAN encapsulation vtep 1 vtep 20 vtep 2 vtep 3 vtep 30 vtep 4 As H2 is behind a VPC interface, the anycast VTEP IP is the target H VLAN 10 (vpc) H VLAN 10 (vpc) BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 100

101 vpc with ACI

102 Nexus APIC = ACI APIC APICAPIC BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 102

103 ACI uses a policy based approach that focuses on the application. QoS Filter Web QoS Service App QoS Filter DB External Network BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 103

104 vpc and ACI ACI fabric utilised for control-plane No dedicated peer-link between vpc peers: Fabric itself serves as the peer-link No out-of-band mechanism to detect peer liveliness: Due to rich fabric-connectivity (leaf-spine), it is very unlikely that peers will have no active path between them CFS (Cisco Fabric Services) is replaced by Zero Message Queue (ZMQ) As ACI fabric is VXLAN-based, an anycast VTEP is shared by both leaf switches in a vpc domain vpc Domains vtep 1 vtep 3 vpc vtep 2 ACI fabric vpc vpc peers BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 104

105 vpc with FCoE

106 CRC EOF FCS Fiber Channel over Ethernet - FCoE Fiber Channel traffic over Ethernet Ethernet Header FCoE Header FC Header FC Payload Ethernet standards to support FCoE: Priority Flow Contol PFC Enhanced Transmission Selection ETS Data Center Bridging Exchange DCBX BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 106

107 vpc with FCoE Unified Fabric Design vpc with FCoE is supported between hosts Nexus 9000, Nexus 7X00, Nexus 5X00 and N5X00 & N2X00 pairs. vpc and FCoE only on the first hop Each vpc peer must be part of separate fabric. Best Practice: Use static port channel rather than LACP with vpc and boot from SAN. VLAN 10,20 LAN Fabric Nexus 5000 FCF-A Fabric A VLAN 10 ONLY HERE! VLAN 10,30 Fabric B Nexus 5000 FCF-B STP Edge Trunk vpc contains only 2 X 10GE links one to each Nexus 5X00 BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 107

108 vpc with FabricPath

109 FabricPath: an Ethernet Fabric Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00 FabricPath Spanning Three Protocol independence High MAC address scalability with conversation learning on Edge ports Unique Switch ID (SID) identifies switches in FabricPath fabric IS-IS for control plane information exchange Multi destination Trees for BUM traffic Loop mitigation with TTL Simple CLI configuration Switch(config-if)# switchport mode fabricpath BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 109

110 vpc vs vpc+ Architecture of vpc and FabricPath with vpc+ CE FP CE Port FP Port CE VLAN s FP VLAN s vpc vpc+ Physical architecture of vpc and vpc+ is the same from the access edge Functionality/Concepts of vpc and vpc+ are the same Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port vpc+ is not supported on Nexus 9X00 & Nexus 3X00 Series BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 110

111 Dynamic Routing over vpc+ Layer 3 devices can form routing adjacencies with both the vpc+ peers over vpc The peer link ports and VLAN are configured in FabricPath mode PIM-SSM multicast L3 peering with vpc+ plus devices is not supported on N7X00 P FabricPath vpc P N55xx, N56xx, N6000 Router/ Firewall P Fabricpath Link Dynamic Peering Relationship Routing Protocol Peer P BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 111

112 Agenda Feature Overview vpc Terminology and Roles Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Key Takeaways

113 Key Takeaways Full bandwidth of the network without Spanning Three High availability and improved convergence for downtime reduction Dynamic routing over vpc for appliance connectivity Dual-homing device in ACI, VXLAN, FCoE and FabricPath environment WHO HAS LEARN SOMETHING NEW TODAY? BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 113

114 Related Sessions Session Id BRKACI-1001 BRKDCN-2404 BRKDCN-2304 BRKDCN-2458 Session Name Your first 7 days of ACI VXLAN deployment models - A practical perspective L4-L7 Service Integration in Multi-Tenant VXLAN EVPN Data Center Fabrics Nexus 9000/7000/6000/5000 Operations and Maintenance Best Practices BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 114

115 Complete Your Online Session Evaluation Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card. Complete your session surveys through the Cisco Live mobile app or on Don t forget: Cisco Live sessions will be available for viewing on demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

116 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Lunch & Learn Meet the Engineer 1:1 meetings Related sessions BRKDCN Cisco and/or its affiliates. All rights reserved. Cisco Public 116

117 Thank you

118