Everyone in this room is a GENIUS

Transcription

1

2 Everyone in this room is a GENIUS 2

3 What are Best Practices? Learning from Others Mistakes 3

4 Learning from your mistakes makes you SMART Learning from others mistakes makes you GENIUS 4

5 vpc Best Practices and Design on NXOS Nazim Khan, CCIE#39502 (DC/SP) Network Consulting Engineer, Data Center Group BRKDCT-2378

6 Session Focus Best Practices and Designs for vpc Nexus 2000 (FEX) will only be addressed from vpc standpoint Fabricpath / vpc+ Overview vpc with FCOE vpc with VXLAN vpc with ACI.

7 Pick the great from the good

8 We Are Not Covering vpc troubleshooting Scalability Fabricpath vpc+ VXLAN FCoE ACI

9 Related Sessions at Cisco Live Berlin Session Id BRKDCT-2404 BRKDCT-3313 BRKDCT-2458 BRKACI-2601 BRKDCT-2333 Session Name VXLAN deployment models - A practical perspective Fabricpath Operations and Troubleshooting Nexus 9000/7000/6000/5000 Operations and Maintenance Best Practices Real World ACI Deployment and Migration Data Centre Network Failure Detection 9

10 Agenda Feature Overview Configuration Best Practices Design Best Practices vpc Operations and Upgrade vpc with Fabric Technologies Reference Material 10

11 Data Center Technology Evolution MPLS, OTV, LISP MPLS, OTV, LISP FabricPath with vpc+ FEX with vpc VPC STP 2010 VXLAN ACI

12 Why vpc?

13 13

14 there s something about vpc 14

15 Role of vpc in the Evolution of Data Center vpc launched in 2009 Deployed by almost 95% of Nexus customers Used to redundantly connect network entities at the edge of the Fabric Dual-homed servers (bare metal, blades, etc.) Network services (Firewalls, Load Balancers, etc.) Unified Fabric 15

16 Agenda Feature Overview Concepts and Benefits Terminology 16

17 vpc Feature Overview vpc Concept & Benefits S1 S2 S1 S2 S1 S2 STP S3 S3 vpc Physical Topology S3 vpc Logical Topology No Blocked Ports, More Usable Bandwidth, Load Sharing Fast Convergence 17

18 Feature Overview vpc Terminology vpc Peer Layer 3 Cloud vpc Domain Peer-Link vpc Peer Keepalive Link Orphan Port S1 CFS S2 vpc vpc Member Port Orphan Device S3 18

19 vpc Failure Scenario vpc Peer-Keepalive Link up & vpc Peer-Link down For Your Reference vpc peer-link failure (link loss): vpc peer-keepalive up Status of other vpc peer known Both peers Active Secondary vpc peer disables all vpc s Traffic from vpc primary. Orphan devices connected to secondary peer will be isolated S1 P vpc1 SW3 vpc Peer-keepalive vpc_plink vpc2 SW4 S S2 Suspend secondary vpc Member Ports Keepalive Heartbeat P S Primary vpc Secondary vpc 19

20 vpc Failure Scenario Dual Active vpc Peer-Keepalive down followed by vpc Peer-Link down For Your Reference 1. vpc peer-keepalive DOWN 2. vpc peer-link DOWN 3. DUAL-ACTIVE or SPLIT BRAIN vpc primary peer remains primary and secondary peer becomes operational primary role S1 P vpc1 vpc Peer-keepalive vpc_plink Traffic Loss / Uncertain Traffic Behavior vpc2 PS S2 Result in traffic loss / uncertain traffic behavior SW3 SW4 When links are restored, the operational primary (former secondary) keeps the primary role & former primary becomes operational secondary P S Primary vpc Secondary vpc 20

21 Agenda vpc Configuration Best Practices Building a vpc domain Domain-ID Peer-Link Peer-Keepalive Link Spanning-Tree Peer-switch Private VLAN (PVLAN) Auto-recovery Object tracking 21

22 vpc Configuration Best Practices Building a vpc domain Configuration Steps 1. Define domains 2. Establish Peer Keepalive connectivity S1 S2 3. Create a Peer link CFS 4. Create vpcs 5. Make Sure Configurations are Consistent (Order does Matter!) S3 22

23 vpc Configuration Best Practices vpc Domain-ID vpc Domain 10 The vpc peer devices use the vpc domain ID to automatically assign a unique vpc system MAC address You MUST use unique Domain id s for all vpc pairs defined in a contiguous layer 2 domain! Configure the vpc Domain ID It should be unique within the layer 2 domain NX-1(config)# vpc domain 20! Check the vpc system MAC address NX-1# show vpc role <snip> vpc system-mac : 00:23:04:ee:be:14 S1 S3 S2 S4 vpc Domain 20 S5 23

24 vpc Configuration Best Practices vpc Peer-Link S1 S2 S1 S2 S3 S3 vpc Peer-link should be a point-to-point connection Peer-Link member ports can be 10/40/100GE interfaces Peer-Link bandwidth should be designed as per the vpc vpc imposes the rule that peer-link should never be blocking 24

25 vpc Configuration Best Practices vpc Peer-Keepalive link Recommendations (in order of preference): Preference Nexus 7X00 / 9500 series 1 Dedicated link(s) (1GE/10GE LC) Nexus 9300 /6000 / 5X00 / 3X00 series mgmt0 interface 2 mgmt0 interface Dedicated link(s) (1GE/10GE LC) 3 L3 infrastructure L3 infrastructure 25

26 vpc Configuration Best Practices vpc Peer-Keepalive link Dual Supervisors For Your Reference Management Switch Management Network When using dual supervisors and mgmt0 interfaces to carry the vpc peer-keepalive, DO NOT connect them back to back between the two switches vpc_pkl vpc_pkl Only one management port will be active a given point in time and a supervisor switchover may break keepalive connectivity vpc1 vpc_pl vpc2 Use the management interface when you have an outof-band management network (management switch in between) Standby Management Interface Active Management Interface 26

27 vpc Configuration Best Practices Spanning Tree (STP) STP is running to manage loops outside of vpc domain, or before initial vpc configuration! S1 S2 S3 S4 S5 All switches in Layer 2 domain should run either Rapid-PVST+ or MST Do not disable spanning-tree protocol for any VLAN Always define the vpc domain as STP root for all VLAN in that domain 27

28 vpc Configuration Best Practices vpc Peer-Gateway Allows a vpc switch to act as the active gateway for packets addressed to the peer router MAC Keeps forwarding of traffic local to the vpc node and avoids use of the peer-link Allows Interoperability with features of some NAS or load-balancer devices S1 S3 S4 S2 N7k(config-vpc-domain)# peer-gateway 28

29 vpc Configuration Best Practices vpc Peer-switch Without Peer-switch Primary vpc Secondary vpc STP for vpcs controlled by vpc primary. vpc primary send BPDU s on STP designated ports vpc secondary device proxies BPDU s to primary BPDUs With Peer-switch Peer-Switch makes the vpc peer devices to appear as a single STP root BPDUs processed by the logical STP root formed by the 2 vpc peer devices N7k(config-vpc-domain)# peer-switch Primary vpc Secondary vpc 29

30 vpc Configuration Best Practices PVLAN on vpc PVLAN configuration across both VPC switches should be identical PVLAN configuration not supported on Peer-Link vpc Primary vpc Secondary Type-1 Compatibility Check Port mode is a type-1 check vpc leg brought down if PVLAN port mode different on vpc legs Type-2 Compatibility Check PVLAN will bring down mismatched tuple S1 PVLAN- PROMISC (3500, 3501) P C P S2 PVLAN- PROMISC (3500, 3501) Community VLAN Note : This feature is currently not supported on N9X00 30

31 vpc Configuration Best Practices PVLAN VPC type 1 Consistency Check vpc Primary vpc Secondary vpc Primary vpc Secondary S1 P P S2 S1 I I S2 Pvlan Promiscuous trunk S3 Pvlan Isolated trunk S3 vpc Primary vpc Secondary S1 I T S2 Type 1 Consistency Failure S3 31

32 vpc Configuration Best Practices PVLAN VPC type 2 Consistency Check vpc Primary vpc Secondary vpc Primary vpc Secondary S1 P P S2 S1 I I S2 PVLAN- PROMISC (10, 201) S3 PVLAN- PROMISC (10, 201) Secondary Trunk (2,31) (3,30), (4,100) S3 Secondary Trunk (2,31) (3,30), (4,100) vpc Primary vpc Secondary Type 2 Consistency Failure S1 I I S2 Secondary Trunk (3,31) (2,30), (4,100) S3 Secondary Trunk (2,31) (3,30), (4,100) 32

33 vpc Configuration Best Practices vpc auto-recovery Operational Primary P S P S P S1 S2 S1 S2 S1 S2 S3 S3 S3 1. vpc peer-link down : S2 - secondary shuts all its vpc member ports 2. S1 down : vpc peer-keepalive link down : S2 receives no keepalives 3. After 3 keepalive timeouts, S2 changes role and brings up its vpc P S vpc Primary vpc Secondary 33

34 vpc Configuration Best Practices vpc auto-recovery For Your Reference Auto-recovery addresses two cases of single switch behavior Peer-link fails and after a while primary switch (or keepalive link) fails Both VPC peers are reloaded and only one comes back up How it works If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives will trigger auto-recovery After reload (role is none established ) auto-recovery timer (240 sec) expires while peer-link and peer-keepalive still down, autorecovery kicks in Switch assumes primary role VPCs are brought up bypassing consistency checks Nexus(config)# vpc domain 1 Nexus(config-vpc-domain)# auto-recovery 34

35 vpc Configuration Best Practices Why Object-Tracking? S4 S5 Modules hosting peer-link and uplink fail on the vpc primary Peer-Link is down and vpc Secondary shut all its vpc Primary Secondary Auto-Recovery does not kick in as peerkeepalive link is active S1 S2 Traffic is black holed S3 35

36 vpc Configuration Best Practices Object-tracking vpc object tracking, tracks both peer-link and uplinks in a list of Boolean OR Object Tracking triggered when the track object goes down Suspends the vpcs on the impaired device Traffic forwarded over the remaining vpc peer! Track the vpc peer link track 1 interface port-channel11 line-protocol! Track the uplinks track 2 interface Ethernet1/1 line-protocol track 3 interface Ethernet1/2 line-protocol S4 S1 S5 S2! Combine all tracked objects into one.! OR means if ALL objects are down, this object will go down track 10 list boolean OR object 1 object 2 object 3! If object 10 goes down on the primary vpc peer,! system will switch over to other vpc peer and disable all local vpcs vpc domain 1 track 10 S3 36

37 vpc Configuration Best Practices Spanning Tree Bridge Assurance Stopped receiving BPDUS! Root Network Network BPDUs BA Inconsistent BPDUs Network Network Malfunctioning switch BPDUs Network Network Blocked BA Inconsistent Edge Edge Stopped receiving BPDUS! %STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48 VLAN0700 switch# show spanning vl 700 in -i bkn Eth2/48 Altn BKN* Network P2p *BA_Inc

38 Spanning Tree Bridge Assurance Almost like a routing protocol For Your Reference Turns STP into a bidirectional protocol Ensures spanning tree fails closed rather than open All ports with network port type send BPDUs regardless of state If network port stops receiving BPDUs, port is placed in BA-Inconsistent state (blocked) %STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48 VLAN0700. switch# sh spanning vl 700 in -i bkn Eth2/48 Desg BKN* Network P2p *BA_Inc 38

39 vpc Configuration Best Practices vpc & Bridge Assurance (BA) STP Bridge Assurance is enabled by default on vpc Peer-Link DON T disable Bridge Assurance on vpc Peer-link NO Bridge Assurance on vpc member ports (even with peer-switch) 39

40 vpc Configuration Best Practices Unidirectional Link Detection (UDLD) Light-weight Layer 2 failure detection protocol Designed for detecting: One-way connections due to physical or soft failure Mis-wiring detection (loopback or triangle) Cisco proprietary, but listed in informational RFC 5171 Runs on any single Ethernet link, even inside bundle Centralized implementation in switching platforms Message interval: 7-90 sec (default: 15 seconds) Detection: 2.5 x interval + timeout value (4 sec) ~ 41 sec Rx Rx Tx Tx For Your Reference 40

41 vpc Configuration Best Practices UDLD with vpc UDLD NOT recommended on vpc peer-link UDLD NOT recommended on vpc member ports if LACP is used UDLD only in normal mode on vpc member ports if required 41

42 Agenda vpc Design Best Practices Mixed Hardware across vpc Peers FHRP with vpc Hybrid topology (vpc and non-vpc) vpc and Network Services vpc Fex Supported Topologies Physical port vpc vpc as Data Center Interconnect (DCI) Dynamic Routing over VPC vpc and Multicast 42

43 Design Best Practices Mixed Hardware across vpc Peers : Line Cards Always use identical line cards on either sides of the peer link and VPC legs! Examples vpc Primary vpc Secondary vpc Primary vpc Secondary S1 N7000 F3 vpc Peer-link F2E F2E vpc F3 S2 N7700 S1 vpc Peer-link M1 M2 vpc S2 43

44 Design Best Practices Mixed Hardware across vpc Peers : Nexus 9500 S1 N9500 vpc Primary X X vpc Peer-link vpc vpc Secondary Y Y S2 N9500 X Y vpc N9K-X9636PQ N9K-X9432PQ N9K-X9564PX N9K-X9464PX N9K-X9564TX N9K-X9464TX N9K-X9536PQ N9K-X9736PQ 44

45 Design Best Practices Mixed Hardware across vpc Peers : Chassis & Supervisors N7000 and N7700 in same vpc Construct -Supported VDC type should match on both peer device vpc peers can have mixed SUP version* (SUP1, SUP2, SUP2E) N5500 and N5600 in same vpc Construct Not Supported vpc Primary vpc Secondary vpc Primary vpc Secondary S1 N7000 S2 N7700 S1 N5500 N5600 S2 *Recommended only for short period such as migration 45

46 Design Best Practices FHRP with vpc FHRP Active : Active for shared L3 MAC FHRP Standby : Active for shared L3 MAC S1 S2 S3 S4 FHRP in Active/Active mode with vpc No requirement for aggressive FHRP timers Best Practice : Use default FHRP timers 46

47 Design Best Practices Backup Routing Path Use one transit vlan to establish L3 routing backup path over the vpc peerlink in case L3 uplinks were to fail, all other SVIs can use passive-interfaces Point-to-point dynamic routing protocol adjacency between the vpc peers to establish a L3 backup path to the core through PL in case of uplinks failure P S3 OSPF/EIGRP S4 P Define SVIs associated with FHRP as routing passive-interfaces in order to avoid routing adjacencies over vpc peer-link A single point-to-point VLAN/SVI (aka transit vlan) will suffice to establish a L3 neighbor L3 L2 P S1 Primary vpc VLAN 99 OSPF/EIGRP Secondary vpc P S2 Alternatively, use an L3 point-to-point link between the vpc peers to establish a L3 backup path P S5 Routing Protocol Peer 47

48 Hybrid topology (vpc and non-vpc) Bridge Priority VLAN 1 4K VLAN 2 8K STP Root VLAN 1 S1 vpc Primary STP Root VLAN 1 VLAN 2 vpc Secondary S2 STP Root VLAN 2 Bridge Priority VLAN 1 8K VLAN 2 4K S3 vpc1 peer-switch S4 VLAN 1 (blocked) VLAN 2 (blocked) Supports hybrid topology where vpc and non-vpc are connected to the same vpc domain Need additional configuration parameters : spanning-tree pseudo-information STP pseudo configuration takes precedence over global STP configuration 48

49 Design Best Practices ASA Cluster Cluster Control Link Cluster Data Link ASA Cluster Mode Use unique vpc for ASA Cluster Data Links to vpc domain Use vpc per ASA device for Cluster Control Link (CCL) to vpc domain Leverage peer-switch configuration 49

50 Nexus 2000 (FEX) Straight-Through Deployment with VPC Port-channel connectivity from the server Two Nexus switches bundled into a vpc pair Suited for servers with Dual NIC and capable of running Port-Channel S1 Fabric Links S2 HIF Fex 100 HIF Fex 101 VPC 50

51 Nexus 2000 (FEX)Active-Active Deployment with VPC S1 S2 Fabric Extender connected to two Nexus 5X00 / 6000 Suited for servers with Single NIC or Dual NIC not having port-channel capability. Scale implications of less FEX per system and less VPC Fabric Links Note : This design is currently not supported on Nexus 9X00 Nexus 7X00 will support this from release 7.2 Fex 100 HIF HIF Fex

52 Nexus 2000 (FEX) Active-Active Scale & Limitations (N7X00) N7X00 can support up to 64 FEXs N7X00 supports only 15 Active-Active FEX in 7.2(0)D1(1) Straight-Through FEX and Active-Active FEX cannot exist on the same ASIC instance Layer 3 HIF ports are not supported with Active-Active FEX Active-Active FEX is not supported with vpc+

53 Nexus 2000 (FEX) - Enhanced VPC Port-channel connectivity to dual-homed FEXs From the server perspective a single access switch with port-channel support each line card supported by redundant supervisors Ideal design for a combination of single NIC and Dual NIC servers with portchannel capability Scale implications of less FEX per system and less VPC Note : This design is currently not supported on N7000 / N7700 and N9X00 S1 S2 Fabric Links Fex 100 Fex 101 HIF HIF 53

54 Nexus 2000 (FEX) Active-Active (Unsupported) 54

55 Physical Port vpc vpc domain vpc domain FEX101 e101/1/1 Po1 VPC1 VPC1 Po1 FEX102 e102/1/1 FEX101 e101/1/1 VPC1 VPC1 FEX102 e102/1/1 Port-channel vpc interface e101/1/1 switchport vpc 1 lacp mode active Physical port vpc vpc configuration on a physical Layer 2 port as opposed to a port-channel Front panel ports and FEX ports connected to F2/F2e/F3 only Improves scaling as separate PC interface not created for single-link VPC leg Key benefit: more than 1000 host facing VPCs with FEX 55

56 ACCESS AGGR CORE vpc - Data Center Interconnect(DCI) DC 1 DC 2 vpc domain 11 Long Distance Dark Fiber vpc domain 21 N E Network port Edge or portfast - Normal port type - N R N - R - - N N E E F F F F E E N N - - R - N R - N CORE AGGR B F R BPDUguard BPDUfilter Rootguard 802.1AE (Optional) - - R R vpc domain 10 vpc domain R R - E B E B - ACCESS Server Cluster Server Cluster 56

57 Design Best Practices vpc as Data Center Interconnect (DCI) PROS vpc is easy to configure and it provides robust and resilient interconnect solution CONS Maximum of only two Data Centers can be interconnected Layer 3 peering between Data Centers cannot be done through vpc and separate links are required 57

58 Design Best Practices vpc -Data Center Interconnect (DCI) vpc Domain id for vpc layers should be UNIQUE BPDU Filter on the edge devices to avoid BPDU propagation STP Edge Mode to provide fast Failover times No Loop must exist outside the vpc domain No L3 peering between Nexus 7000 devices (i.e. pure layer 2) 58

59 Dynamic routing over vpc? 59

60 Dynamic routing over vpc Use Case 1 : Firewall at Aggregation layer Peering Firewalls in routed mode over vpc L3 Cloud Firewalls may be in active-standby mode Static routing / L3 P2P links NOT required External and internal traffic traverse same port channel to firewall. S1 S2 FW-A FW-B Dynamic Peering Relationship 60

61 Dynamic routing over vpc Use Case 2 : Remote Orphan Site Peering in DCI Deployment vpc as Data Center Interconnect (DCI) Remote Site 1 Remote Site 2 Each Switch has routing adjacency with both vpc device in other DC Each DC connected to a remote site by orphan port Remote sites forms routing adjacency with both peers of its directly connected DC S1 S2 S3 S4 61

62 Dynamic Routing over vpc New Supported Designs

63 Dynamic routing over vpc Supported Designs Layer 3 services devices with vpc Layer 3 over DCI - vpc P P P P P P P Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2. Supported on Nexus 9X00 in ACI mode Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card 63

64 Dynamic routing over vpc Supported Designs STP inter-connection using a vpc VLAN Orphan device with vpc peers over vpc VLAN P P P P P P Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2. Supported on Nexus 9X00 in ACI mode Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card 64

65 Dynamic routing over vpc Supported Designs Peering with vpc peers over FEX vpc host interfaces P P P Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2(0)D1(1) Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card 65

66 Dynamic Routing over vpc Unsupported Designs 66

67 Dynamic routing over vpc Unsupported Design Peering across vpc interfaces with unequal L3 metrics The routing metric on S1 is less than the routing metric on S2 (preferred path using S1). B Router2 Int VLAN 20 Po2 SVI Traffic from A to B may hash to S2. This traffic will need to traverse to peer-link to get to B through S1. Due to the vpc loop avoidance rule S1 will not allow traffic to flow to B. Int VLAN 20 S2 Int VLAN 10 Metric 20 Po100 Po1 Router1 Int VLAN 10 Int VLAN 20 S1 Int VLAN 10 Metric 10 SVI A 67

68 Dynamic routing over vpc Configuration L3 over vpc Configuration on Nexus 7x00 platform Command: Layer3 peer-router Mode: config-vpc-domain Default: Disabled Need to configure on BOTH the peers Requirements Command configured on both the peers. Peer-Gateway should be enabled. Peer link should be up. Both peer should run image supporting L3 over vpc feature. Auto Enabling Peer-Gateway If Layer3 peer-router command is enabled without Peer-Gateway a syslog will be displayed to enable Peer-Gateway. 68

69 Dynamic routing over vpc Example Configuration and Verification on Nexus 7x00 vpc domain 200 peer-keepalive destination source peer-gateway layer3 peer-router P P vpc domain 200 peer-keepalive destination source peer-gateway layer3 peer-router show vpc brief Peer Gateway : Enabled Operational Layer3 Peer : Enabled (output truncated for display) show vpc brief Peer Gateway : Enabled Operational Layer3 Peer : Enabled (output truncated for display) P 69

70 Benefits of Dynamic Routing over vpc No Static routes No Parallel links No design changes and loss of business Route peering across vpc s over existing infrastructure Routing between vpc DCI Most wanted by majority vpc customers 70

71 Dynamic Routing over vpc Devices without L3 over vpc support Don t attach routers to VPC domain via L2 port-channel Common workarounds: Individual L3 links for routed traffic Static route to FHRP VIP A B SVI 1 IP Y VIP A SVI 1 IP Z VIP A SVI 1 IP Y VIP A SVI 1 IP Z VIP A SVI 1 IP Y VIP A SVI 1 IP Z VIP A S1 S2 L3 ECMP S1 S2 S1 S2 SVI 2 IP X Router SVI 2 IP X Router SVI 2 IP X Router Static Route to VIP A 71

72 Design Best Practices vpc and Multicast S1 Source S2 vpc supports PIM-SM only vpc uses CFS to sync IGMP state Sources in vpc domain both vpc peers are forwarders Duplicates avoided via vpc loop-avoidance logic Sources in Layer 3 cloud Active forwarder elected on unicast metric vpc Primary elected active forwarder in case metric are equal Source Receivers 72

73 Agenda vpc Operations and Upgrade vpc Self Isolation vpc Shutdown Graceful Insertion and Removal ISSU / ISSD with vpc 73

74 P vpc Configuration Best Practices vpc Self-Isolation Error Triggered S P Self- Isolate S ISOLATED Operational Primary P S1 S2 S1 S2 S1 S2 S3 S3 S3 1. Error Triggered : All Line cards Fail or All Vlans s down on peer-link 2. S1 sends self-isolation message through the peer-keepalive 3. S2 takes over as operational Primary and S1 is isolated from the vpc domain P S vpc Primary vpc Secondary 74

75 vpc Configuration Best Practices Example Configuration and Verification on Nexus 7x00 vpc domain 100 peer-keepalive destination peer-gateway self-isolation vpc domain 100 peer-keepalive destination peer-gateway self-isolation sh vpc brief vpc domain id : 100 Self-isolation : Enabled (output truncated for display) sh vpc brief vpc domain id : 100 Self-isolation : Enabled (output truncated for display) 75

76 vpc Configuration Best Practices vpc Self-Isolation vpc self-isolation is turned OFF by default No Impact on vpc operation if sellf-isolation enabled Functional only when enabled on both vpc peers. Not part of vpc type-1 and type-2 consistency checks 76

77 vpc Configuration Best Practices vpc Shutdown Isolates a switch from the vpc complex Isolated switch can be debugged, reloaded, or even removed physically, without affecting the vpc traffic going through the non-isolated switch Primary vpc Secondary S1 S2 switch# configure terminal switch(config)# vpc domain 100 switch(config-vpc)# shutdown S3 77

78 Graceful Insertion and Removal Change window begins vpc system mode maintenance vpc One command! Pre-change System Snapshot 78

79 Graceful Insertion and Removal Change window complete vpc vpc system mode normal One command! Pre/Post-change Snapshot Comparison 79

80 Graceful Insertion and Removal Flexible framework providing a comprehensive, systemic method to isolate a node. Configuration profile foundation in NX-OS Initial support for: vpc/vpc+ ISIS OSPF EIGRP BGP Interface Per VDC on Nexus 7x00 Platform Release Nexus 5x00/6000 NX-OS 7.1 Nexus 7x00 NX-OS 7.2 Nexus 9000 NX-OS 7.X 80

81 ISSU / ISSD with vpc ISSU is the recommended system upgrade in a multi-device vpc environment vpc system can be independently upgraded with no disruption to traffic Upgrade is serialized and must be run one peer at a time (config lock will prevent synchronous upgrades) Configuration is locked on other vpc peer during ISSU Similar process of downgrades (ISSD) Check ISSU / ISSD compatibility matrix & ensure ISSU is supported from current to target release 5.2(x) / 6.2(x) 81

82 Agenda vpc with Fabric Technologies vpc with Fabricpath (vpc+) vpc with FCOE vpc with VXLAN vpc with ACI 82

83 FabricPath: an Ethernet Fabric Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00 FabricPath Eliminates Spanning tree limitations High resiliency, fast network re-convergence Any VLAN, Anywhere in the Fabric Connect a group of switches using an arbitrary topology With a simple CLI, aggregate them into a Fabric N7K(config)# interface ethernet 1/1 N7K(config-if)# switchport mode fabricpath 83

84 VPC vs VPC+ Architecture of vpc and FabricPath with vpc+ CE FP CE Port FP Port CE VLAN s FP VLAN s vpc vpc+ Physical architecture of vpc and vpc+ is the same from the access edge Functionality/Concepts of vpc and vpc+ are the same Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port vpc+ is not supported on Nexus 9X00 & Nexus 3X00 Series 84

85 Dynamic Routing over vpc+ Layer 3 devices can form routing adjacencies with both the vpc+ peers over vpc The peer link ports and VLAN are configured in FabricPath mode. Fabricpath Core vpc N55xx, N56xx, N6000 support this design with IPv4/IPv6 unicast and PIM-SM multicast This design is not supported on N7X00 P P N55xx, N56xx, N6000 Router/ Firewall P Fabricpath Link Dynamic Peering Relationship Routing Protocol Peer P 85

86 vpc with FCoE Unified Fabric Design vpc with FCoE is ONLY supported between hosts and N5X00 or N5X00 & N2232 pairs. Must follow specific rules: A vfc interface can only be associated with a single-port port-channel. While the port-channel configurations are the same on both switches, the FCoE VLANs are different. FCoE VLANs are not carried on the vpc peer-link (automatically pruned): FCoE and FIP ethertypes are not forwarded over the vpc peer link. vpc carrying FCoE between two FCF s is NOT supported. Best Practice: Use static port channel rather than LACP with vpc and boot from SAN. [If NX-OS is prior to 5.1(3)N1(1)] VLAN 10,20 LAN Fabric Nexus 5000 FCF-A Fabric A VLAN 10 ONLY HERE! VLAN 10,30 Fabric B Nexus 5000 FCF-B STP Edge Trunk vpc contains only 2 X 10GE links one to each Nexus 5X00 86

87 Why VXLAN? Problems being addressed: VLAN scale VXLAN extends the L2 segment ID field to 24-bits, potentially allowing for up to 16 million unique L2 segments over the same network Layer 2 segment elasticity over Layer 3 boundary VXLAN encapsulates L2 frame in IP-UDP header High Level Technology Overview: MAC-in-UDP encapsulation. Leverages multicast in the transport network to simulate flooding behavior for broadcast, unknown unicast and multicast in the same segment Leverage ECMP to achieve optimal path usage over the transport network 87

88 Dst. VXLAN Packet Format Outer Mac Header Outer IP Header UDP Header VXLAN Header Original L2 Frame FCS FCS MAC Addr. Src. MAC Addr. VLAN Type 0x8100 VLAN ID Tag Ether Type 0x0800 IP Header Misc Data Protocol 0x11 Header Checksum Outer Src. IP Outer Dst. IP UDP Src. Port VXLAN Port UDP Length Checksum 0x0000 VXLAN RRRR1RRR Reserved VNID Reserved For Your Reference 14 Bytes (4 bytes optional) 20 Bytes 8 Bytes 8 Bytes VXLAN is a Layer 2 overlay scheme over a Layer 3 network. VXLAN uses Ethernet in UDP encapsulation VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments 88

89 VXLAN Terminology VTEP Virtual Tunnel End Point Transport IP Network VTEP IP Interface VTEP IP Interface Local LAN Segment Local LAN Segment End System End System End System End System VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point). VTEP has two interfaces : 1. Bridging functionality for local hosts 2. IP identification in the core network for VXLAN encapsulation / de-encapsulation. 89

90 vpc VTEP When vpc is enabled an anycast VTEP address is programmed on both vpc peers Multicast topology prevents BUM traffic being sent to the same IP address across the L3 network (prevents duplication of flooded packets) vpc peer-gateway feature must be enabled on both peers VXLAN header is not carried on the vpc Peer link vpc VTEP vpc VTEP VLAN VXLAN 90

91 VXLAN & VPC VPC Configuration Map VNI to VLAN Source Interface individual IP is used for single attached Hosts anycast IP is used for VPC attached Hosts VTEP1 vlan 10 vn-segment For Your Reference interface loopback 0 ip address <VTEP individual IP orphan) ip address <VTEP anycast IP per VPC domain> secondary! interface nve1 source-interface loopback0 member vni mcast-group VXLAN Tunnel Interface vtep 1 vtep 2 vtep 3 vtep 4 VTEP2 vlan 10 vn-segment interface loopback 0 ip address <VTEP individual IP - orphan> ip address <VTEP anycast IP per VPC domain> secondary! interface nve1 source-interface loopback0 member vni mcast-group H VLAN 10 (vpc) H VLAN 10 (vpc) 91

92 VXLAN & VPC VPC Configuration For Your Reference VTEP1 vlan 10 vn-segment VTEP3 vlan 10 vn-segment interface loopback 0 ip address /32 ip address /32 secondary! Interface nve1 source-interface loopback0 member vni mcast-group interface loopback 0 ip address /32 ip address /32 secondary! Interface nve1 source-interface loopback0 member vni mcast-group VTEP2 vlan 10 vn-segment vtep 1 vtep 2 vtep 3 vtep 4 VTEP4 vlan 10 vn-segment interface loopback 0 ip address /32 ip address /32 secondary! Interface nve1 source-interface loopback0 member vni mcast-group interface loopback 0 ip address /32 ip address /32 secondary! Interface nve1 source-interface loopback0 member vni mcast-group H VLAN 10 (vpc) H VLAN 10 (vpc) 92

93 VXLAN & VPC Dual attached Host to dual attached Host (Layer-2) Host 1 (H1) and Host 2 (H2) are dual connected to a VPC domain As H1 is behind a VPC interface, the anycast VTEP IP is the source for the the VXLAN encapsulation vtep 1 vtep 20 vtep 2 vtep 3 vtep 30 vtep 4 As H2 is behind a VPC interface, the anycast VTEP IP is the target H VLAN 10 (vpc) H VLAN 10 (vpc) 93

94 Nexus APIC = ACI APIC APICAPIC 94

95 ACI uses a policy based approach that focuses on the application. QoS Filter Web QoS Service App QoS Filter DB External Network 95

96 vpc and ACI ACI fabric utilised for control-plane No dedicated peer-link between vpc peers: vpc Domains vpc peers Fabric itself serves as the MCT No out-of-band mechanism to detect peer liveliness: Due to rich fabric-connectivity (leaf-spine), it is very unlikely that peers will have no active path between them vtep 1 vtep 3 vtep 2 ACI fabric CFS (Cisco Fabric Services) is replaced by Zero Message Queue (ZMQ) As ACI fabric is VXLAN-based, an anycast VTEP is shared by both leaf switches in a vpc domain vpc vpc 96

97 Agenda Reference Material 97

98 Reference Material For Your Reference vpc Best Practices Design Guide: e.pdf vpc design guides: vpc and VSS Interoperability white Paper: VXLAN Overview : Fabrcipath whitepaper : ACI Overview 98

99 Key Take-Aways vpc in 2016 VXLAN, ACI, Fabricpath vpc Benefits No Blocked Ports High availability Fast Convergence Fabricpath Eliminates Spanning-Tree * High resiliency vpc+ for legacy switches, servers, hosts VXLAN L2 segment scalability VTEP redundancy with vpc ACI Policy Based Fabric for vpc control plane FCoE Unified Fabric for LAN & SAN 99

100 Call to Action Visit the World of Solutions for Cisco Campus Walk in Labs Technical Solution Clinics Meet the Engineer Lunch and Learn Topics DevNet zone related sessions 100

101 Complete Your Online Session Evaluation Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt. All surveys can be completed via the Cisco Live Mobile App or the Communication Stations 101

102 Many Things there s Something About vpc 102

103 Thank you 103

104