Evolution of the Data Centre Access Architecture

Transcription

1 Evolution of the Data Centre Access Architecture BRKDCT

2 Session Goals This session will provide a design level discussion on the best practices and use of the Cisco Nexus family of switches in the Data Centre focusing on the access layer and the connecting of servers to the edge of the Data Centre fabric This session will provide an understanding of how the Nexus 1000v, 2000, 5000/5500 and 7000 switch capabilities and features are most effectively used in building out the Data Centre access layer A detailed design discussion on the best practices for fabric extension using the Nexus 5000/5500/7000 parent switches with the 2000 Fabric Extender (FEX) will be included as well as an examination of the impact of new Data Centre technologies such as FCoE and FabricPath/TRILL to these designs The session will include an introduction and best practices for port extension (Adapter FEX) and embedded virtual bridging (Nexus 1000v) as elements in the switching architecture 2 2

3 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy FEXLink (Phase 2) Adapter-FEX Adapter-FEX Implications of the Next Gen Fabrics Next Generation Fabrics Virtual Machines & VM-FEX FEX Design Considerations 1K Cisco Nexus x86 3

4 Evolving Data Centre Architecture What is attached at the Edge? Spectrum of Design Evolution blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 blade1 blade2 slot 1 blade3 slot 2 blade4 slot 3 blade5 slot 4 blade6 slot 5 blade7 slot 6 blade8 slot 7 slot 8 Ultra Low Latency HPC/GRID Virtualized Data Center MSDC High Frequency Trading Layer 3 & Multicast No Virtualization Limited Physical Scale Nexus 3000 & UCS 10G edge moving to 40G Layer 3 & Layer 2 No Virtualization iwarp & RCoE Nexus 2000, 3000, 5500, 7000 & UCS 10G moving to 40G SP and Enterprise Hypervisor Virtualization Shared infrastructure Heterogenous 1G Edge moving to 10G Nexus 1000v, 2000, 5500, 7000 & UCS Layer 3 Edge (ibgp, ISIS) 1000 s of racks Homogeneous Environment No Hypervisor virtualization 1G edge moving to 10G Nexus 2000, 3000, 5500, 7000 & UCS 4

5 Evolving Data Centre Architecture Where Is the Edge? Eth 2/12 Edge of the Network and Fabric NIC HBA FC 3/11 Eth 2/12 pnic HBA FC 3/11 Still 2 PCI Addresses on the BUS Eth 2/12 Ethernet 10GbE Fibre Channel 10GbE Link vfc 3 PCIe Converged Network Adapter provides virtualisation of the physical Media VETH veth 1 Eth 1 vfc 2 vfc 4 SR-IOV adapter provides 10GE - VNTag multiple PCIe resources FC 2 vf C 3 Eth 3 FC 4 vfc 126 Eth 126 PCI-E Bus VETH VNIC VMFS SCSI PCI-E Bus Edge of the Fabric VETH VNIC VMFS SCSI PCI-E Bus Edge of the Fabric Edge of the Fabric Pass Thru VMF S SCSI PCI-E Bus Operating System and Device Drivers Hypervisor provides virtualisation of PCI-E resources Hypervisor provides virtualisation of PCI-E resources VNIC Hypervisor provides virtualisation of PCI-E resources Compute and Fabric Edge are Merging 5

6 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 6

7 Cisco FEXlink: Virtualised Access Switch Nexus 2000 Fabric Extender Cisco Nexus 5500 Cisco Nexus Distributed High Density Edge Switching System (up to 4096 virtual Ethernet interfaces) Cisco Nexus 2000 FEX Cisco Nexus 2000 FEX 7

8 Nexus 2000: Virtualised Access Switch Changing the device paradigm De-Coupling of the Layer 1 and Layer 2 Topologies Simplified Management Model, plug and play provisioning, centralised configuration Line Card Portability (N2K supported with Multiple Parent Switches N5K, 6100, N7K) Unified access for any server (100M 1GE 10GE FCoE): Scalable Ethernet, HPC, unified fabric or virtualisation deployment... Virtualised Switch 8

9 FEXLink Virtualised Access Switch Fabric Extender Terminology Parent Switch: Acts as the combined Supervisor and Switching Fabric for the virtual switch Fabric Links: Extends the Switching Fabric to the remote line card (Connects Nexus 5000 to Fabric Extender) Host Interfaces: (HIF) FET: Cost-effective transceiver to interconnect Nexus 2000 and Nexus 5000 and 7000 parent switch FEX100 Nexus 5000/5500/7000 FET supported only on Fabric Interfaces FEX101 dc # show interface fex-fabric Fabric Fabric FEX FEX FEX Port Port State Uplink Model Serial Eth1/17 Active 1 N2K-C2148T-1GE JAF1311AFLL 100 Eth1/18 Active 2 N2K-C2148T-1GE JAF1311AFLL 100 Eth1/19 Active 3 N2K-C2148T-1GE JAF1311AFLL 100 Eth1/20 Active 4 N2K-C2148T-1GE JAF1311AFLL 101 Eth1/21 Active 1 N2K-C2148T-1GE JAF1311AFMT 101 Eth1/22 Active 2 N2K-C2148T-1GE JAF1311AFMT 9

10 ucode image Nexus 2000 Fabric Extender Inband Management Model Line Card Model Fabric extender is discovered by switch using an L2 Satellite Discover Protocol (SDP) that is run on the uplink port of fabric extender NX5K checks software image compatibility, assign an IP address and upgrade the fabric extender if necessary N5K pushes programming data to Fabric Extender Satellite Control Protocol (SCP) used to manage the running state of the line card SCP Active control of running FEX SDP Discovers FEX 10

11 Nexus 2000 Fabric Extender Pre-Provisioning the remote line card FEX pre-provisioning allows the definition of a FEX prior to the FEX being physically connected to the Nexus 5000/5500 Prior to NXOS 5.0(2)N1(1) a FEX port configuration for a FEX that is not yet connected and recognised, the following configuration would fail: fex 198 pinning max-links 1 description "FEX0198" type N2224TP switch(config)# int eth198/1/1 With pre-provisioning this portion of the configuration can be pasted together with the fex 198 configuration even if the discovery process is not completed yet FEX is not yet attached, FEX ports are not initially visible Pre-Provisioned FEX and associated configuration is built FEX is attached and discovered via SDP FEX ID is assigned FEX HIF port configuration is applied to active FEX ports 11

12 Nexus 2000 Fabric Extender FEXLink - Fabric Extension Architecture The FEXLink Architecture provides the ability to extend the bridge (switch) interface to downstream devices FEXLink associates the Logical Interface (LIF) to a Virtual Interface (VIF) LIF LIF Bridges that support Interface Virtualisation (IV) ports must support VNTag and the VIC protocol FEX uplink ports must connect to a FEXLink capable bridge or an FEX Downlink FEXLink downlink ports may be connected to an FEX uplink port, bridge or NIC VIF FEX may be cascaded extending the port extension one additional level FEXLink downlink ports are assigned a virtual identifier (VIF) that corresponds to a virtual interface on the bridge and is used to forward frames through FEX s Hypervisor VIF FEXLink capable adapters may extending the port extension Note: Not All Designs Supported in the FEXLink Architecture Are Currently Implemented 12

13 Nexus 2000 Fabric Extender VN-Tag Port Extension Nexus 2000 Fabric Extender operates as a remote line card and does not support local switching All forwarding is performed on the Nexus 5000/5500 UPC or Nexus 7000 EARL VNTag is a Network Interface Virtualisation (NIV) technology that extends the Nexus 5000/7000 port down (Logical Interface = LIF) to the Nexus 2000 VIF referred to as a Host Interface (HIF) LIF Logical Interface (LIF) on the ingress UPC is used to forward the packet Packet is forwarded over fabric link using a specific VNTag VNTag is added to the packet between Fabric Extender and Nexus 5000/5500/7000 DA[6] VNTag is stripped before the packet is sent to hosts HIF SA[6] VNTag allows the Fabric Extender to act as a data path of Nexus 5000/5500/7000 for all policy and forwarding VNTAG[6] 802.1Q[4] Frame Payload l VNTAG Ethertype source virtual interface d p destination virtual interface N2K ASIC maps specific VNTag to HIF interface CRC[4] 13

14 Nexus Virtualised Access Switch Nexus 2000 Multicast Forwarding Nexus 2000 supports egress based Multicast replication Each fabric link has a list of VNTag s associated with each Multicast group A single copy of each multicast frame is sent down the fabric links to the Nexus 2000 Extended Multicast VNTag has an associated flooding fan-out on the Nexus 2000 built via IGMP Snooping Nexus 2000 replicates and floods the multicast packet to the required interfaces Note: When the fabric links are configured using static pinning each fabric link needs a separate copy of the multicast packet (each pinned group on the Nexus 2000 replicates independently) Port Channel based fabric links only require a single copy of the multicast packet 1. MCAST packets is received 2. MCAST frame is tagged with a unique VNTag 3. N2K ASIC has a mapping table of VNTag to IGMP Fan-Out 14

15 Nexus Virtualised Access Switch Nexus 2200 Port Channels Nexus 2248/2232 FEX support local port channels All FEX ports are extended ports (Logical Interfaces = LIF) A local port channel on the N2K is still seen as a single extended port Extended ports are each mapped to a specific VNTag HW hashing occurs on the N2K ASIC Number of local port channels on each N2K is based on the local ASIC 2. Packet is forwarded over fabric link using a specific VNTag for the destination N2K LIF (port channel) 1. Packet is received and lookup forwards out a LIF (N2K) interface 3. N2K ASIC hashes locally and transmits packet on one HIF interface 2148T T VM VMK SC 15

16 Nexus 5000/5500 and 2000 Virtual Switch Switching Morphology Is this Really Different? Nexus 2000 Nexus 5000/5500 Nexus 2000 Nexus 2000 NIV ASIC Ingress UPC Unified Crossbar Fabric Egress UPC Nexus 2000 NIV ASIC Line Card Ports, Buffers, Egress MCAST replication Distributed Forwarding ASIC X-Bar Fabric Internal Packet Header used across the Fabric (Constellation Header VNTag) Port ASIC & Buffers DFC Fabric ASIC Port ASIC & Buffers PFC 67xx - DFC Sup720 67xx - CFC 16

17 Cisco Nexus 2000 Series Platform Overview N2148T 48 Port 1000M Host Interfaces 4 x 10G Uplinks N2248TP 48 Port 100/1000M Host Interfaces 4 x 10G Uplinks N2232PP 32 Port 1/10G FCoE Host Interfaces 8 x 10G Uplinks N2224TP 24 Port 100/1000M Host Interfaces 2 x 10G Uplinks N2232TM 32 Port 1/10GBASE-T Host Interfaces 8 x 10G Uplinks (Module) N2248TP-E 48 Port 100/1000M Host Interfaces 4 x 10G Uplinks 32MB Shared Buffer FET-10G Cost Effective Fabric Extender Transceiver B22HP 16 x 1/10G Host Interfaces 8 x 10G Uplinks 17

18 10G NFS Nexus 2248TP-E 32MB Shared Buffer Speed mismatch between 10G NAS and 1G server requires QoS tuning Nexus 2248TP-E utilises a 32MB shared buffer to handle larger traffic bursts Hadoop, NAS, AVID are examples of bursty applications You can control the queue limit for a specified Fabric Extender for egress direction (from the network to the host) You can use a lower queue limit value on the Fabric Extender to prevent one blocked receiver from affecting traffic that is sent to other non-congested receivers ("head-of-line blocking ) VM #2 10G Attached Source (NAS Array) VM #3 VM #4 1G Attached Server NAS iscsi N5548-L3(config-fex)# hardware N2248TPE queue-limit rx N5548-L3(config-fex)# hardware N2248TPE queue-limit tx N5548-L3(config)#interface e110/1/1 N5548-L3(config-if)# hardware N2348TP queue-limit tx Tune 2248TP-E to support a extremely large burst (Hadoop, AVID, ) 18

19 Nexus 2248TP-E Buffer Allocations N5596-L3-2(config-if)# sh queuing interface e110/1/1 Ethernet110/1/1 queuing information: Input buffer allocation: Qos-group: 0 frh: 2 drop-type: drop cos: xon xoff buffer-size Ingress queue limit(configurable) Queueing: queue qos-group cos priority bandwidth mtu WRR Queue limit: bytes Egress queue limit(configurable) Queue Statistics: Que Received / Tail Drop No Buffer MAC Error Multicast Queue No Transmitted Tail Drop Depth rx tx Egress queues: CoS to queue mapping Bandwidth allocation MTU Per port per queue counters <snib> Drop due to oversubscription 19

20 Ports (000's) DC Design Migration from 1G to 10G servers 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 10G Adoption on Servers 2008A 2009A 2010E 2011E 2012E 2013E 2014E 1 Gbps Ethernet 10 Gbps Ethernet Source: Crehan Research (Q4CY10) 10GBaseT Key Benefits: 10 Gigabit bandwidth requirements for 1 Gig NIC consolidation at the server access and virtual environments Ease of 1GBASE-T to 10GBASE-T migration Flexible, scalable cabling with standard RJ- 45 connector, at distances up to 100m Reuse of existing structured cabling Economics of 1Gigabit Ethernet versus 10 Gigabit Ethernet Prepare for Server LOM Investment protection with 1/10G capabilities 20

21 DC Design Details 10GBaseT Power and EMI Considerations Undesired coupling of signal between adjacent cables Main electrical parameter limiting the performance of 10G Cannot be cancelled Re-Training is the major barrier to use of 10GBaseT for block level storage (FCoE) Can be prevented or mitigated by: Space (Cat6a solution) Shield (Cat6/Ca6a/Cat7 shielded solutions) Technology Cable Distance Power (each side) Transceiver Latency 2232PP SFP+ CU Copper Twinax 1-10m ~0.1-1W ~0.25ms 2232TM 10GBASE-T 65nm Cat6/6a/7 Cat6/6a/7 100m 30m ~6W ~4-5W ~3ms ~3ms 21

22 DC Design Details Blade Chassis Nexus B22 Series Fabric Extender B22 extends FEX connectivity into the HP blade chassis Cisco Nexus 5000 Switch is a single management point for all the blade chassis I/O modules 66% decrease in blade management points* Cisco Nexus B22 Series Blade FEX Blade & rack networking consistency Interoperable with Nexus 2000 Fabric Extenders in the same Nexus parent switch End-to-end FCoE support Support for 1G & 10G, LOM and Mez Dell supports Pass-Thru as an alternative option to directly attaching Blade Servers to FEX ports Nexus B22 (HP FEX) 22

23 Evolutionary Fabric Edge Mixed 1/10G, FC/FCoE, Rack and Blade Consolidation for all servers both rack and blade onto the same virtual switch Support for 1G, migration to 10G, FC and migration to FCoE 1G server racks are supported by 1G FEX (2248TP, 2224TP) or future proofed with 1/10G FEX (2232PP or 2232TM) 10G server racks are supported by the addition of a new 10G FEX (2232PP or 2232TM) Support for direct connection of HBA to Unified Ports on Nexus 5500UP 1G, 10G and FCoE connectivity for HP Blade Chassis Support for NPV attached blade switches during FC to FCoE migration 24

24 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 25

25 Nexus Virtualised Access Switch Nexus 2000 Design Considerations N2K HIF ports have BPDU Guard enabled by default (it is not possible to disable currently) If a BPDU is received port will transition to err-disable state Global BPDU Filter compliments BPDU Guard On link up port will send BPDUs and then stop (in order to reduce CPU load) If BPDU is received the port will err-disable This is NOT interface level BPDU Filtering dc # show spanning-tree int eth 155/1/25 detail Port 1945 (Ethernet155/1/25, vpc) of MST0000 is designated forwarding Port path cost 200, Port priority 128, Port Identifier <snip> The port type is edge Link type is point-to-point by default, Internal Bpdu guard is enabled Bpdu filter is enabled by default PVST Simulation is enabled by default BPDU: sent 11, received 0 1. X-Connected patch cable 2. BPDU Sent on Link- Up 26 E 3. BPDU Guard errdisables edge port and prevents loop E E 4. BPDU are not sent once link is up and active

26 Nexus Virtualised Access Switch Nexus 2000 Design Considerations STP Logical Ports Logical Ports = (# Trunks) x (# VLANs per trunk) STP Network Ports Nexus 7000 STP logical port scaling: Rapid-PVST+ limit = 16,000* STP Edge Ports MST limit = 90,000* Nexus 5500 STP logical port scaling: Rapid-PVST+ limit = 32,000* MST limit = 32,000* Key Point: Ensure you count EDGE & EDGE TRUNK ports too * Not a HW (Line Card, ASIC) limitation, SW improvements will increase this number, CPU upgrades will also increase scalability 27

27 Nexus 7000 Parent Switch System High Availability Nexus 7000 provides chassis based high availability All physical components physically redundant NX-OS high availability Fabric Port channel between a Nexus 2248 to a single Nexus 7000 The port channel can span several I/O Modules for redundancy Component level redundancy is similar to dual homed Nexus 5000/5500 with dual homing to 2 x Nexus 7000 Dual Sup and Chassis HA Fabric Port Channel Spans line cards 28

28 Virtualised Access Switch Supervisor Redundancy vpc Single Supervisor Based Virtual Switch Single Supervisor 1 x N5K Dual Supervisor Based Virtual Switch 4.1(3)N1 Dual Supervisor 2 x N5K 1 12 Line Cards per virtual switch Gbps Fabric allocated per line card (N2K) 1 12 Line Cards per virtual switch Gbps Fabric allocated per line card (N2K) 29

29 Virtual Port Channel vpc Multi-chassis Etherchannel (MCEC) vpc allows a single device to use a port channel across two neighbour switches (vpc peers) Eliminate STP blocked ports Layer 2 port channel only Provide fast convergence upon link/device failure Available in NX-OS 4.1(4) on the Nexus 7000 (Shipping)* Available in NX-OS 4.1(3)N1 on the Nexus 5000 (Shipping)* vpc Peers MCEC vpc Peers MCEC * Currently Recommended Releases N7K 4.2(6) or 5.1(3) N5K 5.0(3)N2(2a)! Enable vpc on the switch dc (config)# feature vpc! Check the feature status dc (config)# show feature include vpc vpc 1 enabled Please see session - BRKDCT-2048 Deploying Virtual Port Channel in NX-OS for more detailed information on how vpc works 30

30 Nexus Virtualised Access Switch Redundant Switches vpc provides two redundancy designs for the virtualised access switch Option 1 - MCEC connectivity from the server Logically a similar HA model to that currently provided by VSS dc # sh mac-address-table int port-channel 50 VLAN MAC Address Type Age Port f.275e.2918 dynamic 0 Po f.275e.7f98 dynamic 300 Po50 CFS dc # sh mac-address-table in port-channel 50 VLAN MAC Address Type Age Port f.275e.2918 dynamic 300 Po f.275e.7f98 dynamic 10 Po50 Port Channel #50 N5K vpc Port Channels 4.2(1) (2) 768 VM VMK SC 5.0(3)

31 Nexus Virtualised Access Switch Redundant Supervisors vpc Option 2 Fabric Extender connected to two Nexus 5000 From the server perspective a single access switch with each line card supported by redundant supervisors dc # sh vpc <snip> vpc status id Port Status Consistency Reason Active vlans <snip> Eth155/1/13 up success success 105 dc # sh vpc <snip> vpc status id Port Status Consistency Reason Active vlans <snip> Eth155/1/13 up success success 105 CFS Port Channel #50 FEX 155 Ethernet 155/1/13 N5K vpc HIF Ports 4.2(1) (2)

32 vpc Virtual Port Channel Consistency Checks - Improvements NX-OS 5.0(2)N1(1) enhances the global consistency checks behaviour (Nexus 5000 & 5500), NX- OS 5.2 for Nexus 7000 Several global features have the misconfiguration type lowered from Type 1 to Type 2 Type 1 inconsistencies prevent the network from being catastrophically broken (new definition) tc-nexus5010-1# show vpc consistency-parameters global Name Type Local Value Peer Value QoS 2 ([], [3], [], [], [], ([], [3], [], [], [], []) []) Network QoS (MTU) 2 (1538, 2240, 0, 0, 0, (1538, 2240, 0, 0, 0, 0) 0) Network Qos (Pause) 2 (F, T, F, F, F, F) (F, T, F, F, F, F) Input Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0) Input Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F) Priority) Output Queuing (Bandwidth) 2 (50, 50, 0, 0, 0, 0) (50, 50, 0, 0, 0, 0) Output Queuing (Absolute 2 (F, F, F, F, F, F) (F, F, F, F, F, F) 33

33 Nexus Virtualised Access Switch vpc - Graceful Resolution With Graceful Resolution only ports on the vpc secondary are suspended if a Type-1 global inconsistency occurs Starting from 5.0(3)N1(1) on Nexus 5000/5500 and 5.2 on Nexus 7000 dc (config)# vpc domain 10 dc (config-vpc-domain)# graceful consistency-check dc (config)# vpc domain 10 dc (config-vpc-domain)# graceful consistency-check VM VMK SC 34

34 Nexus Virtualised Access Switch vpc Config-Sync Config-sync allows administrators to make configuration changes on one switch and have the system automatically synchronise to its peers. Config-sync and Graceful conflict resolution are complementary features Config-sync traffic is carried over the Mgmt. 0 interface (CFSoIP) VM VMK SC Config-Sync allows changes to common resource on both parent switches via one CLI change 35

35 vpc Design Guidelines Config Sync N5000-1# feature vpc vpc domain 10 peer-keepalive destination N5000-1#sh run switch-profile Switch-profile Apple sync-peers destination N5000-1(config-if)# config sync N5000-1(config-sync)# switch-profile Apple N5000-1(config-sync-sp)# int ethernet 100/1/3 N5000-1(config-sync-sp-if)# switch mode trunk N5000-1(config-sync-sp-if)# verify Verify Successful N5000-1(config-if)# config sync N5000-1(config-sync)# switch-profile Apple N5000-1(config-sync-sp)# commit N5000-2# feature vpc vpc domain 10 peer-keepalive destination N5000-2#sh run switch-profile Switch-profile Apple sync-peers destination NOTE: Verify Does Not Push the Config to Peer, User Must Issue Commit for Sync to Take Place If Sync Fails, then the Config Is in the BUFFER Commit Successful N5000-1#sh run switch-profile interface ethernet 100/1/3 switchport mode trunk N5000-2#sh run switch-profile interface ethernet 100/1/3 switchport mode trunk Problem: When a port-channel s configuration is modified which results in the bundle member s configuration also to change, this change is not captured by config-sync - CSCti63620 Solution: Restrict port-channels to be configured either completely within switch-profile or completely from conf-t mode (similar to how acls, qos are subject to mutual-exclusion) Behaviour starting with 5.1(3)N1 36

36 vpc Design Guidelines Why vpc and not VSS or Stackwise? WAN Core FC Core L3 L2 Aggregation Core Access Edge LAN and SAN utilise different High Availability Models SAN is dual fabric, LAN is fully meshed fabric vpc enables both architectures at the edge (single device models not acceptable to SAN engineers) 37

37 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 38

38 Nexus Virtualised Access Switch Server Team view of HA NIC Teaming Options 6 Options 39

39 Nexus Virtualised Access Switch Nexus 2248/2232 Design NIC Teaming Nexus 2248/2232 support the same configuration options as the 2148T with the addition that they both support local port channels Nexus 2248 supports 24 port local port channels on each Nexus 2248 Up to 8 interfaces bundled into a single local port channel Nexus 2232 supports 16 port local port channels on each Nexus 2232 Up to 8 interfaces bundled into a single local port channel 802.3ad & vpc 802.3ad & vpc 802.3ad Nexus 2248 and 2232 support both local port channel and vpc for distributed port channels 40

40 FEXLink & vpc - Virtualised Access Switch Nexus 5000/5500 Topologies prior to 5.1(3)N1 Straight Through Dual Homed FCoE Adapters supported on 10G N2K interfaces vpc Supported with up to 2 x 8 links Local Etherchannel with up to 8 links Redundancy model Dual Switch with redundant fabric Provides isolation for Storage topologies (SAN A and B ) Port Channel and Pinning supported for Fabric Link Redundancy model Single switch with dual supervisor for fabric, data control & management planes No SAN A and B isolation (VSAN isolation sufficient in the future?) 41

41 FEXLink & vpc - Virtualised Access Switch Nexus 7000 Topologies supported as of 5.2 Nexus 7000 vpc Nexus 7000 vpc FCoE Adapters supported on 10G N2K interfaces vpc is not currently supported vpc Supported with up to 2 x 8 links Redundancy model Dual Switch (each switch supports redundant supervisors) vpc Supported with NX-OS 5.2 Local Etherchannel with up to 8 links Fabric links supported on N7K-M132XP-12, N7K- M132XP-12L & N7K-F248XP-25 Port Channel only supported for Fabric Links 42

42 FEXLink & vpc - Virtualised Access Switch Server NIC Teaming Topologies A vpc orphan port is an non-vpc interface on a switch where other ports in the same VLAN are configured as vpc interfaces vpc orphan ports have historically been problematic for mixed server topologies Prior to release 5.0(3)N2 on Nexus 5000/5500 and 5.2 on Nexus 7000 an orphan port was not shut down on loss of vpc peer-links With the latest NX-OS release the orphan ports on the vpc secondary peer will also be shut down triggering NIC teaming recovery for all teaming configurations (identical to VSS behaviour) Configuration is applied to the physical port vpc Supported Server fails over correctly vpc Active/Standby Server does not fail over correctly N5K-2(config)# int eth 100/1/1 N5K-2(config-if)# vpc orphan-ports suspend 43

43 Enhanced vpc Server NIC Teaming Topologies In an Enhanced vpc (EvPC)configuration any and all server NIC teaming configurations will be supported on any port (NX-OS 5.1(3)N1 - shipping Q4 CY11) No orphan ports in the design All components fully redundant in a MCEC environment Supported with Nexus 5500 only Requires support for Multipath LID (LIF port channels) Not required to support a mixed NIC teaming environment, use case is restricted to a mix of all three server NIC configurations (single NIC, ALB/TLB and 802.3ad) Single NIC Dual NIC 802.3ad Dual NIC Active/Standby 44

44 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 45

45 Cisco FabricPath NX-OS Innovation Enhancing L2 with L3 Switching Easy Configuration Plug & Play Provisioning Flexibility Routing Multi-pathing (ECMP) Fast Convergence Highly Scalable FabricPath FabricPath brings Layer 3 routing benefits to flexible Layer 2 bridged Ethernet networks 46

46 Cisco FabricPath A new control plane - ISIS Plug-n-Play L2 IS-IS manages forwarding topology IS-IS assigns addresses to all FabricPath switches automatically Compute shortest, pair-wise paths Support equal-cost paths between any FabricPath switch pairs FabricPath Routing Table S10 S20 S30 S40 Switch IF S10 S20 S30 S40 S200 L1 L2 L3 L4 L1, L2, L3, L4 L1 L2 L3 L4 FabricPath S400 L1, L2, L3, L4 S100 S200 S300 S400 47

47 Cisco FabricPath A New Data Plane The association MAC address/switch ID is maintained at the edge S10 S20 S30 S40 Switch ID space: Routing decisions are made based on the FabricPath routing table A B S100 S300 FabricPath (FP) S100 S200 S300 Switch S100 S300: FabricPath Routing Table IF L1, L2, L3, L4 MAC adress space: Switching based on MAC address tables A 1/1 Classical Ethernet (CE) 1/2 B S300: CE MAC Address Table MAC IF B 1/2 A S100 Traffic is encapsulated across the Fabric 48

48 FabricPath Encapsulation 16 Byte Mac-N-Mac Header Classical Ethernet Frame DMAC SMAC 802.1Q Etype Payload CRC 16 bytes Original CE Frame Cisco FabricPath Frame Outer DA (48) Outer SA (48) FP Tag (32) DMAC SMAC 802.1Q Etype Payload CRC (new) 6 bits bits bits 8 bits 16 bits 16 bits 10 bits 6 bits Endnode ID (5:0) U/L I/G Endnode ID (7:6) RSVD OOO/DL Switch ID Sub Switch ID LID Etype 0x8903 Ftag TTL Switch ID Unique number identifying each FabricPath switch Sub-Switch ID Identifies devices/hosts connected via VPC+ LID Local ID, identifies the destination or source interface Ftag (Forwarding tag) Unique number identifying topology and/or distribution tree TTL Decremented at each switch hop to prevent frames looping infinitely 49

49 FabricPath Key Concept #1 Conversational MAC Learning S10 S20 S30 S40 S300: FabricPath Routing Table Lookup A: Hit Learn source B B A S300 S100 FabricPath S100 S200 S300 Lookup A: Hit Send to S100 Switch S100 IF L1, L2, L3, L4 S100: CE MAC Address Table MAC IF A 1/1 B S300 A 1/1 S200: CE MAC Address Table 1/2 S300: CE MAC Address Table MAC IF MAC IF B B 1/2 A S100 Classical Ethernet Conversational Learning 50

50 FabricPath Key Concept #2 It s a Routed Network Describes shortest (best) paths to each Switch ID based on link metrics Equal-cost paths supported between FabricPath switches FabricPath Routing Table on S100 S10 S20 S30 S40 One best path to S10 (via L1) Switch S10 IF L1 S20 L2 S30 L3 S40 L4 Four equal-cost paths to S300 S200 S300 L1, L2, L3, L4 L1, L2, L3, L4 S100 S200 FabricPath S300 51

51 FabricPath Key Concept #3 It s a multi-topology network Root for Root for Tree 1 S10 S20 S30 Tree 2 S40 Multi-destination traffic constrained to loopfree trees touching all FabricPath switches Root switch elected for each multidestination tree in the FabricPath domain Loop-free tree built from each Root assigned a network-wide identifier (Ftag) Support for multiple multi-destination trees provides multipathing for multi-destination traffic Two multi-destination trees supported in NX- OS release 5.1 FabricPath S100 S200 S300 S100 S20 S100 S10 S10 S200 S30 S40 S200 S20 Root Logical Tree 1 S300 S40 Root Logical Tree 2 52 S300 S30

52 Where is the Layer 2 Boundary? Layer 2 Routing to the Access FabricPath provides a dykstra link state control plane supporting a layer 2 forwarding model Routed Access for Layer 2 L3 Core L2+L3 FabricPath Core FabricPath POD vpc POD vpc+ POD vpc+ POD Please see sessions, TECDCT Cisco FabricPath or BRKDCT-2081 Cisco FabricPath Technology and Design for more detailed information on FabricPath 53

53 What is FCoE? It s Fibre Channel From a Fibre Channel standpoint it s FC connectivity over a new type of cable called Ethernet From an Ethernet standpoints it s Yet another ULP (Upper Layer Protocol) to be transported FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FC-1 Encoding FC-0 Physical Interface FC-4 ULP Mapping FC-3 Generic Services FC-2 Framing & Flow Control FCoE Logical End Point Ethernet Media Access Control Ethernet Physical Layer 54

54 Why do we care about FCoE? Converged Fabrics FCoE SAN iscsi Appliance iscsi Gateway NAS Appliance NAS Gateway Host/ Server Computer System Computer System Computer System Computer System Computer System Application File System Volume Manager SCSI Device Driver FCoE Driver NIC Application File System Volume Manager SCSI Device Driver iscsi Driver TCP/IP Stack NIC Application File System Volume Manager SCSI Device Driver iscsi Driver TCP/IP Stack NIC Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC Application File System I/O Redirector NFS/CIFS TCP/IP Stack NIC Storage Transport Converged Storage Fabric Block I/O File I/O NIC NIC NIC NIC TCP/IP Stack TCP/IP Stack TCP/IP Stack TCP/IP Stack Storage Media FCoE iscsi Layer Bus Adapter iscsi Layer FC HBA FC File System Device Driver Block I/O File System FC HBA FC 55

55 Converged Fabric - Access CNA: Converged Network Adapter Nexus Edge participates in both distinct FC and IP Core topologies What impact does this migration have to the access layer? Converged Network Adapter (CNA) presents two PCI address to the Operating System (OS) OS loads two unique sets of drivers and manages two unique application topologies FCF FCF Server participates in both topologies since it has two stacks and thus two views of the same unified wire SAN Multi-Pathing provides failover between two fabrics (SAN A and SAN B ) NIC Teaming provides failover within the same fabric (VLAN) Unified Wire shared by both FC and IP topologies Ethernet Driver bound to Ethernet NIC PCI address 10GbE 10GbE Link Ethernet Fibre Channel Nexus Unified Edge supports both FC and IP topologies FC Driver bound to FC HBA PCI address PCIe Ethernet Drivers Fibre Channel Drivers Operating System 56

56 Converged Fabric - Access Unified Edge - Attaching an Initiator DCBX negotiation discovers DCB capable devices and negotiates lossless Ethernet capabilities/configs FIP Process discovery and negotiation of FCoE devices and characteristics FCoE VLANs are treated differently than native Ethernet VLANs: no flooding, broadcast, MAC learning, etc. The FCoE VLAN must not be configured as a native VLAN Shared Wires connecting to HOSTS must be configured as trunk ports and STP edge ports Note: STP does not run on FCoE vlans between FCFs (VE_Ports) nor on VLAN s with direct attach servers but does run on FCoE VLANs if a downstream DCB only bridge is attached VLAN 10,20 LAN Fabric Fabric A VSAN 2 VLAN 10 VSAN 3 FCF VN VF FCoE VLAN 10,30 FC Fabric B FCF Direct attach VN_Port to VF_Port! VLAN 20 is dedicated for VSAN 2 FCoE traffic (config)# vlan 20 (config-vlan)# fcoe vsan 2 57

57 Enhanced vpc Isolating SAN A and SAN B How is this achieved? Configuration associates FCoE traffic to a specific fabric link SAN A SAN B switcha(config)# fex 101 switcha(config-fex)# fcoe switchb(config)# fex 101 FCoE Storage FCoE Nexus 5000 (San A) Nexus 5000 (San B) Nexus 2000 Fabric Extender (FEX) CNA 58

58 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 59

59 FEX HIF Ports = Parent Switch LIF Ports FEX architecture Data Plane: Forwarding is performed on the Parent Switch ASIC s Port Extension allows the Fabric Extender to act as a data path extension of Parent Switch Capabilities of the HIF port on the FEX are based on the capabilities of the parent switch ASIC s forwarding the traffic for those HIF ports Future proofed architecture allows feature upgrade without swapping out the line cards LIF HIF Logical Interface (LIF) parent switch performs lookup and policy on the frame Packet is forwarded over fabric link using a specific VNTag N2K ASIC maps specific VNTag to HIF interface Application Payload access control & forwarding policy TCP IP Ethernet VNTAG 60

60 Nexus 5000 Parent Switch Supported Nexus 2000 Parent Switch FEX Supported For Your Reference # FEX FEX HIF Capabilities Nexus 5020 Nexus 5010 N2K-C2148T, N2K-C2248TP N2K-C2248TP-E, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM, B22-HP N2K-C2148T, N2K-C2248TP N2K-C2248TP-E, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM, B22-HP 12 STP Edge Ports FCoE VF ports 12 STP Edge Ports FCoE VF ports Nexus 5548P/UP N2K-C2148T, N2K-C2248TP N2K-C2248TP-E, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM, B22-H 24 L2 8 L3 STP Edge Ports FCoE VF ports FabricPath Edge Ports CTS Edge Ports Nexus 5596UP N2K-C2148T, N2K-C2248TP N2K-C2248TP-E, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM, B22-HP 24 L2 8 L3 STP Edge Ports FCoE VF ports FabricPath Edge Ports CTS Edge Ports 61

61 Nexus 7000 Parent Switch Supported Nexus 2000 Parent Switch FEX Supported # FEX FEX HIF Capabilities N7K-M132XP-12 N2K-C2248TP, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM* 32 - L L3 STP Edge Ports CTS Edge Ports For Your Reference N7K-M132XP-12L N2K-C2248TP, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM* 32 - L L3 STP Edge Ports CTS Edge Ports N7K-M108X2-12L N.A. (FEX not Supported) N.A. N.A. (FEX not Supported) N7K-F132XP-15 N.A. (FEX not Supported) N.A. N.A. (FEX not Supported) F2 48 x 10G N2K-C2248TP, N2K-C2224TP, N2K- C2232PP, N2K-C2232TM* 32 - L L3 STP Edge Ports, FabricPath Edge Ports CTS Edge Ports, FCoE VF ports (CY12) M2 40G & 100G N.A. (FEX not Supported VNTAG Capable) N.A. N.A. (FEX not Supported VNTAG Capable) * Supported with next release of NX-OS Q4CY11 62

62 Virtualised Access Switch Supported FEX Topologies (as of Q4CY11) Nexus 5000/ Nexus 2000 Topologies Supported Routed Access (5500 only) FabricPath (5500 only) FCoE (FCF, NPV & FCoE NPV) Adapter-FEX and VM-FEX vpc vpc+, EvPC & EvPC+ (5500 only) 24 FEX supported Nexus Nexus 2000 Topologies Supported Routed Access (F1/M1 or F2) FabricPath (F2 line cards only) vpc 32 FEX supported Future Capabilities vpc+ (F2 only planned CY12) FCoE (F2 only planned CY12) 63

63 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 64

64 Data Centre Architecture Evolution Embedded 802.1Q Bridging Nexus 1000v Embedded Virtual Bridge - Nexus 1000V 802.1q standards based bridge Performs packet forwarding and applies advanced networking features Policy Based port profile applies port security, VLAN, and ACLs, policy maps for QoS treatment for all systems traffic including VM traffic, Console & Vmotion/Vmkernel Generic adapter on generic x86 server Standard 802.1q based upstream switch Leveraging standard switch to switch links (QoS, trunking, channelling,..) Policy on upstream switch looks like standard aggregation configuration VM VM VM VM Nexus 1000V VEM Hypervisor Generic Adapter VNIC VETH 802.1Q Switch 65

65 Data Centre Access Architecture Connecting Nexus 1000V to MCEC capable upstream devices Prior to the 4.2(1)SV1(4) release (Jan 2011) it was best practice to leverage static configuration of 802.3ad port channels (no LACP) Prior to this release LACP bootstrap problem could impact certain topologies VEM brings up system VLAN and uses one uplink to communicate with VSM VSM sends LACP PDU to the VEM inside packet channel VEM sends and receives LACP PDU with upstream switch Port Channel negotiated with upstream switch If VEM to VSM communication had issues uplink port channel did not come up Post 4.2(1)SV1(4) use of active mode for port channels is preferred Pre 4.2(1)SV1(4) VM VMK SC Post 4.2(1)SV1(4) LACP PDU Sourced by the VSM Nexus1000(config)#port-profile type ethernet sys-uplink Nexus1000(config-port-prof)#no shut Nexus1000(config-port-prof)#channel-group auto mode active <snip> VM VMK SC LACP Sourced by the VEM 66

66 vpath Service Redirection Mechanism for VM s In the non-virtualized model services are inserted into the Data Path at choke points You always knew where a server was in relation to the services device You always knew how traffic got to that services device Virtualized workload may require a reevaluation of where the services are applied and how they are scaled Where VM s are provisioned and where VM s may be moved to Virtualized Services associated with the Virtual Machine (Nexus 1000v &vpath) 67 VM #2 VM #3 Client VM #4 Virtualized Services Nexus 1000v & vpath VSG, vwaas

67 What is vpath? Nexus 1000V vpath VMware ESX/ESXi Server Intelligence build into Virtual Ethernet Module (VEM) of Cisco Nexus 1000V virtual switch (version 1.4 and above); vpath has the following main functions: 1. Intelligent Traffic interception for Virtual Service Nodes (VSN): vwaas, vasa & VSG; 2. Offload the processing of Pass-through traffic VSN Server VM VEM VPATH Interception In/Out 3. ARP based health check; 4. Maintain Flow entry table. vpath is Multitenant Aware Leveraging vpath can enhance the service performance by moving the processing to hypervisor Upstream Switch VSM 68

68 vpath Header L2 or L3 Encapsulation / Decapsulation Upto: Uplink MTU ( ) L2 Mode LLC snap VPath-Hdr VPath-PDU Original packet.2 LLC: snap(dsap:0xaa ssap:0xaa), ctrl:0x03, oui:0x00000c, vendor:0x0136 Packets are redirected by vpath by encapsulating the original frame with Mac-in-Mac SNAP frame. vpath Redirection/Return traffic is sent in Nexus 1000v Service VLAN VPATH Header Overhead : 78 bytes TCP MSS Adjusted (in vwaas, for instance) to account for this overhead to avoid fragmentation. L3 vpath encapsulation original packet L3 Mode Ethernet IP UDP vpath hdr + PDU Ethernet Payload No change in L2 vpath encapsulation IP header added in addition VEM sends ARP request for VSG IP address then uses the response as destination MAC Proxy ARP is needed on the upstream router Fragmentation is not supported when redirecting to L3 VSG. Uplink MTU must be set properly 69

69 vpath Interception Packet Flow Server VSN 1 6 CM 8 VPATH 4 7 VPATH 3 VMware ESX 1 Nexus 1000V VEMs 5 VMware ESX 2 VEM: Virtual Ethernet Module VSM: Virtual Supervisor Module VSN: Virtual Service Node 2 Nexus 1000V VSM 1 vcenter Server Packet Flow Packet destined towards the Server entering the Physical Switch. Physical Switch forwards the packet to VMWare ESX1 (where the destination Server Virtual Machine resides) VEM determines that the packet has to be forwarded to Server VM But the Server VM s Port Profile has vpath interception. The incoming L2 frame is encapsulated with vpath header which says redirect the packet to VSN 1. The interception packet is sent in different VLAN (Nexus 1000v Service VLAN) Since the VSN is located in another ESX host, the encapsulated VPATH packet is switched from ESX 1 to ESX2. VSN 1 receives vpath encapsulated packet, decapsulates the outer vpath header, processes the inner frame and sends the services processed packet out with vpath encapsulation (with dest mac as ESX1 Nexus1000v vswitch vpath MAC Addr) in Service VLAN. The vpath Encapsulated return packet switched from ESX2 to ESX1. The vpath Header is decapsulated by VEM and inner packet is forwarded to the specified destination without further vpath interception (in the Server VLAN) BRKDCT-2023 Server VM receives the packet Cisco and/or its affiliates. All rights reserved. Cisco Public 70

70 Virtual Extensible Local Area Network (VXLAN) VM Scaling and Isolation Ethernet in IP overlay network Entire L2 frame encapsulated in UDP 50 bytes of overhead Include 24 bit VXLAN Identifier 16 Million logical networks VXLAN can cross Layer 3 (IPv4 currently) Tunnel between VEMs VMs do NOT see VXLAN ID IP multicast used for L2 broadcast/multicast, unknown unicast Technology submitted to IETF for standardization VXLAN Encapsulation Original Ethernet Frame Outer MAC DA Outer MAC SA Outer 802.1Q Outer IP DA Outer IP SA Outer UDP VXLAN Header (8 bytes) Inner MAC DA InnerM AC SA Optional Inner 802.1Q Original Ethernet Payload CRC Flags 8 bits Reserved (Nonce/Map-Version) 24 bits VXLAN Networker Identifier (VIN) 24 bits Res. (Locator Status Bits ) 8 bits 71

71 VXLAN Forwarding Basics Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn VEM learns VM s Source (MAC, Host VXLAN IP) tuple Broadcast, Multicast, and Unknown Unicast Traffic VM VM VM VM VM broadcast & unknown unicast traffic are sent as multicast Unicast Traffic Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM) VEM 1 VEM 2 72

72 Broadcast, Multicast & Unknown Unicast Using IP Multicast for Transport: Control Plane Web VM DB VM DB VM Web VM Join Multicast Group Join Multicast Group Join Multicast Group Join Multicast Group

73 Broadcast, Multicast & Unknown Unicast Using IP Multicast for Transport: Data Plane Forwarding Web VM DB VM DB VM Web VM Encapsulate with Blue VXLAN ID Multicast to Servers Registered for Encapsulate with Red VXLAN ID Multicast to Servers Registered for

74 Configuring VXLAN VXLAN-to-VLAN L2 Edge Gateway vshield Edge Virtual ASA (future) Scaling restrictions apply on number of VLANs supported currently (2K with the 4.2(1)SV1(5.1) release) VXLAN utilizes Bridge Domains Bridge-domain <name> Segment id < > Group <segment_mcast_group_ip> Look at the BD on the VEM: Vemcmd show bd bd-name <name> Configure the BD in a port-profile Switchport access bridge-domain <name> VM VM VM VXLAN/VLAN Edge Gateway VXLAN External VLAN External Service 75

75 VXLAN & vpath Infrastructure Prerequisites Multicast & Encapsulation IP Multicast forwarding is required More multicast groups are better Multiple segments can be mapped to a single multicast group If VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLAN If VXLAN transport is traversing routers, multicast routing must be enabled. Increased MTU needed to accommodate VXLAN encapsulation overhead Physical infrastructure must carry 50 bytes more than the VM VNIC MTU size. e.g MTU on VNIC -> 1550 MTU on switches and routers. Leverage 5-tuple hash distribution for uplink and interswitch LACP Encapsulation will generate a source UDP port based on a hash of inner packet 5-tuple If VXLAN traffic is traversing a router, proxy ARP must be enabled on first hop router 76

76 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 77

77 FEXLink Architecture 802.1BR Port and Fabric Extension The FEXLink Architecture provides the ability to extend the bridge (switch) interface to downstream devices LIF LIF Bridges that support Interface Virtualisation (IV) ports must support VNTag and the VIC protocol FEXLink associates the Logical Interface (LIF) to a Virtual Interface (VIF) FEXLink uplink ports must connect to an FEX capable bridge or an FEX Downlink FEXLink downlink ports may be connected to an NIV uplink port, bridge or NIC VIF FEXLink may be cascaded extending the port extension one additional level FEXLink downlink ports are assigned a virtual identifier (VIF) that corresponds to a virtual interface on the bridge and is used to forward frames VIF Hypervisor Adapter-FEX capable adapters may extending the port extension Note: Not All Designs Supported in the FEXLink Architecture Are Currently Implemented 78

78 Data Centre Architecture Evolution Adapter FEX 802.1BR Adapter-FEX presents standard PCIe virtual NICs (vnics) to servers Adapter-FEX virtual NICs are configured and managed via Nexus 5500 Forwarding, Queuing, and Policy enforcement for vnic traffic by Nexus 5500 Adapter-FEX connected to Nexus 2000 Fabric Extender - Cascaded FEX-Link deployment Forwarding, Queuing, and Policy enforcement for vnic traffic still done by Nexus 5500 vnic vnic vnic User Definable vnics Eth FC 10GbE/FC oe FC Eth vhba vhba 79 PCIe x16

79 Adapter FEX Pre-standard 802.1BR Adapter-FEX and VM-FEX are simply extending the switch port to the PCIe BUS The OS sees the same NIC driver, the switch port sees packets from the attached NIC Not a virtualised adapter on a shared wire (e.g. Neterion, ) vnic 1 VIF Multiple MAC addresses on a shared interface Cisco VIC Unique MAC addresses on a dedicated interface veth1 LIF Virtual NIC Nexus 5500 Adapter FEX 80

80 Adapter FEX Association of a vnic to a veth Virtual NIC (vnic): Refers to a hardware partition of a physical NIC as seen by an Operating System. Virtual Ethernet interface (veth): Refers to a virtual network port (vnic) as seen by the Nexus veth1 veth2 1A 2B vnic 1 vnic 2 OS 81

81 Adapter FEX Adapter Failover With Adapter Failover each vnic uses two VIFs (A and B) for redundancy Active veth1 Standby veth2 Standby veth1 Active veth2 Active/Standby Operation, no Port-channelling, i.e. 1A is active, and 1B is standby Half of the vnics use path A, and half of the vnics use path B, so all paths are used If one path fails the vnic automatically fails over to the alternate path without OS noticing it (and it updates the L2 forwarding tables with gratuitous ARP and IGMP general leave) NIC redundancy without OS teaming 1A 2A 1B 2B vnic 1 vnic 2 OS 82

82 Virtualised Access Switch High Density Edge 4096 veth ports Up to16 static vnic per adapter Up to 2 vhba (A-FEX) veth1 veth1 veth16 Up to 96 dynamic vnics (VM-FEX) Maximum supported interfaces = 4096 with the 5.1(3) release feature adapter-fex port-profile type vethernet user_data switchport trunk allowed vlan switchport trunk native vlan 2 switchport mode trunk state enabled port-profile type vethernet user_management switchport access vlan 1 state enabled 1A port-profile type vethernet user_backup switchport mode trunk switchport trunk allowed vlan switchport trunk native vlan 2 state enabled interface Ethernet100/1/5 description ucs_vic2/0 switchport mode vntag vnic 1 vnic 2 vnic 16 OS 83

83 Nexus 5500 Adapter-FEX P81E Virtual Interface Card & BCM57712 CNA vnics are presented to the host like standard PCIe devices In A-FEX mode: supports up to 16 Eth vnic and 2 FC vhba Adapter Failover feature: in failure scenarios, the vnic is mapped to the other port transparently to the OS In VM-FEX mode: supports up to 96 Virtual Interfaces ( vnics + vhbas) No need of trunking all VLANs to the server interface (improving security and scalability) 3 rd Party adapter supporting VN-TAG vnics are presented to the host like standard PCIe devices In A-FEX mode supports up to 8 Virtual Interfaces total Max of 8 veth Max of 2 vhba No adapter failover 84

84 802.1BR & 802.1Qbg Standards IEEE P802.1BR 85

85 802.1BR Status Working Group Ballot of Bridge Port Extension (P802.1BR), the IEEE standard for VNLink, has reached 100% approval by the voting members of the IEEE committee. November 10, the IEEE committee passed a motion to advance the draft standard to Sponsor Ballot. This is the final stage for ratification of the standard.the first Sponsor Ballot is expected to take place in late November Ratification of the standard is currently predicted for March 2012 The same is true for P802.1Qbg, which is the standard the includes some of the protocols that support Bridge Port Extension as well as the VEPA device being promoted by HP Both standards are expected to be ratified in March. 86

86 VNTag and 802.1BR Differences and Migration BR is based on ETAG vs. Cisco s VNTAG ETAG offers larger address field for VM s VNTAG has single control plane BR has an individual control plane for each data plane NIC s vendor are fine with the dual TAG Implementation Nexus and UCS TAG translation 87

87 Evolution of the Data Centre Access Architecture Agenda The Evolving Data Centre FEXLink (Phase 1) Nexus 2000 Nexus 2000 Architecture Design Considerations - vpc Design Considerations Server Redundancy Implications of the Next Gen Fabrics Next Generation Fabric FEX Design Considerations FEXLink (Phase 2) Virtual Machines Nexus 1000v Design Considerations Adapter-FEX, VM-FEX The Evolving Fabric and the Edge 1K Cisco Nexus x86 88

88 Nexus Port Profiles Nexus 7000, 5000/5500, 1000v Enables the application of common configuration across groups of ports A port-profile can inherit attributes from other portprofiles (nested profiles) A change to a port-profile automatically updates configuration of all member ports Any interface command available on a Nexus interface can be a part of a port-profile e.g. ACL, L3, VLAN, etc. Configuration precedence/order: Default config. < Port-profile < Manual config. foo Speed/Duplex 100 Mbps Full Duplex QoS Service Policy Input Layer 3 OSPF 300 OSPF Area 0 OSPF Hello 1s port-profile foo speed 100 duplex full service-policy input xyz ip router ospf 300 area 0 ip ospf hello-interval 1 Interface e2/1,e7/9,e11/4 port-profile foo E2/1 E7/9 E11/4 89

89 Nexus 1000v Port Profiles Common abstraction for physical and virtual Coordinated Management State between Network and Compute Coordinated Control Plane state between Network and Compute Transition to real time coordination between fabric and compute vcenter VSM n1000v(config)# port-profile WebServers n1000v(config-port-prof)# switchport mode access n1000v(config-port-prof)# switchport access vlan 100 n1000v(config-port-prof)# no shut VM #2 VM #3 VM #4 ESX & VEM 90

90 Adapter-FEX Port Profiles Common abstraction for physical and virtual Up to16 static vnic per adapter Up to 2 vhba (A-FEX) Up to 96 dynamic vnics (VM-FEX) Maximum supported interfaces = 2048 with the 5.1(3) release veth1 veth1 veth16 feature adapter-fex port-profile type vethernet user_data switchport trunk allowed vlan switchport trunk native vlan 2 switchport mode trunk state enabled port-profile type vethernet user_management switchport access vlan 1 state enabled port-profile type vethernet user_backup switchport mode trunk switchport trunk allowed vlan switchport trunk native vlan 2 state enabled interface Ethernet100/1/5 description ucs_vic2/0 switchport mode vntag 1A vnic 1 vnic 2 vnic 16 OS 91

91 Nexus 5500 VM-FEX Port Profiles Common abstraction for physical and virtual Coordinated Management State between Network and Compute Coordinated Control Plane state between Network and Compute Port Profiles applied via vcenter rather than directly on server and switch config Nexus 5500 vcenter N5500(config)# port-profile WebServers N5500(config-port-prof)# switchport mode access N5500(config-port-prof)# switchport access vlan 100 N5500config-port-prof)# no shut VM #2 VM #3 VM #4 VM-FEX 92

92 Nexus Port Profiles Consistent Policy Tools across VM and Physical MSFT Hyper-V VMware ESX A-FEX Port Profiles provides an object oriented interface provisioning model enabled consistently across physical & virtual (ESX, Hyper-V, VM-FEX) 93

93 Evolving Data Centre Architecture Where is the Edge? VM VM VM VM Nexus 1000V VEM Hypervisor Generic Adapter 802.1Q Switch Adpater-FEX Nexus 5500 Bare Metal Use Cases 802.1q Embedded Bridge - Nexus 1000v 802.1q Embedded Bridge + Adapter-FEX Nexus 1000v & Nexus 5500 VM-FEX Nexus 5500 Virtual Machine Use Cases 94

94 Q & A

95 Complete Your Online Session Evaluation Complete your session evaluation: Directly from your mobile device by visiting and login by entering your username and password Visit one of the Cisco Live internet stations located throughout the venue Open a browser on your own computer to access the Cisco Live onsite portal Don t forget to activate your Cisco Live Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit 96