UCS Networking Deep Dive

Transcription

1

2 UCS Networking Deep Dive Steve McQuerry, CCIE # 6108, UCS Technical 2

3 UCS LAN Deep Dive Agenda High-level system overview review of different Fabric Interconnects, IOM and Cisco adapters unified ports IO module port-channel Ethernet modes of operation end-host mode vs switching dynamic and static pinning concepts vnic failover demo Server Connectivity options M81KR vs VIC 1280 Recommend designs disjoint L2 VLANs C-series Integration 3

4 UCS LAN details System Overview

5 System Components: High-level Overview FABRIC INTERCONNECT 6120 & UP CHASSIS IO MODULE 2104XP 2208XP Cisco (M81KR; VIC1280) 3 rd party INTERFACE CARDS 5

6 UCS Fabric Portfolio UCS Fabric Interconnect UCS x 10GE Ports 1 RU 40x 10GE Ports 2 RU Ethernet or FC Expansion Modules UCS Fabric Interconnect UCS x Unified Ports (Eth/FC) 1 RU 32x base and 16x expansion UCS 6100 UCS 6200 UCS Fabric Extender UCS x 10GE Downlinks to Servers 4x 10GE Uplinks to FIs UCS Fabric Extender UCS 2208/ x 10GE Downlinks to Servers 8x 10GE Uplinks to FIs UCS 2104 UCS 2208/2204 IOM Adapters - M81KR VIC, M71KR, etc. Up to 2x 10GE ports M81KR: Up to 128 virtual interfaces Adapter - UCS VIC 1280 Up to 8x 10GE ports Up to 256 virtual interfaces Generation 1 Generation 2 6

7 UCS 6100 Series Fabric Interconnects Fabric Interconnects 20-Port L2 Interconnect 20 fixed ports 10GE/FCoE, fixed 1 Expansion Module 40-Port L2 Interconnect 40 fixed ports 10GE/FCoE, fixed 2 Expansion Modules Expansion Modules (2)Fibre Channel 8 Ports 1/2/4G FC 6 ports 2/4/8G FC Combo FC + Ethernet 4 Ports 10GbE/FCoE 4 Ports 1/2/4G FC Ethernet 6 Ports 10GE/FCoE 7

8 UCS 6248: Unified Ports Dynamic Port Allocation: Lossless Ethernet or Fibre Channel Native Fibre Channel Benefits FC Simplify switch purchase - remove ports ratio guess work Increase design flexibility Remove specific protocol bandwidth bottlenecks Eth Lossless Ethernet: 1/10GbE, FCoE, iscsi, NAS Use-cases Flexible LAN & storage convergence based on business needs Service can be adjusted based on the demand for specific traffic 8

9 UCS 6248 Hardware Diagram 10 Gig 12 Gig Carmel 1 Carmel 2 Carmel 3 12 Gig Carmel 1 Carmel 2 Carmel 6 Carmel cpu Sunnyvale NVRAM Serial South Bridge Memory Flash Intel Jasper Forest PCIe x4 PCIe x8 PEX port PCIE Switch PCIe x4 PCIe x4 DDR3 x2 Unified Crossbar Fabric Sunnyvale Carmel CPU 0 1 PCIE Dual Gig PCIE Dual Gig PCIE Dual Gig N/C 12 Gig 12 Gig Xcon1 Mgmt Carmel 4 Carmel 5 Carmel 6 Xcon2 Console 10 Gig 9

10 Generic Expansion Module (GEM) Unified Port GEM for UCS 6200 Series UCS-FI-E16UP 16 Unified Ports Ports can be configured as either Ethernet or Native FC Ports Ethernet operations at 1/10 Gigabit Ethernet Fibre Channel operations at 8/4/2/1G Uses existing Ethernet SFP+ and Cisco 8/4/2G and 4/2/1G FC Optics Flexibility AND Simplicity 10

11 Cisco UCS 6000 Series Fabric Interconnects Differences between Gen 1 and Gen 2 Product Features and Specs UCS 6120XP UCS 6140XP UCS 6248UP Flexibility Scalability Multipurpose Switch Fabric Throughput 520 Gbps 1.04 Tbps 960 Gbps Switch Footprint 1RU 2RU 1RU 1 Gigabit Ethernet Port Density Gigabit Ethernet Port Density G Native FC Port Density Port-to-Port Latency 3.2us 3.2us 2.0us # of VLANs * Layer 3 Ready (future) 40 Gigabit Ethernet Ready (future) Virtual Interface Support 15 per Downlink 15 per Downlink 63 per Downlink Unified Ports (Ethernet or FC) *1024 at FCS 11

12 Cisco UCS 6000 Series Fabric Interconnects Differences between Gen 1 and Gen 2 Feature 61x0 62xx Flash 16GB eusb 32GB isata DRAM 4GB DDR3 16GB DDR3 Processor Single Core Celeron 1.66 Dual Core Jasper Forest 1.66 Unified Ports No Yes Number of VIF s / UPC 128 / port fixed 4096 programmable Buffering per port 480KB 640KB VLANs 1k 1k (4k future) Active SPAN Session 2 4 (w/dedicated buffer) Latency 3.2uS 2uS MAC Table 16k 16k (32k future) L3 Switching No Future IGMP entries 1k 4k (future) Port Channels (96 in 6296) FabricPath No Future 12

13 UCS 6248: Unified Ports Dynamic Port Allocation: Lossless Ethernet or Fibre Channel Ports on the base card or the Unified Port GEM Module can either be Ethernet or FC Only a continuous set of ports can be configured as Ethernet or FC Ethernet Ports have to be the 1st set of ports Port type changes take effect after next reboot of switch for Base board ports or power-off/on of the GEM for GEM unified ports. Base card 32 Unified Ports GEM 16 Unified Ports Eth FC Eth FC 13

14 Configuring Unified Ports 14

15 Unified Port Screen Slider based configuration Only even number of ports can be configured as FC Configured on a per FI basis 15

16 Putting the Components Together SAN A ETH 1 ETH 2 SAN B MGMT MGMT Uplink Ports OOB Mgmt Fabric Switch Server Ports 6100 Fabric A Cluster 6100 Fabric B Fabric Extenders I O M I O M I O M Chassis 1 Chassis 20 I O M Virtualized Adapters Compute Blades Half / Full width A CNA B200 B A CNA CNA B250 B 16

17 UCS LAN Details Chassis Connectivity Options

18 Nexus 5000/7000 p10 Uplink or border link 1/ /19 HA links network port To SAN Fabric Interconnect A server ports Fabric B fabric links IOM 1 IOM 2 north These links represent backplane traces. e1/1/7 There are always two of those (one goes to Fab A, one to B) Chassis Server1/7 Server1/8 Operating System s (Ethernet and FC) as defined in Service Profile 18

19 UCS Fabric Topologies: Chassis Bandwidth Options 2x 1 Link 20 Gbps per Chassis 2x 2 Link 40 Gbps per Chassis 2x 4 Link 80 Gbps per Chassis 2x 8 Links 160 Gbps per Chassis 19

20 IO Modules 2104XP (Generation 1) FLASH DRAM EEPROM 4 Physical ports to Fabric Interconnect Up to 80Gbps per chassis Chassis Management Controller Control IO Redwood ASIC Chassis Signals Switch Up to 8 internal backplane ports to Blades Up to 20Gbps per slot No Local Switching ever! Traffic goes up to FI 20

21 IOM Connections 2104XP A 2104 IOM (sometimes called Fabric Extender ) provides 8+1 internal IO channels (8 slots + 1 internal mgmt network) 4 external ports (10Gbps each; no Etherchannel in the 1st release) The servers mezz cards use those IO channels for external connectivity Servers with one mezz card use one IO channel per IOM 1 can for instance use IOM 1 while 2 uses IOM2 This -to-iom routing is flexible and user-configurable Servers with two mezz cards use two IO channels per IOM Each IOM actually provides a 9th internal IO channel for internal management connectivity 21

22 Blade Northbound Ports Naming Convention with 2104XP These interfaces (show int brief - NXOS prompt) are backplane traces Eth X/Y/Z where X = chassis number Y = mezz card number (always 1 with half-width blades) Z = satellite port number (slot where the blade server resides) Eg: eth1/1/7 is one of the two northbound ports on server 7 s mezz card in chassis 1 22

23 2208-XP Architecture FLASH DRAM EEPROM 8 Physical ports to Fabric Interconnect Up to 160Gbps per chassis Chassis Management Controller Control IO Woodside ASIC Chassis Signals Switch Up to 32 internal backplane ports to blades No Local Switching ever! Traffic goes up to FI Up to 80Gbps per slot (with VIC 1280) 23

24 UCS I/O Modules Two generations, three IO modules Feature ASIC Redwood Woodside Woodside Host Ports Network Ports CoSes 4 (3 enabled) Support No Yes Yes Latency ~800nS ~500nS ~500nS Adapter Redundancy 1 mlom only mlom and Mezz mlom and Mezz 24

25 Block Diagram: Gen 2 UCS Fabric Details Fabric Interconnects UCS x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ UCS 6248 Expansion Module Double the Fabric Uplinks IO Modules 2208XP 2208XP Midplane Quadruple the Downlinks Mezzanine Mezz Card x16 Gen 2 x16 Gen 2 Server Blade IOH CPU CPU UCS Blade Chassis 25

26 IO Modules Connectivity Static Pinning Concepts

27 UCS 2104 (1st Gen FEX) Server to Fabric Pinning slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 1 link NIF Fabric Interconnect Server slots pinned to uplink Uplink: slots 1,2,3,4,5,6,7,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 2 links NIF Fabric Interconnect Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 4 links NIF Fabric Interconnect Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8 27

28 UCS 2208 (2 nd Gen FEX) Server to Fabric Pinning slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 1 link NIF Fabric Interconnect Server slots pinned to uplink Uplink: slots 1,2,3,4,5,6,7,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 2 links NIF Fabric Interconnect Uplink 1: slots 1,3,5,7 Uplink 2: slots 2,4,6,8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X 4 links NIF Fabric Interconnect Uplink 1: slots 1,5 Uplink 2: slots 2,6 Uplink 3: slots 3,7 Uplink 4: slots 4,8 28

29 UCS 2208 (2 nd Gen FEX) Server to Fabric Pinning slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X F E X 8 links NIF Fabric Interconnect Server slots pinned to uplink Uplink 1: slot 1 Uplink 2: slot 2 Uplink 3: slot 3 Uplink 4: slot 4 Uplink 5: slot 5 Uplink 6: slot 6 Uplink 7: slot 7 Uplink 8: slot 8 Server slots channeled across all uplinks 8 links Uplink 1: slots 1-8 NIF Fabric Interconnect Uplink 2: slots 1-8 Uplink 3: slots 1-8 Uplink 4: slots 1-8 Uplink 5: slots 1-8 Uplink 6: slots 1-8 Uplink 7: slots 1-8 Uplink 8: slots

30 Server to Fabric Interconnect Port Pinning 20 Gb 40 Gb 80 Gb _Port 1 _Port 1 11 Slot 1 Slot Slot 1 Slot Slot 1 Slot 2 2 Slot 3 Slot 4 Slot 3 Slot 4 3 Slot 3 Slot 4 4 Slot 5 Slot 6 Slot 5 Slot 6 Slot 5 Slot 6 Slot 7 Slot 8 Port 2 Slot 7 Slot 8 Port 2 _Port 1 _Port 1 Slot 7 Slot 8 Port 2 Port to 2104 or to 2104 or to 2104 or to 2104 or to 2104 or to 2104 or

31 Individual Links Static Pinning (IOM-FI) Static Pinning done by the system based on number of fabric ports 1,2 4, 8 (2^x) are valid links for initial pinning Applicable to both 6100 / 6200 and 2104XP/2208XP Blade 1 Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade 7 Blade 8 Server Ports Fabric Ports IOM Fabric Interconnect 31

32 Individual Links Fabric Port Failure Pinned HIFs are brought down Fabric Interconnect Other blades unaffected Blade 1 Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade 7 Server Ports Fabric Ports Blade 8 IOM 32

33 Individual Links Re-ack of chassis Blades re-pinned to valid number of links 1,2,4 or 8 Pinned blade connectivity affected HIF s brought down/up for re-pinning May result in unused links Server Ports Fabric Ports IOM Fabric Interconnect 6100/6200 Unused Link Addition of links requires re-ack of chassis. Blade 1 Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade7 Blade 8 33

34 IOM to Fabric Interconnect Port Pinning Server-to-Fabric Port Pinning Configurations 160 Gb (Discrete Mode) 160 Gb (Port Channel Mode) Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot to to to

35 Fabric Ports: Discrete Vs. Port Channel Mode 160 Gb (Discrete Mode) 160 Gb (Port Channel Mode) Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Slot 1 Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Servers can only use a single 10GE IOM uplink Bandwidth range per blade - 0 to 20 Gb A blade is pinned to a discrete 10 Gb uplink FabricFailover if a single uplink goes down Per blade traffic distribution Suitable for traffic engineering use case Addition of links requires chassis re-ack. Servers can utilize all 8x 10GE IOM uplinks Bandwidth range per blade 0 to 160 Gb A blade is pinned to a logical interface of 80 Gbps FabricFailover if all uplinks on same side go down Per flow traffic distribution with-in a port-channel Suitable for most environments Addition of links does not require chassis re-ack. Recommended with VIC

36 Port-Channel (IOM-FI) Only possible between XP Fabric Interconnect 6200 HIFs pinned to port-channel Port-Channel Hash IP L2 DA, L2 SA, L3 DA, L3 SA,VLAN FCoE L2 SA, L2 DA, FC SID,FC DID Server Ports Fabric Ports Blade 1 IOM 2208XP Port Channel Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade7 Blade 8 36

37 Port-Channel (IOM-FI) Link Failure Blades still pinned to Port-channel on a link failure HIF s not brought down till all members fail Server Ports Fabric Ports Fabric Interconnect 6200 IOM 2208XP Port Channel Blade 1 Blade 2 Blade 3 Blade 4 Blade 5 Blade 6 Blade7 Blade 8 37

38 UCS LAN details Virtual Interfaces (VN-Tag)

39 How Do Servers Communicate? We know servers with one mezz card present two (M81KR and non-cisco adapters) or 2 x 4 x 10G Base-KR external or northbound interfaces The OS knows nothing of this The OS sees PCI devices on the bus and loads device drivers for those devices In UCS, the Service Profile controls what the OS sees E.g.: a blade can be shown 6 x 10GE NICs and 2 x HBAs while another sees 8 x 10GE NICs and no HBAs This means the northbound physical interfaces between the adapter and the IOM can carry both Ethernet and FC traffic for several s. We need a mechanism to identify the origin server Concept of Virtual Interface or VIF (see next slide) 39

40 Virtual Interfaces Blade 1 eth2 eth0 hba0 OS eth3 eth1 hba1 Southbound or OS-side virtual Interfaces assigned through Service Profile Virtual interface tag to associate frames to a Fabric VIF (i.e. vmnic0, vmnic2, fc0, etc) 0 1 External 10GE physical port that connects to the backplane IOM 1 Backplane Eth X/Y/Z interface IOM 2 IOM-to-FI link Vif 1 Vif 2 Vif 3 Vif 4 Fabric A Fabric B 40

41 VN-Tag: Instantiation of Virtual Interfaces Virtual interfaces (VIFs) help distinguish between FC and Eth interfaces They also identify the origin server VIFs are instantiated on the FI and correspond to frame-level tags assigned to blade mezz cards A 6-byte tag (VN-Tag) is preprended by Palo and Menlo as traffic leaves the server to identify the interface VN-Tag associates frames to a VIF VIFs are spawned off the server s EthX/Y/Z interfaces (examples follow) 41

42 VN-Tag at the Adapter (Mezz Card) Level Connect to a server s adapter and use show-vifs 42

43 VIFs Ethernet and FC are muxed on the same physical links concept of virtual interfaces (vifs) to split Eth and FC Two types of VIFs: veth and vfc Veth for Ethernet and FCoE; vfc for FC traffic Each EthX/Y/Z interface typically has multiple vifs attached to it to carry traffic to and from a server To find all vifs associated with a EthX/Y/Z interface, do this: 43

44 Another Way to Find VIFs: 44

45 45

46 UCS LAN Details Server Connectivity

47 UCS 1280 VIC Customer benefits Dual 4x 10 GE (80 Gb per host) VM-FEX scale, up to 112 VM interfaces /w ESX 5.0 Feature details Dual 4x 10 GE port-channels to a single server slot Host connectivity PCIe Gen2 x16 PCIe Gen 2 x16 bandwidth limit is 32 Gbps HW Capable of 256 PCIe devices OS restriction apply PCIe virtualization OS independent (same as M81KR) Single OS driver image for both M81KR and 1280 VIC FabricFailover supported Eth hash inputs : Source MAC Address, Destination MAC Address, Source Pprt, Destination Port, Source IP address, Destination, P address and VLAN FC Hash inputs: Source MAC Address Destination MAC Address, FC SID and FC DID UCS 2208 IOM Side A UCS 1280 VIC 256 PCIe devices Side B UCS 2208 IOM 47

48 Cisco 1280 VIC Adapter Presents up to 116 Interfaces to the OS NICs or HBAs UCS 6248 Fabric Interconnect A IOM (2208) UCS 6248 Fabric Interconnect B IOM (2208) U C S B L A D E C H A S S I S S E R V E R B L A D E Physical Port 1 Physical Port 2 Physical Port 3 Physical Port vhba VIC Mezzanine Adapter OS Physical Port 5 Physical Port 6 Physical Port 7 Physical Port vhba

49 Connectivity IOM to Adapter Up to 32 Gbps throughput per using flow based port-channel hash 2208 IOM 2208 IOM Implicit Port-channel between UCS 1280 VIC adapter and UCS 2208 IOM Side A 1 VM Side B UCS 1280 VIC VM Flows Gb FTP traffic Gb UDP traffic 7-Tuple Flow based hash A is active on side A or B. A have access to up to 32 Gbps throughput. 49

50 Block Diagram: Next Gen UCS Fabric Details UCS 6248 UCS 6248 Fabric Interconnects 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module IO Modules 2208XP 2208XP Midplane Mezzanine Server Blade 1280 VIC IOH x16 Gen 2 x16 Gen 2 4x10 Gbps Ether channel from VIC 1280 to 2208 IO Modules No user configuration required flows are 7-tuple Load Balanced across links Each individual flow limited to 10Gb Fabric Failover available CPU CPU UCS Blade Chassis 50

51 Topology Designs For Maximum Bandwidth UCS 6248UP or UCS 6100 UCS 6248UP UCS 6248UP or UCS 6100 UCS 6248UP UCS 2104 IOM UCS 2208 IOM UCS 2208 IOM UCS 2208 IOM Side A 1280 VIC or M81KR Side B Side A M81KR Side B Side A UCS 1280 VIC Side \B Side A UCS 1280 VIC Side B Shared IOM uplink bandwidth of 10Gbps Burst up to 10Gbps Shared IOM Uplink with 1 server Host port pinned to a discrete IOM uplink Shared IOM uplink bandwidth of 80Gbps Burst up to 10Gb Shared IOM Port-Channel with 8 servers Host port pinned to a discrete IOM port-channel Dedicated IOM uplink bandwidth of 10Gbps Burst up to 10Gbps *(IOM uplink limitation) Dedicated IOM Uplink Host port-channel pinned to discrete IOM uplink Shared IOM uplink bandwidth of 80Gbps Burst up to 32Gbps *(PCIe Gen 2 limitation) Shared IOM Port-Channel with 8 servers Host port-channel pinned to the IOM port-channel 51

52 UCS Fabric: Backwards Compatibility and Interoperability

53 Block Diagrams: Gen 1 and Gen 2 UCS Fabrics UCS 6120 UCS 6120 UCS 6248 UCS x 10GE Expansion Module 20x 10GE Expansion Module 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module 2104XP 2104XP 2208XP 2208XP M81KR VIC x16 Gen VIC x16 Gen 2 x16 Gen 2 x16 Gen 2 IOH IOH CPU CPU UCS Blade Chassis CPU CPU UCS Blade Chassis 53

54 UCS Fabric Component Interoperability Complete hardware inter-operability between Gen 1 and Gen 2 Fabric Interconnect IOM Adapter Supported Min Software version required UCS M81KR UCSM 1.4(1) or earlier UCS M81KR UCSM UCS UCS1280 VIC UCSM UCS UCS1280 VIC UCSM UCS UCS M81KR UCSM UCS UCS M81KR UCSM UCS UCS1280 VIC UCSM UCS UCS1280 VIC UCSM UCS

55 Block Diagram: Mixed Configurations UCS 6120 UCS 6120 UCS 6248 UCS x 10GE Expansion Module 20x 10GE Expansion Module 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module 2208XP 2208XP 2104XP 2104XP M81KR VIC x16 Gen VIC x16 Gen 2 x16 Gen 2 x16 Gen 2 IOH IOH CPU CPU UCS Blade Chassis CPU CPU UCS Blade Chassis 55

56 Mixed Hardware Support For Upgrades Mixed hardware generation between Fabric A and B is only supported during hardware upgrade. Mixed hardware is not supported in production. Hardware Fabric A Fabric B Support FI Only during HW upgrade IOM Only during HW upgrade 56

57 UCS LAN Details Ethernet: Modes of Operation

58 Ethernet Switching Modes of Operation Default: end-host mode No spanning-tree protocol; no blocked ports ever Admin differentiates between server and network ports Using dynamic (or static) server to uplink pinning No MAC learning except on server ports; no flooding! Fabric failover for Ethernet s (not available in switch mode!) Highly recommended User-configurable: switch mode Fabric Interconnects behave like regular ethernet switches but no config mode accessible Very, very few reasons to adopt that mode Pre 1.4 release: direct attach of IP-based storage filers / arrays Microsoft Windows Network Load Balancing in unicast mode 58

59 Ethernet End Host Mode FI A Spanning Tree Fabric A VNIC 0 Server 2 VLAN 10 L2 Switching LAN veth 3 veth 1 VNIC 0 Server 1 MAC Learning MAC Learning Server pinned to an Uplink port No Spanning Tree Protocol Reduces CPU load on upstream switches Reduces Control Plane load on 6100 Simplified upstream connectivity UCS connects to the LAN like a Server, not like a Switch Maintains MAC table for Servers only Eases MAC Table sizing in the Access Layer Allows Multiple Active Uplinks per VLAN Doubles effective bandwidth vs STP Prevents Loops by preventing Uplink-to-Uplink switching Completely transparent to upstream LAN Traffic on same VLAN switched locally 59

60 End Host Mode Unicast Forwarding Server to server traffic on the same VLAN is locally switched Uplink port to uplink port traffic not switched Each server link is pinned to an uplink port / port-channel Network to server unicast traffic is forwarded to server only if it arrives on pinned uplink port. This is termed as the Reverse Path Forwarding (RPF) check No flooding of unknown unicast and multicast! Packet with source MAC belonging to a server received on an uplink port is dropped (Deja-Vu Check) Uplink Ports FI LAN VLAN 10 Server 2 Deja-Vu RPF veth 1 veth 3 VNIC 0 VNIC 0 Server 2 Server 1 60

61 End Host Mode Multicast Forwarding UCSM 2.0 Broadcast traffic for a VLAN is pinned on exactly one uplink port (or port-channel) i.e., it is dropped when received on other uplinks Server to server multicast traffic is locally switched RPF and deja-vu check also applies for multicast traffic Uplink Ports FI B LAN Broadcast Listener per VLAN veth 1 veth 3 B B VNIC 0 Server 2 VNIC 0 Server 1 61

62 End Host Mode Individual Uplinks Dynamic Re-pinning of failed uplinks FI-A Fabric A veth 3 VLAN 10 Sub-second re-pinning veth 1 Pinning Switching All uplinks forwarding for all VLANs GARP aided upstream convergence No STP Sub-second re-pinning No server NIC disruption L2 Switching VNIC stays up VNIC 0 MAC A vswitch / N1K ESX HOST 1 VM 1 VM 2 VNIC 0 Server 2 MAC B MAC C 62

63 End Host Mode Port Channel Uplinks Recommended: Port Channel Uplinks No disruption No GARPs needed FI-A Fabric A veth 3 VLAN 10 Sub-second convergence veth 1 Pinning Switching More Bandwidth per Uplink Per flow uplink diversity No Server NIC disruption Fewer GARPs needed Faster bi-directional convergence Fewer moving parts RECOMMENDED L2 Switching VNIC 0 NIC stays up Server 2 VNIC 0 MAC A vswitch / N1K ESX HOST 1 VM 1 VM 2 MAC B MAC C 63

64 Switching Modes: Switch Fabric Interconnect behaves like a normal Layer 2 switch Server traffic follows VLAN forwarding Spanning tree protocol is run on the uplink ports per VLAN Rapid PVST+ Configuration of STP parameters (bridge priority, Hello Timers etc) not supported VTP is not supported currently MAC learning/aging happens on both the server and uplink ports like in a typical Layer 2 switch Upstream links are blocked per VLAN via Spanning Tree logic FI-A Fabric A VNIC 0 Server 2 VLAN 10 L2 Switching Root LAN veth 3 veth 1 VNIC 0 Server 1 MAC Learning 64

65 End Host Mode Operation Primary Root Secondary Root External LAN Forwarding Layer 2 links Active/Active Border Ports Fabric InterConnect A Fabric InterConnect B Server Ports Server Ports 65

66 UCS LAN Details Fabric Failover

67 End Host Mode Fabric Failover for Ethernet Fabric provides NIC failover capabilities chosen when defining a service profile Traditionally done using NIC bonding driver in the OS Provides failover for both unicast and multicast traffic Works for any OS on bare metal and hypervisors as of UCS 1.4 Chassis UCS Fabric Interconnects LAN Fabric Extender Adapter CiMC SAN A SAN B Fabric Extender Adapter CiMC Half Width Blade Half Width Blade 67

68 Fabric Failover Bare Metal OS Works for any OS Upstream Switch UCS FI-A Uplink Ports Upstream Switch Uplink Ports UCS FI-B MAC-A garp VLAN 10 VLAN 20 VLAN 10 VLAN 20 Backplane Ports stays UP Server Ports 1 2 Fabric Ports 3 4 FEX-A Eth 1/1/4 Eth 0 HA Links UCS Blade Chassis Adapter MAC A MAC B Server Ports Fabric Ports 3 4 FEX-B Eth 1/1/4 Eth 1 Blade Server Backplane Ports PCI Bus Bare Metal Operating System Windows / Linux 68

69 Fabric Failover Hypervisors Not Recommended with version UCSM 1.3 or lower UCS FI-A Upstream Switch VLAN Web VLAN NFS VLAN VMK VLAN COS Uplink Ports Server Ports HA Links Uplink Ports VLAN Web VLAN NFS VLAN VMK VLAN COS Upstream Switch Server Ports UCS FI-B MAC-A garp Backplane Ports 1 2 Fabric Ports 3 4 FEX-A UCS Blade Chassis 1 2 Fabric Ports 3 4 FEX-B Backplane Ports Eth 1/1/4 Adapter Eth 1/1/4 Blade Server Veth10 Profile Web MAC C Eth 0 Veth11 Profile Web Veth20 Profile NFS Eth 1 MAC A MAC B MAC D MAC E Veth5 Profile VMK Kernel Veth10 Profile COS Service Console Hypervisor Switch 69

70 End Host Mode Recommended Topology, Disjoint VLANs, etc.

71 Recommended Topology for Upstream Connectivity Access/Aggregation Layer vpc/vss Forwarding Layer 2 links Fabric InterConnect A Fabric InterConnect B 71

72 Traditional Layer 2 Switching Single Layer 2 domain and no physical boundary between organizations. LAN: Production Public and Backup Vlan Trunk Vlan

73 End Host Mode Disjointed L2 Domains Version 1.4 and below Broadcast Link Each FI picks ONE uplink for incoming Broadcast/Mcst Processing DMZ 1 DMZ 2 EHM FI-A EHM FI-B Cannot see DMZ 2 Broadcasts DMZ 1 Server DMZ 2 Server 73

74 Disjointed Layer 2 Domains - Version 2.0 DMZ 1 VLAN DMZ 2 VLAN All Links Forwarding Prune VLANs EHM FI-A EHM FI-B DMZ 1 Server DMZ 2 Server Assumption: No VLAN overlap between DMZ 1 & DMZ 2 74

75 Disjoint Behavior with UCS 2.0 A can only Production VLAN Backup VLAN participate in one L2 network upstream Both dynamic and static Fabric Interconnect - A End Host Mode Fabric Interconnect - B End Host Mode pinning methods are IOM-A IOM-B supported VLAN 11 Adapter VLAN 30 -A -B Half Width Blade 75

76 Inter-Fabric Traffic Example VNIC 0 on Fabric A VNIC 1 on Fabric B VM1 Pinned to VNIC0 VM4 Pinned to VNIC1 VM1 on VLAN 10 VM4 on VLAN 10 EHM FI-A L2 Switching EHM FI-B VM1 to VM4: 1) Leaves Fabric A 2) L2 switched upstream 3) Enters Fabric B VNIC 0 ESX HOST 1 VNIC 1 VNIC 0 ESX HOST 2 VNIC 1 vswitch / N1K Mac Pinning vswitch / N1K Mac Pinning VM1 VM2 VM3 VM4 76

77 End Host Mode or Switch Mode That is the Question

78 Scalability Spanning Tree protocol not used in EHM Relieves control plane EHM is least disruptive to upstream network BPDU Filter/Guard, Portfast enabled upstream No MAC learning on uplink ports Current MAC address limitation on the 6100 ~14.5K. Recommendation: End Host Mode 78

79 Pinning for Deterministic Traffic Flows Dynamic pinning Server ports pinned to an uplink port/port-channel automatically Static pinning Specific pingroups created and associated with adapters Static pinning allows traffic management if required for certain applications / servers Recommendation: End Host Mode A veth 3 veth 1 Fabric A VNIC 0 VNIC 0 Server X Oracle DEFINED: PinGroup Oracle APPLIED: PinGroup Oracle Pinning Switching 79

80 Fabric Failover FI-A L1 L2 L1 L2 FI-B veth 1 veth 1 Fabric Failover is only applicable in EHM. Physical Cable IOM IOM NIC teaming software required to provide failover in Switch mode. Virtual Cable 10GE 10GE PHY Adapter Cisco VIC Menlo M71KR 1 VIRT Adapter Recommendation: End Host Mode OS / Hypervisor / VM 80

81 Active/Active Use of Uplinks End Host Mode Switch Mode Primary Root Secondary Root Primary Root Secondary Root LAN LAN Active/Active Border Ports Border Ports Blocking FI-A FI-B FI-A FI-B Server Ports Server Ports Recommendation: End Host Mode 81

82 Application Specific Scenarios Certain applications such as Microsoft Windows Server Network Load Balancing in Unicast mode have the need for unknown unicast flooding There is no flooding in EHM Certain network topologies provide better network path out of the Fabric Interconnect due to STP root placement and HSRP L3 hop. Switch Mode is catch all for different scenarios. Recommendation: Switch Mode 82

83 C-Series Integration into UCSM

84 C-Series UCSM Integration Phase Simplified topology and management Scalable up to 160 servers (target scale) 6X00 running UCSM 6X00 running UCSM Mix of B & C Series is supported (no B Series required) Nexus 2232 Nexus LOM ports exclusive CIMC connectivity C200M2, C210M2, C220M3, C240M3, C250M2, C260M2 or C460M2 GE LOM CIMC PCIe Adapter CPU Mem OS or Hypervisor Adapter support: Emulex CNA Qlogic CNA Intel 10g NIC Broadcom 10g NIC Cisco VIC Mgmt Traffic Data Traffic 84

85 What Have We Learned?

86 Recap UCS is first and foremost a server, not a switch Gen 1 vs Gen 2 components 2208 and VIC 1280 allow port-channels VIC 1280 with 2208XP for maximum bandwidth End-host mode forwarding rules: dynamic pinning Preferred mode of operation should always be end-host mode Very much plug and play, scalability, L2 multipathing, fabric failover Switch mode: spanning-tree, practically no user configuration possible Operational consistency: C-series integration 86

87 Complete Your Online Session Evaluation Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center. Don t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit 87

88 Final Thoughts Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042 Come see demos of many key solutions and products in the main Cisco booth 2924 Visit after the event for updated PDFs, ondemand session videos, networking, and more! Follow Cisco Live! using social media: Facebook: Twitter: LinkedIn Group: 88

89