UCS Networking Deep Dive

Size: px

Start display at page:

Download "UCS Networking Deep Dive"

Della Poole
6 years ago
Views:

2 UCS Networking Deep Dive Michael Ciesla, Customer Support Engineer #clmel

3 Agenda UCS Overview Chassis Connectivity Server Connectivity Fabric Forwarding Topologies

4 Cisco Unified Computing System (UCS) Single Point of Management Logical Building Blocks Hardware/Software Abstraction (Service Profiles) 4

5 UCS Components LAN MGMT SAN Fabric Interconnect UCS Chassis Heartbeat link (No Data) 5

6 UCS Components LAN MGMT SAN Fabric Interconnect UCS Chassis Heartbeat link (No Data) IO Module 6

7 UCS Components LAN MGMT SAN Fabric Interconnect 4 KR lanes to each half width blade slot UCS Chassis IO Module Heartbeat link (No Data) 7

8 UCS Components LAN MGMT SAN Fabric Interconnect 4 KR lanes to each half width blade slot UCS Chassis IO Module Heartbeat link (No Data) 8

9 UCS Components Fabric Interconnect Cisco VIC IO Module Heartbeat link (No Data) UCS Blade 9

UCS 6248 Hardware Diagram 10 Gig 12 Gig Carmel 1 Carmel 2 Carmel 3 Unified Crossbar Fabric Sunnyvale 12 Gig Carmel 1 Carmel 2 Carmel 6 Carmel cpu Sunnyvale NVRAM Serial Carmel CPU South Bridge 0 1

10 UCS 6248 Hardware Diagram 10 Gig 12 Gig Carmel 1 Carmel 2 Carmel 3 Unified Crossbar Fabric Sunnyvale 12 Gig Carmel 1 Carmel 2 Carmel 6 Carmel cpu Sunnyvale NVRAM Serial Carmel CPU South Bridge 0 1 Memory Flash PCIE Dual Gig PCIe x4 Intel Jasper Forest PEX port PCIE Switch PCIe x4 PCIE Dual Gig PCIe x8 PCIe x4 PCIE Dual Gig DDR3 x N/C 12 Gig 12 Gig Xcon1 Mgmt Carmel 4 Carmel 5 Carmel 6 Xcon2 Console 10 Gig 10

11 Cisco UCS 6200 Series Fabric Interconnects Flexibility Product Features and Specs UCS 6248UP UCS 6296UP Switch Fabric Throughput 960 Gbps 1.92 Tbps Scalability Multipurpose Switch Footprint 1RU 2RU 1 Gigabit Ethernet Port Density Gigabit Ethernet Port Density G Native FC Port Density Port-to-Port Latency 2.0us 2.0us Active # of VLANs

12 UCS Mini: 6324 Fabric Interconnect UCS B + IO Modules + UCS 5108 Chassis Supports existing and future blades 6248 or 6296 Fabric Fabric Interconnects UCS Mini Fabric Interconnect UCS 5108 Chassis Supports existing and future blades

13 TAC Tip: Carmel ASIC Port Mapping

14 Chassis Connectivity Options

15 Californian Octopus

16 UCS Fabric Topologies Chassis Bandwidth Options 2208XP only 2x 1 Link 20 Gbps per Chassis 2x 2 Link 40 Gbps per Chassis 2x 4 Link 80 Gbps per Chassis 2x 8 Links 160 Gbps per Chassis 16

17 UCS 2200 IO Module (FEX) UCS-IOM-2204XP UCS-IOM-2208XP 40G to the Network 160G to the Hosts 2x10G Half width slot 4x10G Full width slot 80G to the Network 320G to the Hosts 4x10G Half width slot 8x10G Full width slot

18 220x-XP Architecture FLASH DRAM EEPROM Chassis Management Controller Control IO Switch Fabric Ports to FI Woodside ASIC Feature 2204-XP 2208-XP ASIC Woodside Woodside Fabric Ports (NIF) Host Ports (HIF) Chassis Signals Internal backplane ports to blades No Local Switching ever! Traffic goes up to FI 18 CoS 8 8 Latency ~ 500ns ~ 500ns

19 NIF/HIF Interfaces Output from show fex detail of the NXOS shell FI ports connecting to the FEX Backplane ports connecting to server 1/3 Link between the FI and IOM/FEX which the server will be using Additional interface?

20 UCS Internal Block Diagram Fabric Interconnects UCS x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ UCS 6248 Expansion Module Fabric Uplinks (NIFs) IO Modules 2208XP 2208XP Midplane Backplane Ports (HIFs) Adapter mlom Mezz x8 Gen 3 x8 Gen 3 Server Blade CPU 0 CPU 1 QPI Link UCS Blade Chassis

21 CLI Block Diagram Output from show platform software woodside sts of the IOM (FINAL POSITION TBD) Uplink #: Link status: SFP: [$][$][$][$][ ][ ][ ][ ] N N N N N N N N I I I I I I I I NI (0-7) HI (0-7) HI (8-15) HI (16-23) HI (24-31) H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I I [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] [ ][ ][ ][ ][ ][ ][ ][ ] \ \ / / \ \ / / \ \ / / \ \ / / \ \ / / \ \ / / \ \ / / \ \ / / blade8 blade7 blade6 blade5 blade4 blade3 blade2 blade1 Eth1/1/1 Port # from the FEX Port 21

22 CLI Command Output from show platform software woodside rate of the IOM Blade 1 fex-1# sh platform software woodside rate Port Tx Packets Tx Rate Tx Bit Rx Packets Rx Rate Rx Bit Avg Pkt Avg Pkt (pkts/s) Rate (pkts/s) Rate (Tx) (Rx) Err BI Kbps Kbps CI Kbps Kbps NI Kbps Kbps NI Kbps Kbps NI Kbps Kbps NI Kbps Kbps HI Kbps bps HI Kbps bps HI Kbps bps HI Kbps Kbps HI bps bps HI bps bps

23 Fabric Link Connectivity 23

24 Chassis Connectivity Policy 24

25 IO Module HIF to NIF Pinning 2208XP 1 Link HIF1-4 Slot 1 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 HIF9-12 Slot 3 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF

26 IO Module HIF to NIF Pinning 2208XP 2 Links HIF1-4 Slot 1 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 NIF2 HIF9-12 Slot 3 HIF9-12 NIF2 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF

27 IO Module HIF to NIF Pinning 2208XP 4 Links HIF1-4 Slot 1 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 NIF2 NIF3 NIF4 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 NIF2 NIF3 NIF4 27

28 IO Module HIF to NIF Pinning 2208XP 8 Links Slot 1 HIF1-4 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 NIF2 NIF3 NIF4 HIF9-12 HIF13-16 Slot 3 Slot 4 HIF9-12 HIF13-16 NIF2 NIF3 NIF4 NIF5 Slot 5 NIF5 NIF6 NIF7 NIF8 HIF17-20 HIF21-24 HIF25-28 Slot 6 Slot 7 Slot 8 HIF17-20 HIF21-24 HIF25-28 NIF6 NIF7 NIF8 HIF29-32 HIF

29 CLI Mapping FEX Level command Output from show fex detail of the NXOS shell <output truncated> Fabric interface state: Eth1/9 - Interface Up. State: Active Eth1/10 - Interface Up. State: Active Eth1/11 - Interface Up. State: Active Eth1/12 - Interface Up. State: Active Fex Port State Fabric Port Eth3/1/1 Up Eth1/9 Eth3/1/2 Up Eth1/10 Eth3/1/3 Down Eth1/11 Eth3/1/4 Up Eth1/12 Eth3/1/5 Up Eth1/9 Eth3/1/6 Down None Eth3/1/7 Down Eth1/11 Eth3/1/8 Down Eth1/12 Eth3/1/9 Up Eth1/12 Interface Level command Output from show interface ethernet <mod/port> fex-intf Fabric FEX Interface Interfaces Eth1/9 Eth3/1/1 Eth3/1/5 <output truncated> 29

30 IOM Link Failure Scenario HIF1-4 Slot 1 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 Link Failure NIF2 NIF3 NIF4 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 NIF2 NIF3 NIF4 30

31 IOM Link Failure Scenario HIF1-4 Slot 1 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 NIF2 NIF3 NIF4 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 NIF2 NIF3 NIF4

32 IOM Link Failure Scenario HIF1-4 Slot 1 HIF1-4 NIF1 HIF5-8 Slot 2 HIF5-8 NIF1 NIF2 NIF3 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 HIF9-12 HIF13-16 HIF17-20 HIF21-24 HIF25-28 HIF29-32 NIF2 NIF3 NIF4

33 Increased Bandwidth Access to Blades 4 links, Discrete - Today 8 links, Discrete Up to 8 links, Port-channel slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X Available bandwidth per blade 10Gb Statically pinned to individual fabric links Deterministic Path Fabric Interconnect slot 1 slot 2 slot 3 slot 4 slot 5 slot 6 slot 7 slot 8 F E X Available bandwidth per blade 20Gb Statically pinned to individual fabric links Deterministic Path Guaranteed 10Gb to each blade Fabric Interconnect F E X Available bandwidth per blade up to 160Gb Statically pinned to Portchannel Increased and shared bandwidth Higher Availability Fabric Interconnect 33

34 Port-channel Pinning No slot based pinning No invalid link count for NIF ports (no power of 2 rule) VIC 1200/1300 adaptor with DCE links in Port-Channel HIFs 2200-IOM Pinned to Po NIF Gen-1 adaptor with single 10G link HIF 34

35 Server Connectivity

Dual 10Gb Dual 40Gb (4 x 10Gb) 16x PCIe Gen1 16x PCIe Gen 2

36 Cisco Virtual Interface Cards (VIC) 1 st Gen 2 nd Gen 3 rd Gen M81KR, P81E 1240, 1280, 12xx 128 PCIe devices 256 PCIe Device Dual 10Gb Dual 40Gb (4 x 10Gb) 16x PCIe Gen1 16x PCIe Gen , 1380 Dual 8x PCIe Gen 3 Native 40Gb Support VXLAN & NVGRE RoCE

37 UCS Cisco 1200/1300 VIC Adapter UCS 2208 IOM UCS 2208 IOM Side A Side B UCS 1200/1300 VIC 256 PCIe devices 1240 Sereno 1280 Sereno 1340 Cruz 1380 Cruz 37

(Sereno) Fits in the Mezzanine slot of B200 M3/M4 Port

38 VIC 1240/1340 Plus Port Expander Card Base option supports dual 2x10Gb Option to enable all port of ASIC (Sereno) Fits in the Mezzanine slot of B200 M3/M4 Port Expander has no PCIe presence It is a passive connector device 38

39 Connectivity IOM to Adapter 2208 IOM 2208 IOM Implicit Port-channel between VIC 1200/1300 and UCS 2200 IOM 7-Tuple Flow based hash, 10 Gbps per flow Side A Side B A vnic is active on side A or B UCS 1200/1300 VIC vnic1 VM VM Flows Gb FTP traffic Gb UDP traffic 39

40 VIC 1240/1340 to IOM Connectivity MLOM only Fabric Interconnects UCS x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ UCS 6248 Expansion Module IO Modules 2208XP 2208XP Midplane Adapter 1340 VIC Empty Dual 2x10 Gb port-channel from VIC 1240/1340 to 2208 IO Modules x8 Gen 3 x8 Gen 3 Server Blade CPU 0 CPU 1 QPI Link B200 M3/M4 UCS Blade Chassis 40

41 VIC 1240/1340 to IOM Connectivity MLOM plus Port Expander UCS 6248 UCS 6248 Fabric Interconnects 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module IO Modules 2208XP 2208XP Midplane Port Channel 1 Port Channel 2 Adapter 1340 VIC x8 Gen 3 Port Exp x8 Gen 3 Port Expander Passive Increase BW to 80Gbps Dual 4x10Gbps Port-channel Server Blade CPU 0 QPI Link CPU 1 B200 M3/M4 41 UCS Blade Chassis

42 What Does The OS See? 42

VIC 1x40 & 1x80 to IOM Connectivity Fabric Interconnects UCS 6248 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ UCS 6248 Expansion Module IO Modules 2208XP 2208XP Midplane

43 VIC 1x40 & 1x80 to IOM Connectivity Fabric Interconnects UCS x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ UCS 6248 Expansion Module IO Modules 2208XP 2208XP Midplane Adapter Server Blade 1340 VIC VIC1380 x8 Gen 3 x8 Gen 3 CPU 0 CPU 1 QPI Link Adapter Redundancy Split vnic across adapters 4 2x10 Gb Port-channels B200 M3/M4 UCS Blade Chassis 43

44 Full Width Blade to IOM Connectivity MLOM, Port Expander, VIC1x80 UCS 6248 UCS 6248 Fabric Interconnects 16x SFP+ 16x SFP+ Expansion Module 16x SFP+ 16x SFP+ Expansion Module IO Modules 2208XP 2208XP Midplane Port Channel 1 Port Channel 2 Adapter 1340 VIC Port Exp 4x10 4x10 VIC1380 Total BW is 160G Four 40G port-channels x8 Gen 3 x8 Gen 3 x8 Gen 3 Server Blade CPU QPI Link CPU QPI Link CPU B420 M3 / B260 M4 44 UCS Blade Chassis

45 UCS Mini: Fabric to Server Connectivity Same server-side connectivity as the 2204XP IOM 20G per half width blade

46 Topology Designs For Maximum Bandwidth UCS 6248UP UCS 6248UP UCS 6248UP UCS 6248UP UCS 2104 IOM UCS 2208 IOM UCS 2208 IOM UCS 2208 IOM Side A Side B 1240 or M81KR Side A M81KR Side B Side A Side \B 12xx / 13xx Side A Side B 12xx / 13xx Shared IOM uplink bandwidth of 10Gbps vnic Burst up to 10Gbps Shared IOM uplink bandwidth of 80Gbps vnic Burst up to 10Gb Dedicated IOM uplink bandwidth of 10Gbps vnic Burst up to 10Gbps Shared IOM uplink bandwidth of 80Gbps 46

47 Fabric Interconnect VIF Calculation FI-A UPC UPC 1 2 UPC 3 UPC 6 UPC 1 UPC 2 UPC 3 UPC 6 UCS 2208XP UCS 2208XP UCS 2208XP UCS 2208XP IOM-A 2208 XP IOM-B 2208 XP IOM-A 2208 XP IOM-B 2208 XP Recommended Not recommended Maximise number of available VIFs to the host Minimal number of VIFs to the host

48 Virtual Interfaces

49 Cisco UCS: Infrastructure Virtualisation Switchport Virtualisation (veth, vfc) Fabric Interconnect vfc 1 veth 1 vfc 2 veth 2 Cable Virtualisation (VNTag) DCB Ethernet Eth 1/1 Eth 1/2 Individual Ethernets Service Profile # Adapters Identity (MAC / WWN) Firmware Settings Blade or Rack CPU MEM I/O PCIe PCIe Adapter Individual Storage (iscsi, NFS, FC) Server Abstraction 49 Adapter Virtualisation (NIV)

50 Abstracting the Logical Architecture Physical Logical 6200-A 6200-A 6200-A Switch Eth 1/1 vfc 1 veth 1 vfc 1 veth 1 Dynamic, Rapid Provisioning Cable IOM A 10GE A IOM A 10GE A State abstraction Location Independence Blade or Rack Adapter Blade vhba 1 vnic 1 Service Profile (Server) vhba 1 (Server) vnic 1 Physical Cable Virtual Cable (VN-Tag) 50

51 Logical Switch Fabric Extension (FEX) Concept Virtualising the Network Port Legacy multi-tier architecture LAN FEX architecture LAN Switch port extended over Fabric Extender Switch Switch Switch FEX

52 Fabric Extender Evolution Netw ork Administrator Many applications require multiple interfaces VN-TAG/IEEE 802.1BR FEX Cisco VN-TAG is the prestandard to IEEE 802.1BR Port Extension The 802.1BR Architecture provides the ability to extend the bridge (switch) interface to downstream devices Legacy

53 VN-TAG FEX architecture Switch FEX LAN Frame VNTAG Frame Application Payload TCP IP VN-TAG Ethernet VN-TAG Ethertype d p destination virtual interface l r ver source virtual interface

54 Fabric Extender Evolution Virtual Interfaces Netw ork Administrator LIF 802.1BR associates the Logical Interface (LIF) to a Virtual Interface (VIF) FEX VIF Adapter FEX

55 VN-Tag at the Adapter (Mezz Card) Level 55

56 56

between VM and PCI vnic Each VM gets a dedicated switch port on the Fabric Interconnect

57 Fabric Extender Evolution (VM-FEX) Netw ork Administrator VN-TAG/IEEE 802.1BR* FEX Each VM assigned dedicated NIC on ESXihosts PCIE bus Hypervisor arbitrates mapping between VM and PCI vnic Each VM gets a dedicated switch port on the Fabric Interconnect IEEE 802.1BR* VN-TAG/IEEE 802.1BR* Hypervisor VM network managed by Server administrator Legacy Adapter FEX VM-FEX

58 vnic vnic vnic vnic Fabric Failover End Host Mode (only) LAN SAN A SAN B Fabric provides NIC failover capabilities chosen when defining a service profile Chassis UCS Fabric Interconnects Fabric Extender Fabric Extender Traditionally done using NIC bonding driver in the OS Provides failover for both unicast and multicast traffic Works for any OS on bare metal and hypervisors Adapter CiMC Adapter CiMC UCS-6200-A /chassis/server/adapter/host-eth-if # show vif VIF: ID Fabric ID Transport Tag Status Oper State A Ether 0 Allocated Active 1202 B Ether 0 Allocated Passive Half Width Blade Half Width Blade 58

59 Fabric Forwarding - Ethernet

60 Ethernet Fabric Forwarding Mode of Operations LAN Switch mode: User configurable Fabric Interconnects behave like regular ethernet switches STP parameters are lock VLAN/Mac based forwarding End-host mode (EHM): Default mode No spanning-tree protocol (STP) Active/Active for all links and VLANs Port definitions Policy based forwarding No unknown unicast forwarding 60

61 End Host Mode Spanning Tree FI A Fabric A VLAN 10 L2 Switching LAN veth 3 veth 1 MAC Learning MAC Learning Completely transparent to the network Presents itself as a bunch of hosts to the network No STP simplifies upstream connectivity All uplinks ports are forwarding never blocked VNIC 0 VNIC 0 Server 2 Server 1 61

62 End Host Mode Unicast Forwarding FI Uplink Ports LAN VLAN 10 Server 2 Deja-Vu RPF veth 1 veth 3 MAC/VLAN plus policy based forwarding Server pinned to uplink ports Policies to prevent packet looping déjà vu check RPF No uplink to uplink forwarding No unknown unicast VNIC 0 VNIC 0 Server 2 Server 1 62

63 End Host Mode Multicast Forwarding Uplink Ports FI B LAN Broadcast Listener per VLAN veth 1 veth 3 B Broadcast traffic for a VLAN is pinned on exactly one uplink port (or port-channel) i.e., it is dropped when received on other uplinks Server to server multicast traffic is locally switched RPF and déjà vu check also applies for multicast traffic B VNIC 0 VNIC 0 Server 2 Server 1 63

64 Switch Mode VNIC 0 VLAN 10 L2 Switching Root VNIC 0 LAN veth 3 veth 1 MAC Learning Fabric Interconnect behaves like a normal L2 switch Rapid-STP+ to prevent loops STP parameters are not configurable Server vnic traffic follows STP forwarding states Use VPC to get around blocked ports VTP is not supported MAC address learning on both uplinks and server links Server 2 Server 1 64

65 Uplink Pinning

66 End Host Mode - Dynamic Pinning LAN FI A veth 2 veth 3 veth 1 VLAN 10 Pinning Switching UCSM manages the veth pinning to the uplink UCSM will periodically veth distribution and redistribute the veths across the uplinks VNIC 0 VNIC 0 VNIC 0 Server 2 Server 3 Server 1 66

End Host Mode Individual Uplinks Dynamic Re-pinning of failed uplinks FI-A Fabric A veth 3 VLAN 10 Sub-second re-pinning veth 1 Pinning Switching All uplinks forwarding for all VLANs

67 End Host Mode Individual Uplinks Dynamic Re-pinning of failed uplinks FI-A Fabric A veth 3 VLAN 10 Sub-second re-pinning veth 1 Pinning Switching All uplinks forwarding for all VLANs GARP aided upstream convergence No STP Sub-second re-pinning No server NIC disruption L2 Switching VNIC 0 VNIC stays up Server 2 VNIC 0 MAC A vswitch / N1K ESX HOST 1 VM 1 VM 2 MAC B MAC C 67

68 End Host Mode Port Channel Uplinks Recommended: Port Channel Uplinks No disruption No GARPs needed FI-A Fabric A More Bandwidth per Uplink Per flow uplink diversity No Server NIC disruption Fewer GARPs needed Faster bi-directional convergence Fewer moving parts RECOMMENDED veth 3 Sub-second convergence VLAN 10 L2 Switching VNIC 0 68 veth 1 NIC stays up Server 2 VNIC 0 MAC A Pinning Switching vswitch / N1K ESX HOST 1 VM 1 VM 2 MAC B MAC C

69 End Host Mode Static Pinning LAN FI A veth 2 veth 3 veth 1 VLAN 10 VNIC 0 VNIC 0 VNIC 0 Server 2 Server 3 Server 1 Pinning Switching Administrator Pinning Definition veth Interfaces veth 1 veth 2 veth 3 Administer controls the veth pinning Deterministic traffic flow Uplink Blue Blue Purple Pinning configuration is done under the LAN tab -> LAN Pin groups and assigned under the vnic No re-pinning with in the same FI Static and dynamic pinning can coexist 69

70 Which uplink is the servers veth pinned to? TME-UCS6100-A(nxos)# sh pinning border-interfaces Border Interface Status SIFs Po1 Active Veth1093 Veth1094 Veth1099 Po2 Active sup-eth2 Veth1103 Eth1/6 Down Eth1/7 Down Eth1/8 Down Total Interfaces : 5 70

71 Fabric Forwarding - Multicast

72 IGMP Querier? Three Options: 1. Upstream IGMP Querier / PIM Router 2. Fabric Interconnect IGMP Querier 3. IGMP Snooping disabled

73 UCS Multicast IGMP Querier Upstream LAN IGMP Querier / PIM Router 1. IGMP Querier Uplink Ports FI Broadcast Listener per VLAN 3. IGMP Report veth 1 veth 3 2. IGMP Report VNIC 0 Server 2 VNIC 0 Server 1

74 G-pinned?

75 UCS Multicast Internal Querier FI Uplink Ports IGMP Querier veth 1 veth 3 2. IGMP Report LAN Broadcast Listener per VLAN 1. IGMP Querier VNIC 0 Server 2 VNIC 0 Server 1

76 UCS Multicast Configuration (2.1+)

77 UCS & Microsoft Network Load Balancing (NLB) Unicast Mode Ethernet Switching mode only Nexus 1000v (no mac auto-static-learn) Multicast Mode IGMP Multicast Requires igmp querier

78 Fabric Forwarding - QoS

79 UCS Congestion Management 8 Classes: 1 FCoE, 1 best effort, 4 user-definable, 8 Classes: 2 reserved for control

80 UCS QoS Marking / Classification

81 Pause! LAN PFC or 802.3x Pause UCS 6200 PFC Pause PFC or 802.3x Pause UCS IOM Side A Adapter Side B

82 ssless rubric will continue leveraging their QoS/CoS semantics to ensure reliability. Priority Flow Control elow displays the differences in the format of the legacy PAUSE frame with that defined in bb. Note how the PFC frame now has fields targeting different traffic classes. Classical Ethernet Pause vs. Data Centre Ethernet PFC Pause Priority Flow Control Transmit Queues Ethernet Link Receive Buffers One One Two Two Three Three Four Four Five Five Six STOP PAUSE Six Seven Seven Eight Eight Eight Virtual Lanes Enables lossless Fabrics for each class of service PAUSE sent per virtual lane when buffers limit exceeded 82

83 PFC Pause: What does it look like?

84 UCS QoS Identifying Congestion UCSB-2-B(nxos)# show interface priority-flow-control ============================================================ Port Mode Oper(VL bmap) RxPPP TxPPP ============================================================ Ethernet1/1 Auto Off 0 0 Ethernet1/2 Auto Off 0 0 Ethernet1/3 Auto Off Ethernet1/4 Auto Off Ethernet1/5 Auto Off 0 0 fex-1# show platform software woodside loss frm_to Port Extra RMON Drop S SS Loss Counters COS XOFF S Port Tx Pause Rx Pause Errors Counters x RX SS Tx SS SS Total NI HI fex-1#

85 Fabric Forwarding - Storage

86 SAN End Host NPV Mode N-Port Virtualisation Forwarding SAN A SAN B FLOGI FDISC NPIV VSAN 1 F_Port N_Proxy 6100-A 6100-B NPIV F_Port VSAN 1 N_Proxy vfc 1 vfc 2 vfc 1 vfc 2 F_Proxy F_Proxy vhba 0 N_Port Server 1 VSAN 1 vhba 1 vhba 0 N_Port vhba 1 Server 2 VSAN 1

87 SAN End Host NPV Mode N-Port Virtualisation Forwarding with MDS, Nexus 5000 SAN A SAN B F_ Port Channel & Trunk NPIV VSAN 1,2 N_Proxy 6100-A 6100-B vfc 1 vfc 2 vfc 1 vfc 2 F_Proxy F_Port NPIV VSAN 1,2 vhba 0 N_Port Server 1 VSAN 1 vhba 1 vhba 0 vhba 1 Server 2 VSAN 2

88 SAN FC Switch Mode Direct Attach FC & FCoE Storage to UCS FC FCoE SAN Optional N_Port VSAN 1 VSAN 2 F_Port MDS TE_Port MDS 6100-A FC Switch 6100-B FC Switch vfc 1 vfc 2 vfc 1 vfc 2 F_Port vhba 0 N_Port Server 1 VSAN 1 vhba 1 vhba 0 vhba 1 Server 2 VSAN 2

89 Multi-Hop FCoE Supports MDS, N5K & N7K Unified Uplink port type FI in ENM Switching Mode VNP port type FI in FC Switching Mode VE port type MDS/N5K/N7K FCoE/Ethernet FCoE STORAGE FCoE FCoE MDS/N5K/N7K Unified Uplinks FCoE/Ethernet NPV/EHM UCS FI UCS FI UCS B-Series

90 Unified Appliance Ports Direct attach FCoE, iscsi, NFS & CIFS storage Storage Unified Appliance Port FCoE iscsi NFS CIFS UCS FI UCS FI UCS B-Series

91 FCoE with Adapter FEX VFC interface bound to 802.1BR / VN-Tag virtual ethernet interface SAN A vfcx vethx Fabric Interconnects vfcx vethx SAN B Can be FC or FCoE Can be FC or FCoE 802.1BR / VN-Tag IOM IOM 802.1BR / VN-Tag vhba A FC0 Adapter vhba B FC1 Binding OS

92 Topologies 92

93 Recommended Topology for Upstream Connectivity Access/Aggregation Layer vpc/vss Forwarding Layer 2 links Fabric Interconnect A Fabric Interconnect B 93

vnic-a vnic-b Layer 2 Disjoint Topology Production VLAN 10-20 Backup VLAN 30-40 A vnic can only participate in one L2 network upstream Both dynamic and static

94 vnic-a vnic-b Layer 2 Disjoint Topology Production VLAN Backup VLAN A vnic can only participate in one L2 network upstream Both dynamic and static pinning methods are supported Fabric Interconnect - A End Host Mode Fabric Interconnect - B End Host Mode IOM-A IOM-B VLAN 11 Adapter VLAN 30 Half Width Blade 94

95 Inter-Fabric Traffic Example VNIC 0 on Fabric A VNIC 1 on Fabric B VM1 Pinned to VNIC0 VM4 Pinned to VNIC1 VM1 on VLAN 10 VM4 on VLAN 10 EHM FI-A L2 Switching EHM FI-B VM1 to VM4: 1) Leaves Fabric A 2) L2 switched upstream 3) Enters Fabric B VNIC 0 ESX HOST 1 VNIC 1 VNIC 0 ESX HOST 2 VNIC 1 vswitch / N1K Mac Pinning vswitch / N1K Mac Pinning VM1 VM2 VM3 VM4 95

96 UCS & ACI Fabric vcenter APIC 1. FI sends CDP/LLDP to leaf, CDP to ESXi blade. 2. ESXi blade & leaf send discovery data to vcenter & APIC CDP / LLDP 3. APIC receives discovery data from vcenter CDP IOM IOM 4. APIC downloads policy on all leafs providing path to the ESXi blade

97 97 VIC 1300 New Features

98 Native 40 Gigabit Ethernet 10 Gb Encoding 40 Gb Encoding = 4 x 10 Gb

99 VIC 1300 VXLAN & NVGRE Offload Hardware based TCP segmentation & checksum verification Overlay encap/decapperformed by hypervisors: Increases CPU utilisation Decrease network throughput Inner & outer packet hardware processing: TCP segmentation TCP/UDP checksum IP checksum

100 VXLAN Performance Test 2 x ESXi 5.5 GA, VMware DVS UCS B200 M3, VIC 1380 FI 6248, IOM 2208 VM guest OS: RHEL bit, 1GB RAM, 1vCPU Unicast iperf test between two VMs iperf s, iperf c , TCP window size 23.2 KB Locally switch path on FI

101 VXLAN Offload Performance - CPU VXLAN stateless offloads disabled Host 1 Host 2 VXLAN stateless offloads enabled Host 1 Host 2

102 VXLAN Offload Performance - Throughput VXLAN stateless offloads disabled VXLAN stateless offloads enabled

103 VXLAN Offload Configuration

104 VIC 1300 RoCE Support Remote Direct Memory Access over Converged Ethernet (RoCE) Access remote node s memory w/o CPU interruption Lower latency, better CPU use "RoCE does for InfiniBand what FCoE did for Fibre Channel * * Scott Lowe -

105 RoCE vs Protocol InfiniBand Stack Protocol Stack RoCE Based Applications written over IB Transport Layer Socket applications Standard Ethernet applications written over Sockets API RDMA applications ULP L4 IB transport TCP L3 IB GRH IPv4 L2 IB Ethernet L1 IB (S/D/Q) XAUI XFI SGMII 2011 MELLANOX TECHNOLOGIES 8

106 Packet Format RoCE vs InfiniBand Packet Format InfiniBand LRH (L2 Hdr) GRH L3 Hdr BTH+ (L4 Hdr) InfiniBand Payload ICRC VCRC No Changes RoCE Eth L2 Header GRH L3 Hdr BTH+ (L4 Hdr) InfiniBand Payload ICRC FCS

107 Microsoft SMB Direct (SMB over RMDA) Initial use case for RoCE is Microsoft Windows 2012 SMB Direct 3.0 User Kernel Network w/ RDMA support Network w/ RDMA support NTFS SCSI Jose Barreto, Tech Ed, 2013, B335#fbid=

108 109 New Features in 2.2

109 IPv6 Management Support UCS FI management can be configured with IPv6 address IPv6 capable external services (i.e., NTP, SSH, TACACs, HTTP/HTTPs, etc) 110

110 Inband Management for CIMC Separate server management (CIMC) traffic from UCSM Designating servers CIMC into different groups Higher bandwidth 10G vs 1G 111

111 Faster and Better: Link Layer Enhancements Uplink ports only Faster link failure detection with UDLD Layer 2 protocol that runs on top of the physical layer to help detect mis-wiring and unidirectional communication UDLD Interval 7-90 seconds, with a detection time of 3x the interval LACP (pertain to uplinks only) Choices between different timers, slow (30 sec) or fast (1 sec). Default is slow. Provides very fast failure detection Enable suspend-individual link. Default is disable Match settings of the LAN Switch 112

112 PVLAN Enhancements Promiscuous on Appliance Port Community Support PVLAN trunking on the vnic (extend PVLAN to the virtual switches) 113

113 Netflow+ Improve workload visibility Capacity planning Security Troubleshooting 114

114 Other Sessions BRKCOM UCS Fundamentals BRKCOM Cisco UCS Network Performance Optimisation and Best Practices for VMware BRKCOM UCS Systems Management Deep Dive with UCS Foundational Software BRKCOM Hyper-Converged Computing BRKCOM Next Generation Computing Architectures for Cloud Scale Applications BRKCOM UCS C-Series Deployment Options, Best Practice and UCSM Integration BRKCOM UCS Performance Troubleshooting BRKVIR Multi-Hypervisor Networking - Compare and Contrast

115 Q & A

Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2015 T-Shirt! Complete your Overall Event Survey and 5 Session Evaluations.

116 Complete Your Online Session Evaluation Give us your feedback and receive a Cisco Live 2015 T-Shirt! Complete your Overall Event Survey and 5 Session Evaluations. Directly from your mobile device on the Cisco Live Mobile App By visiting the Cisco Live Mobile Site Visit any Cisco Live Internet Station located throughout the venue T-Shirts can be collected in the World of Solutions on Friday 20 March 12:00pm - 2:00pm Learn online with Cisco Live! Visit us online after the conference for full access to session videos and presentations.

117 Thank you.

118

UCS Networking Deep Dive. Neehal Dass - Customer Support Engineer

UCS Networking Deep Dive. Neehal Dass - Customer Support Engineer UCS Networking Deep Dive Neehal Dass - Customer Support Engineer Agenda Chassis Connectivity Server Connectivity Fabric Forwarding M-Series Q & A Cisco Unified Computing System (UCS) Single Point of Management