Ethernet Services over IPoIB

Transcription

1 Services over Ali Ayoub, Mellanox Technologies March,

2 Background What is? encapsulation over InfiniBand RFC 4391/4392 Provides services over InfiniBand fabric Benefits: Acts like an data-link within the TCP/ Stack Socket-based apps run transparently Allow users to run -based unmodified applications on InfiniBand Supports NIC offloads: CSUM, TSO, etc.. And other performance features such as: NAPI, TSS, RSS, etc.. 2

3 Packet Format integrated as Data Link layer Sockets API is not affected Sockets API TCP/ Stack L5 L4 L3 L2 () L1 (IB) Packet frame: IB Payload 3

4 Limitations is limited for applications only No encapsulation network interface doesn t act like a standard Network Device Non-standard interface utilities & characteristics 20 Bytes link-layer address link-layer (MAC) address is 6 Bytes DHCP requires using dhcp-client identifier Host administrator must be aware of PKEYs While interfaces support VLANs vconfig command cannot be used 4

5 Limitations cont. Link Layer setting, modification, migration Link Layer address is based on QPN/GID Cannot be controlled by the user Host administrator cannot set or re-configure the link layer address (MAC) Link Layer (RFC 4391) 5

6 Limitations cont. cannot be used in fully-virtualized or para-virtualized environments Hypervisor networking model normally use bridged mode Virtual Switch (vswitch) is L2 switch NIC cannot be enslaved to a vswitch Promiscuous mode is unsupported Simplifies vswitch functionality 6

7 & Network Virtualization Hypervisor VM sees an Network Device (e1000, virtio, vmxnet, etc..) Virtual Interface(s) (vifx) Packet sent out by the VM includes an header Virtual Switch(s) (vswitchx) Uplink(s) (pifx) vswitch0 pif0 vswitch1 pif1 vswitch forwards the packet to the host Network Interface Network Virtualization Model Native driver serves only packets (doesn t expect ) Reference 7

8 Solution Create a new Network Device over interface Managed by e kernel module TCP/ Registers a standard Interface into the Operating System Push Same Look & Feel as an NIC ifconfig, vconfig, ethtool, promiscuous, etc.. Strip e Shim Layer Strip e is a shim layer driver between the TCP/ stack and the native Push IB layer 8

9 e vs. Interface ifconfig ib0 9

10 e vs. Interface ifconfig eth3 10

11 e Operations in a Nutshell Initialization: Register network interface (eth0) Map it to native interface (ib0) Transmit: Packet to be sent has e strips the Pass the packet to the underlying interface. Non /ARP/RARP packets are dropped Native interface sends the packet out Receive: Packet received is an packet Native handovers the packet to e layer e pushes the header Packet forwarded to upper layers as regular frame Strip Push TCP/ e Shim Layer Push Strip IB layer 11

12 e Flow Data TCP/UDP TCP/ Data Data TCP/UDP e Shim Layer Data Strip InfiniBand Data CRC Push IB layer 12

13 e/ Interoperability Same wire protocol e talks TCP/ Push TCP/ e Shim Layer Strip Strip IB layer InfiniBand Data CRC Push IB layer 13

14 e Networking Model (KVM) eth0 ib0 Reference

15 e Design (KVM) vm1 vm2 vm.x vif0.1 vif0.2 vif0.3 vif0.x QEMU TCP/ tap0.1 tap0.2 tap0.3 tap0.x br0 eth0 CMA ib0.0 ib0.1 ib0.2 ib0.3 ib0.x ib0 IB port

16 e Closer Look Promiscuous mode MAC Translation 16

17 Promiscuous Mode Normally vswitch uplink is put in promiscuous mode InfiniBand doesn t support promiscuous mode Promiscuous mode is simulated by: a. Snooping the src.mac and vlan of outgoing packets b. OS notifies the driver when a new MAC/VLAN need to be served. For example: the driver can get a notification from libvirt library (available for KVM/XEN) when a new VM virtual NIC is created Multicast promiscuous support requires IGMP/MLD Snooping in e level 17

18 MAC Translation Requirements: e exposes an MAC Local MAC (LEMAC) 6 bytes length e neighbors are seen as neighbors Remote MAC (REMAC) 6 bytes length Local MAC used on the wire Local MAC (LIMAC) 20 bytes neighbors MAC used on the wire Remote MAC (RIMAC) 20 bytes 18

19 MAC Translation cont. Receive Flow Receiver QP => dst.mac SQPN/SLID => src.mac Remember QPN/LID to -MAC (QPN/GID) mapping header => ethertype Replace header by header Transmit Flow src.mac => QP (child interface) Source MAC normally controlled by the host/hypervisor admin dst.mac => -MAC header ethertype => Strip header, and handover packet to ARP/NDP packets are modified in TX/RX flow, so the MAC addresses in the packet payload are updated accordingly 19

20 Questions? 20