ETHOS A Generic Ethernet over Sockets Driver for Linux

Transcription

1 ETHOS A Generic Ethernet over Driver for Linux Parallel and Distributed Computing and Systems Rainer Finocchiaro Tuesday November CHAIR FOR OPERATING SYSTEMS

2 Outline Motivation Architecture of ETHOS Experimental Results Conclusions and Outlook ETHOS, A Generic Ethernet over Driver for Linux 2 Chair for Operating Systems

3 Motivation Software Kerrighed Kernel-level cluster integration user similar to Mosix, OpenSSI Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet other ETH interface Ethernet NIC driver Ethernet NIC ETHOS, A Generic Ethernet over Driver for Linux 3 Chair for Operating Systems

4 Motivation Software Kerrighed Kernel-level cluster integration similar to Mosix, OpenSSI network layer used by Kerrighed Ethernet (only) new or ETH interface user Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet other ETH interface Ethernet NIC driver Ethernet NIC ETHOS, A Generic Ethernet over Driver for Linux 3 Chair for Operating Systems

5 Motivation Software Kerrighed Kernel-level cluster integration similar to Mosix, OpenSSI network layer used by Kerrighed Ethernet (only) new or ETH interface ETH interface user Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet other ETH interface Ethernet NIC driver Ethernet NIC ETHOS, A Generic Ethernet over Driver for Linux 3 Chair for Operating Systems

6 Motivation Hardware user Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet ETH int. SCI DX other IRM -specific lower DX layers SA/CM Verbs IB ETHOS, A Generic Ethernet over Driver for Linux 4 Chair for Operating Systems

7 Motivation Hardware We need -space UDP sockets! user Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet ETH int. SCI DX other IRM -specific lower DX layers SA/CM Verbs IB ETHOS, A Generic Ethernet over Driver for Linux 4 Chair for Operating Systems

8 Motivation Hardware We need -space UDP sockets! SCI (Scalable Coherent Interface) UDP sockets no IP device Dolphin DX (switched PCIe) UDP sockets IP device in development InfiniBand only TCP sockets IP device user Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet ETH int. SCI DX other IRM -specific lower DX layers SA/CM Verbs IB ETHOS, A Generic Ethernet over Driver for Linux 4 Chair for Operating Systems

9 Motivation Hardware We need -space UDP sockets! SCI (Scalable Coherent Interface) UDP sockets no IP device Dolphin DX (switched PCIe) UDP sockets IP device in development InfiniBand only TCP sockets IP device sockets = least common denominator user Applications Higher Kerrighed Layers Kerrighed Comm-Layer Ethernet ETH int. SCI DX other IRM -specific lower DX layers SA/CM Verbs IB ETHOS, A Generic Ethernet over Driver for Linux 4 Chair for Operating Systems

10 Architecture of ETHOS Linux Ethernet Networking stack Application user Standard Other Standard native/ Other ETH interface Ethernet NIC ETHOS, A Generic Ethernet over Driver for Linux 5 Chair for Operating Systems

11 Architecture of ETHOS Linux Ethernet Networking stack Application Application user Standard Other user Standard Other Standard native/ Other ETH interface Standard native/ Other ETH interface (ETHOS) Ethernet NIC SCI IPoIB Standard ETH Other SCI NIC Infiniband NIC Ethernet NIC Other NIC ETHOS, A Generic Ethernet over Driver for Linux 6 Chair for Operating Systems

12 Architecture of ETHOS Implementation of ETHOS two sockets per interface lower IP-address reach peers upper IP-address ifconfig NIC could be IB, SCI,... user node... n node 1 node 2 Application eth2: ethif (ETHOS) Send Socket NIC Receive Socket Application eth2: ethif (ETHOS) Send Socket NIC Receive Socket ETHOS, A Generic Ethernet over Driver for Linux 7 Chair for Operating Systems

13 Experimental Results Latency and bandwidth measurements TCP Performance NPtcp Performance tipcbench Focus on latency! ETHOS, A Generic Ethernet over Driver for Linux 8 Chair for Operating Systems

14 Experimental Results Latency (NPtcp) Round-Trip Latency (RTT/2) measured with NPtcp Round-Trip/2 [us] IPoIB ETHOS on IPoIB ETHOS on SCI Gigabit Ethernet ETHOS on Gigabit Ethernet Message size [Byte] ETHOS, A Generic Ethernet over Driver for Linux 9 Chair for Operating Systems

15 Experimental Results Bandwidth (NPtcp) Bandwidth measured with NPtcp Throughput [Mbit/s] IPoIB 0.25 ETHOS on IPoIB ETHOS on SCI Gigabit Ethernet ETHOS on Gigabit Ethernet Message size [Byte] ETHOS, A Generic Ethernet over Driver for Linux 10 Chair for Operating Systems

16 Experimental Results Latency (tipcbench) ETHOS on IPoIB ETHOS on SCI Gigabit Ethernet Latency measured with tipcbench Round-Trip/2 [us] Message size [Byte] ETHOS, A Generic Ethernet over Driver for Linux 11 Chair for Operating Systems

17 Experimental Results Bandw. (tipcbench) 2048 Bandwidth measured with tipcbench Throughput per direction [Mibit/s] ETHOS on IPoIB ETHOS on SCI Gigabit Ethernet Message size [Byte] ETHOS, A Generic Ethernet over Driver for Linux 12 Chair for Operating Systems

18 Conclusions and Outlook Ethernet device for interconnects that offer sockets over high-speed interconnects First Eth/IP-device for SCI Performance very promising Next steps: Find out about bandwidth issues with /tipcbench Measure and test other interconnects (e.g. Dolphin DX) Reduce latency: use lower SW layers of interconnects ETHOS, A Generic Ethernet over Driver for Linux 13 Chair for Operating Systems

19 Conclusions and Outlook Ethernet device for interconnects that offer sockets over high-speed interconnects First Eth/IP-device for SCI Performance very promising Next steps: Find out about bandwidth issues with /tipcbench Measure and test other interconnects (e.g. Dolphin DX) Reduce latency: use lower SW layers of interconnects ETHOS, A Generic Ethernet over Driver for Linux 13 Chair for Operating Systems

20 Thank you! Questions? ETHOS, A Generic Ethernet over Driver for Linux 14 Chair for Operating Systems