Санкт-Петербургский Государственный Политехнический Университет Факультет Технической Кибернетики Кафедра Информационных Измерительных Технологий

Transcription

1 Санкт-Петербургский Государственный Политехнический Университет Факультет Технической Кибернетики Кафедра Информационных Измерительных Технологий Доклад по HPC Тема: Interconnection technologies of Cluster Networks Выполнил: Ван Юйцзюань Гр. 6085/1 Проверил: Солнушкин К. С. Санкт-Петербург

2 Chapter 1. Cluster Networks Concept Commodity cluster solutions are viable today due to a number of factors such as the high performance commodity servers and the availability of high speed, low-latency network switch technologies that provide the inter-nodal communications. Commodity clusters typically incorporate one or more dedicated switches to support communication between the cluster nodes. The speed and type of node interconnects vary based on the requirements of the application and organization. With today s low costs per-port for Gigabit Ethernet switches, adoption of 10-Gigabit Ethernet and the standardization of 10/100/1000 network interfaces on the node hardware, Ethernet continues to be a leading interconnect technology for many clusters. In addition to Ethernet, alternative network or interconnect technologies include Myrinet, Quadrics, and Infiniband that support bandwidths above 1Gbps and end-to-end message latencies below 10 microseconds (usec). 1.1 Network Characterization There are two primary characteristics establishing the operational properties of a network: Bandwidth -- is measured in millions of bits per second (Mbps) and/or billions of bits per-second (Gbps). Peak bandwidth is the maximum amount of data that can be transferred in a single unit of time through a single connection. Bi-section bandwidth is the total peak bandwidth that can be passed across a single switch. Latency -- is measured in microseconds (usec) or milliseconds (msec) and is the time it takes to move a single packet of information in one port and out of another. For parallel clusters, latency is measured as the time it takes for a message to be passed from one processor to another that includes the latency of the interconnecting switch or switches. The actual latencies observed will vary widely even on a single switch depending on characteristics such as packet size, switch architecture (centralized versus distributed), queuing, buffer depths and allocations, and protocol processing at the nodes. Tab.1 Examples of some interconnection technologies 2

3 Table.1. gives a summary of some interconnection technologies whick are then compared as shown in table.2. The comparisons examine factors that include: bandwidth, latency, hardware availability, suppot for Linux, muximum number of custer nodes supported, how the protocol is implemented, support for Virtual Interface Architecture(VIA), and support for Message Passing Interface(MPI). Comparison Criteria Gigabit Ethernet 10G Ethernet Infiniban d 4X Interface PCI PCI PCIe PCIe Myrinet QsNet-II SCI PCI-X 1.0/PCIe 1.0a Peak Bandwidth (MB/s) < <320 MPI Latency (us) < <7 10 <3 1-2 Hardware Availability Now Now Now Now Now Now Linux Support Now Now Now Now Now Now Max. No. of Nodes 1000 s 1000 s >1000 s 1000 s s Protocol Firmware on Firmware on Firmware Hardware Hardware Hardware Implementation adaptor adaptor on adaptor VIA Support NT/Linux Software Linux None Software MVICH over MPI Support MPI/Pro 3 rd Party Quadrics 3 rd Party M-VIA, TCP Tab.2. Comparison of some interconnection technologies ( Peak bandwidth of 10G Ethernet got from MPI latency of 10G Ethernet got from PCI Chapter 2. Classify of Interconnection of Cluster Networks 2.1 Gigabit Ethernet and 10 Gigabit Ethernet Gigabit Ethernet (GbE or 1 GigE) Gigabit Ethernet is a term describing various technologies for transmitting Ethernet frames at a rate of a gigabit per second, as defined by the IEEE standard. Half duplex gigabit links connected through hubs are allowed by the specification but in the marketplace full duplex with switches is the norm. Network topolopy: Star. Speed: 1000 Mbit/s 10 Gigabit Ethernet Here, we will discuss the performance of ther first Quadrics 10 Gb/s Ethernet (10 GbE) switch with Myricom Myri-10G adapters, also using socket and MPI microbenchmarks. In order to evaluate the 3

4 performance of MPI, we used one of the most well-known sets of micro-benchmarks: the Ohio State University (OSU) MPI Benchmarks. During all testing, we can get the conclusions: the network while using sockets is able to deliver a UDP bandwidth up to 9.92 Mb/s and TCP bandwidth of 9.89 Gb/s, that is full available bandwidth utilization. In terms of latency, TCP latency goes as low as 9.89 us on top of which a switching latency as low as 470 ns on the shortest switch path and up to 1.6 us on the longest has to be added. Using MPI only slightly decreases bandwidth down to 9.61 Gb/s and 8.93 Gb/s for 128 KB messages, whilst latency is degraded to 12 us. The switch adds the same extra low latency, between 470 us and 1.6 us. The data can be got from Here, I want to introduce an Ethernet switch QS-TG108. It is an 8U chassis providing up to Gb/s Ethernet ports, and utilises the new low cost 10 Gb/s Ethernet specification 10Gbase-CX4. This manufacture is qualified for use with 10 Gb/s Ethernet adapters from Intel, Chelsio, Myricom and Neteffect. 10 Gigabit Ethernet can support many standards, which are used to transmit information to different distance. See their features in table. ( Nomenclaturee Distance Media Note 10GBASE-CX4 15 m Copper Used for InfiniBand 4x adapter and CX4 cable 10GBASE-SR 26 82m Multimode Fiber 300m Multimode Fiber (2GHz) GBASE-LX m Multimode Fiber 10km Singlemode Fiber GBASE-LR/ER 10km/40km Singlemode Fiber GBASE-SW/LW/EW Same as 10GBASE- Fiber SR/ LR/ER GBASE-T 55 m Category 6 cabling 100 m Category 6 or 6a Tab.3. 10G Ethernet suit standards 2.2 Scalable Coherent Interface(SCI) ( Scalable Coherent Interface, is the oldest of the networks discussed here. SCI was largely designed to avoid the usual bus limitations which resulted in a ring structure to connect the hosts. This is because of one of the design principles of SCI: signals flow continuously in one direction enabling a very high signal rate without noise interference. SCI nodes are interconnected through unidirectional point-to-point links in a ring/ringlet topology. Switches are used to connect multiple independent SCI ringlets. A special feature of SCI is its ability keep the caches of the processors it connects to coherent. In clusters SCI is always used as just an internode network. Because of the ring structure of SCI this means that the network is arranged as a 1-D, 2-D, or 3-D torus as shown in Figure 4

5 Figure.1. Topology of SCI ( The torus network has some risks in comparison to other networks like the fat tree or the Clos network: when a node adaptor fails it incapacitates all the nodes that share the ring on which it lies. So, for instance, Dolphin networks, one of the SCI vendors provides software to reconfigure the torus in such a way that the minimal number of nodes become unavailable. Furthermore, it is not possible to add or remove an arbitrary number of nodes in the cluster because of the torus topology. Performances of the SCI-based clusters: Network topolopy: 1-D, 2-D, or 3-D torus Bandwidth: up to about 320 MB/s for a Ping-Pong experiment MPI latency: 1 2 µs for small messages. 2.3 Myrinet Nowadays, the interconnect technology that applies most extensive is -- Myrinet. At present, Myricom supplies Myrinet components and software in two series: Myrinet-2000 and Myri-10G. Myrinet-2000 is a superior alternative to Gigabit Ethernet for clusters, whereas Myri-10G offers performance and cost advantages over 10-Gigabit Ethernet. Myri-10G uses the same physical layers (PHYs: cables, connectors, signaling) as 10- Gigabit Ethernet, and is highly interoperable with 10-Gigabit Ethernet. In fact, Myri-10G NICs are both 10- Gigabit Myrinet NICs and 10-Gigabit Ethernet NICs. Performances of Myrinet: can suit any topology. compare the bandwidth, latency and other features of the two series: 5

6 Tab.4. Performance Comparing of the series Myrinet-2000 and Myri-10G ( Prices of Myrinet NICs (data from For Myrinet-2000, the prices of NICs can be found out in figure. Tab.5. Prices of NICs for Myrinet-2000 And the prices of NICs for Myri-10G can be found out in next tables: 6

7 1. For 10-Gigabit Ethernet solutions. Tab.6. Prices of NICs for Myri-10G by 10-Gigabit Ethernet solutions 2. For 10-Gigabit Myrinet and HPC solutions. Tab.7. Prices of NICs for Myri-10G by 10-Gigabit Myrinet and HPC solutions Prices of Myrinet Switches (data from 1. For Myrinet-2000 products Small switches Tab.8. Small Myrinet-2000 Switches 7

8 Intermediate-size switch networks Tab.9. Intermediate-size switch networks Switch networks for large clusters Tab.10. Switch networks for large clusters Clos256: 256 host ports, no inter-switch ports. Clos : 256 host ports, 256 interswitch ports on 64 quad-fiber ports. Spine1280: 10 quad (40) 32-port switches presented on 320 quad-fiber ports (1280 ports total). This configuration may be used as the spine for 5 Clos units. 2. For Myri-10G products 8

9 Tab.11. Myri-10G Switches Myrinet-2000 Hardware Latency ( 2000/) The latency measurements presented on these pages were, unless otherwise noted, between NICs whose ports are connected through short cables through a single switch. The measured latency includes ~0.5µs total hardware latency in the circuitry between the ports of the Lanai-X chips in the two NICs. This hardware latency is introduced by circuitry that converts to and from the Fiber Physical layer, and by the switch itself. Networks for more than 512 and up to 8192 hosts with XBar32- based switches have hardware latencies up to ~1.1µs, but the hardware latency is still small compared with the software latency. The "time of flight" latency of a fiber cable is ~0.0065µs per meter, i.e., ~1.3µs for a maximal-length, 200m fiber cable, but only 0.03µs for a 5m cable. 2.4 Quadrics Quadrics is a supercomputer company formed in 1996 as a joint venture between Alenia Spazio and the technical team from Meiko Scientific. They produce hardware and software for clustering commodity computer systems into massively parallel systems. The products of Quadrics Hardware: (from QsNet I - Quadrics interconnect based around the elan3/elite3 ASICs 5us MPI latency) QsNet II - Quadrics interconnect based around the elan4/elite4 ASICs (912MB/s on SR1400 EM64T and 1.26us MPI latency on HP DL145G2) QsNet II E-Series- a range of small-medium configurations (8-128-way) at less than USD 1,800 price/per port QsTenG - 10 Gigabit Ethernet switch for up to 96 ports. The newest performance data that the company supply is shown in table. ( 9

10 MPI Latency Bandwidth Peak Bandwidth Interface QsNet-I 5us 350MB/s 528MB/s PCI 2.1 QsNet-II 1.22 us 912MB/s 1064MB/s PCI-X 1.0 Tab.12. Features comparison of QsNet Nowadays, the most popular product is QsNet-II, so we will describe its performance in details. QsNet- II uses the fat-tree topology. It is possible that connecting most 4096 nodes using QsNet-II (connecting most 1024 nodes with QsNet-I). The components of QsNet-II network are Elan4 network card and Elite4 exchange machine. Basic performance ( The bandwidth of QsNet-II was measured in two ways: a simple ping test where two nodes communicate between each other in turn; a batched test where a number of MPI sends and receives are queued up together. The results are shown in following picture. Figure.2. QsNet-II MPI Bandwidth Figure.3. QsNet-II MPI Latency 10

11 QsNet-II could make two systems: ( 7) small system(up to 8, 32 or 128 nodes) using a single switch. larger system( nodes) using a federated network consisting of multiple switch modules. QsNet-II uses a fat tree topology, this permits scaling up to 4096 nodes. QsNet-II hardware is just one part of a complete family of products for building high performance clusters. The basic building block of QsNet-II switch networks is the QS5A switch chassis. A single chassis can be configured to provide up to 64 prots of switching implemented as a 3 stage fat tree networks. For switches of greater than 64 ports, multiple switch chassis are sed in a federated network. Federated switching is a packaging solution, which enables very large networks to be imlemented with two stages of switch chassis. For instance, configuration of a 256-way system with 4 node levels switches each porting 64 nodes and 2 top level switches (full bandwidth) is shown at figure.4.( 0053D9DA) Figure.4. Configuration of 256-way system In this figure, number 32 means a 32 way standalone switch, which can provide double the bandwidth for the high CPU count SMP nodes. And other configurations are listed in tab. Tab.13. Function to configurate different system In following figures showed the adapters and switches used for QsNet-II technology: 11

12 Fig.5. QsNet-II adapters Fig.6. QsNet-II swiches 2.5 Infiniband InfiniBand is a technology standard which components include IBM, Intel, Sun, Voltaire, Mellanox, Topspin, etc. It is a switched fabric communications link primarily used in high-performance computing. Infiniband uses a switched fabric topology, as opposed to a hierarchical switched network like Ethernet. 12

13 The unidirectional MPI bandwidth may approach to 860Mb/s, and the lowest MPI latency is 4.5us (evaluated by Mellanox company, using network card with PCI-X interface. The latency will increase to 5.9us if it is applied on MPI application layer.) Like Fibre Channel, PCI Express, Serial ATA, and many other modern interconnects, InfiniBand is a point-to-point bidirectional serial link intended for the connection of processors with high speed peripherals such as disks. It supports several signalling rates and, as with PCI Express, links can be bonded together for additional bandwidth. Signalling rate ( The serial connection's signalling rate is 2.5 gigabit per second (Gbit/s) in each direction per connection. InfiniBand supports double (DDR) and quad data (QDR) speeds, for 5 Gbit/s or 10 Gbit/s respectively, at the same data-clock rate. Links use 8B/10B encoding every 10 bits sent carry 8bits of data so that the useful data transmission rate is four-fifths the raw rate. Thus single, double, and quad data rates carry 2, 4, or 8 Gbit/s respectively. Links can be aggregated in units of 4 or 12, called 4X or 12X. A quad-rate 12X link therefore carries 120 Gbit/s raw, or 96 Gbit/s of useful data. Most systems today use either a 4X 2.5Gb/s (SDR) or 5Gb/s (DDR) connection. InfiniBand QDR was already demonstrated during 2007, with expectations of productions systems during Larger systems with 12x links are typically used for cluster and supercomputer interconnects and for interswitch connections. Effective theoretical throughput in different configurations ( Single Double Quad 1X 2 Gbit/s 4 Gbit/s 8 Gbit/s 4X 8 Gbit/s 16 Gbit/s 32 Gbit/s 12X 24 Gbit/s 48 Gbit/s 96 Gbit/s Tab.14. Effective theoretical throughput in different configurations Latency ( The single data rate switch chips have a latency of 200 nanoseconds, and DDR switch chips have a latency of 140 nanoseconds. However, due to the larger effect of the end-points, the total message delivery latency is much larger, from 1.26 microseconds MPI latency (Mellanox ConnectX HCAs) to 1.29 microseconds MPI latency (Qlogic InfiniPath HTX HCAs) to 2.6 microseconds (Mellanox InfiniHost III HCAs). Nowadays, the products of Mellanox company are usually used. Mellanox offers a dual-port 10Gb/s PCI-X card, and dual-port and single-port 20Gb/s PCI Express cards designed to drive the full performance of high-speed InfiniBand fabrics: ConnectX IB Adapter Cards ConnectX EN Adapter Cards InfiniHost Adapter Cards InfiniHost III Ex Adapter Cards InfiniHost III Lx Adapter Cards Their detailed data could be found at website: 13

14 ( Infiniband switches for HPC are normally offered with ports and presently mostly at a speed of 1.25 GB/s. The switches can be configured in any desired topology but in practice a fat tree topology is almost always preferred. Now we introduce some switches in detail: 1. Flextronics 288-port 4X InfiniBand DDR Switch ( Fig.7. Features of Flextronics 288-port 4X InfiniBand DDR Switch 2. Switches Mellanox InfiniScale - Eight- Port 10 Gb/s 4X InfiniBand Switch Mellanox InfiniScale III 24- port 4X (or 8-port 12X) InfiniBand Switch supporting up to 60Gb/s per port KEY FEATURES Eight 10Gb/s InfiniBand 4X Ports 160Gb/s of Available Bandwidth Scalable to 32 Non-Blocking Full Wire Speed 1X Ports Scalable to 192 Ports 1X (2.5Gb/s) Supports Multi-Protocol Applications for Clustering, Communication, and Storage Integrated SMA and GSA Inbound and Outbound Partition (P_KEY) Checking 32 Integrated 2.5Gb/s SerDes Cut-Through Switching InfiniPCI Technology Support Programmable Port Mirroring Large On Chip Port Buffering Twenty-four 10 or 20Gb/s InfiniBand 4X ports or eight 30 or 60Gb/s InfiniBand 12X ports (or any combination) 480Gb/s (SDR version) or 960Gb/s (DDR version) of total switching bandwidth Scalable to thousands of Ports 96 integrated 2.5Gb/s (SDR version) or 5Gb/s (DDR version) SerDes interfaces (physical layer) Auto-negotiation of port link speed Ultra low latency cut-through switching (less than 200 nanoseconds) MTU Size from 256 to 2K bytes 14

15 Mellanox InfiniScale IV - Fourth Generation InfiniBand Switch Supports Multi-Protocol Applications for Clustering, Communication, and Storage Integrated Subnet Management Agent (SMA) 36 40Gb/s InfiniBand 4X ports 12 80Gb/s InfiniBand 8X ports Gb/s InfiniBand 12X ports Port speed auto-negotiation Line rate switching 2.88 Tb/s total switching capacity 144 integrated SerDes interfaces, each operate up to 10+10Gb/s Tab.15. Features of switches made with Mellanox company 2.6 Marketable Distribution of several cluster networks According the report of Top500 in 2007, the development of infiniband and gigabit ethernet are very fast, and there is a downtrend of myrinet. We also can see the distributed instance: Figure.8. Interconnect Family share for 11/2007 Conclusions How can we evaluate if the interconnection of cluster networks is good or not? There are some primary factors need to be considered: Latency, Bandwidth, Price, and Supports. Generally speaking, price is direct ratio with bandwidth, and inverse ratio with latency. For saving money, we need to configurate interconnection according different application. In a word, the highest bandwidth is Infiniband, the lowest latency is SCI, but SCI and QsNet have the expensive prices, followed by Infiniband, Myrinet, Gigabit Ethernet. The all interconnection support MPI in function, higher efficienct communication protocols had been realized except for Gigabit Ethernet. SCI and QsNet can support sharing memory, but the latency of accessing remote nodes is still on level millisecond. Seeing the developing trend, Infiniband will be the primary interconnection equipment because of its scaling benefits. 15

16 Список литературы [1] Cluster Computing: High-Performance, High-Availability, and High-Throughput Processing on a Network of computers [2] QsNet-II Performance results [3] Quadrics QsTenG Ethernet Switch Performance Report [4] [5] [6] [7] /3A912204F DD C7 [8] /3A912204F DD C7 [9] /2B3D9C399FA9BED D9DA [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] 16