Performance monitoring in InfiniBand networks

Transcription

1 Performance monitoring in InfiniBand networks Sjur T. Fredriksen Department of Informatics University of Oslo May 2016 Abstract InfiniBand has quickly emerged to be the most popular interconnect used in high performance computing and enterprise data centers. Deployment of InfiniBand networks has grown in terms of size and scale, and it is becoming a challenging task to predict the behavior of these networks. Monitoring the performance of an InfiniBand network is very important for the behavior prediction. There exists some tools that can visualize link usage, but many has efficiency issues. In this essay I will present the InfiniBand Architecture, take a look at current tools and propose a way of utilizing parts of the specification to efficiently monitor the performance of a fabric. 1

2 1 Introduction Development of modern CPUs has since 1970 followed Moore s law[1], a law that dictates that the number of transistors in a microprocessor should double approximately every two years. Since 2007 the Intel x86_64 architecture has been the dominant CPU architecture in the list of the top 500 supercomputers in the world[2]. Intel x86_64 has over the years become very affordable compared to older architectures. Because of this evolution in processing capacity and the lower cost of developing and building CPUs[3], the number of supercomputers in the world has grown tremendously. Many of these supercomputers are clusters with a vast number of smaller computing nodes interconnected. The largest supercomputer today is named Tianhe-2 and is located in China. Performance of supercomputers are measured in the number of floating-point operations per second (FLOPS) they are able to compute. Tianhe-2 can do Peta Flop/s[4]. If we look back only ten years ago the leading supercomputer could do Terra flop/s. This corresponds to more than a 100 times increase in computing power over the last decade[2]. The amount of computing power needed by researchers and enterprises in the world is expected to grow in the future, as problems become more complex. Even though modern CPUs are getting new routines and both CPU speed and memory capacity are increasing every year it still is physical limitations on how powerful one computing node can be. To gain the extra needed computing power the best way is to add more computing nodes to a cluster. This growth puts high pressure on the infrastructure around the computing nodes to utilize the computing power efficiently. Interconnecting the computing nodes has become a central part of the supercomputer as the number of individual computing nodes still increases. In these distributed environments factors such as high bandwidth and low latency is the most important requirements in the communication equipment. In the most known communication network - the Internet, the TCP/IP stack are used on top of variating layer two networks, but Ethernet has long been the default standard in local area networks. Ethernet was in 2005 the dominating interconnect standard in supercomputers listed in the top500 list[5], and widely in use the years before. Due to the high latency[6] of a transmission signal between two nodes in Ethernet a new standard with lower latency was highly needed. Work on a new standard started already in the late 90s, and in 2000 the first version of the InfiniBand (IB) standard was released. The IB architecture 2

3 has emerged to become the most used interconnect standard in High Performance Computing (HPC). Of the top500 supercomputers in the world, IB is today used in 47.4% of the systems[4]. IB has also started to take marked share in enterprise computing and storage systems[7]. IB offers much lower latency compared to Ethernet[6] and supports Remote Direct Memory Access (RDMA). RDMA permits memory access in a remote computer over the network whiteout using the operating system (OS) of the hosts. By directing the signal outside of the OS buffers and processing, and by not using shared buses, it ensures low latency and high bandwidth which is very much needed in these environments. To support larger clusters the IB topologies are becoming bigger and more complex. Building and operating these networks are thus more complicated and detecting problems within the network is becoming more important to ensure that the cluster utilization is optimal. One of the challenges with this is to monitor the performance of the network without wasting CPU cycles or filling the network with overhead traffic. In this essay I will present an overview of the IB architecture and present a method for efficient performance monitoring of an IB cluster. 2 InfiniBand overview The InfiniBand Architecture (IBA)[8] is defined by the InfiniBand Trade Association (IBTA), a group of more than 220 companies, founded in 1999[9]. IBTA has the responsibility of maintaining and furthering the IB specification. The sole purpose behind the design of IBA was to build an new open industry standard that reduced the overhead found in the existing industrystandard I/O systems which uses shared buses. Overhead issues in those systems where related to copying, buffering, checksumming and interrupts. The IBA defines a switch-based I/O interconnect where computing nodes or servers are connected to the fabric using Host Channel Adapters (HCA). This topology is often referred to as a switched fabric. These adapters are used to connect processors and the I/O device of the computing nodes. The HCA is designed to allow direct application level communication and avoiding kernel calls to achieve lower latency operation. The IBA is like other communication systems a layered stack inspired by the OSI model[10]. 3

4 2.1 Subnet Manager The Subnet Manager (SM) is a key part of the IBA. It is the job of the SM to initialize, configure and manage all routers, switches and HCAs on the network. It must be at least one SM present on each subnet for the fabric to function. The SM continuously probes the fabric to get information about the connected IB devices, and is the receiver of traps that IB devices are sending when events occur. When some fault occur, it is the SM s job to reconfigure devices in the fabric so that operation can continue as normal. There can be multiple SMs present on a subnet as long as just one is active at a time. Standby SMs present on the fabric keeps a copy of the state of the active SM. Standby SMs are verifying that they have a valid copy, and that the active one still is active with short intervals. If an active SM fails, a standby SM would take over the responsibilities to ensure that the subnet does not go down together with the SM. The SM communicates with all IB devices using a separate virtual lane on the physical links. All IB devices must have a Subnet Manager Agent (SMA) entity present. The SMA works like a daemon on the switch or HCA and it should apply the received configuration on the local IB device. The SMA is responsible for the communication with the SM, it receives and transmits packets called Subnet Management Packets, which is a sub-class of Management Datagrams (MAD). The SMA should also transmit traps to the SM when IB related events occur on the device. The SM is also responsible for calculating all paths between all pairs of source and destination nodes and distribute the calculated forwarding tables to all involved devices. The IB specification does not force any particular algorithm for this, it is left for the network administrator to decide what to implement. Many different algorithms are available to choose from. 2.2 Physical layer At the physical layer it is specified how bits are placed on wires and fiber optic cables. It defines symbols for the bit stream and delimiters between frames from the link layer. These symbols are converted to a signal that can be transmitted over the medium. The IBA supports three different link width, 1x, 4x and 12x. Each link width needs a pair of wires / fibers for each direction to support full duplex communication. Thus a 4x link needs four different pairs each direction, eight in total. 4

5 Together with the different link width there are multiple options in link speeds that can be used to establish a link. As of today the IBA supports five different link speeds, all can be used together with all the link widths. Single Data Rate (SDR), Double Data Rate (DDR), Quad Data Rate (QDR), Fourteen Data Rate (FDR-10/FDR-14) and Enhanced Data Rate (EDR). EDR has a signaling rate of 25 Gbit/s and gives a theoretical throughput of Gbit/s on 1x link width. EDR is the link speed with the highest data rate available today. A 12x EDR link would give 300 Gbit/s of link rate. The IBTA are currently working on the next link speed called High Data Rate (HDR) which are planned to be released in 2017 and aims for a signaling rate of 50Gbit/s. 2.3 Link layer Many of the IBA core functions are located at the link layer. The link layer handles all point-to-point link operations including switching within a subnet. IBA has support for two different types of packets; MADs and data packets. Management packets are for example used by the SM to send configuration to connected IB devices. Data packets are used by computing nodes to transport data. The supported maximum transmission unit is 4k bytes. Within a subnet, switching is handled at the link layer. All devices within the subnet has a 16 bit Local ID (LID) assigned to it by the SM. All IB devices inside a subnet use this LID for addressing when packets are transmitted. The LID is stored inside a Local Route Header (LRH), and IB switches are forwarding packets based on the LID field Quality of Service An important feature at the link layer is Quality of Service (QoS). To achieve QoS in IB networks each physical link in the network has multiple Virtual Lanes (VL) with differentiated transmit priority. VLs are logically separated channels on a link using their own set of dedicated transmit and receive (tx/rx) buffers for each port. VLs are also implemented with individual flow-control mechanisms. The IB specification allows for a total of 16 different VLs. VL0-14 are used to carry application traffic and VL15 are used exclusively for management traffic and has no flow-control. The management VL has the highest priority of all the lanes. 5

6 IBA provides a four bit header field in the LRH to mark packets with their QoS level. This is normally described as the Service Level (SL) and it is also the name of the header field. This field may be arbitrarily used to indicate a class of service. The mapping between the SL and forwarding behavior is not defined by the IBA, but left to be set as a policy by the network administrator. In addition to the SL field, the LRH also has a VL field which indicates the VL number from which VL the packet was transmitted on. When a switch receives a packet the VL field is checked, and the packet is placed on the corresponding VLs rx buffer. All switches in the fabric has a SLtoVL mapping table. By looking up in this table a switch will know which VL to forward the packet on. The switch uses the SL field, the port it was received on, and the port it should be forwarded on to determine the VL to transmit the packet on. The process a switch uses on an output port to find which VL to transmit from is called VL arbitration. IBA has specified a dual priority weighed round robin scheme for this. As mentioned before, each VL has a different transmitting priority. Packets from the high priority VLs are always transmitted before packets on lower priority ones. The VL arbitration is specified using a VL arbitration table on each IB port where each list entry contains a VL number and a weighting value. The weighting value specifies the number of 64-byte units that can be sent from that VL before moving to the next VL. 2.4 Network layer The IB network layer handles routing of packets between different IB subnets. It uses IPv6 as known from the Internet as the address schema. Addresses are 128 bits long and are stored in the Global Route Header (GRH) of network packets. It should be noted that the Network layer is not required to operate within one subnet which is the likely scenario for a IB network. When the network layer is not in use, the network layer header can be dropped, this ensures that overhead traffic is kept as low as possible. 2.5 Transport layer In IB networks the transport layer is responsible for in-order packet delivery, partitioning, channel multiplexing and transport services. IBA uses a trans- 6

7 port header on all packets which contains information required by the end node to handle the incoming packets and deliver it to the correct application. Applications running on a computing node are communicating with the transport layer using work queues for receiving and transmitting operations. These queues are referred to as the Queue Pair (QP), they can be seen as the IB consumer and worker interface to the fabric. In general the transmit queue holds instructions that cause the hardware to transfer data between the requesters memory, and another applications memory. The receive queue is holding information about where to store received data. When a QP is created, it must be associated with one of the five transport protocols defined by the IBA. IBA supports five different transport protocols; Unreliable Datagram (UD), Reliable Datagram (RD), Unreliable Connected (UC), Reliable Connected (RC) and Raw Datagram. These different services gives different services for data reliability. To support a scalable reliable service between multiprocessor computing nodes, RD must be used. In RC the QP is keeping track of the reliability context for each communication channel. The reliability context is the various state information needed to provide reliable service, such as sequence numbers. In a fabric with N multi processor nodes with P processors, (N 1)P 2 QPs are needed to keep the context for all nodes. Since that does not scale, the developers of IB moved the reliability context out of the QP and established a separate entity called End-to-End context (EE context) - this is where RD differentiates from RC. The RD EE context solution is using P QPs plus N EE context per node. As part of both the transmitting and receiving instructions supported we find Remote Direct Memory Access (RDMA). IBA supports both reading and writing to another applications memory over the fabric. IBA RDMA is zerocopy, which means that the reads/writes can be done without copying the data multiple times at either hosts before it can be transmitted or handled when received. IBA RDMA is done without interrupting the CPU of the remote host. This ensures low latency operations and minimizing the CPU time used on computing nodes for overhead computing. 2.6 Performance counters The IBA specifies a set of performance counters that must be present on all IB ports. These counters provide basic performance and exception statistics for all IB ports. When the system is initialized by a SM these counters are set 7

8 to zero on all devices. When they reach their maximum value they are defined to stop and not overflow. Writing zero to the counter will reset it, writing any other value has undefined behaviour. Some of the available counters are: LinkDownedCounter, PortRcvErrors, PortXmitDiscards, PortRcvData, PortXmitPkts, PortRcvPkts, PortXmitWait. Many of these counters has explanatory names, for example; PortRcvErrors counts the number of received packets that had an error. Normally this is caused by a CRC fail due to a bit error inside the packet. On modern hardware this counter is 64 bit long, on older hardware it where only 32 bit long. 3 Monitoring the fabric As IB subnets expand in size, scale and in adoption, the need for fabric monitoring and understanding of the behavior in these networks are becoming critical. Monitoring of the network should be as efficient as possible, not wasting computing power or network resources. For the monitoring tools it is important that they in a user friendly way can show the network administrator how well the network are performing and what faults that are occurring. It is essential to know when congestion emerge in the network. These tools should also help the network administrator plan for future expansion and upgrading of switches and links. 3.1 Current monitoring tools Some tools, both open and closed source are already proposed, developed and currently in use. Popular tools like Ganglia[11] and Nagios[12] are both open-source and has support for IB using plugins. These tools offer much of the same information in total, and for the IB level they are based on the same binaries to collect the performance counters. Ganglia and Nagios, with their respective plugins are using binaries from Open Fabrics Enterprise Distribution (OFED) to collect data from IB devices. OFED offers multiple utilities to read information from IB devices. One of the binarys that these plugins use are perfquery. When run, perfquery will report back the Xmt- Data, RecvData and XmtWait among other things. The tools that OFED offers and that Ganglia and Nagios use are restricted to these counters, and thus these tools has no knowledge of links or the topology of the fabric. To get further information these tools are daemon based which means that each monitored device in the IB cluster must run a software daemon on 8

9 every monitored node in the fabric to collect data about the device. The data collected by the daemon is transmitted to a central daemon and stored in a database. When the daemons are executing they steal computing time from other applications running on computing nodes. When the daemons transmits data they also generate extra traffic on the network which is transmitted together with the application traffic. The developers of these tools has tried to limit the overhead generated by not having the daemon sample data constantly, but rather in intervals. Longer time intervals between each sampling reduces overhead, but it also reduces the liveness of the monitoring tool. This monitoring solution also has another problem, the data collecting daemons cannot run on switches and routers where the network administrator are not allow to launch his own processes. Another available tool is FabricIT[13], a IB management solution developed by Mellanox. A performance monitoring software among many other features are built into their Subnet Manager called FabricIT. This software is not based on a host agent such as Ganglia or Nagios and thus the overhead issues from these tools are not present in this solution. FabricIT scales up to 648 nodes. One issue with this package is that it does not offer long term data storage of performance counters. The biggest issue with FabricIT is that it is a closed-source software package and Mellanox proprietary, and it will only run on Mellanox produced switches. In 2011, Nishanth Dandapantuhula wrote a thesis at The Ohio State University called InfiniBand Network Analysis and Monitoring using OpenSM[14]. He looked into how to build a low overhead monitoring tool for IB clusters that is capable of depicting the communication matrix of target applications and the link usage of various links in the IB network. He proposed a system with two distinct modules, the InfiniBand Network Querying Service (INQS) and the Web-based Visualization Interface (WVI). The INQS uses MADs to collect data from the performance counters of IB devices in the fabric and stores them in a MySQL database. The WVI presents the data to users using HighCharts JS[15]. 3.2 Getting the counters efficient To get the IB performance counters we propose a querying system that is using the OFED libraries for communication over the IB fabric. Libibmad provides low level IB functions that can be used for monitoring tools. Using 9

10 this library all traffic generated on the fabric related to performance monitoring will use VL15 and not mix with general application traffic and take up space in application traffic buffers. We could not find any research that has examined how well IB devices handles rapid querying of performance counters or querying of multiple counters in one query. We plan to examine how well the IB devices handles these requests and how the network is affected. We don t know how the hardware chips in IB devices will react to rapid querying of all ports or how optimized the hardware in switches are to respond to this types of messages. We need to perform testing of factors such as how often we can query devices, and how many counters at a time we can query. We also need to look at the scalability of the querying. The IBA has specified at least 320 bits total for all performance counters that must be present. The specification provides 32 bit for many of the counters, but today hardware producers have upgraded these to 64 bit due to the much higher bandwidth links we have today. If we where to query the 21 mandatory counters one-by-one very many small packets will be generated on VL15. To much traffic on VL15 could slow down application traffic as the tx and rx queues on VL15 has the highest priority. 4 Using the counters We plan to integrate the querying service with the Fabriscale SM[16]. The performance monitor will act as a plugin to the SM collecting performance data. Fabriscale aims to simplify fabric management; they aim to build a "smartsm" where most of the time the system administrator does not need to get involved. Both the monitoring tool and the SM is benefiting for this consolidation. One benefit is that the network administrator will have only one system to relate to for both management and monitoring of the subnet, making the job more convenient. By integrating the performance monitor into the SM, the monitoring tool can be topology aware and provide a full view of the network with performance statistics for all links and nodes. Fabriscale is based on OpenSM and will support all IB vendors in contrast to FabricIT. As mentioned above, the goal of the Fabriscale SM is to be "smartsm" - the SM could benefit from these performance metrics when calculating 10

11 routing decisions. The Fabriscale SM could be able to predict congestion and reroute traffic to avoid the congestion and potential retransmissions and latency increase in the fabric. In my masters thesis we will look closer at how machine learning on the data collected by the proposed performance monitor can benefit the SM. 5 Problem statement InfiniBand deployments has grown in terms of size and scale, and the task of predicting behavior in these networks is becoming a challenging task for the system administrators. Monitoring of these networks are essential to understand their behavior. Network administrators need to understand and predict the growth to plan for future expansion and to optimize flow in the network. Giving the SM access to performance statistics of links it can better calculate routing decisions. Some plugins for Ganglia and Nagios has been developed, but those tools are not efficient and uses CPU cycles in computing nodes. They do not give a full topology view and can not easily be integrate into a SM. It also exist a closed source tool only supporting one hardware vendor. It is unknown how well hardware tackle rapid querying of performance counters. A better monitoring software package that supports all InfiniBand hardware is necessary for networks to be driven efficiently. To address these problems we will answer the following research questions: RQ1: How well can IB devices handle rapid querying of performance counters, and how much bandwidth will such querying take up in the network? RQ2: Can a robust, scalable and efficient performance monitor be written so that all IB devices can be monitored without wasting CPU cycles and take up to much bandwidth in network? RQ3: Can we integrate such a monitor into the Fabriscale SM so that topology data from the SM can be used in the monitoring, and data from the monitor be used to calculate routing decisions? RQ4: To what extent can machine learning methodology be applied on the performance data collected, and what can it be used to? 11

12 References [1] Gordon E. Moore. Cramming more componets onto integrated circuits. In: Electronics Magazine 38.8 (Apr. 1965). [2] Top 500 Supercomputer systems. Jan url: [3] Ryan Smith. Intel s 14nm Technology in Detail. Aug. 11, url: 14nm- technology- indetail. [4] Topp 500 November Mar. 10, url: lists/2015/11. [5] Interconnect Family statistics, November Mar. 10, url: [6] HPC Advisory Council. Interconnect Analysis: 10GbE and Infiniband in High Performance Computing. Tech. rep. HPC Advisory Council, [7] Jeff Byrne. InfiniBand networking for storage and converged data centers. Apr. 5, url: InfiniBand-networking-for-storage-and-converged-data-centers. [8] InfiniBand Architecture Specification Volume 1. Release 1.3. InfiniBand Trade Association. Mar [9] Mellanox Technologies. Introduction to InfiniBand. Revision 1.9. [10] Open Systems Interconnection. Reference Model (ISO7498) [11] Ganglia. Mar. 26, url: [12] Nagios. Mar. 26, url: [13] FabricIT. Apr. 6, url: http : / / www. mellanox. com / page / ib _ fabricit_efm_management. [14] Nishanth Dandapanthula. InfiniBand Network Analysis and Monitoring using OpenSM. MA thesis. United States of America: Ohio State University, [15] HighChart JS. May 30, url: [16] Fabriscale. Apr. 5, url: 12