Initial studies of SCI LAN topologies for local area clustering

Transcription

1 Prepared for the First International Workshop on SCI-Based Low-Cost/High-Performance Computing, Santa Clara University Initial studies of SCI LAN topologies for local area clustering Haakon Bryhni * and Bin Wu **, University of Oslo, Norway * Department of Informatics. bryhni@ifi.uio.no ** Department of Physics, bin.wu@fys.uio.no Abstract A Local Area Network (LAN) can be built using the ANSI/IEEE Standard Scalable Coherent Interface (SCI) as the underlying hardware protocol. For the LAN, SCI will act as a new physical layer, and traditional protocols (as e.g. TCP/IP) may be run on top of SCI. In a first approach, SCI is used to implement message passing while new protocols and functions that take advantage of the shared memory functionality can be developed. In this paper we do initial studies of throughput and latency at the physical layer for some SCI LAN candidate topologies. This performance will be a hard upper bound for application performance. SCI LANs based on ring, switched star and a mixed topology of hubs and a switched backbone are considered, and simulations are done to show how the performance metrics throughput and latency varies as function of topology and the physical transmission distance of the network. The new point in this study is to focus on the effect of increasing transmission distance to show that a LAN as well as a closely coupled system can be built by using the SCI interconnect. 1. Introduction The ANSI/IEEE Scalable Coherent Interface is a standard giving computer-bus like service to nodes in a distributed environment. SCI scales well, and avoids the limitations of buses by using many point-to-point links and a hardware-embedded packet protocol to provide coherent and non-coherent shared memory to the nodes. The nodes in SCI may be processors, memories and I/O units or complete workstations (WS) connected to SCI by means of a Cluster Adapter () doing protocol conversion between the WS bus and the SCI interconnect as shown in figure 1 []. cpu mem I/O Figure 1: workstation[n] workstation[1] cpu mem I/O SCI LAN local-area bus with topology X Local area computing environment The flexibility of the interconnect allows a wide range of topologies ranging from tightly to loosely coupled systems. In this paper, we examine some candidate topologies for a loosely coupled system of SCI-connected workstations forming a Local Area Computing Environment (LACE). In a LAN environment, distances between nodes are typically in the range of 10 to 1000m. Most studies on topologies for SCI so far has been done on tightly coupled systems [e.g. Bothner93], where the effect of varying physical distance between the nodes are not considered in detail. The new point in this study is to focus on the effect of increasing transmission distance to show that a LAN as well as a closely coupled system can be built by using the SCI interconnect. A Local Area Network is characterized by a number of factors, such as the coverage of a small geographic region, the communication channels between the interconnected computers are usually privately owned (one administrative domain, often trusted nodes), the channels have a low bit error rate and are of relatively high capacity, etc. SCI matches all these requirements, and adds features such as hardware support for coherent shared memory, high-end throughput of Gbyte/s and good architectural support for design of multiprocessor systems. For computer communication, there exists two distinct methods of communication, message passing and shared memory [Tanenbaum87]. The idea of SCI LAN is closely connected to the distinction between these communication paradigms. All Local Area Networks of widespread use are based on message passing, but the advent of hardware-supported coherent shared memory (implemented by transactions in the interconnect) allow computers in a LACE to communicate by using shared memory in stead of message passing. The complexity of providing coherent shared memory in such a system is hidden by hardware, and the communication protocols may exploit the simpler shared memory communication paradigm. Use of the coherence mechanisms may further increase both performance and functionality. Using shared memory for interprocess communication is well known from multiprocessor systems [Delp88, Cheriton94], but is it possible to get the benefit of these features also in a loosely coupled LAN-like system? This will require high throughput and low latency even when transmission distance increases. In our opinion, latency is the most crucial parameter that will be discussed in the following sections. We use a simulation model to compare the candidate topologies. Our modelling and model assumptions are discussed in section 2. In section 3 the selected topologies are presented, and in section 4 the simulation results are discussed. Section 5 contains our conclusions. 2. Modelling and Model Assumptions In this section, we briefly discuss model assumptions for the simulations. 2.1 Simulation model The simulation model is a discrete-event simulator, implemented in the Modsim II programming language [MODSIM]. 2.2 Traffic We choose one common arrival intensity λ, and a negative exponential arrival distribution is used. λ is selected to give a heavy traffic load, but low enough to give reasonable latency. We use a simple traffic model, where burstiness, variations in application frame lengths and the properties of segmentation and reassembly of packets are not considered. All traffic is modelled as memory accesses with a 64-byte cache line granularity. Since we do not know the locality of SCI LAN accesses, we limit our study to two cases as shown in figure 2, 1) uniform distributed targetid s, and 2) the central server case where one server takes 90% of total traffic (typically a multimedia server), while the rest of the traffic is uniformly distributed over all nodes. 1

2 We model the traffic between the workstations and the server as SCI transactions DMOVE64. This is an SCI transaction moving 64 byte data with 16 byte overhead. This transaction does not require the SCI response subaction. 2.3 Physical interface We have chosen to use simulation parameters from the Node- Chip TM SCI interface from Dolphin Inc., since this chip can be used in early SCI LAN applications. We simulated the 62.5 MHz CMOS chip, with 125 Mbyte/s link speed. Results for other implementations may be obtained by scaling the results to other interface chip speeds. We note, however, that the overhead introduced in higher layer protocol processing in the end-systems usually dominate over latencies introduced in the physical layer. Different approaches may be taken to overcome the protocol processing bottleneck, but in this paper we study latency and throughput on the physical layer only, since this gives us a hard upper bound of application performance. A switch can be made by N node interface chips interconnected by e.g. a non-blocking switch [Wu94]. We do not go into detail about the switch architecture, but assume that a 16 port switch can be designed by e.g. interconnecting 2x2 or 4x4 switches. We give some rough parameters in Table 1. Hubs interconnect several SCI links, and may be implemented as both rings or switches. Simulations will show the benefits of different topologies. For the physical interface, we use a propagation speed of m/s which is derived from physical measurements, and we assume error free transmission. 2.4 Parameters The parameters are set as realistic as possible: 16 nodes is a reasonable (but still small) number of workstations in a LACE and we believe that both ring and switch topologies can be realized by using CMOS technology. Table 1 summarize model parameters used in the simulation. 2 1) Figure 2: Table 1: Two cases of traffic distribution Parameter Simulation parameters Value Number of nodes N 16 (3.1, 3.2) and 32 (3.3) NodeChip clock speed C Transmission speed R on serial fiber-optical link 62.5 MHz 1 Gbit/s Transaction arrival rate λ 10 6 Distance L Signal propagation velocity SCI Transactions used 2) m 2 * 10 8 m/s DMOVE64 parameters τ 400 (WS bus response time) Nodechip Bypass FIFO delay # 4 * 80 byte buffers (input and output, request and response) of the node interface Switch parameters Switch architecture Routing strategy τ τ Table 1: Decode (Switch addr. decoding) 10 Switch (internal port-port delay) 100 # 4 * 80 byte buffers (input and output, request and response) of the switch ns 16 ns The parameter τ is the memory cycle time in the receiver. This parameter gives an upper bound on receive performance for each node. Cut-through routing [Bertsekas92] give the advantage of pipelining the switching process. 2.5 Performance metrics Throughput X and latency τ are simulated as experienced on the bus between an SCI interface and the rest of the logic (typically local bus glue logic). We assume a can follow maximum link throughput. There are two common definitions of latency, including and excluding the queuing delay in the network adapter (here: ). In the first definition, latency is calculated from the generation of a packet until it is received in the destination processor or memory. In this case, both network latency, interface speed and processor/memory latency are considered. This is interesting from a users points of view, since the user experience only application performance. From a network designers view, the performance of the interconnect itself is an important upper bound for the overall system performance. To study the interconnect isolated from the interconnected systems, we exclude the queuing delay in the network adapter, and measure from the time a packet enters the SCI interface until it is received in the interface at the destination (see figure 3). workstation A A LOGIC bus A Parameter SCI IF Figure 3: Simulation parameters SCI LAN 1 Non-blocking crossbar Virtual Cut-trough ns ns 2 SCI IF Definition of throughput and latency Point of measurement Value workstation B B LOGIC bus B

3 We have chosen to show overall mean throughput and the mean latency experienced by all nodes. X is then the sum of all packets on all nodes sent successfully per second, τ is the average time from a packet enters the SCI interface in a workstation until it is received in the SCI interface in the target workstation. N N 1 X = X i τ = --- τ N i i = 1 i = 1 For all measurements, gross throughput X are given, net throughput (SCI user payload) are always 64/80 = 80% of this figure. Demands for throughput and latency are given by the applications and will vary for the different traffic sources. Generally, we would like to maximize X and minimize τ. Isochronous channels (e.g. audio and video) in particular, demands low latency and little variation in latency (jitter) but can to some extent tolerate loss of packets. Asynchronous channels (like e.g. memory accesses) can tolerate jitter but want high throughput, low average τ and no packet loss. The fact that the transfer of large data blocks are optimal when the transmission distance increases, is considered, and leads to the use of SCI move transactions that do not have response subactions. Protocols to ensure guaranteed delivery is left to the upper layers of the protocol as an end-to-end responsibility to minimize network processing of each packet and maximize throughput. 3. SCI LAN Topologies In traditional LANs, there is a development away from bus-based topologies as used in e.g. IEEE (CSMA/CD LAN). New infrastructure in office buildings are based on structured cabling systems using a star topology with twisted pair or fiber optical cabling for point-to-point connections between central hubs and the distributed workstations. Even Ethernet is now preferred as a point to point link, avoiding the inherent bottleneck of shared media. Physical star topologies (as with structured cabling systems) easily allows logical ring topologies. Ring based topologies have a number of benefits and are often a good compromise between throughput, latency and reliability. A ring uses point-to-point connections and shares the transmission capacity on the ring. The ever-increasing demand for throughput, however, will in the long run force a development towards switched networks on the cost of switch-capable hubs. We have chosen to consider tree topologies, a ring, a star using a central switch, and a mixed topology of hierarchically interconnected hubs. A hub is a component for transparently connecting two or more similar buses [HUB]. The hubs can be designed as e.g. rings or stars. 3.1 SCI LAN ring-based hub We use the ring topology in figure 4 as the starting point of our simulation, since this is the default SCI interconnection, and can be realized as a patch-panel-hub in a wire center. The new point in this simulation compared to other simulations of the SCI ring topology is again the focus on the transmission distance. We show how the performance metrics vary as a function of the workstation to hub cable length L. The SCI ring topology have been studied in a number of different simulations and have also been treated analytically [Scott92]. For small values of L, this experiment also serves as an informal verification of the simulation model. workstation[1] Figure 4: SCI LAN ring topology 3.2 SCI LAN Switch-based hub In figure 5, the hub is designed by means of a switch. Each connection is now a point-to point dedicated communication link. workstation[1] Figure 5: SCI LAN star topology 3.3 SCI LAN hubs and backbone In figure 6, a mixed topology of interconnected hubs are presented. We assume no contention in the switches, and do not simulate the shaded part of the backbone network. Contention in the central switches will of course degrade overall throughput X and increase latency τ. Figure 6: HUB ws[1] L 2 ws[2] L ws[2] HUB SCI LAN hubs and switched backbone L L 1 ws[1] 3

4 4. Results and Analysis We start with the two suggested implementations of a hub, a ring and a switched scheme, and discuss this for the random traffic and for the central server case. The mixed topology of interconnected hubs are then considered for the two traffic distributions random and central server case, respectively. 4.1 Ring and switch topology with random traffic Figure 7 shows the raw throughput X of the system versus the workstation to hub length L. When the traffic pattern in the systems is totally random, i.e. every node sends packets randomly to the other nodes with a uniform distribution, we can see that the system with 16 nodes connected by a switch has much higher throughput than the ring based topology as might have been expected. The results of the ring topology conform with analytical work and the simulations that has been performed by other simulators [Scott92, Bothner92]. The maximum throughput can not exceed approx. 1.5*125 Mbyte/s, i.e. 190 Mbyte/s for low values of L. It is also clear from the figure that even for a system with a Workstation to hub distance up to 1000 meters, the total raw throughput of the switch based system can still approach 150 Mbyte/s. The latency for the two cases (figure 8) shows an approximate linear increase for a switch based system, which can be interpreted by the physical line delay. Remember that in our model, the speed of the transmission is 20 cm/ns and a long distance will limit the system from saturation. The echo will have to travel long way back to acknowledge the success of sending, and then free the FIFO in the sender s interface. The latency for the switch based system is only of tens of micro seconds. Figure 8: Average system latency versus WS to hub distance, random traffic taken by the server, while the rest 10% is evenly distributed among the remaining 15 nodes (16 nodes together). We are not too surprised to see that even the switch based system will not have a high throughput in this configuration, the 220 Mbyte/s maximum in figure 9 is reasonable. The 90% of total traffic can not be taken by the server, 125 Mbyte/s is the peak throughput that can be consumed on this single link, while the rest 95 Mbyte/s is contributed by other links. The ring based system does not suffer very much from the client-server traffic pattern. The observed throughput reduction is due to the lower utilization of the ring after the server, because most of the packets are received by the server. This leads to a bottleneck on the input link of the server, while the output link has a very low utilization. Figure 7: Raw system throughput versus WS to hub distance, random traffic 4.2 Ring and switch topology, central server case For the central server case, we suspect the input and output links of the server to be the bottleneck. We are interested to study how the ring based system and the switch based system can tolerate a traffic pattern where 90% of the total traffic is Figure 9: Raw system throughput versus WS to hub distance, 90% load on server We are surprised to see the much longer latency for the switch based system in figure 10, while the ring based system enjoy the same order of latency as it has in random traffic. The 4

5 explanation is that when a switch is used, each client send its packets to the server via the switch, the packets are buffered in the queues of the switch and an echo will be returned immediately. This echo frees the client queue and results in a new packet being stored in the queue. However, the former packets that are stored in the switch queues can not be forwarded to the destination, because of the contention on the server. Thus, the new packets in the clients queues will get echo.busy all the time until the corresponding queue in the switch is freed (it will get through at last, a round robin scheme is used). So the reason of the high latency for the switch based system is due to our definition of latency as described in section 2.5 since busied packets in the switch are contributing to the latency, while busied packets in the s do not contribute. We could achieve better throughput and latency if the central server was situated close to the switch. The link to the server would be a bottleneck anyway, but we would get faster access to the server. Figure 11: Raw system throughput versus distance between two switches Figure 10: Average system latency versus WS to hub distance, 90% load on server 4.3 Mixed topology of interconnected hubs It is likely that each hub-based system will be contained in a building, while remote sites will be connected using a bidirectional link. We discussed such a model in section 3.3. For the mixed topology, we compare two traffic scenarios as earlier: 1) random destination is selected, and 2) the traffic generated in each node consist of 90% traffic to a local server (local hub), and 10% traffic to a destination connected to the remote hub. In both cases, we choose the local hub to be switch-based. There is only one server in each local hub-based network and 15 clients. The workstation to hub distance is set to 10 meter. The throughput of the two scenarios compared in figure 11 shows that a system with the random traffic pattern has a very high throughput for short switch to switch distance L 2, but this throughput decrease very fast with increasing L 2. Figure 12 shows that a system with the random traffic pattern experience the highest latency. This is because the random traffic is random over all nodes in the system, which means that fifty percent of the traffic goes across the long distance connection, which makes the average latency larger than in the 90% local traffic case where most of the packets are sent to a local destination. Figure 12: Average system latency versus distance between two switches 5. Conclusion Generally, all topologies give poor performance when the workstation to hub distance L increases. This is due to a low link utilization when transmission distance increases. The throughput could be increased by means of larger FIFO buffers, but this will again increase latency. The detailed conclusions for the simulated topologies are given in the following sections. 5.1 Ring/switch based hubs, random traffic pattern With a workstation to hub distance of 1000 m, the switch-based hub throughput is more than 150 Mbyte/s with a latency of ~ 14 µs. In contrast, the ring-based hub give a very low throughput for this distance, but the mean latency experienced is still fairly low (~ 70 µs). 5

6 A switch based hub gives much better throughput than a ring when the traffic is randomly distributed. For short workstation to hub distances, X is over 1.4 Gbyte/s while the ring based system cannot achieve more than 190 Mbyte/s. This is still a high total throughput for the 16 nodes in the system. 5.2 Ring/switch based hubs, central server case With a workstation to hub distance of 1000 m, the latency experienced is about 160 µs for the switch and about 70 µs for the ring case. Corresponding system throughput is 30 Mbyte/s and 10 Mbyte/s. This is much lower than in the random traffic case, but a central server case are thought to be a more realistic traffic scenario. Again, switch based hubs give the best throughput performance, but get a latency penalty because of server contention (see section 4.2 for discussion). For short workstation to hub distances, latency decrease to less than 30µs and throughput increase to more than 200 Mbyte/s and 130 Mbyte/s, for the switch and ring case, respectively. One solution to the bottleneck problem in the central server case, would be to distribute the server responsibility among the attached nodes. This way of sharing server responsibility would lead to a more even traffic pattern, and thus increase performance. 5.3 Mixed topology with interconnected hubs The analysis of the mixed topology with interconnected hubs leads to the following conclusions: Traffic must be kept as local as possible to utilize local troughput and avoid the bidirectional link interconnecting the hub-based local systems. A random traffic pattern leads to much traffic between the two hubs, and as L 2 increase, both X and τ are strongly affected by the interconnection bottleneck. In the central server case, more traffic is local and the interconnection bottleneck is not as obvious as in the random traffic case. 5.4 General conclusion A general conclusion would be to keep L as low as possible in all topologies. In some cases, we would prefer rings to keep L short, but this can be in conflict with modern cabling principles (e.g. structured cabling are star-based) and as we have seen, switched hub solutions generally give us better throughput and latency. The simulated throughput and latencies in a distributed SCI interconnect using the candidate topologies show that a SCI LAN can be designed and high performance can be achieved. The main challenge is now to find a good architecture for cluster adapters to give applications in the end-systems access to this estimated interconnect performance. Acknowledgements We give special thanks to professor Stein Gjessing, University of Oslo, for comments on the paper. Both authors are supported by the Norwegian Research Council. Terminology CSMA/CD FIFO LACE LAN Carrier Sense Multiple Access/Collision Detect A buffer implementing a First In, First Out queue. Local Area Computing Environment Local Area Network NodeChip TM SCI Interface Chip from Dolphin Interconnect Solutions (Dolphin ICS AS, Norway) SCI TCP/IP References Scalable Coherent Interface Transport Control Protocol / Internet Protocol [Bertsekas93] Bertsekas, D., Data Networks, 2nd edition, Prentice Hall, 1992, ISBN , pp 373. [Bothner93] [Cheriton94] [] [Delp88] [HUBS] [MODSIM] Bothner, J.W., Hulaas, T. I., Topologies for SCI-based systems with up to a few hundred nodes, Master Thesis, University of Oslo, Norway Cheriton, R., Kutter, R. A., Optimized Memory-Based Messaging: Leveraging the Memory System for High-Performance Communication, Technical Report, CS-TR , Stanford University, SCI Cluster Adapter (a P1596 study-group activity), SCI 2-page summary from the hplsci.hpl.hp.com server, 21. april Delp, G., The Architecture and Implementation of Memnet: a High-Speed Shared-Memory Computer Communication Network. PhD thesis, University of Delaware, Department of Electrical Engineering, Newark, DE, Hubs: Collections of SCI Bridges (a P1596 study-group activity), SCI 2-page summary from the hplsci.hpl.hp.com server, 21. april Modsim II Reference Manual, CACI Inc., USA, [Scott92] Scott, S.L., Goodman, J. R., Vernon, M. K: Performance of the SCI Ring, International symposium on Computer Architecture, May [Tanenbaum87]Tanenbaum, A., Operating Systems - Design and Implementation, Prentice Hall, 1987, ISBN , pp [Wu94] Wu, B, Bogaerts, A, Kristiansen, E.H., Muller, H, Ernesto, P, Skaali, B, Applications of the Scalable Coherent Interface in Multistage Networks, to be appeared on IEEE TENCON 94, Frontiers of Computer Technology, Aug , 1994, Singapore. (paper available on anonymous ftp server: fidibus.uio.no, under /incoming/sci). 6