The Switcherland Distributed Computing System

Transcription

1 4th GI/ITG-Fachtagung Arbeitsplatz-Rechensysteme, Koblenz, ay 21-22, 1997, pp The witcherland Distributed Computing ystem ichaela Blott, Hans Eberle, Erwin Oertli, eter Ryser wiss Federal Institute of Technology Institute of Computer ystems CH-8092 Zurich, witzerland {michaela.blott, eberle, oertli, ryser}@inf.ethz.ch Abstract: witcherland is a distributed computing system optimized for running applications that process continuous data such as video and audio. For this purpose, witcherland implements a communication model that offers quality of service (Qo) for a distributed shared memory architecture. Qo is provided in that memory can be accessed at guaranteed rates and in bounded time. witcherland is based on a scalable interconnection structure that can serve as an I/O interconnection structure of a workstation as well as a network interconnection structure of a cluster of workstations. This paper gives an overview of the architecture and communication model implemented by witcherland and describes the system components we are currently building. Keywords: interconnection structure, quality of service, distributed shared memory, cluster computing. 1 Introduction rocessing continuous data such as video and audio data will become an essential task of future computing systems. Current computing platforms use specialized subsystems for processing such data. The main reasons for using specialized subsystems rather than general-purpose system resources are I/O bandwidth limitations and a lack of real-time guarantees. oon, general-purpose processors will be able to process continuous data in realtime, e.g. to compress and decompress audio and video data [8]. To handle the high data rates of uncompressed video data, interconnection structures with adequate bandwidth are needed. Further, when processing continuous data real-time support in the form of communication channels with guaranteed bandwidths and bounded end-to-end transmission delays is of particular importance. Otherwise, data is produced, transmitted and reproduced at unpredictable rates and with high delay jitter.

2 4th GI/ITG-Fachtagung Arbeitsplatz-Rechensysteme, Koblenz, ay 21-22, 1997, pp odern networking technologies such as asynchronous transfer mode (AT) networks recognize the need for real-time support. Unfortunately, the real-time properties of AT networks do not extend into the workstations connected to them: the real-time guarantees are lost as soon as the data enters the workstations. Therefore, we want to investigate how such guarantees can be provided at all system levels including the hardware and software of the workstations connected to such an interconnection structure. Other projects with goals similar to ours are the Desk-Area Network [5], the calable Coherent Interface [4], Telegraphos [7] and TNet [6]. The common objective of these efforts is scalability and a simple communication model based on a distributed shared memory architecture. We are, however, not aware of any project that offers Qo for such a communication model. This paper is organized as follows. Chapter 2 gives an overview of the main characteristics of witcherland. Chapter 3 gives a status report on its implementation. Finally, Chapter 4 contains the conclusions. 2 Overview Figure 1 shows an example of a distributed computing system based on the witcherland interconnection structure. A port of a switch () can be connected either to a node or to another switch. A node corresponds to a processor () with local memory () or to an I/O device (I/O). Examples of I/O devices are frame buffers and secondary storage devices. As shown, a workstation can contain one or several switches. The switches have two purposes. Within a workstation, they serve as an I/O interconnection structure and, within a cluster, they are used as a network interconnection structure. By using the same interconnection structure for inter- and intra-workstation communication the boundary of the workstation is less clear than in a traditional networked system since the logical grouping of nodes can easily be different from the physical arrangement. By cascading switches any arbitrary topology is possible. The aggregate bandwidth of such a system can be many times the bandwidth of a single switch. By adding switches and links, the aggregate bandwidth can be increased as required. Also, availability can be improved by adding redundant links and switches. The following paragraphs summarize the main characteristics of the witcherland interconnection structure. ore details can be found in [1,2]. calability: In most workstations the components are interconnected by a single shared link or bus. The main attraction of the bus is its low cost and low complexity thanks to the use of a shared medium. The drawbacks are limited extensibility and scalability. The maximum number of nodes is fixed and only relatively short distances can be spanned. These limitations become even more evident at high speeds due to the electrical properties of busses. Further, the bandwidth is fixed and cannot be increased when new nodes are added. A possible alternative to a bus is an interconnection structure based on switches. ince such an interconnection structure uses multiple links rather than one shared link, it cannot compete in terms of cost. However, an interconnection structure based on switches is attractive

3 4th GI/ITG-Fachtagung Arbeitsplatz-Rechensysteme, Koblenz, ay 21-22, 1997, pp Figure 1: A Distributed ystem based on the witcherland Interconnection tructure because of its high bandwidth, scalability and availability. Compared with busses, switches provide higher bandwidth because multiple links can be used simultaneously. ince switches can be clustered easily, the aggregate bandwidth can be many times the link bandwidth and also many times the switch bandwidth. Availability is higher for a switch than for a bus since there is no single point of failure. With a bus all communication fails if the shared link fails. If a link of a switch-based interconnection structure fails, typically only a subset of the nodes is affected. Further, redundant links can be added to achieve higher availability. Load/tore Architecture: witcherland implements a load/store architecture. This means that local communication within a workstation as well as remote communication within a cluster of workstations translate to load and store operations. These operations are applied to a distributed memory which resides in a single address space. Compared with message passing, shared memory offers a simpler communication model since the mechanisms to access local and remote memory are the same. With it, the programmer is presented a uniform model for inter- and intra-workstation communication. Of course, the latency differs with local accesses taking a fraction of the time required by accesses to remote memory. ince our interconnection structure is intended for running loosely-coupled distributed applications rather than tightly-coupled parallel applications, there is little hardware support provided for synchronizing processors and keeping memories consistent. The only synchronization primitive implemented in hardware is an atomic compare_and_swap operation. This operation is typically used for implementing critical regions. There is, however, no

4 4th GI/ITG-Fachtagung Arbeitsplatz-Rechensysteme, Koblenz, ay 21-22, 1997, pp hardware support for keeping main memories and cache memories consistent. If memory consistency is required, it needs to be implemented in software. Traffic Classes: imilar to an AT network witcherland distinguishes between constant bit rate (CBR) traffic and variable bit rate (VBR) traffic. The traffic classes differ in the guarantees provided by the switches. For CBR traffic, the switches provide bandwidth guarantees and bounded transmission delays. For VBR traffic, a certain amount of buffering is preallocated in the switches, thereby guaranteeing that cells belonging to this class are never dropped due to buffer overflow. When a switch becomes heavily loaded, the delivery of VBR cells, however, may be arbitrarily delayed without an upper bound while the delivery of CBR cells remains unaffected. Typically, CBR connections are used for transferring continuous data. uch transfers need to be characterized by low latency and little jitter. An example is the transmission of video frames at a given rate. VBR connections are used for transmitting non-continuous data. The transfer of such data is characterized by variable delays. An example might be an interactive application with a user generating requests to read sectors on a disk. In this scenario, data is transmitted in bursts making it difficult to specify any bandwidth and latency requirements. Fixed-size Cells: witcherland transfers data in the form of cells, that is, small packets with a fixed length. The length of a cell is 64 bytes or sixteen 32-bit words. Cells contain one control word, one address word and fourteen data words. The control word specifies the routing path, the traffic class and the type of memory operation while the address and data words make up the actual memory operation. ulticast: ulticast is offered for transferring CBR data from one source node to many destination nodes. It is implemented by the switches in that they can forward data from one input port to multiple output ports. This feature allows to save communication bandwidth when executing applications such as video conferencing. End-to-end Flow Control: ince we limit the diameter of our interconnection structure to about ten switches, flow control is done end-to-end rather than link-by-link. This simplifies the design of the switches significantly since they do not have to be involved in controlling the flow of data. Different mechanisms are used to control the flow of CBR and VBR traffic. Rate-based flow control is used for CBR traffic in that a node may inject data into the interconnection structure at a rate specified for the corresponding connection. Credit-based flow control is used for VBR traffic in that a node injects data as long as the buffer space, which was reserved along the path of the connection, has not been used up. A description of how flow control works in detail can be found in [3]. 3 Implementation We are currently working on a prototype system consisting of switches, processor nodes, frame buffer nodes, disk nodes, audio/video input/output nodes and standard workstations connected through CI adapters. We use off-the-shelf components only and, in particular,

5 4th GI/ITG-Fachtagung Arbeitsplatz-Rechensysteme, Koblenz, ay 21-22, 1997, pp field-programmable gate arrays since rapid prototyping and flexibility are our main concerns. witch: The implemented prototype switch supports four ports. Each port connects to a 266 bit/s full-duplex serial link. Deducting the overhead of the 8B/10B encoding scheme [9] used for transmitting data over the links the switch provides a bandwidth of 0.85 Gbit/s. The links use shielded twisted-pair wiring and can span distances of up to 50 m. Logically, the implementation of the switch represents a non-blocking crossbar switch with output buffering. The switching fabric and output buffers provide enough bandwidth so that data is never lost due to congestion. rocessor Node: The processor node consists of a I R4700 CU, synchronous DRA to reduce memory access time and serial interfaces for connecting a keyboard, a mouse and other devices. Its link interface provides support in hardware for efficiently handling remote load and store operations. Frame Buffer Node: In addition to a bitmap memory and display refresh controller, this node provides hardware support for clipping and windowing. This feature will, for example, allow direct transfer of multiple video streams between sources of video data and frame buffers without the need for processor intervention. Disk Node: The disk node is based on the processor node which, for this purpose, is extended with a CI interface. We use audio/video hard disk drives which allow tighter control of access times than traditional drives. To complement the real-time guarantees of the interconnection structure, the disk node will allow disk data to be accessed at guaranteed rates. CI Adapter: The CI adapter connects a C or workstation based on the CI bus to the witcherland interconnection structure. It is designed to support the full link bandwidth, i.e. it can simultaneously receive and transmit data at 266 bit/s. With it, witcherland can also be used as a high performance cluster interconnection structure for commercial platforms. Audio/Video Input Node: The audio/video input node digitizes analog audio and video signals and sends the corresponding digital signals to a witcherland link. The video input signal can be taken from up to three different external sources or an on-board TV tuner. ultichannel Audio Input/Output Node: This node simultaneously samples eight audio input channels and plays back two audio channels. The corresponding digital signals are sent and received by a witcherland link. The analog signals are sampled at a rate of 44.1 khz and quantized at 18 bit/sample. EG-2 Decoder Node: The EG-2 decoder node inputs a EG-2 encoded stream from a witcherland link, decodes it and outputs it either as an analog TV baseband signal available at a board connector or as a digital signal sent to the link. 4 Conclusions

6 4th GI/ITG-Fachtagung Arbeitsplatz-Rechensysteme, Koblenz, ay 21-22, 1997, pp To process continuous data such as video and audio data, real-time support is required at all system levels. odern local area networks, such as AT networks, provide support for processing such data in the form of connections with guaranteed bandwidths and bounded delays. These guarantees, however, do not extend into current workstations. witcherland fills this gap. The presented interconnection structure offers high bandwidth and support for processing continuous data. It is the basis for a uniform treatment of continuous and noncontinuous data. Video and audio data will eventually become standard data types and, therefore, need to be able to pass through a system by using standard communication channels. witcherland provides a uniform communication model for a distributed computing environment. hysically, uniformity is achieved in that the interconnection structure can be used not only for connecting the components of a workstation but also for interconnecting a cluster of workstations. In terms of the programming model, uniformity is provided in that all nodes are part of a single address space, and there is no distinction between local and remote memory accesses. References [1] H. Eberle, A calable Interconnect for Distributed ultimedia ystems. 2nd IATED/I Int. Conference on Distributed ultimedia ystems and Applications, tanford, CA, August 7-9, 1995, pp [2] H. Eberle, witcherland - A calable Interconnection tructure for Distributed Computing. 3rd Int. Conf. of the Austrian Center for arallel Computation, Klagenfurt, eptember 22-25, In L. Böszörmenyi, Editor, arallel Computation, Lecture Notes in Computer cience, vol. 1127, pringer-verlag, 1996, pp [3] H. Eberle, E. Oertli, Flow Control in the witcherland Interconnection tructure, ICE 96, 11th Int. Conference on ystems Engineering, Las Vegas, July 9-11, [4] D. Gustavson, The calable Coherent Interface and Related tandard rojects. IEEE icro, vol. 12, no. 1, February 1992, pp [5]. Hayter, D. cauley, The Desk-Area Network. AC Operating ystems Review, vol. 25, no. 4, October 1991, pp [6] R. Horst, TNet: A Reliable ystem Area Network. IEEE icro, vol. 15, no. 1, February 1995, pp [7]. Katevenis, Telegraphos: High-peed Communications Architecture for arallel and Distributed Computer ystems. Technical Report 123, Foundation for Research and Technology, Heraklio, Crete, [8] R. Lee, Accelerating ultimedia with Enhanced icroprocessors. IEEE icro, vol. 15, no. 2, April 1995, pp [9] A. Widmer,. Franaszek, A DC-Balanced, artioned-block, 8B/10B Transmission Code, IB Journal on Research Developments, vol. 27, no. 5, eptember 1983, pp