CHAPTER 1 Introduction This chapter provides the background knowledge about Multistage Interconnection Networks. Metrics used for measuring the performance of various multistage interconnection networks are explained. The need, objectives, research methodology and scheme of the study have also been described. A parallel system means a set of different processors or computers interconnected together to improve the performance. High performance computing systems can be designed using parallel processing. With the rapid advances in technology, it is now possible to build a system consisting of hundred or thousands of processors. The general-purpose parallel/distributed computer systems are divided into two categories: multiprocessors and multicomputers. The main difference between them lies in the level at which interactions between the processors occur (Bhuyan, 1987). Multiprocessors: A multiprocessor must permit all processors to directly share the main memory (Bhuyan et al., 1989). All the processors address a common main memory space. Multiprocessors can be further divided as tightly coupled and loosely coupled. In a tightly coupled system, the main memory is situated at a central location so that the access time from any processor to the memory is the same. In addition to this central memory (also called main memory, shared memory, global memory etc.), each processor might consist of some local memory or cache. In a loosely coupled system, the main memory is partitioned and attached to the processors, although the processors share the same memory 1
address space. A processor can directly address a remote memory, but the access time is much higher compared to a local memory access. Multicomputers: In a multicomputer, each processor has its own memory space and sharing between the processors occurs at a higher level as with complete file or data set (Bhuyan et al., 1989). A processor cannot directly access another processor s local memory. The interaction between the processors relies on message passing between the source and destination processors (nodes). The message passes over a link that directly connects two nodes and might have to pass through several such nodes in a store and forward manner before it reaches its destination. Therefore, each interaction involves a lot of communication overhead, and only those applications that need less interprocessor communication are well suited to multicomputers. 1.1 Interconnection Networks (INs) The performance of a multiprocessor rests primarily on the design of its interconnection network (Bhuyan et al., 1989). An interconnection network is a complex connection of switches and links permitting processors in a multiprocessor system to communicate among themselves or with memory modules or I/O devices (Bhuyan, 1987). It is the path, in which the data must travel in order to access memory in a shared memory computer or to communicate with other processes in a distributed memory environment or to use any I/O devices. 2
Figure 1.1 : An Interconnection Network The switching element may be viewed as a very small network. These switches are the devices having multiple inputs and multiple outputs. A switch box has any one of the following four states, i.e., straight, exchange, upper broadcast and lower broadcast as shown in Figure 1.2. Straight Exchange Lower Broadcast Upper Broadcast Figure 1.2 : Types of Switching Elements 3
1.1.1 Design Dimensions of Interconnection Networks According to Feng (1981), the key design dimensions for interconnection networks are: a) Switching Methodology b) Operational Mode c) Control Strategy d) Network Topology. a) Switching Methodology Two major switching methodologies are: Circuit Switching: In circuit switching an end-to-end physical path is actually established between a source and a destination (Feng, 1981). This path exists as long as the data transmission is not complete. Circuit switching is suitable for bulk transmission of data. Packet Switching: In packet switching data is divided into packets and routed through the interconnection network without establishing a physical connection (Feng, 1981). Packet switching is more efficient for short messages. b) Operational Modes Operational modes can either be synchronous or asynchronous or a combination of the two. Synchronous: Synchronous control techniques are characterized by a global clock, which broadcasts clock signals to all devices in a system so that the entire system operates in lock-step fashion (Bhuyan et al., 1989). 4
This mode is useful for either a data manipulating function or for a data instruction broadcast. Asynchronous: Asynchronous techniques do not utilize a single global clock, rather distribute the control function throughout the system, often utilizing many individual clocks for timing. Asynchronous communication is needed for multi-processing in which connection requests are issued dynamically (Feng, 1981). c) Control Strategy A typical interconnection network consists of a number of switching elements and interconnection links. Interconnection functions are realized by proper control of switching elements (Feng, 1981). The control strategy can be of two types: Centralized Control Strategy: In this strategy, a centralized controller manages all the switching elements. Distributed Control Strategy: In this, individual switching element manages all control settings. d) Network Topology A network can be represented by a graph in which nodes indicate switches and edges represent communication links (Feng, 1981). Topology is the pattern in which the individual switches are interconnected to other elements such as processors, memories and other switching elements. The topologies as shown in Figure 1.3 can be categorized into two groups: 5
Figure 1.3 : Types of Network Topology Static: In static topology, links between two processors are passive and have dedicated buses, which cannot be reconfigured for direct connection with other processors. Static networks are generally used in messagepassing architectures. The following network topologies are commonly used: 1) Ring Network 2) Star Connected Network 3) Completely Connected Network 4) Tree Network 5) Mesh Network 6) Hypercube Network. 6
Dynamic: On the other hand, links in dynamic topology are reconfigured by setting network s active switching elements. Dynamic interconnection networks implement one of the following interconnection techniques: 1) Crossbar Network 2) Bus based Network 3) Multistage Interconnection Network. Static Topologies Ring Network: In ring network, every device has two adjacent neighbors for communication. In a ring network, all the communication messages travel in the same direction whether clockwise or anti-clockwise. Damage of a cable or a device can result in the breakdown of the whole network. Star Connected Network: In computer networking, the most commonly used topology is the star topology. All the computers in the star topology are connected to central device like hub, switch or router. The main disadvantage of this kind of topology is that if central device stops working then there will be no transmission between any nodes. Completely Connected Network: In this topology, all nodes are directly connected to every other node with a point-to-point link. In this, message sent to the destination can take any possible shortest, easiest route to reach its destination. In the star and ring topologies, messages are usually broadcast to every computer. Tree Network: Tree topology integrates multiple star topologies together onto a bus as shown in Figure 1.4 (d). Only the hub devices can connect directly with the tree bus and each hub functions as a root of a tree of the network devices. 7
This bus/star/hybrid combination supports future expandability of the computer networks, much better than a bus or star. Mesh Network: A mesh simply connects one processor to four other processors, as shown in Figure 1.4 (e). Processors along the top and bottom edges are connected to the processor in the same column along the opposite edge. The processors on the last column are connected to the first processor on the next row. The processor in the bottom right corner would have a connection to the processor in the top left corner. With this layout, there are 2N connections, but it takes at most N-1 shifts to get data from one processor to any other. Hypercube Network: A zero dimensional hypercube is a single processor and a one-dimensional hypercube connects two zero dimensional hypercubes, i.e., a line connecting two nodes defines a one-dimensional cube. A square with four nodes is a two-dimensional cube. Generally, a hypercube of dimension d+1 is constructed by connecting corresponding processors in two hypercubes of dimension d. In a hypercube, two processors are connected if and only if the binary representation of their labels differs in a single position. It may be considered as a mesh, with additional long distance connections. A disadvantage of the hypercube interconnection is that it is more complex than the mesh. In Figure 1.4 (f), processors in the cubes of dimension 1, 2, and 3 are labeled with integers, represented as binary numbers. Two processors are neighbors in dimension d if and only if their binary labels differ only in the d th place. (a) Ring (b) Star (c) Completely connected 8
(d) Tree (e) Mesh (f) Hypercube of dimension zero to three Figure 1.4 : Various Types of Static Network Topologies Dynamic Topologies Crossbar Networks: The crossbar makes a connection from a given vertical bus to the appropriate horizontal bus and allows traffic to flow along this path. In crossbar network, the other horizontal or vertical buses can be supporting a flow of traffic at the same time. For example, if each horizontal bus needs to talk to a separate vertical bus then they all can move data at the same time. This completely eliminates the single-shared-resource limitation of the system bus. The crossbar is a preferable approach for high performance multiprocessors. Figure 1.5 shows a crossbar network. 9
Figure 1.5 : A Crossbar Network Bus Network: Bus topology uses a common backbone to connect all the network devices in a network in a linear shape as shown in Figure 1.6. Some of the simplest and earliest parallel machines used bus topology. All processors access a common bus for exchanging data. The distance between any two nodes is O(1) in a bus. The bus also provides a convenient broadcast media. However, the bandwidth of the shared bus is a major bottleneck. A bus is highly non-scalable architecture, because only one processor can communicate on the bus at a time. A bus network design offers minimum bandwidth. It is highly inefficient and unreliable because of a single bus, the failure of which will make it unusable. Buses are commonly used in shared memory parallel computers to communicate read and write requests to a shared global memory. A single bus organization is simple and inexpensive. Addition of more processors or memory modules increases the bus contention, thus decreases the bus throughput. 10
Figure 1.6 : A Bus Network Multistage Interconnection Networks: Multistage Interconnection Networks (MINs) consist of more than one stage of small interconnection elements called switching elements and links interconnecting them. MINs are used in multiprocessing systems to provide cost-effective, high-bandwidth communication between processors and/or memory modules. A MIN normally connects N inputs to N outputs and is referred as an N N MIN. The parameter N is called the size of the network. Figure 1.7 illustrates structure of a multistage interconnection network, which is representative of a general class of networks. This figure shows the connection between p inputs and b outputs via n number of stages. A multistage interconnection network is actually a compromise between crossbar and shared bus networks (Bhuyan et al., 1989). Table 1.1 describes the properties of various types of multiprocessor interconnection networks. 11
Figure 1.7 : A Multistage Interconnection Network (MIN) Table 1.1 : Properties of Different Networks Property Bus Crossbar Multistage Speed Low High High Cost Low High Moderate Reliability Low High High Configurability High Low Moderate Complexity Low High Moderate 1.1.2 Classification of Multistage Interconnection Networks Multistage interconnection networks can be classified according to different categories. The main classification categories are: a) According to number of paths b) According to number of switches c) According to control d) According to availability of path. 12
a) Classification according to Number of Paths According to number of paths, MINs can be classified as unique and multi-path networks, as described below: Unique Path Networks: These networks provide a unique path between every source and destination. The failure of any switching element along the path disconnects some source-destination pairs, so adversely affecting the capabilities of existing network. These networks are not reliable for a large multiprocessor system, as they cannot tolerate even a single fault. In case of multiple requests, a source destination connection may be blocked by a previously established connection, thus providing a poor performance. Multi-path Networks: These networks provide more than one path between a given source and destination. In case of failure of any switching element in the path, the request is routed through some alternative path. Unique path multistage interconnection networks can be made multi-path by adding redundancy in the form of extra switching elements, links, stages, sub networks, by increasing the size of switching elements or by using multiple networks. Multi-path multistage interconnection networks can be either static or dynamic. For static networks, if a fault is encountered, then data has to backtrack to the source or some fixed point to select an alternate path in the network. The implementation of backtracking is expensive in terms of the hardware. In dynamic networks, if a fault is encountered in a particular stage, a switching element in preceding stage will re-route data through an alternate available path. 13
b) Classification according to Number of Switches MINs can be classified according to number of switches as regular and irregular networks, as described below: Regular Networks: Regular multistage interconnection networks have an equal number of switching elements at each stage (Mittal et al., 1995), as a result they may impose equal time delay to all the requests passing through them. Irregular Networks: Irregular multistage interconnection networks have unequal number of switching elements at each stage (Mittal et al., 1995). For a given source destination pair multiple paths with different path lengths are available. c) Classification according to Control Flip Controlled Networks: Flip controlled multistage interconnection networks have a common control signal for switching in various switching elements at a given stage. These networks are less complicated due to lesser number of control signals but have lesser bandwidth. Distributed Control Networks: Distributed control multistage interconnection networks have a separate control signal for every switching element. These have higher bandwidth due to selection of source destination pair at a given time and are quite complex. d) According to Availability of Path Blocking Networks: A network is called a blocking network if simultaneous connections of more than one terminal may result in conflict in use of network communication links. Example of blocking network is Omega network. 14
Non-blocking Networks: A network is called non-blocking if it is possible to route data from any source to any destination, in presence of other established source-destination routes, provided no two sources have same destination. In other words, a network that can handle all possible connections without blocking is called non- blocking network. 1.1.3 Performance Parameters The performance parameters used for MINs are: a) Permutation Passibility b) Fault Tolerance c) Bandwidth d) Throughput e) Probability of Acceptance f) Processor Utilization g) Reliability. a) Permutation Passibility: A permutation for a network is a pairing of its inputs and outputs such that each of the input and output appears in exactly one pair. In other words, a permutation is a full one-to-one mapping between the network inputs and outputs (Subramanyam et al., 2006). If a number of requests occur simultaneously at the source then the permutation passibility behavior of a network shows that how many input requests are able to pass through the given network, and how many of them will successfully mature, i.e., reach their destination. The request always passes through the most favorable path (generally the path with minimum length), if such path is busy or faulty then the request is 15
passed through an alternate path. If no alternate path is available, then the request is simply dropped (Sharma et al., 2008a, 2008b). So, some of the requests will pass through the most favorable path, whereas others have to be routed through available alternate path. b) Fault Tolerance: A fault tolerant multistage interconnection network provides services even under faults. A fault can be permanent or transient in nature. Fault tolerance is a criteria that must be met for the network which has tolerated a given fault or faults. A network is called single fault tolerant if it can tolerate or function in case of a single fault. In general, if a network can tolerate i-faults, then network is called i-fault tolerant. c) Bandwidth (BW): It is the most common performance parameter used in analyzing a synchronous interconnection network. It is defined as the number of input requests honored per unit of time. A bandwidth (BW) also takes into account the memory access conflicts caused by the random nature of the input requests (Bhuyan et al., 1989). BW can be defined as the expected number of requests reaching destination in any given cycle, i.e., the total number of requests matured. A high bandwidth is often desired at reasonably low cost of the network. d) Throughput (TP): Throughput is the maximum number of traffic accepted by the network per unit time. In other words, it can be defined as average number of packets delivered from source to destination by a network in unit time. It is measured in terms of packets per node per cycle. Another definition of throughput 16
is the number of cells delivered by the network per unit time per unit line (Bhuyan et al., 1989). e) Probability of Acceptance (P a ): It is defined as the ratio of expected bandwidth to the expected number of requests generated per cycle (Bhuyan et al., 1989). Expected means, the request generated by the source in a random access environment. f) Processor Utilization (PU): It is the expected percentage of time a processor is active. A processor is said to be active when it is doing internal computation without accessing the global memory (Bhuyan et al., 1989). g) Reliability: Reliability of a system is the probability that it will successfully perform its intended operations for a given time under stated operating conditions. There are three main measures of reliability: Terminal Reliability (TR): Terminal Reliability is the probability that at least one path exists from a particular input port to a particular output port (Sivakumar and Venkatesan, 1996). TR is always associated with a Terminal Path (TP), which is a one-to-one connection between an input port (source) and an output port (destination). A network is considered failed if it is not able to establish a connection from a given source to a given destination. Broadcast Reliability (BR): Broadcast Reliability is the probability that at least one path exists from a particular input port to all the output ports. BR is always associated with a Broadcast Path (BP), which is a connection from one source to all destinations in the network. BR is usually referred to as the source-to-multiple terminal (SMT) reliability (Sivakumar and Venkatesan, 1996). 17
Network Reliability (NR): The ability of interconnecting all inputs to all outputs can be demonstrated by the Network Reliability. NR is the probability of maintaining full access capability throughout the network. NR is associated with network path (NP) which is a many-to-many connection linking sources to many destinations (Sivakumar and Venkatesan, 1996). h) Cost Effectiveness: To estimate the cost of a network one common method is to calculate the switch complexity, with the assumption that the cost of a switch is proportional to the number of gates involved, which is roughly proportional to the number of cross points within a switch. For example a 4x4 switch has 16 units of hardware cost whereas a 2x2 switch has 4 units. For the multiplexers and demultiplexers it is roughly assumed that each of Kx1 multiplexers or 1xK demultiplexers has K units of cost. 1.2 Need for the Study General goals for the design of fault-tolerant MINs has good performance even in the presence of faults, maintain high reliability while avoiding exorbitant additional cost, along with routing of messages around failed components to uphold high computational speed. Despite tremendous achievements in attaining this aim, there is inadequacy of suitable designs for the increasing parallel system applications. Most regular MINs proposed in the literature have path length of O(logN), which is a limitation on their computational speed, whereas their irregular counterparts have variable path lengths. Generally, 2x2 crossbar switches have been adopted as the basic switching element (SE) and investigations of the benefits of SEs with bigger size have not been analyzed enough. So, a need for the study was felt to design new fault-tolerant 18
static and dynamic MINs, which are regular and irregular, displaying better faulttolerant capabilities with improvement in the performance as well as the reliability. 1.3 Objectives of the Study: The specific objectives of the study are as follows: To study the existing MINs and propose new MINs/modify the existing regular and irregular MINs in order to improve their fault-tolerance and performance. To design a simulator for the analysis of permutation passibility behavior of various MINs. Reliability analysis of regular and irregular MINs. Bandwidth analysis of regular and irregular MINs. 1.4 Research Methodology: A comprehensive literature survey has been conducted to understand the strengths and weaknesses of the existing static and dynamic, regular and irregular MINs. The study led to the development of three new MINs (namely, IASEN, IFTN and IABN) having full access capability, fault-tolerance and reasonably good performance parameters. All these three MINs have irregular structure and dynamic routing. Developed and implemented algorithms in C/C++ for the simulation of permutation passibility behavior of proposed and corresponding existing MINs. During coding MINs have been realized through equivalent digraphs. 19
Computed and analyzed mathematical equations for upper and lower bounds of reliability for Proposed MINs using the series-parallel probabilistic models. The reliability equations have been solved by appropriate numerical techniques. Bandwidth and associated parameters have been worked out and computed for the proposed MINs, viz. IASEN, IFTN and IABN. Cost functions have been derived and compared by plotting graphs. In order to reach the conclusion the performance parameters of proposed MINs have been compared with the other existing MINs (namely, ASEN-2, FT and ABN). 1.5 Scheme of the Study This research work is an attempt to investigate the issues and tradeoffs for the design of high-speed, reliable, fault-tolerant MINs with high performance and low cost. Chapter 2 presents the literature survey. It includes a review of different multistage interconnection networks and their design issues. This is useful for designing new interconnection networks. In this context, various techniques made to improve performance, latency and cost-effectiveness have been identified which helped to construct new MINs. Chapter 3 presents the design and routing scheme of all the three newly proposed MINs. The first MIN, an Irregular Augmented Shuffle Exchange Network (IASEN), has been derived from Augmented Shuffle Exchange Network (ASEN-2) by removing four switches from stage 1. In IASEN, 1x4 demultiplexers have been used in lieu of 1x2. 20
Besides, there are some changes in existing loops and inter-connections also. IASEN provides eight paths between a source destination pair. The second MIN, an Improved Four Tree Network (IFTN), has been designed from Four Tree (FT) network by removing one stage. It contains multiplexers and demultiplexers same as that of FT network. An IFTN being an irregular network supports multiple paths of different path lengths. The main advantage of this network is that if both the switches in a loop are simultaneously faulty, even then some sources are connected to the destination. Therefore, FT is more fault-tolerant. Here, the maximum path length is 4 which is one less compared to FT. The third MIN is an Irregular Augmented Baseline Network (IABN) designed to improve the Augmented Baseline Network by adding an additional stage, auxiliary links and enhancing the size of demultiplexers. IABN is a dynamically re-routable irregular MIN that provides multiple paths of varying lengths between any arbitrary sourcedestination pair. In an IABN, there are six distinct paths between any source-destination pair, whereas ABN has two paths. More paths between every source-destination pair have been facilitated through the additional stage. Chapter 4 shows the permutation passibility behavior of six multi-path faulttolerant MINs with and without faults. The request is first tried through the most favorable path, generally the path with minimum path length. The request is routed through an alternative path only if the most favorable path is not available due to faulty switch or being used by another request. If no alternative path is available, the request is dropped. For the analysis, the concept of directed graph is used to simulate the topology of the network, which is implemented in C language. 21
Chapter 5 presents the reliability analysis of IASEN, IFTN and IABN in terms of mean time to failure using the series-parallel model. The reliability has been evaluated for both upper and lower bounds. The comparative analysis made with other related networks is shown with the help of graphs. Chapter 6 carries the Bandwidth analysis of various multistage interconnection networks. The computation of some other performance parameters, namely, Probability of acceptance, Throughput and Processor Utilization are also presented. Chapter 7 summarizes this work by discussing the contributions of the proposals and drawing the conclusions. The scope for future research is also outlined. 1.6 Limitations of the Study 1) The thrust of the study is on conventional MINs using simple switches and links. The optical MINs have not been taken into the purview. 2) The Reliability of MINs in terms of MTTF has been computed with the assumption that there is no repair between failures. 1.7 Chapter Summary This chapter describes why and how the present study was designed and conducted. The discussion on the contemporary networking concepts is linked to the design of regular and irregular multistage interconnection networks. The focus of this research is to design cost-effective fault-tolerant, reliable MINs, which provide multi- 22
path routing. Further, it describes the relevant terminology, the various concepts of MINs concepts along with objectives of study and its limitations. In the next chapter, an endeavour has been made to examine the different uniquepath and multi-path Multistage Interconnection Networks, having regular or irregular topology through the review of relevant literature available on the subject. 23