High-speed network switch RHiNET-2/SW and its implementation with optical interconnections

Transcription

1 High-speed network switch RHiNET-2/SW and its implementation with optical interconnections S. Nishimura (1), T. Kudoh (2), H. Nishi (2), J. Yamamoto (2), K. Harasawa (3), N. Matsudaira (3), S. Akutsu (3), K. Tasyo (4), and H. Amano (5) (1) RWCP Optical Interconnection Hitachi Laboratory. (c/o Central Research Laboratory, Hitachi, Ltd.) Higashi-Koigakubo, Kokubunji, Tokyo , JAPAN (2) Real World Computing Partnership Tsukuba Research Center (3) Hitachi Communication Systems, Inc. (4) Synergetech, Inc. (5) Dept. of Information and Computer Science, Keio University Abstract--RHiNET-2/SW is a network switch that enables high-performance parallel computing in a distributed environment. We have produced a prototype network-switch board (RHiNET-2/SW) for the RHiNET-2 parallel-computing system. Eight pairs of 800-Mbit/s 12-channel optical interconnection modules and a one-chip CMOS ASIC switch LSI (a 784-pin BGA package) are mounted on a single board. This switch allows high-speed 8-Gbit/s/port parallel optical data transmission over a distance of up to 100 m, and the aggregate throughput is 64 Gbit/s/board. By using a large amount of embedded memory on the switching LSI, RHiNET-2/SW allows low-latency, free-topology network performance. We evaluated the reliability of each optical port by measuring BER: no errors were detected during bit packet data transmission at a data rate of 880 Mbit/s 10 bits (BER: < ). This test result shows that the RHiNET-2/SW can provide highthroughput, long-transmission-length, and highly reliable data transmission. I. INTRODUCTION Network-based parallel processing using commodity components, such as personal computers, has received attention as an important parallel-computing environment [1-3]. In most of today's offices and laboratories, there are tens of personal computers and workstations, which are not always in use. If the processing power of such idle computers could be combined, the resulting processing power might be comparable to that of a supercomputer. However, most high performance cluster systems consisting of personal computers or workstations use a system area network (or a server area network: SAN) such as Myrinet [4] as their interconnection. SAN provides low latency high bandwidth communication without discarding any packet. It also provides high bisection bandwidth, which is required for high performance parallel computing. Since they are designed to connect dedicated computers in a small place, both the link length and topology are restricted. On the other hand, highspeed LANs with more than 1 Gbit/s link bandwidth are becoming available [5-6]. LANs provide relatively flexible topology choices and longer length of links. Nevertheless, the communication latency of most of today's commodity LANs tends to be larger than that of SANs because of their store-and-forward routing strategy. Moreover, today's LANs support the IP protocol, consisting of a lot of layers, which introduce overhead. LASN (Local Area System Network) is a new class of networks which has the advantages of both SANs and LANs. As shown in Fig. 1, an LASN assumes an environment in which personal computers and workstations are distributed within one or more floors of a building (i.e. a LAN environment).

2 RHiNET (RWCP High performance network) is the first network designed with the concepts of an LASN in mind. cables are used for electrical interconnection, the transmission length is limited to about 10 m by electrical circuit drivability). To meet the requirements of RHiNET, RHiNET-2/SW provides large internal memory blocks and supports topology free, reliable, low-latency and highbandwidth communication. To achieve such highspeed optical transmission in RHiNET, 8.8-Gbit/s (800-Mbit/s 11-bit) synchronized parallel optical interconnection is used for each data link. Synchronized parallel optical interconnection allow s high-speed, long-transmission-length, low-latency node-to-node interconnection [9-13]. III. OPTICAL INTERCONNECTION MODULE Fig. 1: Schematic structure of RHiNET II. CONCEPT OF THE RHINET We have developed the firstprototype called RHiNET-1 using 1.33-Gbit/s optical interconnections [7] and second prototype called RHiNET-2 using 8.8-Git/s optical interconnections. In RHiNET-2, PCs are interconnected via highthroughput 8 8 network switches (the RHiNET- 2/SW) [8]. RHiNET-2/SW has eight optical input and eight optical output ports (Fig. 2). Each port has an 8-Gbit/s transmission capacity, and the aggregate throughput is 64 Gbit/s. A 12-bit synchronized parallel optical signal is converted to a 12-bit electrical signal in the optical receiver, switched by the SW-LSI, and re-converted to a 12- bit parallel optical signal in the optical transmitter. The transmission length is limited to 100 m by the skew of the fiber ribbon (however, when copper We use synchronized parallel 12-channel optical transmitter and receiver modules in RHiNET-2/SW [9, 10] (Fig. 3). The optical transmitter modules consist of a 1.3-µm edge-emitting laser-diode (LD) array and a single-mode-fiber (SMF) array. The channel configuration is made up of 11 lowvoltage-differential-signaling (LVDS)/positive-level emitter-coupled-logic (P-ECL) non-return-to-zero (NRZ) data signals and one LVDS/P-ECL clock signal. The P-ECL output signals from the receiver module are converted to LVDS signals with a level converter. The input clock signal is used to latch the 11 data signals in the transmitter and receiver modules in order to eliminate skew caused by the logic LSIs. A transmission length is up to 100 m and total throughput is up to 8.8 Gbit/s/module (800-Mbit/s 11-bit data and 1-bit clock). To achieve high-density implementation with highspeed signaling devices, we have to overcome many complex problems, as such as crosstalk, skew and propagation-loss. optical input 800 Mbits 10 bits/port; with clock & framing 12-channel optical receiver RX 10+2 SW-LSI 8 8 switch Internal memory: 512 kbyte 12-channel optical transmitter TX optical output 800 Mbit/s 10 bits/port; with clock & framing printed circuit board electrical signals (differential) LVDS 800 Mbit/s 10 bits/port; with clock & framing Fig. 2: Schematic structure of RHiNET-2/SW

3 Fig. 3: 12-channel parallel optical interconnection modules [9, 10] IV. SWITCHING LSI A. Overview We developed a 64-Gbit/s/chip high-throughput CMOS switching LSI for the RHiNET-2/SW (Figs. 2, 4 and 5). This switching LSI has eight input and eight output ports. Both input and output ports consist of 10-bit data signals, a clock signal, and a framing signal of 800 Mbit/s. The core switch logic operates at a clock rate of 100 MHz. Therefore, 1:8 demultiplexers are provided at the input ports and 1:8 multiplexers are provided at the output ports bit incoming data are transformed to the 80- bit data by the demultiplexer. ECC decoders and encoders are provided at the input and output ports respectively. The ECC decoder decodes the 80-bit data to a 66-bit data and is handled by the core logic. Input signals synchronized with the transmission clock are retimed to be in-phase with the baseclock (200 MHz) in the elastic buffer. Since the source synchronous clocking is used, an elastic buffer is provided at each input port to compensate the difference between the transmission clock and the baseclock of up to 100-ppm. All electrical I/O interfaces are 2.5-V LVDS- CMOS devices. To achieve high-speed I/O, rise and fall times (< 0.3 ns) and a signal skew (< 0.3 ns) must be very small. We used 0.18-µm technology to fabricate the LSI. The LSI-package is a 784-pin ball grid array (BGA), the pin-pitch of the package is 1.27 mm, and the package size is mm. There are 384 high-speed signal pins (12 bits/port 8 ports 2 pins [differential] for input and output; data rate: 800 Mbit/s/pin). We customized the assignment of the LSI pins to achieve highspeed, low-crosstalk data I/O with a compact, highdensity circuit board. Fig. 4: Block diagram of the switch core in the SW-LSI for RHiNET-2/SW Fig. 5: Floor plan of the SW-LSI B. Switching functions RHiNET-2/SW has the following features (Figs. 4 and 5): 1) Asynchronous wormhole routing Store and forward routing, which is commonly used in conventional LAN switches/routers, yields a large latency. Wormhole routing achieves low latency switching, since a switch can simultaneously transmit the first part of a packet if possible, even while receiving the latter part of the same packet [2]. However, the performance of pure wormhole routing is severely degraded when a message is multicast in a loaded network. To cope with this problem, asynchronous wormhole

4 routing (which provides a certain size of packet buffer) is adopted. 2) No packet discarding The switch never discards packets even when the network is severely congested. 3) In-order delivery The network ensures in-order delivery of packets. 4) Free topology design while avoiding deadlock The switch avoids deadlock by providing a number of VCs (virtual channels) at each input port. By using a different VC as a packet travels through the switches, no cyclic dependency is generated. RHiNET-2/SW has 16 VCs at each input port. This means the diameter of the network can be up to 16. Since each switch has eight ports, this number of VCs provides virtually free topology of the network. 5) Supports up to 100-m links RHiNET-2/SW supports optical links up to100 m long. Since an optical signal propagates in the fiber at the speed of 5 ns/m, the round-trip delay of a 100-m-long optical link is about 1 µs. The handshake logic of the switch also yields some delays. Therefore, handshake will produce up to 1.5-µs delay. When data rate is 8 Gbit/s, 1.5-µs delay corresponds to the time to transfer 1.5 kbytes. Therefore, to receive data without discarding anything, the receiver side should send a handshake message to stop transmission when it does not have enough usable memory space (less than 1.5 kbytes + maximum packet size) in the input buffer. Such a flow-control mechanism is called the slack buffer [7]. In RHiNET-2/SW, each of the 16 VCs of an input port provides a 4-kbytes slack-buffer mechanism. Multiple slack buffers are therefore provided for each input port. 6) Multiple-bit-rate support Each port can be set to the bit rate of 8 Gbit/s, 2 Gbit/s or 1 Gbit/s. The slower bit rates are provided to support slower network interfaces. The maintenance processor sets the bit rate. C. Packets Figure 6 shows the packet format of RHiNET- 2/SW handled in core logic (64-bit data). RHiNET- 2/SW supports variably sized packets. A data packet contains a maximum data size of 2 kbytes. The hop counter is incremented when a packet goes through a switch and is used to detect an irregularly routed packet caused by wrong routing table or damaged header. A handshake packet includes programmable almost full flags of all VCs. A ping/pong packet reports its own logical ID, physical ID, and port ID. Command packets are used to exchange information between maintenance processors of adjacent nodes. They are immediately forwarded to the maintenance processor when received. The payload of the command packet is the message to the maintenance processor. Fig. 6: RHiNET-2/SW packet format D. Routing Routing is done according to the routing information statically stored in the routing table of each switch. Each switch has a routing table with 65,536 entries. An entry is a 9-bit full bitmap of the outputs (8 bits correspond to the output ports, and one bit corresponds to the maintenance processor), and setting multiple bits of an entry provides multicasting. And the routing ID of a header of a data packet is used as an entry id of the routing table. The maintenance processor sets the entries of the routing table. For example, if destination routing is used and there is no multicasting, an entry of the routing table can correspond to a node; thus, a maximum of 65,536 nodes can be supported. E. Maintenance processor and hot-plug support An on-chip maintenance module and an off-chip maintenance processor are provided to configure routing tables and support dynamic link detection. While a link has not been established, RHiNET- 2/SW continuously transmits ping packets. When a switch receives a ping packet, it replies with a pong

5 packet and then the link between the two switches is established. The ping-and-pong packet includes the sender's physical ID (Fig. 6). By receiving a ping or a pong packet, the maintenance processor obtains the physical ID of the switch at the other end of the link. Then, the maintenance processors of the switches exchange the necessary information to set the routing table. RHiNET-2/SW transmits handshaking packets in regular intervals during a link is established. It then detects the link disconnection if it receives no handshake packet for a certain period of time. In such a case, RHiNET- 2/SW starts to transmit ping packets again. V. HIGH-DENSITY IMPLEMENTATION OF HIGH- SPEED SIGNALS In RHiNET-2/SW, to realize high-speed, highdensity integration with the optical interconnection module and SW-LSI, we employ a MULTIWIRE TM * interconnect board (MWB TM ) as a printed circuit board. The MWB TM can achieve high wiring density and superior electrical characteristics (low-loss, high-accuracy 50-ohm impedance, and lowreflection). The MWB TM uses copper wires (0.1 mm diameter) that are coated with polyimide insulation and can therefore be cross wired. This also accounts for the high wiring density (a 0.3 mm wire-pitch can be achieved). Since constant diameter wires are incorporated in the MWB TM s, controlled characteristics impedance (Z 0 : 50 Ω) can be easily realized. Furthermore, by utilizing very thin wires, with adequate spacing, crosstalk, and bending-loss are minimized. We measured the physical characteristics of the MWB TM (propagation-loss and crosstalk). In 150- mm-long straight wires, the 3-dB-down bandwidth was greater than 2.4 GHz, and the crosstalk on the receiver side was less than 1.2% at 900 MHz and a wire-pitch of 0.5 mm. We then optimized the layout of the circuit board based on the experimental results to realize low-crosstalk, high-speed, and high-density electrical I/O [8]. LSI socket. This socket was designed specifically for the high-speed LSI (bandwidth: DC to 6 GHz; path inductance: <1 nh; and capacitance [signal-tosignal]: < 1 pf). Each port has 800-Mbit/s 12-bit optical I/O channels and uses one pair of the 12- channel parallel optical interconnection modules. The board size is mm. The eight pairs of optical transmitter and receiver modules are mounted near the SW-LSI. The daughter board has an H8 microprocessor subboard to control the maintenance-signals of the SW-LSI. A crystal oscillator is mounted to generate the 200-MHz internal clock signal. The structure and layout of the circuit board are optimized for high-speed, high-density implementation [8]. Figure 8 shows a photograph of the RHiNET- 2/SW. These are four sockets of the four-by-twelvechannel fiber adapters. The motherboard is mounted here on the upper layer of the cabinet. The power supply unit and maintenance processorcard are packaged here in the lower layer. (a) SW-LSI mounted side VI. RHINET-2/SW We have produced a prototype of the RHiNET- 2/SW eight-by-eight network switch (Fig. 7). In the center of the board, the SW-LSI is mounted in an *: MULTIWIRE is a trademark owned by ADVANCED INTERCONNECTION TECHNOLOGY, INC. (b) Optical modules mounted side Fig. 7: Layout of the motherboard of the RHiNET-2/SW

6 Fig. 8 Photograph of the cabinet the optical receiver module and reconverted into electrical signals and sent to the error-rate detector (ERD). The fiber runs were 50 m long. Figure 10 shows the eye-pattern of the measured electrical output signal and the waveform of the clock signal. A clear eye-pattern was obtained. The signal rise time (Tr) and fall time (Tf) of the electrical output signal were both less than 400 ps. The jitter was less than 100 ps. We evaluated the reliability of each optical port by measuring the BER. We observed no errors during bit packet data transmission at a data rate of 880 Mbit/s 10 bits. (This corresponds to a BER of less than ) We used a pseudo-random word sequence (PRWS) as a data pattern. These test results show that the reliability of the I/O ports in RHiNET-2/SW is sufficient for RHiNET-2 and that our high-speed and high-density circuit-board layout enables us to construct a high-performance network switch. VII. EVALUATION TEST RESULTS We measured the signal eye-pattern by oscilloscope and bit-error (BER) rate by error-rate detector (the measurement setup is shown in Fig. 8). The 800-Mbit/s 12-bit electrical data signals were generated by the data generator (DG) as a clock signal (CLKI), a framing data signal (AI), and 10-bit packet data signals (DI[9..0]). These 12-bit electrical signals were converted to 12-bit optical signals by the optical transmitter module and transmitted through the 12-channel fiber ribbon. The optical signals were input to an RX-port of RHiNET-2/SW. In RHiNET-2/SW, the 12-bit optical input signals were converted to electrical signals in the RX-port, propagated through the SW-LSI, and reconverted to optical signals in the corresponding TX-port, then transmitted from the TX-port as optical signals. The output signals were received by Fig. 10: Measured eye-pattern of an electrically re-converted 0th data bit [D0] and waveform of the clock signal [CLK] (200 mv/div; 250 ps/div; data rate: 800 Mbit/s). To achieve highly reliable (error-free) parallel interconnection, suppressing skew is the most important improvement that must be made. This is even more important than improving the sensitivity and bandwidth. Our system requirement was that the skew be suppressed to within 20% of the clockcycle. In the case of 800 Mbit/s transmission, the Fig. 9: Experimental setup to measure the signal eye-patterns and BER of RHiNET-2/SW.

7 skew should be suppressed to less than 250 ps. To suppress the skew, we used high-speed LVDS electrical circuits, and precisely controlled the lengthwise placement of the wires. To suppress the skew between data signals, the 800-Mbit/s 11-bit synchronized parallel data signals were retimed with an 800-MHz clock signal using gate-latching in the TX and RX modules and at the TX- and RX-ports of the RHiNET-2/SW. The fiber length was 50 m. We measured the skew of the 10-bit data signal at the two points of the RHiNET-2/SW using a setup shown in Fig. 9. These two points were the electrical output pins of SW-LSI, and the optical output port of the optical transmitter module (Fig. 11). In the input port of the SW-LSI, the skew was eliminated by the gate-latching, but the output signal of the SW-LSI had a 141-ps skew caused by nonuniformity of the LSI output port. We eliminated this skew by using gate-latching in the optical TX module, and the skew of the optical output signal from the output port was 19.4 ps. The maximum skew of our 50-m-long 12-channel fiber ribbon was 50 ps. Thus, after 50-m-long fiber transmission, the worst-case fiber skew is 69.4 ps. Therefore, the skew of the data-signal is sufficiently suppressed by the gate-latching, and thus supports high-speed and highly reliable synchronized parallel data transmission. VIII. RELATED WORKS Myrinet [4] is one of the most popular SAN widely used for cluster computing. Myrinet switches never discard any packets, and provides reasonably high link bandwidth (1.28 Gbit/s) and very low latency. However, Myrinet switches support fewer number of virtual channels. Therefore, network topology restricted so as to avoid deadlock by using carefully selected routing paths. GSN [14], is high bandwidth and low latency interconnect standard, which provides 6.4 Gbit/s link bandwidth of error-free, and flow controlled data. Although it provides four virtual channels for each link, it is difficult to support a deadlock free routing in a free topology because of the channel number limitation. A fat tree is used in the cluster using GSN. Compaq uses the SC interconnect [15] for its inter-server connection. The SC Interconnect consists of a high-bandwidth crossbar switch and a PCI adapter for each node. The detailed architecture of the SC interconnects is not disclosed. However, it also uses a fat tree topology to keep a high degree of bisection bandwidth without deadlock. IX. SUMMARY Fig. 10 Skew of 10-bit data based on the edge of the 0th data bit (in I/O port 4). We have developed the RHiNET-2/SW network for high-performance computing using personal computers distributed in an office or floor environment. Optical interconnection allows highspeed, highly reliable data transmission over a long distance. To achieve high-speed and low-latency node-to-node interconnection, we implemented eight pairs of 8.8-Gbit/s optical interconnection modules and a 64-Gbit/s SW-LSI in a compact circuit board. We have produced an optical interconnection module for RHiNET-2/SW that is capable of speeds of up to 8.8 Gbit/s and a onechip CMOS ASIC switch (784-pin BGA). RHiNET- 2/SW has eight input and eight output optical data ports. The bandwidth of each port is 8 Gbit/s (aggregate throughput of the switch is 64 Gbit/s). We developed a high-speed, high-density implementation technology to overcome electrical problems such as signal propagation-loss and crosstalk. All of the electrical interfaces are composed of high-speed CMOS-LVDS logic. The structure and layout of the circuit board is optimized for high-speed, high-density implementation. Our prototype system achieved 880-Mbit/s 10-bit parallel data transmission. We

8 observed no errors during bit packet data transmission at a data rate of 880 Mbit/s 10 bits with a 50-m fiber. (This corresponds to a BER of less than ) We have thus successfully produced a compact high-throughput optical I/O network switch using a one-chip SW-LSI and eight pairs of optical interconnection modules. This switch enables high-performance parallel computing in a distributed computing environment. ACKNOWLEDGEMENT We are grateful for the assistance and advice of Takahiko Takahashi and Kazuyoshi Satoh of the Device Development Center, Hitachi, Ltd., Atsushi Takai and Atsushi Miura of the Telecommunication and Information Infrastructure Systems Group, Hitachi, Ltd., T. Keicho of Hitachi ULSI Systems Co., Ltd., Y. Keikoin and K. Ohsugi of Hitachi Information Technology Co., Ltd., and M. Tanaka of Hitachi Communication Systems, Inc. REFERENCES [1] T. Kudoh, J. Yamamoto, F. Sudoh, H. Amano, Y. Ishikawa, and M. Sato: "Memory based light weight communication architecture for local area distributed computing'', Innovative architecture for future generation highperformance processors and systems, IEEE Computer Society Press, pp , [2] L.M. Ni, "Should Scalable Parallel Computers Support Efficient Hardware Multicast", Proceeding of 1995 Int'l Conference on Parallel Processing Workshop on Challenges for Parallel Processing, pp. 2-7, August [3] T. Horie, H. Ishihara, T. Shimizu, and M. Ikesaka, "AP1000 Architecture and Performance of LU Decomposition", Proceedings of 1991 Int'l Conference on Parallel Processing, pp , August [4] [5] HIPPI-6400 working drafts, T11.1 maintenance drafts of ANSI NCITS [6] IEEE802.3 Higher Speed Study Group /public/index.html [7] H. Nishi, K. Tasho, T. Kudoh, H. Amano, "RHiNET-1/SW: One-chip switch ASIC for a local area system network", Proc. COOL Chips III, Apr to appear [8] S. Nishimura, T. Kudoh, H. Nishi, K. Harasawa, N. Matsudaira, S. Akutsu, K. Tasyo, and H. Amano, "A network switch using optical interconnection for high performance parallel computing using PCs", pp. 5-12, Anchorage U.S.A., Oct [9] A. Takai, T. Kato, S. Yamashita, S. Hanatani, Y. Motegi, K. Ito, H. Abe, and H. Kodera, "200- Mb/s/ch 100-m Optical Subsystem Interconnections Using 8-Channel 1.3-µm Laser Diode Arrays and Single-Mode Fiber Arrays", J. of Lightwave Technology 12, pp , [10] 4.htm [11] J. W. Goodman F. I. Leonberger, Sun-Yuan Athale, and R. A. Kung, "Optical interconnects for VLSI system", Proceedings of the IEEE 72, pp , July [12] D. A. B. Miller and H. W. Ozaktas, "Limit to the Bit-rate Capacity of Electrical Interconnection from the Aspect Ratio of the System Architecture", Journal of Parallel and Distributed Computing 41, pp , [13] S. Nishimura, H. Inoue, H. Matsuoka, and T. Yokota: "Optical interconnection subsystem used in the RWC-1 massively parallel computer", IEEE Journal of Selected Topics on Quantum Electronics 5, pp , [14] [15] ounce_p3.html