Programmable on-chip and off-chip network architecture on demand for flexible optical intra- Datacenters

Transcription

1 Programmable on-chip and off-chip network architecture on demand for flexible optical intra- Datacenters Bijan Rahimzadeh Rofoee, 1,* Georgios Zervas, 1 Yan Yan, 1 Norberto Amaya, 1 Yixuan Qin, 2 and Dimitra Simeonidou 1 1 High Performance Networks group, University of Bristol, BS8 1TH, UK 2 School of CSEE, University of Essex, CO4 3SQ, UK *Bijan.Rofoee@bristol.ac.uk Abstract: The paper presents a novel network architecture on demand approach using on-chip and-off chip implementations, enabling programmable, highly efficient and transparent networking, well suited for intra-datacenter communications. The implemented FPGA-based adaptable line-card with on-chip design along with an architecture on demand (AoD) based off-chip flexible switching node, deliver single chip dual L2- Packet/L1-time shared optical network (TSON) server Network Interface Cards (NIC) interconnected through transparent AoD based switch. It enables hitless adaptation between Ethernet over wavelength switched network (EoWSON), and TSON based sub-wavelength switching, providing flexible bitrates, while meeting strict bandwidth, QoS requirements. The on and off-chip performance results show high throughput (9.86Ethernet, 8.68Gbps TSON), high QoS, as well as hitless switch-over Optical Society of America OCIS codes: ( ) Networks combinatorial network design; ( ) Fiber optics and optical communications. References and links 1. A. Vahdat, L. Hong, Z. Xiaoxue, and C. Johnson, The emerging optical data center, in conference of Optical Fiber Communication Conference and Exposition (OFC/NOFEC) 2011, pp H. Liu, C. F. Lam, and C. Johnson, Scaling optical interconnects in datacenter networks, opportunities and challenges for WDM. Google Inc, Moumtain view, CA.(2010) L. Peng, C. Qiao, W. Tang, and C. Youn, Cube-Based Intra-Datacenter Networks with LOBS-HC, in international conference on communications (ICC) 2011, pp C. Albrecht, J. Foag, R. Koch, E. Maehle, and T. Pionteck, DynaCORE Dynamically Reconfigurable Coprocessor for Network Processors, (Dynamically Reconfigurable Systems, , 2010). 5. M. Hubner, L. Braun, D. Gohringer, and J. Becker, Run-time reconfigurable adaptive multilayer network-onchip for FPGA-based systems, International Symposium on Parallel and Distributed Processing (IPDPS) pp G. S. Zervas, J. Triay, N. Amaya, Y. Qin, C. Cervelló-Pastor, and D. Simeonidou, Time Shared Optical Network (TSON): a novel metro architecture for flexible multi-granular services, Opt. Express 19(26), B509 B514 (2011). 7. N. Amaya, G. S. Zervas, B. R. Rofoee, M. Irfan, Y. Qin, and D. Simeonidou, Field trial of a 1.5 Tb/s adaptive and gridless OXC supporting elastic 1000-fold bandwidth granularity, in European Conference and Exhibition on Optical Communication (ECOC), 2011, pp Introduction Intra-datacenter [1,2] network interconnection provide high data rate lines preferably using optical fibres among the servers to enable bandwidth aggressive applications with tight QoS requirements of parallel and distributed computing in massive scales. The requirement of highly connected servers is approached by deploying hierarchies of L2/L3 switches in (C) 2013 OSA 11 March 2013 / Vol. 21, No. 5 / OPTICS EXPRESS 5475

2 different levels within and between datacenter racks and clusters [1] with not much concern about resource/spectral efficiency [2]. This is whilst optical networking, specifically sub wavelength networking, supporting efficient optical networking for wide range of bitrates, is an candidate solution for complementing the highly electrical switching architecture of intra data center networks today [3]. In the meantime, the semiconductor industry with innovative FPGA based hardware solutions with increased logical gates and processing power units enable implementation of dynamic and programmable run time reconfiguration of the FPGA boards [4]. The NoC based run time reconfiguration approach can introduce new levels of flexibility to the network architecture, by adapting to different functionalities, through reconfiguration of the on chip components and implemented Intellectual Property (IP) cores connections [5]. In this paper, we propose a novel programmable and reconfigurable network on-chip andoff-chip design and implementation, which enables operation in bitrate ranges of 100Mbps up to ~8.7Gbps using wavelength, and sub-wavelength based technologies, improving networks performance with transparent, highly resource efficient, and ultra low latency communications, well matching with intra datacenter strict requirements. This work is based on open-hardware infrastructure that allows for on-demand programmability that in turn delivers flexibility, agility, efficiency as well as high network performance and high QoS delivery. This design implements FPGA based network line card with runtime reconfigurable on-chip design supporting 10GEthernet and TSON [6] Tx and Rx. The line-card operation of Ethernet or TSON is supported by the flexible and AoD [7] optical transport layer built as offchip network. It supports guaranteed transport of high capacity (9.8 Gbps per 10G NIC) and ultra-low latency (<10 µs) services on EoWSON or slightly lower capacity (8.68 Gbps) yet statistically multiplexed optical sub-wavelength services over TSON. The hardware infrastructure delivers hitless switch-over between the two offered services on a dynamic fashion. Fig. 1. legacy ToR in Fat-Tree intra-datacenter network architecture. Transparent network on-chip and off-chip to replace the legacy architecture. (c) Network off-chip where optical components are selected for Ethernet or TSON connectivity paradigms. (d) FPGA based Network on-chip implementation with modules attached to an electronic back plane. 2. Reconfigurable network of on-chip and off-chip design The novel concept of the Reconfigurable Network of On-chip and Off-Chip design is displayed in Fig. 1. (C) 2013 OSA 11 March 2013 / Vol. 21, No. 5 / OPTICS EXPRESS 5476

3 Fig. 2. network off-chip for optical transport of broadcast and select delivering EoWSON network on-chip for building TSON and Ethernet interfaces (c) network off-chip for TSON operation The proposed solution presumes installing specially designed on-chip network line cards capable of Ethernet and TSON (optical sub-wavelengths) on the servers (or PCI-E to Ethernet and TSON), and replacement of a typical Ethernet Top of the Rack switch with a more flexible all optical switch supporting sub-wavelength switching for TSON and wavelength switching for Ethernet transport, which is built as Network-off-Chip solution (Fig. 1). The Network-on-Chip design is implemented using a High Performance Xilinx Virtex6 HXT FPGA, which operates as a bi-functional line-card supporting 10GEthernet and 10G TSON transport interfaces with 9.8 Gbps and 8.6 Gbps throughput respectively. It enables hitless switching between Ethernet and TSON operation, so the user/application or the server itself can deploy either upon request with no interruption (no packet loss) to the running services. This on-chip design exploits partial reconfigurability of the FPGA chip whilst running aided by MIMO buffers to achieve no packet loss operations. Using TSON the network interface card can optically transmit sub-wavelength data employing time-shared statistical multiplexing; it can support as low as 100Mbps with equal step size over 10Gbps links, offering highly efficient traffic aggregation. Using Ethernet transmission, the server will have a dedicated wavelength to transport its data over wavelength switched optical network (WSON). The TSON configuration allows shared use of on-chip and off-chip resources enabling multiple accesses of several servers over each wavelength channel. The EoWSON allows the Network-on-Chip to operate at maximum throughput and transfer bulk data on a single location per channel. Bit Rate Throughput Comparison 1500B 64B Ethernet Theoretical 100 Ethernet Experimental TSON Experimental 1 Latency(us) Ethernet vs TSON Latency results 1G 3G 5G 7G 9G Ethernet throughoput (bps) Ethernet Node: Pacekt Size 64B Ethernet Node: Packet Size 1500B TSON Node: Packet Size 64B TSON Node: Packet Size 1500B Fig. 3. FPGA-based TSON-Ethernet switching NoC results. Throughput. Ethernet/TSON latency. On the other hand, the Network-off-Chip design provides flexible optical node and network architecture, following the AoD approach. In AoD, an optical backplane, e.g. 3D (C) 2013 OSA 11 March 2013 / Vol. 21, No. 5 / OPTICS EXPRESS 5477

4 MEMS switch, interconnects a pool of components and enables reconfiguration of the transport layer by configuring internal cross-connections. The coordinated operation of the on and off chip networking enables bandwidth flexible, efficient and transparent cross connection within datacenters. Network On-chip design: Network-on-chip design is an approach to design the communication subsystem between sub-modules, so to achieve desired performance and functionality. Several sub-systems (on-chip components, IP cores, etc) for separate functions of Ethernet RX, TSON RX, Ethernet TX and TSON TX have been developed in a modular manner. Using a deployed internal electronic backplane switch, each of the implemented subsystems can be selected using the internal communication network to enable the preferred service and mode of operation. Different sub modules of the Network-on-chip are displayed in Fig. 1. The same fig illustrates how with a select-and-place approach using different subsystems of TSON TX and RX, Ethernet MAC and buffers, the line card is able to operate as Ethernet to TSON or Ethernet to Ethernet interface. The implemented on-chip design allows a hitless switch over between TSON and Ethernet modules, by incorporating Ethernet or TSON related blocks in the operation. The electronic switch-over is dynamically controlled 7.2 by a server that is able to update the FPGA mode (ETH or TSON) LUT on-demand. The selection of any of the combinations, for example Ethernet-TSON-Ethernet, is supported by the reconfiguration of the transport plane as well. Fig. 4. FPGA-based TSON-Ethernet NoC results. switch-over. Tx/Rx frames. Off-chip network design: The off-chip networking between different FPGA based line cards happens through an AoD node [7] which is able to implement the required optical node architecture for EoWSON or to re-configure it to support TSON aggregation and switching in 20 msec (optical backplane reconfiguration time). For transparent transport of Ethernet channels, wavelength switching can be adopted using broadcast and select topology by choosing couplers and AWG components and the back plane in-between as in Fig. 2. For TSON operations fast optical switches (PLZT) need to be involved, as shown in Fig. 2(c), in order to multiplex time-slice data sets from different servers and map them on specific time slices over one or few wavelengths. Fibre switching is also available by using the ports on the back plane MEMS switch. 3. Results The FPGA-based Network-on-chip experimental performance carried out using a 10G data analyzer for applying different data rates to the FPGA, reaching the maximum throughput of the implemented design. Figure 3 shows the maximum theoretical and achieved experimental Ethernet and sub-wavelength TSON throughput. Fig. 5. 1Gbps slice allocation patterns. P1: contiguous allocation. P2: distributed allocation. (C) 2013 OSA 11 March 2013 / Vol. 21, No. 5 / OPTICS EXPRESS 5478

5 TSON delivers 87.96% of the maximum measured Ethernet throughput (9.868 Gbps) for 1500 Byte Ethernet frame size, and 94.38% of maximum 64 Byte Ethernet throughputs (7.69 Gbps). Figure 3 shows a comparison of the measured ultra-low latency results for Ethernet (<6.7 µs) and very-low latency for TSON (<160.6 µs). TSON latency decreases with higher bitrates as the data aggregation and time-slice TX take less time. Latency(us) Byte packet Latency results 1G 2G 3G 4G 5G 6G 7G Ethernet Bitrate (bps) P2 P1 Latency(us) Byte packet Latency results G 2G 3G 4G 5G 6G 7G 8G Ethernet Bitrate (bps) P2 P1 Fig. 6. FPGA-based TSON latency for different time slice allocation. for 64 Byte Ethernet frame size, for 1500 Byte Ethernet frame size. Figure 4 shows an inside FPGA TSON to/from Ethernet switch-over which is triggered via an Ethernet control interface. At point A the operation mode is switched from TSON to Ethernet, and at point B it is switched from Ethernet to TSON. Figure 4, shows the measured throughput result when switching between TSON and Ethernet (Input frame size 64B). When switching from TSON to Ethernet, the bit rate increases for a short duration because the data inside the TSON aggregation FIFO are released immediately. Also, when switching from Ethernet to TSON, the bit rate decreases for a short duration, as it is necessary to wait for at least one time-slice to aggregate the data in burst and then transmitting it. The delay in switching from TSON to Ethernet can last as long as it takes to empty the transmission buffer, which can get as large as 5 time slices (8x1500 Byte packets ~50μs, with regards to the available on-chip memory (524Kb) and 1ms frame size in TSON). The switchover from Ethernet to TSON is almost immediate since the last Ethernet packet is released right away. Figure 4 shows that the number of received frames matches the number of transmitted frames, which proves the hitless feature of the design. Since the TSON design is based on process of aggregating Ethernet packets in an optical burst, and transmitting on preallocated time-slices the pattern of the allocation can cause different delays of optical bursts in TX FIFOs. Figure 5, shows two sample time-slice allocation patterns: pattern 1 (P1) with contiguous allocation and pattern 2 (P2) with distributed slice allocation ( 1 for allocated and 0 for not allocated). To reach 1Gbps of bitrate, P1 uses the first 11 time- out of the available 91 slices (includes the overheads), whilst P2 uses distributed 11 time-slices across the 91 time-slices (as shown on the first row of either pattern). The next rows with allocations in each pattern show how the rest of the time-slices are allocated to reach higher data rates. Percentage above 1us 0.10% 0.08% 0.06% 0.04% 0.02% 0.00% 64B PDV (P1) 1G 2G 3G 4G 5G 6G Ethernet Bit rate bps 11us- 12us 22us- 23us 30us- 100us Percentage above 1us 0.70% 0.60% 0.50% 0.40% 0.30% 0.20% 0.10% 0.00% 64B PDV (P2) 1G 2G 3G 4G 5G 6G Ethernet Bitrate bps 11us-12us 22us-23us 30us-100us Fig. 7. FPGA-based TSON jitter for 64 Byte Ethernet frames with different time slice allocation patterns. time slice allocation pattern 1, time slice allocation pattern 2. Using P1 and P2, we have benchmarked the latency and jitter of TSON which utilizes the aggregation mechanism. The effect of P1 and P2 with min and max Ethernet frame sizes of (64 and 1500Byte) on the latency is shown in Fig. 6. P2 shows expectedly less latency in comparison to P1 in both and since it uses more distributed available time-slices in (C) 2013 OSA 11 March 2013 / Vol. 21, No. 5 / OPTICS EXPRESS 5479

6 each frame whilst P1 only offers time slices at the beginning of each frame (arriving packets in the middle of the frame should wait). With the increase in the data rate, the buffering and aggregation of the packets and therefore the release of them takes less and less time, so the latency of the implemented TSON designs which uses the aggregation mechanism for higher bitrates drops. The experimental jitter results for TSON under different allocation patterns and frame size are shown in Fig. 7, and Fig. 8. In Fig. 7, we have jitter for the percentage of minimum Ethernet frames size, received in the time intervals above 1μs, as displayed in the legend. It can be seen from the bars how in both Fig with the increase of bitrate, the frames experience less jitter due to the faster aggregation process(at >7Gbps there are no green bars of μs(less jitter) in P2 and P1, and less stretches of red bars in P2 in higher bitrates). Starting with roughly more than 0.60% of traffic at 1Gbps experiencing high jitter values of μs in P2 as an impact of applying more random time slice allocation, P1 introduces low jitter values for <0.10% of ingress traffic. In Fig. 8, we can see the impact of different patterns on the measured jitter for maximum Ethernet size traffic. This fig shows the distribution of the jitter bars along different bitrates for packets arrived with jitters values above 2μs. It can be seen (especially more distinguishable for lower bit rates) that P2 (Fig. 8) shows higher jitter values combining its jitter bars at each bit rate value. Comparing Fig. 7, and Fig. 8, more than 99% of 64 Byte packets are received <1μs, whilst about 87% of 1500 Byte packets are received <2μs. The greater delay variation for the 1500byte packet size has to do with the fixed size FIFOs in TX, which is limited to maximum 8 packets of 1500Byte length, so it is very probable that more packets are deferred to the next time slice for transmission, which can cause higher delay variations compared to smaller packets. Percentage above 2us 20% 10% 0% 1500B PDV (P1) 1G 2G 3G 4G 5G 6G 7G 8G Ethernet Bit rate (bps) 2us-3us 13us-14us 24us-25us above 30us Percentage above 2us 20% 10% 0% 1500B PDV (P2) 1G 2G 3G 4G 5G 6G 7G 8G Ethernet Bit rate (bps) 2us-3us 13us-14us 24us-25us above 30us Fig. 8. FPGA-based TSON jitter for 64 Byte Ethernet frames with different time slice allocation patterns. time slice allocation patterns 1, time slice allocation patterns Conclusions This paper presents the novel on-chip and-off chip network design and implementation as programmable and modular open-hardware solution for fast and flexible bitrate switching solution suitable for intra-datacenter optical networks. The proposed approach uses programmable line-cards capable of operating in 10GEthernet or in TSON sub-wavelength mode supported by a flexible optical switching/transport node based on AoD to perform wavelength switching (for Ethernet) or fast time-sliced wavelength switching (for TSON). Experimental results demonstrate hitless switch-over operation between Ethernet and TSON, very high throughput for TSON (8.68Gbps % of maximum Ethernet), ultra low latency (<6.7 µs for Ethernet and <160.6 µs for TSON) and <2 μs jitter for 87% of the traffic. We have also studied the QoS performance of the proposed design under contiguous or distributed allocation of network time-sliced resources for TSON sub wavelength switching. Acknowledgments This work is supported by the EC through IST STREP project MAINS (INFSO-ICT ) and PIANO + ADDONAS, as well as EPSRC grant EP/I01196X: Transforming the Future Internet: The Photonics Hyperhighway. (C) 2013 OSA 11 March 2013 / Vol. 21, No. 5 / OPTICS EXPRESS 5480