Optics in Data Center: Improving Scalability and Energy Efficiency

Transcription

1 Optics in Data Center: Improving Scalability and Energy Efficiency I. Cerutti, N. Andriolli, P. G. Raponi, P. Castoldi Scuola Superiore Sant Anna, Pisa, Italy O. Liboiron-Ladouceur McGill University, Montreal, Quebec Abstract Interconnection networks within modern data centers suffer from bandwidth and scalability limitations, and important power consumption. Optical switching solutions can help to overcome such issues when properly designed and optimized. his paper discusses three areas of improvement beyond state of the art: i) the use of more energy-efficient optical devices to realize high-capacity and energy-efficient optical space switches; ii) the enhancement of their scalability by combining space switching with time switching into a space-time interconnection network architecture (SIA); iii) the planning of an energyefficient SIA-based network meeting data center bisection bandwidth requirements. Performance assessment shows that i) by combining different switching devices in the same space switch, up to 0% of the power can be saved with respect to conventional SOA-based space switches; ii) SIA allows scalability to increase by a factor of, thanks to the combined use of space and time switching domains, while limiting the energy per bit; iii) SIA allows the realization of data center topologies, such as folded Clos and flattened butterfly, whose optimized design can lead to power saving up to 0% with respect to an energy unaware design. I. INRODUCION he emerging issue in data centers concerns the interconnection network performance. More specifically, the limited capacity of today s electronic networks might cause throughput/latency issues for some applications. Also, power consumption is relevant and is expected to further increase with the capacity []. Finally, scaling of the network with the number of servers is hindered by the non-linear relationship (up to quadratic) between the number of switching elements and the number of servers, leading to a significant increase in cost, power consumption and capacity requirements []. his paper aims to tackle the challenges of capacity, energy efficiency, and scalability from the single switching element to the whole data center network. In particular, the areas requiring improvements are: high-capacity and energy-efficient space switches, scalable switching architectures, and energy-efficient design of data center networks. he fundamental technology to achieve such objectives is optics, which enables switches with high throughput and low power consumption [], []. However, progress beyond state of the art is required to enhance the current switching solutions with higher capacity, energy efficiency, and scalability. Now at NetSpeed Systems, 0 Seely avenue, San Jose, CA. his work was supported by MIUR through FIRB project MINOS, MAE and MFE through the high-relevance bilateral projects NANO-RODIN and DEMO-RODIN, and the Canada Research Chairs program. his paper summarizes the improvements beyond state of the art achieved by i) optimizing the design of a single optical space switch, ii) increasing the scalability of the optical switches, and iii) planning the whole network by connecting multiple switches, while ensuring the required level of capacity and minimizing the power consumption. Each area of improvement is presented and analytically assessed. More specifically, energy-efficient optical devices (such as interferometers) are exploited as switching gates along with conventional gates based on semiconductor optical amplifiers (SOA), for realizing energy-efficient optical space switches, offering large bandwidth and fast switching times. Placement of interferometers in such heterogeneous optical switches needs to be optimized, while accounting for physical layer performance. Also, along with the space domain a second switching domain (i.e., time) is exploited to increase the scalability beyond the existing solutions. he resulting switching architecture named spacetime interconnection architecture (SIA) is assessed for energy efficiency and scalability. Finally, the whole network connecting the numerous servers is designed with the required level of bisection bandwidth and assessed in order to find the most energy-efficient combination. II. SAE OF HE AR Optical space switches are used as switching fabrics connecting multiple inputs to multiple outputs []. hey exploit the space domain to route packets through spatially disjoint paths. Different architectures have been proposed, differing in the way paths are arranged. An M M Spanke architecture requires M switches M, each of them connected to M couplers M :, allowing a trivial routing between input and output ports. Multi-stage architectures are a wide class of architectures realized by cascading elementary switching elements. For instance, an M M Beneš architecture exploits switching blocks arranged in ( log M ) stages, with each stage composed of M/ switching blocks. Multi-stage architectures allow for a strong reduction in the required number of elements, and thus are more suitable for implementing large size space switches []. However, compared to Spanke, more complex routing policies must be implemented. o implement these switching architectures in optics, different switching elements can be exploited. Optical space switches are typically implemented with a single type of elements, such as semiconductor optical amplifiers (SOA) [], [], or interferometer-based devices (e.g., Mach-Zehnder inter-

2 ferometers or micro-rings). SOAs are mature components that can mitigate impairments due to in-band crosstalk thanks to their high extinction ratio, fast switching time, and capability to act both as a switch and as an amplifier. On the other hand, interferometer-based devices are more energy-efficient, but suffer from loss and crosstalk. A -port switch prototype based on a Spanke SOA-based architecture is presented in []. Despite the large number of active components, the power consumption and the cabling are demonstrated to significantly decrease compared to all-electrical switches. A integrated multi-stage architecture is presented in [], stemming from switches requiring just two SOAs. A optical switching fabric realized with Mach-Zehnder interferometerbased switches is shown in []. A hybrid InP/Silicon Beneš architecture in [] uses MZI as switches and SOA as amplifiers only. o further scale the switch size, other optical switching techniques must be pursued in combination with space switching. Optics offer additional degrees of freedom for switching beyond space: wavelength and time are the domains that can be efficiently exploited for optical switching to enable higher scalability. he resulting switching architectures are thus multi-domain and are referred to as multi-domain optical interconnection networks. Multi-domain interconnection networks based on space and wavelength domains have been designed and realized by different research groups [], [] []. he solutions presented in [], [] exploit space and wavelength routing through the use of Arrayed Waveguide Grating Router (AWGR) and tunable wavelength converters. An optical broadcast-and-select interconnection network based on wavelength and space switching is proposed in OSMO- SIS []. Further, the space wavelength (SW) architecture [] is a modular interconnection network, which exploits the space domain to address the destination card and the wavelength domain to address the destination port in a card on a perpacket basis. o realize a network for data center, it is necessary to interconnect a very large number of switching devices with the tens of thousands servers, according to a topology able to offer the desired bisectional bandwidth within a limited number of hops []. raditional types of topologies suitable for data centers include folded Clos, butterfly (and its variations), and trees. A folded Clos topology is a multi-stage topology able to provide path diversity and full bisection bandwidth, at the cost of an increased latency compared to other traditional topologies. o reduce the hop count and the latency, topologies with high port count switches, such as the flattened butterfly, can be exploited, reducing the wiring of folded Clos while achieving better performance and path diversity than a conventional butterfly []. It is derived from a butterfly topology combining multiple switching nodes into a single one to reduce the overall number of elements. Due to the complexity of all-optical solutions for data center applications, several hybrid solutions have been considered in the literature [], where the optical network provides high bandwidth but slowly reconfigurable connections, and the electrical network provides flexibility. III. HE HEEROGENEOUS IMPLEMENAION OF OPICAL SPACE SWICHES Energy-efficient optical space switches can be realized by resorting to both interferometric switching elements (such as MZI) and SOAs. Such implementation is here defined as a heterogeneous implementation, to distinguish it from the conventional homogeneous implementation based on a single type of switching element. Indeed, when SOAs are replaced by MZI, a heterogeneous implementation consumes less power than a homogeneous implementation based solely on SOAs. At the same time, SOA amplification can compensate the power losses introduced by the interferometric devices, enabling scalability to higher port count compared to a homogeneous implementation based solely on MZI. A thorough physical layer analysis is then required for a proper design. he heterogeneous implementation of an optical space switch with M input and M output ports interconnected by a non-blocking architecture can be realized as follows for Spanke and Beneš architectures []. For realizing an energyefficient space switch, SOAs are inserted only when strictly necessary for compensating for power loss. In particular, an SOA is put on the routing path after every s-th passive elements. he periodic placement of SOAs differs in Spanke (s Sp ) and multi-stage architectures (s MS ), due to the different devices in the routing path traversed by the optical signal. Finally, energy efficiency is further improved by enabling SOAs only when necessary (i.e., when an optical packet is gated), through a self-enabling mechanism or switch control. In this way, the overall power consumption of the optical switch is more proportional to the network load, i.e., the average packet arrival rate per time slot. he physical layer performance of the heterogeneous and homogeneous SOA-based implementations is assessed, considering -wavelength signals at Gb/s [], []. Fig. refers to Spanke architecture, showing the bit error rate (BER) of the worst-case channel of both heterogeneous and homogeneous SOA-based implementations. Fig. a shows the BER as a function of the received optical power, for different port counts M. he error floor exhibited by the heterogeneous implementation is due to the crosstalk introduced by the MZI gates, whose extinction ratio (ER) is lower than in SOAs. In particular, a realistic ER of db for the MZI gates, compatible with today s technology, is considered [], allowing a scalability up to more than ports with the BER floor well below. Note that the BER results are intentionally shown at very low values to highlight the noise floor change caused by the extinction ratio. he figure also shows a low dependence of the BER performance on the switch size M. At BER =, the power penalty for ports is. db over ports. he effect of crosstalk can be better understood from Fig. b, showing the BER for different MZI ERs at a fixed number of input/output ports M =. An improvement of db in the ER reduces the required received optical power by. db at a BER of. More importantly, this improvement reduces the error floor. Hence, the maximum size of an heterogeneous

3 -Log(BER) Received Optical Power [dbm] M= SOA-based M=0 SOA-based M= Heterogeneous M=0 Heterogeneous M=0 Heterogeneous M= Heterogeneous (a) (b) Fig.. BER vs received optical power comparison for homogeneous and heterogeneous Spanke architectures for fixed extinction ratio ER = db and different port number M (a) and for M = and different ER (b). -Log(BER) ER=-dB, s MS = ER=-dB, s MS = ER=-0dB, s MS = ER=-0dB, s MS = ER=-dB, s MS = ER=-dB, s MS = 0 Number of ports Fig.. Heterogeneous Beneš architecture: BER vs. port number M at a received optical power of - dbm for different values of ER and s MS. switch with Spanke architecture is mainly limited by the ER of the MZI gates, a trade-off over energy efficiency. Beneš architecture is considered in Fig., which shows the BER as a function of the space switch size M, for different crosstalk values and spacings. he heterogeneous implementation of multi-stage architectures is realized with MZI switching elements and SOA switching elements. Differently from the Spanke architecture, the in-band interference due to the crosstalk introduced by the MZI cannot be blocked and affects the signal along the entire path in the space switch. hus, in-band crosstalk can strongly impair the quality of the signal after crossing few switching blocks, hence multistage architectures are more heavily impacted by MZI ER compared to Spanke, and can scale to lower dimensions for a fixed ER. By comparing the different curves, it can be noticed that the BER degrades more when increasing s MS rather than the port count. With ER = db (considering a dilated configuration, where the same MZI-based gates considered in Fig. a are duplicated to improve the ER, at the expense of twice the loss), simulations show that to ensure a BER < Beneš architecture can support a maximum number of input/output ports M equal to 0 and for s MS =and s MS =, respectively, whereas Spanke with s Sp =can scale up to. -Log(BER) Received Optical Power [dbm] SOA-based ER=-dB Heterogeneous ER=-dB Heterogeneous ER=-dB Heterogeneous ER=-dB Heterogeneous ABLE I POWER CONSUMPION SAVINGS OF DIFFEREN SPACE SWICHES COMPARED O HOMOGENEOUS SOA-BASED SPANKE SWICH. Size (M) 0 0 Spanke het. 0.%.%.%.%.% Beneš hom..%.%.%.0%.% Beneš het., s MS =.%.% 0.%.% Beneš het., s MS =.% he power savings of the different implementations and spacings of Spanke and Beneš space switches normalized to a homogeneous SOA-based Spanke architecture are compared in able I. Given an architecture, substituting power-hungry SOAs with more energy efficient MZI-based interferometers allows the reduction of the power consumption of the space switch. he number of gating elements required in Beneš architecture grows more slowly than in Spanke, hence the difference in power consumption between these two architectures []. Heterogeneous Beneš saves more than 0% of the power, but the crosstalk limits its maximum scalability to M = 0 (s MS =) and M = (s MS =). If the technological advancements enable fabrication of interferometers with ER lower than db [], heterogeneous space switch architectures with higher spacing can be achieved, leading to potentially greater energy saving. IV. INERCONNECION NEWORK SCALING HROUGH MULI-DOMAIN SPACE-IME SWICHING An optical multi-domain interconnection network that exploits both space and time domains is reported here and is referred to as space-time interconnection architecture (SIA) []. SIA, more scalable and energy-efficient than the space-wavelength architecture [], consists of M cards, each one with N input and output ports (Fig. a). o route optical packets from any input port to any output port, the selection of the output card is performed in the space domain (i.e., using a space switch), whereas the selection of the output port on the selected card is performed in the time domain. Space switching is achieved by making use of an M M optical space switch presented in Sec. III. ime

4 i o ime gated by SOA Broadband E/O Passive Wavelength-striped Mapping (PWM) ime-frame ime-slot Packet Serial Packet Packets in MxM Port out, Port out,n Port out, Port out,n M M i M Serial Packet ime-slot o M Broadband O/E Passive Wavelength-striped Mapping (Reversed delay) ime-slot Packet ime gated by SOA (a) SIA architecture with M cards and an M M space switch. (b) ime-compressed WDM packets in a card through passive wavelengthstriped mapping. Fig.. Architecture and packet transmission in Space-ime Interconnection Architecture (SIA) []. switching is achieved by transmitting the optical packet in a pre-determined time-slot within a time-frame. In addition, the wavelength domain is exploited to achieve high throughput by encoding packets on multiple wavelengths (also referred to as WDM packets), as shown in Fig. b. he WDM packets are generated from serial electrical packets of duration : the bits of the electronic packets simultaneously modulate a comb of N optical channels with a single broadband modulator. hen, a passive wavelength-striped mapping (PWM) element delays each modulated channel by /N from each other and the delayed channels are gated in time with an SOA to generate a WDM packet of duration /N []. he serial packet is therefore compressed in time by the number of channels, N, equal to the number of ports. he generated WDM packets are transmitted in time-slots of duration /N and time-multiplexed at the card in a time-frame of duration, by an N :coupler. Each time-slot of a time-frame is assigned to a specific output port of the card. he multiplexed WDM packets are then sent to the M M space switch. After crossing the space switch, the multiplexed WDM packets pass through a : N splitter that broadcasts the routed packets to each port of the card, where an SOA gate selects the corresponding time-slot. he PWM finally appropriately delays each channel and converts the WDM packet into a serial packet through a broadband optical receiver. he scalability and the energy per bit of a SIA versus the total number of ports MN is shown in Fig. a for a utilization of 0%. For the first time, both homogeneous and heterogeneous implementations with Spanke or Beneš architectures have been compared (the parameters are consistent with the simulations in Fig. ). With a Spanke space switch, the power consumption grows significantly with the port count, due to the quadratic relation between switching elements and ports. On the other hand, with a Beneš space switch, the number of elements grows more slowly, but the scalability is limited by physical layer impairments at ports (for a spacing s MS =) or 0 (for s MS =). For less than 0 ports, the advantage of using a Beneš heterogeneous switch over a homogeneous Spanke in the SIA is limited to a saving of up to % in energy efficiency, but savings scale quadratically for higher port count up to MN =. Moreover the scalability is N-time fold the scalability of the corresponding space switch alone. V. ENERGY EFFICIEN DAA CENER NEWORKS he presented SIA is suitable for realizing hybrid data center networks. Servers connected to the same SIA can exploit the full bandwidth of the SIA to communicate among them. However, the communication between servers connected to different SIAs must go through multiple hops exploiting optical-electronic-optical (O/E/O) conversion to overcome synchronization and buffering issues. wo widely utilized data center topologies considered, namely folded Clos and flattened butterfly []. Each node of the topology is a SIA, and server ports are directly connected to level 0 SIA cards. SIA-based folded Clos and flattened butterfly topologies are compared in terms of energy consumption in Fig. b in a data center interconnecting servers. All SIAs utilized in the network are assumed to be identical: Spanke and Beneš space switches with homogeneous and heterogeneous implementations have been compared for the first time, using two different spacing s MS for heterogenous Beneš. he energy consumption is shown as a function of the total number of ports MN at a utilization of 0%. For a fair comparison of the different topologies, the power consumption is normalized to the bisection bandwidth. he figure shows that for very low port count, folded Clos is more energy efficient than flattened butterfly, while the inverse happens for and ports. For larger networks, the energy per bit of folded Clos is again lower. he highest energy efficiency is reached for ports, a limit case in which a single SIA connects all servers, and both the topologies collapse into a single switch. Excluding this case, the most energy efficient topology is folded Clos for 0 and 0 ports. he utilization of

5 Energy per bit [pj/bit] Spanke homog. Benes homog. Spanke heterog. Benes heterog. s MS = Benes heterog. s MS = 0 0 Number of ports (MN) Energy per server [pj/bit] 00 Flat bfly Spanke hom. Folded Clos Spanke het. Folded Clos Spanke hom. Flat bfly Benes het. s MS = Flat bfly Benes hom. Folded Clos Benes het. s MS = Folded Clos Benes hom. Flat bfly Benes het. s MS = Flat bfly Spanke het. Folded Clos Benes het. s MS = 0 0 Number of ports (MN) (a) Energy per bit for a fully equipped SIA. (b) Energy per bisection bandwidth per server for different data center topologies. Fig.. Energy per bit (pj/bit) vs. total number of ports, for different space switch implementations at 0% utilization. Beneš switch inside the SIA keeps the energy per bisection bandwidth per server almost constant with the number of ports instead of a quadratic increase as experienced when the Spanke switch is used. Moreover the use of the heterogeneous architecture allows for additional power saving for any port number, but may restrict the scalability with the number of ports as discussed in Sec. III. VI. CONCLUSIONS In this paper three areas of improvement were identified where optical technologies could overcome the challenges of today s interconnection networks, namely capacity, scalability, and power consumption. Optical space switches, exemplified by Spanke and Beneš architectures, can be implemented with an heterogeneous approach, i.e., resorting to interferometric devices such as MZIs, to alleviate the power consumption problem, at the expense of scalability constraint given by the limited extinction ratio. o scale beyond the physical layer limitations of space switches alone, multiple switching domains can be exploited in a multidomain interconnection network. he space-time interconnection architecture (SIA) is shown to scale the number of ports as a function of the number of wavelengths, while keeping the energy consumption per bit limited. hus it has the potential of becoming a candidate for data center networks, thanks to its modularity, scalability, and energy efficiency. Finally, the whole data center topology can be carefully optimized by selecting the most suitable topology and the SIA size that leads to the most energy efficient network, while ensuring sufficient bandwidth for server communication. Folded Clos offers full bisection bandwidth, with a better power consumption per server per bisection bandwidth when increasing the SIA size. On the other hand, flattened butterfly has slightly better energy efficiency for small port sizes. However, the offered bisection bandwidth is limited to 0% of the full bandwidth, and more complex routing strategies are also required. he results indicate that high scalability of SIA is typically beneficial from the bisection bandwidth and energy efficiency standpoint, provided that the internal space switch is optimized (e.g., by using a more Beneš instead of Spanke architecture). High port count SIAs and heterogeneous implementation of the space switch allow energy efficiency independently from the selected topology. REFERENCES [] D. Abts et al., Energy proportional datacenter networks, in Int. Symp. on Computer architecture (ISCA), 0, pp.. [] O. Liboiron-Ladouceur, I. Cerutti, P. G. Raponi, N. Andriolli, and P. Castoldi, Energy-efficient design of a scalable optical multiplane interconnection architecture, IEEE J. Sel. opics Quantum Electron., vol., no., pp., Mar./Apr. 0. [] C. Kachris and I. omkos, A survey on optical interconnects for data centers, IEEE Comm. Surveys ut., vol., no., pp., 0. [] W. Dally and B. owles, Principles and Practices of Interconnection Networks. Morgan Kaufmann, 00. [] P. G. Raponi, N. Andriolli, I. Cerutti, D. orres, O. Liboiron-Ladouceur, and P. Castoldi, Heterogeneous optical space switches for scalable and energy-efficient data centers, JL, vol., no., pp., 0. [] A. Wonfor, H. Wang, R. V. Penty, and I. H. White, Large port count high-speed optical switch fabric for use within datacenters, J. Optical Commun. and Netw., vol., no., pp. A A, Aug. 0. [] H. Onaka et al., WDM Optical Packet Interconnection using Multi- Gate SOA Switch Architecture for Peta-Flops Ultra-High-Performance Computing Systems, in Proc. ECOC, Sep. 00. [] A. Albores-Mejia et al., Monolithic multistage optoelectronic switch circuit routing 0 Gb/s line-rate data, JL, vol., no. 0, 0. [] Y. Ueda et al., InAlGaAs/InAlAs Optical-Switch Fabric by Cascading Mach-Zehnder Interferometer-ype Optical Switches With Low-Power and Low-Polarization-Dependent Operation, IEEE Photonics echnology Letters, vol., no., pp., May 0. [] L. Chen, E. Hall, L. heogarajan, and J. Bowers, Photonic switching for data center applications, IEEE Photonics Journal, vol., no., pp., Oct. 0. [] X. Ye, Y. Yin, S. Yoo, P. Mejia, R. Proietti, and V. Akella, DOS - A scalable optical switch for datacenters, in ACM/IEEE Symp. on Arch. for Netw. and Comm. Systems (ANCS), Oct. 0, pp.. [] H. Chao et al., Petastar: a petabit photonic packet switch, IEEE J. on Sel. Areas in Comm., vol., no., pp., Sept. 00. [] C. Minkenberg et al., Designing a crossbar scheduler for HPC applications, IEEE Micro, vol., no., pp., May/Jun. 00. [] D. Abts and J. Kim, High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities. Morgan & Claypool, 0. [] O. Liboiron-Ladouceur, P. G. Raponi, N. Andriolli, I. Cerutti, M. S. Hai, and P. Castoldi, A scalable space-time multi-plane optical interconnection network using energy-efficient enabling technologies, J. of Optical Commun. and Netw., vol., no., pp. A A, Aug. 0. [] I. Cerutti, P. G. Raponi, N. Andriolli, P. Castoldi, and O. Liboiron- Ladouceur, Designing energy-efficient data center networks using space-time optical interconnection architectures, IEEE J. Sel. opics Quantum Electron., vol., no., pp. 0, Mar./Apr. 0.