MASTER THESIS. TITLE : Design and Evaluation of Architectures for Intra Data Center Connectivity Services

Transcription

1 MASTER THESIS TITLE : Design and Evaluation of Architectures for Intra Data Center Connectivity Services DEGREE: Master of Science in Telecommunication Engineering and Management (MASTEAM) AUTHOR: Sergio Jiménez Feijóo DIRECTOR: Dr. Salvatore Spadaro DATE: March 7, 2014

2

3 Title : Design and Evaluation of Architectures for Intra Data Center Connectivity Services Author: Sergio Jiménez Feijóo Director: Dr. Salvatore Spadaro Date: March 7, 2014 Overview Emerging applications for Data Centers like Cloud Computing, High Performance Computing and Big Data are based on the cooperation of thousands of servers which are connected through a Data Center Network. All these applications require the servers of the Data Center to exchange huge amounts of data between them. In order to withstand such high throughput the Data Center Network must provide ultra-large capacity. Moreover, low latencies are also mandatory, especially when a tight synchronization between servers is required. Several forecasts predict a threefold increase in the global Data Center traffic between the years 2012 and Furthermore, a 76% of such traffic is expected to be produced inside Data Centers. Unfortunately, current multi-tier hierarchical tree-based Data Center Network architectures which rely on Ethernet or Inifiniband are neither flexible nor scalable enough to meet the expected growth previsions. Such limitations have mandated a renewed investigation into the introduction of ultra-highbandwidth and low-latency optical technologies in Data Center Networks. Several architectures for Data Center Networks have been proposed but none of them provides a high degree of scalability at a reasonable cost. Following these trends, LIGHTNESS, an European Framework Programme 7 project, has the objective to design, implement and experimentally demonstrate a high-performance Data Center Network infrastructure for future Data Centers by employing innovative optical switching and transmission solutions. In this thesis a new all-optical Data Center Network architecture envisioned within the LIGHTNESS project is introduced and its performance is benchmarked against other solutions.

4

5 This thesis is dedicated to: My thesis director Salvatore Spadaro for giving me the opportunity of working in such a challenging project and my partner Albert Pagès for his incommensurable help. All my university teachers for their commitment to their task and for making me enjoy every single day I have spent learning from them. My parents Jose and Sílvia and my brother Mario for all their support.

6

7 CONTENTS INTRODUCTION CHAPTER 1. Intra Data Center Networks Current Architecture Access Level Aggregation Level Core Level Drawbacks of current DCNs architecture Scalability Power Consumption Upgrade Cost Quality of Service Traffic Isolation CHAPTER 2. Optical Networks for Data Centers From Opaque to Transparent Optical Networks Optical Circuit Switching Optical Packet Switching Existing architectures c-through and Helios Proteus DOS, Petabit and IRIS OSMOSIS and Data Vortex LIGHTNESS Architecture CHAPTER 3. Performance Evaluation Off-line Scenario Optical Circuit Switching Optical Packet Switching Hybrid Hybrid With Preliminary Mapping Benchmark Results

8 3.2. On-line Scenario Simulator Design Benchmark Results CHAPTER 4. Conclusions Future Lines of Work Environmental Impact BIBLIOGRAPHY

9 1 INTRODUCTION A new wave of innovative Data Center applications is emerging, generating an unprecedented need for ultra-large throughput and extremely low latency between the servers of a Data Center. Also, several forecasts predict a threefold increase in the global Data Center traffic between the years 2012 and Furthermore, a 76% of such traffic is expected to be produced inside Data Centers. Currently deployed Data Center Networks follow a multi-tier hierarchical tree-based architecture and rely on electronic switching technologies like Ethernet or Infiniband. Due to these characteristics, the scalability of this architecture is limited up to the point it will not be able to meet such growth previsions. This upcoming situation has arisen the interest of the research community which is working towards designing a new scalable and flexible architecture for intra Data Center Networks. Owing to the promising features of optical technology, several optical architectures have been proposed but none of them satisfies all the requirements at a reasonable cost. In this line, a new optical architecture has been envisioned within LIGHTNESS, an European Framework Programme 7 project which has the purpose to design, implement and experimentally demonstrate a high-performance Data Center Network infrastructure based on optical technology. The purpose of this thesis is to introduce such new architecture and to benchmark its performance by matching it against other existing solutions. This document is structured as follows: In chapter 1 the currently deployed intra Data Center Networks architecture is analysed, its drawbacks are identified and several potential solutions for overcoming them are provided. Next, in chapter 2 the use of optical technologies to surpass the limitations of current intra Data Center Networks is discussed. The details of several optical-based architectures are provided and their scalability is evaluated. After that, a new architecture for Data Center Networks envisioned within the LIGHTNESS project is introduced and their details are studied. Then, in chapter 3 the performance of the proposed architecture is benchmarked against other solutions in two different scenarios: the network planning stage and the network operation phase. Finally, in chapter 4 several conclusions are provided. Also, some future lines of work are considered and the environmental impact of this thesis is evaluated.

10 2 Design and Evaluation of Architectures for Intra Data Center Connectivity Services

11 Intra Data Center Networks 3 CHAPTER 1. INTRA DATA CENTER NETWORKS A Data Center (DC) is a facility used to house massive amounts of computing, storage and network resources like servers, hard drives and bandwidth. These resources can either be used by the DC operators to deploy their own services or be rented to their customers. These customers are usually Small and Medium Enterprises (SMEs) which have reduced resource requirements and would find very expensive to deploy and maintain their own infrastructure. Therefore the DC operators provide Infrastructure as a Service (IaaS) [1] to their customers. The main issues which DCs address are related to the fulfilment of the Service Level Agreement (SLA) [2] which their operators sign with the customers and the maximization of the profits they obtain. In such a context, the key challenges DCs operators have to face are: Scalability: The capability of being able to increase the number of housed resources and bandwidth. Fault tolerance: The capability of being able to withstand failures without producing an impact on the service. Cost effectiveness: The capability of reducing the amount of required resources. This can be achieved through the use of several virtualization technologies, abstracting the physical resources into several virtual (or logical) resources. This allows the DC operators to optimise the resource usage by providing to each customer only what he needs and pays for. Power efficiency: The capability of minimizing the power consumption produced by the resources. By taking a closer look into intra Data Center Networks (DCNs) it can be observed that they intend to follow these same principles. An intra DCN s function is to allow resilient, high bit-rate and low-latency communications between the DC s computing and storage resources. This is a critical task since a failure (or congestion) in the network would degrade the performance of the connections or directly block them. If this happens, the outcome (from the user s point of view) is the same as if the computing or storage resources were down since his request cannot be attended. An example of this would be the communication between a virtual machine and a storage server (Figure 1.1). Several forecasts, among them the Cisco GCI 2012 (Figure 1.2), indicate that the global DC traffic will grow from about 2.6 Zettabytes/Year on the year 2012 up to approximately 7.7 Zettabytes/Year on the year 2017 experiencing a threefold increase. Moreover, the expected Compound Annual Growth Rate (CAGR) for global DC traffic is around 25%. Furthermore, in such a period of time, the 76% of the global DC traffic is expected to be intra DC traffic (Figure 1.3). This forthcoming substantial growth in intra DC traffic will push to the limit the scalability and performance of currently deployed intra DCNs architecture.

12 4 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Figure 1.1: Virtual Machine to Storage Successful and Failed Communications [3]. Figure 1.2: Global Data Center IP Traffic Growth. Cisco GCI Figure 1.3: Global Data Center Traffic by Destination. Cisco GCI 2012.

13 Intra Data Center Networks 5 In the next section the currently deployed intra DCNs architecture is analysed in order to determine if it will be able to meet the requirements for future intra DCNs Current Architecture Current DCNs present a Fat Tree [4] (or hierarchical) architecture (Figure 1.4) with three distinguished levels: access, aggregation and core. Figure 1.4: Current Architecture [3]. In the next sections, a bottom-up analysis of the architecture will be performed by starting with the access level, then proceeding with the aggregation level and finishing with the core level Access Level In the access level the servers are stacked and placed into racks. Nowadays, each rack usually contains between 20 and 40 servers. Each server can host around 40 virtual machines (depending on the amount of resources each virtual machine requires). Each rack also contains a Remote Direct Memory Access over Converged Ethernet (RoCE) [5] or an InfiniBand (IB) [6] switch which is also known as Top of The Rack (ToR) switch. ToR switches have a high port density (24 or 48 ports) since each server of the rack is connected to a ToR switch by twisted pair or biaxial copper cables. The most used standards for deploying these types of links are shown in the Table 1.1. Thanks to the ToR switches any server can exchange traffic with any other server of its same rack.

14 6 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Standard Bit-rate Cable Maximum length 1000Base-T 1 Gbps 5e (TIA/EIA-568-B) or higher 100 m 10GBase-T 10 Gbps 6 (TIA/EIA-568-B.2-1) 55 m 10GBase-T 10 Gbps 6a (TIA/EIA-568-B.2-10) or higher 100 m 10GBASE-CX4 10 Gbps InfiniBand biaxial cable 15m InfiniBand SDR 2.5 Gbps InfiniBand biaxial cable 60m InfiniBand DDR 5 Gbps InfiniBand biaxial cable 30m InfiniBand QDR 10 Gbps InfiniBand biaxial cable 15m Table 1.1: Gigabit Ethernet and InfiniBand Access Level Standards Aggregation Level The aggregation level consists on several RoCE or InfiniBand switches which are used to exchange traffic between servers which are located in different racks. The aggregation switches usually have fewer ports than a ToR switch but they can achieve higher bit-rates (10Gbps, 40Gbps or even 100Gbps). This traffic exchange is performed by connecting the ToR switches to the aggregation switches with these higher bit-rate links (also known as uplinks). In order to provide redundancy, each ToR switch has at least two uplinks (each one connected to a different aggregation switch). When working with RoCE the Spanning Tree Protocol (STP) [7] or the Provider Backbone Bridge - Traffic Engineering (PBB-TE) [8] are usually set up to ensure a loop-free topology while also providing fast failure recovery. InfiniBand provides out of the box topology loop control. The aggregation level can also be used to partition the network into clusters (or sectors). By doing so, each aggregation switch is directly connected only to a subset of the ToR switches (instead of all of them). This may help to improve the scalability of the architecture (mainly due to the reduction in the number of ports which each aggregation switch requires). The uplinks are usually optical due to the impossibility of achieving such high bit-rates with copper-based links which have a limited bandwidth. The switches which implement this kind of high capacity links typically are equipped with Small Form Pluggable (SFP) or XFI Form Pluggable (XFP) ports which allow different types of transceivers. Since the distances between the ToR switches and the aggregation switches in a DC environment tend to be short (in the order of few hundreds of meters) there are no significant attenuation or dispersion effects on the optical links. Consequently low power lasers operating at the first and second window (850 nm and 1310 nm respectively) are typically used in conjunction with multi-mode optical fibres. Also, neither amplifiers nor regenerators are needed. The most used standards for deploying this type of links are shown in the Table 1.2.

15 Intra Data Center Networks 7 Standard Bit-rate Fibre type Band Maximum length 10GBASE-USR 10 Gbps Multi-mode 850 nm 100 m 10GBASE-SR 10 Gbps Multi-mode 850 nm 400 m 10GBASE-LRM 10 Gbps Multi-mode 1310 nm 220 m 10GBASE-LR 10 Gbps Single-mode 1310 nm 10 km 40GBASE-SR4 40 Gbps Multi-mode 850 nm 400 m 40GBASE-LR4 40 Gbps Single-mode 1310 nm 10 km 100GBASE-SR4 100 Gbps Multi-mode 850 nm 100 m 100GBASE-SR Gbps Multi-mode 850 nm 400 m 100GBASE-LR4 100 Gbps Single-mode 1310 nm 10 km InfiniBand QDR 10 Gbps InfiniBand 4X QDR 40 Gbps InfiniBand 12X QDR 120 Gbps Table 1.2: Gigabit Ethernet and InfiniBand Aggregation Level Standards Core Level The core level consists on the switches and routers which are responsible for the exchange of the traffic which is headed to (or comes from) outside the DC. This includes communications with other DCs and access to the Internet. The core switches and routers usually have fewer ports than the aggregation and ToR switches but instead their ports can work at huge bit-rates (100 Gbps or more) and their backplane is designed to support a throughput of several Tbps. In order to obtain such high degree of connectivity the DC operators usually need to sign several peering agreements with Tier 1 [9] ISPs. This guarantees that enough bandwidth is available for the customers and end-users Drawbacks of current DCNs architecture To conclude this analysis, several drawbacks of the current intra DCNs architecture and some possible solutions for overcoming them are discussed Scalability The most critical drawback of the current intra DCNs architecture is the existence of a bottleneck in the core level due to its hierarchical structure. The traffic headed to (or coming from) outside the DC goes all through a single node or very few nodes. This concentration of the traffic in such few nodes constitutes a critical failure point and also requires those nodes to be able to manage an extremely high throughput (which is very expensive).

16 8 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Also, due to the forthcoming increase in global DC traffic the number of servers per rack is expected to increase. Since the number of ports in the ToR switches is limited the current architecture will not be able to withstand such growth. A possible solution for this limitation would be to switch to a non-hierarchical architecture (also known as flat architecture) which balances the traffic among more nodes. Also, the need to process the packet headers could be avoided by switching to a transparent optical network architecture (further explained in chapter 2) Power Consumption Another drawback of the current architecture is the high power consumption of the network equipment. All the network equipment need to process the headers of the Ethernet or InfiniBand frames in order to successfully determine the output port they should be sent to. This process implies several calculations. Even though most of them are quite simple (like searching in a look-up table or performing a binary AND operation with a mask) they still constitute a complex computational issue when dealing with high throughputs and require power-hungry hardware. Moreover, this power consumption problem is aggravated since the network equipment consume almost the same amount of power when they are idle as when they are working at full load [10]. Therefore it is clear that the only way to reduce the power consumption is by shutting down as many idle network equipment as possible or by not having electrical network equipment at all. A way to do this is by switching to a transparent optical network architecture (further explained in chapter 2) Heat dissipation As a consequence of this high power consumption a lot of heat is dissipated. In order for the network equipment to work properly and to avoid physical damages due to over-heating it is necessary to deploy a cooling system. Furthermore, it needs to be taken into account that cooling systems do have a pretty high power consumption too. Several studies indicate that in order to estimate real power consumption of a DC network the power consumption of the network equipment should be doubled as a consequence of the cooling costs [11] Upgrade Cost When a network is built some extra capacity is provisioned in order to withstand future traffic increases. But in the long term some old equipment of the network have to be replaced by newer ones in order to increase the available bandwidth.

17 Intra Data Center Networks 9 Since all the optical links of the network are point-to-point the traffic undergoes several optical-to-electrical and electrical-to-optical conversion stages along the path between the source and the destination servers. This implies that enough bandwidth must be available on the path to carry the traffic. Also, traffic grooming is performed (at the electrical level) at each one of the intermediate nodes. This electrical processing of the traffic requires the network equipment to have a powerful enough backplane. Therefore an increase in the available bandwidth between two servers may require the replacement of several transponders and/or network equipment along the path by newer ones which offer a higher capacity. An optical transponder is a device which allows the transmission and reception of optical signals. The optical transmission is achieved though the use of a Light Emitting Diode (LED) [12] or a Light Amplification by Stimulated Emission of Radiation (LASER) [12] device. The optical reception is achieved through the use of a Photodiode [12]. The cost of a transponder highly depends on several parameters. For the transmitter: Its maximum optical power at the output and the spectral width of the optical signal at the output. For the receiver: Its sensitivity, its noise figure and its bandwidth. Each optical link of the network needs two transponders: One at the head node and another at the tail node. A possible solution to reduce the upgrade cost of the network would be to decrease the number of transponders (by avoiding point-to-point links) and electrical network equipment (by avoiding the need to process the frame headers) by switching to a transparent optical network architecture (further explained in chapter 2) Quality of Service In any packet switched network, the frames received by an input interface of a network equipment are processed by inspecting their headers and then transmitted by the selected output interface. But when the rate of the input traffic destined to a certain output interface is higher than its available bandwidth not all the frames can be processed and transmitted at once. Then, the frames which have not been able to be transmitted will be stored in a queue in the same order they reached the input interface. If the queue is full and more frames arrive they will be dropped. A scheduler will then decide in which order the frames of the queue will be processed and transmitted once there is enough bandwidth available. Usually, schedulers apply a First In First Out (FIFO) policy which consists on attending the frames in the same order they reached the queue. Therefore, a frame will need to wait in the queue until the scheduler decides it is time for it to be processed and transmitted. The amount of time a frame stays in a queue waiting for its turn its named queuing delay. The queuing delay and the frame loss ratio increase with the network load because the queues are more populated. The queuing delay and frame loss are inherent to all frame

18 10 Design and Evaluation of Architectures for Intra Data Center Connectivity Services switched networks even though different scheduler policies can be applied to favour certain types of frames (and reduce their queuing delay). Nevertheless, this comes at the cost of increasing the queuing delay of the rest of frames (the mean queuing delay always remains constant). Moreover, the queuing delay and frame loss ratio between two servers not only depends on the traffic they are exchanging but also on the traffic other servers are exchanging between them. This is so because frames from different sources and destinations may share the same queue in a certain network equipment. Also, bandwidth intensive applications which work with TCP/IP may experience delay and jitter due to the processing of the segments through systems calls to the kernel of the operating system. Furthermore, due to the behaviour of TCP when a segment loss is the detected the bandwidth is dramatically reduced during several Round Trip Times (RTT) [13]. Several DC applications like Fibre Channel over Ethernet (FCoE) [14] are extremely sensitive to queuing delay and frame loss. RoCE and InfiniBand solve these issues by implementing several flow control and Quality of Service mechanisms which reduce the congestion on the network. Also, Remote Direct Memory Access (RDMA) [15] is implemented to allow data transfers to directly access the hardware (bypassing the kernel system calls). Despite these improvements introduced by RoCE and InfiniBand, the only way to completely avoid that an increase in the load produced by one customer affects the queuing delay which the rest of customers experience is by switching to a circuit-switched architecture which provides dedicated resources for each customer Traffic Isolation In order to block unauthorized communications between servers rented by different customers and to prevent the broadcast and multicast traffic to spread through the entire network, the current architecture relies on the RoCE Virtual LANs [8] and the InfiniBand Subnets. A VLAN is a virtual slice of a LAN which is constituted by a subset of the LAN s resources (links, equipment, servers, etc). The servers which are assigned to a VLAN can only establish layer 2 communications between them. Several VLANs can coexist over the same LAN sharing its physical resources. There is a limit of VLANs which an RoEC LAN can support. This has to do with the 12 bit VLAN ID (VID) header field which the 802.1Q [8] frame (Figure 1.5) uses to indicate the VLAN number. If there are 12 available bits and each bit can take a 0 or a 1 value then there are 2 12 = 4,096 possible VLANs. The only way to overcome this limitation is by using double-tagged frames (802.1AD or 802.1QinQ) which provide 2 24 = 16,777,216 possible values (more than enough for today s requirements).

19 Intra Data Center Networks bits 3 bits 1 bit 12 bits TCI TPID PCP DEI VID Figure 1.5: 802.1Q Frame. An InfiniBand subnet is the InfiniBand counterpart of a RoCE VLAN. InifiniBand Subnets are set up by using the InfiniBand Subnet Manager. This segmentation of the network is useful to implement security and traffic control functionalities but a proper configuration is required in all the involved network equipment and servers. This can be an important issue in big networks, where a lot of network equipment and servers may have to be reconfigured quite often either manually or by an automated process (running scripts). A way to avoid the need for the configuration of traffic isolation is by switching to a circuitswitched architecture. Summary To conclude this analysis, in the Table 1.3 are summarised all the discussed scalability drawbacks of the current architecture and the proposed potential solutions. Drawback Scalability Power Consumption Upgrade Cost Quality of Service Traffic Isolation Potential Solutions Flat Architecture, Transparent Optical Architecture Transparent Optical Architecture Transparent Optical Architecture Circuit-Switched Architecture Circuit-Switched Architecture Table 1.3: Current Architecture Drawbacks and Solutions Summary. After a brief inspection of the table it is clear that all of the current architecture drawbacks could be avoided by switching to a circuit-switched transparent optical architecture. In the next chapter several optical architectures for intra DCNs are discussed.

20 12 Design and Evaluation of Architectures for Intra Data Center Connectivity Services

21 Optical Networks for Data Centers 13 CHAPTER 2. OPTICAL NETWORKS FOR DATA CENTERS An optical network transports signals in an optical form between a source and one or more destinations. Optical networks offer a very high bandwidth-distance product and therefore are the preferred choice for transporting high bit-rate signals over several kilometres. In order to understand the advantages which optical networks offer the concept of transparency needs to be introduced From Opaque to Transparent Optical Networks The first optical networks which appeared were used to provide higher capacity and lower bit error rate than their contemporary copper-based networks. In this type of networks all the switching and intelligent network functions where handled electronically. All the links of the network were point to point and all the traffic needed to be electronically processed in all of the intermediate nodes. As bandwidth requirements began to increase, this type of networks required expensive and power hungry network equipment. This type of networks perform an optical-to-electrical and electrical-to-optical conversion in all the intermediate nodes. Therefore, the wavelength used to carry an optical signal can be different in each one of the links of the path between the source and the destination nodes. Because of these optical-to-electrical and electrical-to-optical conversions which the signal undergoes along its path, this type of networks require proper transponder upgrading in order to support different bit-rates and protocols. This is why they are known as opaque networks. Opaque networks present a low blocking probability because any available wavelength in a link can be used to establish a connection. On the other hand, this type of networks require a high number of transponders per demand (2 per each link) and also produce a high power consumption. An example of this type of optical networks are Synchronous Digital Hierarchy (SDH) [12] networks. In SDH networks the traffic is multiplexed electrically by using Add and Drop Multiplexers (ADMs) and Digital Cross Connects (DXCs). ADMs allow the multiplexing and demultiplexing of several Virtual Containers (VCs) inside a Synchronous Transport Module (STM) signal. DXCs allow the rearrangement of VCs among several STMs. When optical technology reached matureness, optical networks began to offer more advantages than just a high bandwidth-distance product. Several functionalities which were previously handled electronically were moved to the optical layer in order to remove the need for electrical processing at each node. Also, point-to-point links were substituted by end-to-end optical connections which go through several intermediate nodes without any need for electrical processing. This significantly reduced the load of the nodes which now only have to process the traffic intended to them.

22 14 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Figure 2.1: Fully Tunable OADM [12]. Since the signal is carried in an optical form all the way between the source and the destination, the network is transparent to the data which is being sent over the optical signal. This means the network is compatible with any bit-rate and protocol, even with analogue data. Transparent optical networks do not perform any optical-to-electrical or electrical-to-optical conversion in any of the intermediate nodes. Therefore, the wavelength used to carry an optical signal is always the same along all the links of the path between the source and the destination nodes. This type of networks present a higher blocking probability than opaque networks because no node is able to perform a wavelength conversion. Hence there are a more restrictions in order to establish a connection between the source and destination nodes. On the other hand, this type of networks require a lower number of transponders than opaque networks (2 per each connection) and also produce a lower power consumption. However, in order to provide higher bandwidth than opaque networks, transparent networks require all the transponders to be able to work at very high speeds (100 Gbps or more) resulting in a substantial increase in the transponder cost. An example of this type of optical networks are Optical Transport Networks (OTNs) [12]. In OTN networks the traffic is multiplexed (being the smallest multiplexing unit an entire wavelength) by using Optical Add and Drop Multiplexers (OADMs) and Optical Cross Connects (OXCs). OADMs (Figure 2.1) allow to insert or extract optical signals inside an optical fibre. OXCs allow to rearrange optical signals between two or more optical fibres. Due to the promising features of transparent optical networks, they have been thoroughly studied (and still are) by the research community and have arisen the industry s interest on them. As a result several switching technologies have been proposed, among them Optical Circuit Switching (OCS) and Optical Packet Switching (OPS). At the time being, OCS technology is mature enough and already being deployed in a wide number of optical networks while OPS is still in an early development stage. In the next sections both OCS and OPS technologies are discussed.

23 Optical Networks for Data Centers Optical Circuit Switching OCS networks provide an optical end-to-end circuit (ligthpath) to each connection. Each ligthpath is carried over the same wavelength on each one of the links of the path between the source and destination nodes. Each link of the network has several wavelengths available due to the use of the Dense Wavelength Division Multiplexing (DWDM) technology. Different lightpaths can use the same wavelength as long as they do not share any common links. This allows a wavelength to be used simultaneously in several ligthpaths. These circuits provide a guaranteed amount of bandwidth which is allocated for the exclusive use of the connection during all the time it is active. As a consequence, Quality of Service is guaranteed because different connections never share their bandwidth. However, in OCS networks the bandwidth allocated for a connection is always a full wavelength. This means that connections which require a bandwidth which is close to the bandwidth of a wavelength will make an efficient use of the spectrum while connections which require a low amount of bandwidth will waste a lot of spectrum. For this same reason the OCS technology also presents a poor spectral efficiency when dealing with bursty traffic. The channel spacing between wavelengths in a DWDM link can be set to 100, 50, 25 or 12.5 GHz according to the DWDM frequency grid [16] defined by the International Telecommunication Union (ITU). Such wavelengths have a center frequency which can be obtained from the Table 2.1 according to the selected channel spacing (where n Z). Bandwidth Center Frequency (THz) 100 GHz n GHz n GHz n GHz n Table 2.1: DWDM Frequency Grid [16]. The lower the channel spacing, the higher will be the number of wavelengths which the DWDM link can contain. Nevertheless, this comes at the cost of also reducing the available bandwidth for each wavelength. The use of spectral efficient modulations is required to achieve high bit-rate transmissions over a reduced bandwidth. Figure 2.2 shows an example of a 50GHz channel spacing frequency grid. Figure 2.2: 50 GHz Channel Spacing Example [16].

24 16 Design and Evaluation of Architectures for Intra Data Center Connectivity Services A possible way to increase the spectral efficiency of OCS networks would be to assign to each wavelength the channel spacing which best fits its bandwidth requirements. In Mixed- Line-Rate (MLR) DWDM links, wavelengths with different channel spacings can coexist as long as there is no spectrum overlapping. Despite of this, the rigidity of the frequency grid may lead to spectrum gaps. In order to avoid such situation the ITU has also defined a flexible frequency grid [16]. This flexible grid defines a set of frequency slots which have a central frequency and a width. The central frequency of a slot (in THz) is calculated by using the Equation 2.1 (where n Z). The slot width (in THz) is calculated by using the Equation 2.2 (where m N and 12.5 is the slot granularity in THz). s f = n (2.1) s w = 12.5 m (2.2) Any combination of frequency slots is allowed as long as no two frequency slots overlap. Figure 2.3 shows an example of a flexible frequency grid. Figure 2.3: Flexible Grid Example [16]. However, the flexible grid technology is still at an early development stage and has not been yet implemented in commercial equipment. Furthermore, when a new circuit is established a suitable combination of path and wavelength needs to be found. This requires to maintain an updated database with the current status of all the links and wavelengths of the network and to perform several computations with this data in order to determine a feasible solution. Also, once a path and a wavelength have been selected, some signalling is required between the Reconfigurable Optical Add and Drop Multiplexers (ROADMs) of the network in order to establish the circuit. ROADMs are devices which can dynamically modify their switching behaviour, enabling the network to provide on-demand end-to-end optical circuits (ligthpaths). In currently deployed Automatically Switched Optical Networks (ASONs) this functionalities are provided by the Generic Multi-Protocol Label Switching (GMPLS) protocol. This process significantly increases the required time to establish a connection insomuch that short-lived connections experience high delays.

25 Optical Networks for Data Centers 17 These two drawbacks of the OCS technology (spectral efficiency and connection establishment delay) have motivated the appearance of the OPS technology. In the following section the details of such technology are discussed Optical Packet Switching OPS networks provide a virtual optical end-to-end circuit to each connection. This virtual circuit emulates the behaviour of a ligthpath with the difference that connections can allocate a bandwidth smaller than a full wavelength. This is achieved by breaking the connection s data streams into optical packets and adding them a new header which identifies the optical destination node. Optical packets from different connections are multiplexed in the time domain by using Optical Time Division Multiplexing (OTDM) and sent over a wavelength. By doing so, each connection has available the entire bandwidth of a wavelength during a certain period of time. This type of multiplexing is also known as sub-wavelength switching because streams with bandwidths smaller than a full wavelength are switched (unlike OCS). By performing the multiplexing process directly in the optical domain OPS switches can easily manage high bit-rate streams. The OPS switches of the network read the header of the incoming optical packets and then send them to the appropriate output port. By doing this, the OPS switches are sending optical packets over OCS circuits which are established between them. Figure 2.4 shows the architecture of an OPS switch. OPS nodes also need to handle contention at their output ports. Contention happens when two or more optical packets from different input ports simultaneously need to be sent over the same output port. In this situation, since the optical signals would collide, only one of them can be sent at the same time, the rest are dropped or sent through another output port. The packet losses increase exponentially with the traffic load because the collision probability is higher. This is due to the current non existence of optical buffers (the optical counterpart of elec- Figure 2.4: OPS Switch Architecture [12].

26 18 Design and Evaluation of Architectures for Intra Data Center Connectivity Services tronic buffers) which would be required to store optical packets. The research community is currently studying the use of delay lines (very long optical fibres) as a way to temporarily store optical packets. However, delay lines introduce attenuation and dispersion effects to the optical signals and therefore should be used carefully. Moreover, since electromagnetic waves propagate through optical fibers at a speed of (2/3) c (where c is the speed of light), the required length for a delay line in order to obtain a significant delay would be in the order of thousands of kilometres Existing architectures Future intra DCNs are expected to provide higher bandwidth, scalability, flexibility, power efficiency and cost effectiveness and lower end-to-end latencies. In order to achieve these goals, several optical architectures have been proposed. Over the next sections the technical details of such architectures and their scalability drawbacks are discussed c-through and Helios c-though [17] and Helios [17] are two hybrid optical/electrical architectures designed to enhance currently deployed intra DCNs. The c-through architecture is an evolution of the Fat Tree architecture where the ToR switches are connected both to an electrical packet-switched network (Ethernet or Inifini- Band) and an OCS network. The electrical-packet switched network handles the low bandwidth demands while the OCS network can be configured in such a way that pairs of ToR switches with high bandwidth demands can be connected through optical circuits. The Helios architecture [17] features a 2 tier design with hybrid optical/electrical ToR switches (called pod switches). Such hybrid ToR switches are connected both to electrical packet switches (through colorless optical transceivers) and OCS switches (through DWDM optical transceivers). As in the c-through architecture, the electrical packet switched network is used to handle the low bandwidth demands while the OCS network is used to connect pairs of ToR switches with high bandwidth demands. However, both architectures present scalability issues due to the inherent limitations of electronics. Figure 2.5 shows both the c-through (left) and the Helios (right) architectures Proteus Proteus [17] is an all-optical OCS architecture based on Wavelength Selective Switches (WSSs) [12] and an optical switching matrix which is based on Micro-Electro-Mechanical Switches (MEMSs) [12]. In the Proteus architecture each ToR switch has several optical transceivers operating at different wavelengths. The optical wavelengths are combined using a multiplexer and are routed to a WSS. Each group of wavelengths is connected to

27 Optical Networks for Data Centers 19 Figure 2.5: c-through Architecture (left) [17]; Helios Architecture (right) [17]. a port in the MEMS optical switch which is used to establish point-to-point connections between the ToR switches. Figure 2.6 shows the Proteus architecture. The main drawback of this architecture is the elevated circuit reconfiguration time due to the use of the MEMSs DOS, Petabit and IRIS Datacenter Optical Switch (DOS) [17], Petabit [17] and IRIS [17] are all-optical OPS architectures based on the use of Arrayed Waveguide Grating Routers (AWGRs) [12] and Tunable Wavelength Converters (TWCs) [12]. The optical switch fabric of the DOS architecture consists of an array of TWCs (one TWC for each node), an AWGR and a loopback shared buffer. Each node can access any other node through the AWGR by configuring the transmitting wavelength of the TWC. The switch fabric is configured by the control plane that controls the TWC and the label extractors (LEs). The control plane is used for the contention resolution and TWC tuning. Figure 2.7 shows the DOS architecture. The Petabit [17] architecture adopts a three-stage Clos network where each stage consists of an array of AWGRs which are used for the passive routing of packets. In the first stage, the tunable lasers are used to route the packets through the AWGRs, while in the second and in the third stage TWC are used to convert the wavelength and route accordingly the packets to destination port. The IRIS architecture is based on a three-stage switch. Each node is connected to a port of the first stage using several wavelengths. The first stage consists of an array of Wavelength Switches (WSs). Each WS is based on an array of all-optical Semiconductor Optical Amplifier (SOA) [12] wavelength converters that is used for the wavelength routing. The second stage is a time switch that consists of an array of optical time buffers. The time switch is composed of an array of WC and two AWG interconnected with a number of optical lines, each one with different delays. Based on the delay that needs to be

28 20 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Figure 2.6: Proteus Architecture [17]. Figure 2.7: DOS Architecture [17].

29 Optical Networks for Data Centers 21 Figure 2.8: OSMOSIS Architecture [17]. added, the WC converts the optical signal to a specific wavelength that is forwarded to the AWG with the required time delay. The delayed signals are multiplexed through a second AWG and are routed to the third stage (a second space switch). Based on the final destination port, the signal is converted to the required wavelength for the AWG routing. Nevertheless, the scalability of all three architectures depend on the scalability of the AWGR and the tuning range of the TWC OSMOSIS and Data Vortex OSMOSIS [17], Data Vortex [17] are all-optical OPS broadcast-and-select (B&S) architectures based on the use of couplers, splitters and SOA broadband optical gates. The OSMOSIS architecture is composed of two stages. In the first stage, multiple wavelengths are multiplexed in a common WDM line and are broadcasted to all the modules of the second stage through a coupler. The second stage use SOAs as fiber-selector gates to select the wavelength that will be forwarded to the output. Rather than using tunable filters, this design features a demux-soa-select-mux architecture. However, the main drawback of this architecture is that it is based on power hungry SOA devices which significantly increase the overall power consumption. Figure 2.8 shows the OSMOSIS architecture. The Data Vortex architecture relies on an gate-array of SOAs, which act as photonic switching elements. This topology is composed entirely of 2x2 switching elements arranged in a fully connected, directed graph with terminal symmetry. Such modularity provides an efficient scalability. Nevertheless, the topology becomes extremely complex when it is scaled to large networks. As the number of nodes increase the packets have to traverse several nodes before reaching their destination, resulting in an increase of latency and jitter.

30 22 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Summary To conclude this analysis, all the scalability drawbacks of the discussed architectures are summarised in the Table 2.2. Architecture c-through Helios Proteus DOS Petabit IRIS OSMOSIS Data Vortex Drawbacks Throughput Throughput Reconfiguration Time AWG Size, TWC Range AWG Size, TWC Range AWG Size, TWC Range Power Consumption Delay, Jitter, Complexity Table 2.2: Existing Solutions and Drawbacks Summary. By inspecting the table, it can be concluded that none of the discussed architectures meets all the bandwidth, latency, scalability, flexibility, power efficiency and cost effectiveness requirements of future intra DCNs. In the next section a new all-optical hybrid architecture for intra DCNs envisioned within the Lightness project is presented LIGHTNESS Architecture As it has been shown in the previous section, none of the currently existing optical architectures meet all the requirements of future intra DCNs. By inspecting closely the details of the proposed solutions, it can be appreciated that all of them are based on one single switching technology (either OCS or OPS). Both OCS and OPS technologies have advantages and disadvantages. OCS offers a very low packet loss ratio which is independent of the traffic load but its bandwidth granularity is always an entire wavelength and requires several milliseconds in order to establish a circuit. On the other hand, OPS offers a sub-wavelength bandwidth granularity and a low establishment delay but its packet loss ratio increases exponentially with the load. The Table 2.3 summarizes this comparison. Therefore, the OCS technology provides high efficiency when dealing with high bit-rate demands. Serving a low bit-rate demand with OCS would result in a waste of bandwidth because only a small fraction of the bandwidth of the wavelength would be used. On the OCS OPS Bandwidth Granularity Wavelength Sub-Wavelength Packet Loss Ratio Load Independent Very Load Dependent Establishment Delay High Low Table 2.3: OCS and OPS Technologies Comparison.

31 Optical Networks for Data Centers 23 High Bandwidth Low Packet Loss Ratio Low Establishment Delay Strong OCS OCS OPS Soft OPS OPS OCS Table 2.4: Best Technology Selection Based on Demand Requirements. other hand the OPS technology is suited for dealing with low bit-rate flows, also providing high efficiency through statistical multiplexing gain. Serving a high bit-rate demand with OPS would require to split its bandwidth and to use several wavelengths (in order to avoid an increase in the packet loss ratio) resulting also in a waste of bandwidth. Moreover, OCS is best suited for serving demands which require a low packet loss ratio while OPS is the best choice to serve demands which require a low establishment delay. The table 2.4 summarizes which is the most efficient switching technology based on the requirements of the traffic. However, a demand can have several strong requirements (i.e. high bandwidth and low establishment delay) which cannot be all fulfilled by the same switching technology. In such a case, there is a trade-off between the weight of each one of the requirements. The architectures which feature only one of the two switching technologies lack the flexibility required to achieve a high efficiency when dealing with an heterogeneous traffic profile. Therefore, the advantages of a hybrid architecture in front of only OCS or OPS networks are evident: hybrid networks can choose which of both technologies (OCS or OPS) is the best suited to serve a demand by taking into account its bandwidth, packet loss ratio and delay requirements. For example, high bit-rate demands which require a bandwidth close to the capacity of a full wavelength can be served with OCS while low bit-rate demands which require a small fraction of the bandwidth of a full wavelength can be served with OPS. A new hybrid all-optical architecture for Intra Data Center Networks has been envisioned within the LIGHTNESS project. The LIGHTNESS project [3] is an European FP7 project with the objective of designing, implementing and evaluating high performance interconnects for DCNs through the use of optical technologies. Such architecture features a transparent optical network with a flattened structure which combines both OCS and OPS technologies in order to provide high bandwidth and scalability and reduced latency and power consumption. The LIGHTNESS architecture is depicted in the Figure 2.9. The main novelty introduced by the LIGHTNESS architecture is a hybrid OCS/OPS ToR switch which connects each rack simultaneously to an OCS network and an OPS network. As a result, the architecture is able to provide a high degree of flexibility by being able to serve each traffic demand with the technology which best suits its bandwidth and latency requirements while also maximizing the efficiency of the network resources. Also, the LIGHTNESS architecture features a new Architecture on Demand (AoD) OCS switch which provides full mesh OCS and OPS connectivity between the ToR switches. Moreover, the scalability of the architecture is increased by partitioning the network into clusters of ToR switches. Figure 2.10 shows the intra-cluster configuration.

32 24 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Figure 2.9: LIGHTNESS General Architecture [3]. Figure 2.10: LIGHTNESS Intra-Cluster Configuration [18].

33 Optical Networks for Data Centers 25 Figure 2.11: LIGHTNESS Inter-Cluster Configuration [18]. An inter-cluster AoD OCS switch provides full mesh connectivity between all the cluster AoD OCS switches. Figure 2.11 shows the inter-cluster configuration. Also, a Software Defined Network (SDN) [19] [20] control plane has been defined based on the OpenFlow [21] protocol. The Software Defined Network paradigm allows to decouple the control and data planes of the network by transferring the control logic from the network equipment to a centralized controller which has a global perspective of the network. Since the network controller has full knowledge of the status of the network it can take optimal decisions which maximize the performance of the network. Each network equipment is equipped with an OpenFlow agent. An OpenFlow agent is a software client which connects to the controller server through a southbound interface. OpenFlow agents provide information to the controller regarding the current status of the equipment. Also, OpenFlow agents translate the instructions received by the OpenFlow network controller into hardware-specific instructions which target the equipment s data plane. This SDN-based control plane performs several critical functions like network monitoring, topology discovery, deciding whether a demand should be served with OCS or OPS, path computation, signalling during the establishment of a circuit and integration with GMPLS, SNMP and other protocols. Also, the control plane has a northbound interface which allows communication with the DC management system. This, allows the DC operators to provide a management API to their customers which facilitates on-demand reconfiguration of virtual slices. The Figure 2.12 shows the LIGHTNESS control plane.

34 26 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Figure 2.12: LIGHTNESS Control Plane [22]. In the next chapter the performance of the LIGHTNESS architecture is evaluated.

35 Performance Evaluation 27 CHAPTER 3. PERFORMANCE EVALUATION In order to evaluate the performance of the proposed hybrid architecture, two different scenarios (off-line and on-line) have been considered. The off-line scenario reproduces the conditions encountered during the network planning stage while the on-line scenario emulates the behaviour of the network during its operation phase. In the next sections both scenarios are detailed and the performance of the network is evaluated Off-line Scenario The off-line scenario reproduces the conditions which are given during the network planning stage. When a new network is designed (or an existent one is upgraded) the network operator needs to perform an estimation of the amount of resources which are required for a proper operation of the network when it is under a certain traffic load. In the case of optical networks, such resources are the amount of optical fibres, wavelengths per fibre and transponders. A useful parameter which allow us to evaluate the performance of a network under a certain traffic load is the Blocking Probability (BP). The BP is expressed as the ratio between the blocked and the attempted connections. This ratio gives us an idea about how many demands can be served with a certain amount of resources. An ideal BP value would be 0% but this would require the network to be equipped with a lot of resources that might be underused during most of the time. On the other hand, if the network does not have enough resources to serve the demands the BP would rise showing a degradation in the performance of the network. Therefore there is a trade-off between the number of equipped resources in a network and the maximum BP which can be tolerated. This maximum BP may widely depend on several factors, among them the Quality of Service requirements of the application which generated the traffic and how restrictive is the SLA signed between the customer and the network operator. The network operator s task during the network planing stage is to properly dimension the network in order to achieve a low BP while also reducing the amount of required resources. Therefore, a lightpath (a wavelength over a path) has to be found for each demand while also minimizing the number of required wavelengths. This type of issues are known as Routing and Wavelength Assignment (RWA) problems and have been (and still are) widely studied by the research community. A way to solve RWA problems is through the use of Integer Linear Programming (ILP). The RWA problem is modelled as an ILP problem which consists on an objective function which is to be minimized or maximized and several constraints which need to be fulfilled. Once the problem is mathematically formulated, several methods (relaxations, heuristics, branch and bound, simplex, etc.) are applied in order to obtain the optimal solution.

36 28 Design and Evaluation of Architectures for Intra Data Center Connectivity Services In the next sections several RWA problems which allow us to evaluate the performance of the proposed architecture are discussed and modelled as ILP problems. For benchmarking reasons, the proposed hybrid architecture is matched against an only OCS architecture and an only OPS architecture. First, the RWA problem for an OCS architecture is considered. Next, the RWA problem for an OPS architecture is analysed. Finally the RWA problem for the hybrid architecture which features both OCS and OPS technologies is discussed. To conclude, the results of the benchmark are presented Optical Circuit Switching This RWA problem focuses on OCS networks. Its goal is to find the optimal combination of ligthpaths which should be assigned to each one of the demands in order to successfully establish them all while also minimizing the number of required wavelengths per link Problem Formulation The OCS RWA problem can be stated as Given: 1. a transparent optical network characterized by a graph G = (N, E), where N denotes the set of nodes and E = {(i, j),( j,i) : i, j N,i j} the set of physical links; 2. a set of wavelengths per physical link denoted as W ; 3. a set of demands D to be served, each one of them characterized by a source node R d and a destination node S d. Find a ligthpath between the source and destination node of each demand of the demand set subject to the following constraints: 1. traffic constraint: all the demands of the demand set must be served; 2. non-bifurcated flows: each demand must use one single lighpath (the bandwidth of a demand cannot be split); 3. wavelength clashing: each wavelength of each link can serve only one demand; 4. wavelength continuity: the same wavelength must be used in all the links of the lightpath between the source and the destination nodes of a demand with the objective to minimize the number of required wavelengths per physical link of the network. Next is provided a link-path [23] based ILP model for the OCS RWA problem.

37 Performance Evaluation ILP Model Let us define P as the set of paths in the physical network, P d as the set of p P associated with the demand d D, and P d,e as the set of p P d which traverse the edge e E. The decision variables of the ILP model are: Z w : binary; 1 if the wavelength w W is used to serve any demand, 0 otherwise. J d,e,w : binary; 1 if the wavelength w W is used to serve the demand d D in the edge e E, 0 otherwise. X d,p,w : binary; 1 if the path p P and the wavelength w W are used to serve the demand d D, 0 otherwise. The ILP formulation is stated below: d D p P d w W min Z w,s.t. (3.1) w W X d,p,w = 1, d D (3.2) p P d,e X d,p,w 1, e E,w W (3.3) Z w J d,e,w, d D,e E,w W (3.4) J d,e,w = p P d,e X d,p,w, d D,e E,w W (3.5) Objective function 3.1 aims at minimizing the number of wavelengths required per physical link. Constraints 3.2 are the traffic and non-bifurcated flow constraints which ensure that all the demands are served and that their bandwidth cannot be split. Constraints 3.3 are the wavelength clashing constraints which guarantee that every wavelength of every link can serve only one demand. The wavelength continuity constraint is enforced due to the use of a link-path formulation. Constraints 3.4 and 3.5 are auxiliary. Constraints 3.4 assure that a wavelength is used in the network if it is used to serve any demand over any link. Constraints 3.5 ensure that a wavelength of a link is used to serve a demand if any path which goes through it is serving that demand and vice-versa Complexity Analysis The proposed ILP Model for the OCS RWA problem requires W + D E W + D P d W variables and D + E W + 2 D E W constraints (where P d is the average path number for the demands of the demand set). The number of variables can be

38 30 Design and Evaluation of Architectures for Intra Data Center Connectivity Services approximated by O( D E W ) and the number of constraints to O(2 D E W ) under the assumption that the network is composed by a small number of clusters Optical Packet Switching This RWA problem focuses on OPS networks. Its goal is to find the optimal combination of lightpaths (paths and wavelengths) which should be assigned to each one of the demands in order to successfully establish them all while also minimizing the number of required wavelengths per link Problem Formulation The OPS RWA problem can be stated as Given: 1. a transparent optical network characterized by a graph G = (N, E), where N denotes the set of nodes and E = {(i, j),( j,i) : i, j N,i j} the set of physical links; 2. a set of wavelengths per physical link denoted as W ; 3. a set of demands D to be served, each one of them characterized by a source node R d, a destination node S d, a required bandwidth B d and a maximum bandwidth per lightpath (due to its packet loss ratio requirements) Bm d ; 4. a real number M close to infinity. Find a set of ligthpaths between the source and destination node of each demand of the demand set subject to the following constraints: 1. traffic constraint: all the demands of the demand set must be served; 2. QoS constraint: the used bandwidth of a wavelength in a link cannot exceed the maximum bandwidth of any of the demands which undergo it; 3. wavelength continuity: the same wavelength must be used in all the links of a ligthpath between the source and the destination nodes of a demand with the objective to minimize the number of required wavelengths per physical link of the network. Next is provided a link-path based ILP model for the OPS RWA problem.

39 Performance Evaluation ILP Model Let us define P as the set of paths in the physical network, P d as the set of p P associated with the demand d D, and P d,e as the set of p P d which traverse the edge e E. The decision variables of the ILP model are: Z w : binary; 1 if the wavelength w W is used to serve any demand, 0 otherwise. K d,e,w : binary; 1 if the wavelength w W is used to serve the demand d D in the edge e E, 0 otherwise. Y d,p,w : real; amount of bandwidth of the demand d D which is being served by the path p P d and the wavelength w W. The ILP formulation is stated below: d D p P d w W min Z w,s.t. (3.6) w W Y d,p,w = B d, d D (3.7) p P d,e Y d,p,w Bm d K d,e,w + (1 K d,e,w ), d D,e E,w W (3.8) Z w K d,e,w, d D,e E,w W (3.9) K d,e,w Y d,p,w, d D,e E, p P d,e,w W (3.10) M p P d,e Y d,p,w K d,e,w, d D,e E,w W (3.11) Objective function 3.6 aims at minimizing the number of wavelengths required per physical link. Constraints 3.7 are the traffic constraints which ensure that all the demands are served (and allows the bandwidth of a demand to be split among several lightpaths). Constraints 3.8 are the QoS constraints which assure that the used bandwidth of a wavelength in a link does not exceed the maximum bandwidth of any of the demands which undergo it. The wavelength continuity constraint is enforced due to the use of a link-path formulation. Constraints 3.9, 3.10 and 3.11 are auxiliary. Constraints 3.9 assure that a wavelength is used in the network if it is used to serve any demand over any link. Constraints 3.10 and 3.11 guarantee that a wavelength of a link is used to serve a demand if any path which goes through him is serving that demand and vice-versa.

40 32 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Complexity Analysis The proposed ILP Model for the OCS RWA problem requires W + D E W + D P d W variables and D + 3 D E W + D E P d,e W constraints (where P d is the average path number for the demands of the demand set). The number of variables can be approximated to O( D E W ) and the number of constraints to O(4 D E W ) under the assumption that the network is composed by a small number of clusters. By comparing the complexity of this ILP model with the one of the OPS RWA problem it can be observed that the complexity of this ILP model is higher because it requires the double of constraints than the OCS RWA ILP model Hybrid This RWA problem focuses on hybrid optical networks which combine both OCS and OPS technologies. Its goal is to successfully establish all the demands by properly choosing whether they should be served with OCS or OPS technology and finding the optimal combination of ligthpaths which minimize the number of required wavelengths per link Problem Formulation The Hybrid RWA problem can be stated as Given: 1. a transparent optical network characterized by a graph G = (N, E), where N denotes the set of nodes and E = {(i, j),( j,i) : i, j N,i j} the set of physical links; 2. a set of wavelengths per physical link denoted as W ; 3. a set of demands D to be served, each one of them characterized by a source node R d, a destination node S d, a required bandwidth B d and a maximum bandwidth per lightpath (due to its packet loss ratio requirements) Bm d when it is served by OPS; 4. a real number M close to infinity. Find an OCS ligthpath or a set of OPS ligthpaths between the source and destination node of each demand of the demand set subject to the following constraints: 1. traffic constraint: all the demands of the demand set must be served; 2. demand mutual exclusion: a demand cannot be served by OCS and OPS simultaneously; 3. wavelength mutual exclusion: each wavelength of each link cannot serve OCS and OPS demands simultaneously;

41 Performance Evaluation OCS non-bifurcated flows: each demand which is served with OCS must use one single lighpath (the bandwidth of a demand cannot be split); 5. OCS wavelength clashing: each wavelength of each link can serve only one demand served with OCS; 6. OPS QoS constraint: the used bandwidth of a wavelength in a link cannot exceed the maximum bandwidth of any of the demands served with OPS which undergo it; 7. wavelength continuity: the same wavelength must be used in all the links of a ligthpath between the source and the destination nodes of a demand with the objective to minimize the number of required wavelengths per physical link of the network. Next is provided a link-path based ILP model for the Hybrid RWA problem ILP Model Let us define P as the set of paths in the physical network, P d as the set of p P associated with the demand d D, and P d,e as the set of p P d which traverse the edge e E. The decision variables of the ILP model are: Z w : binary; 1 if the wavelength w W is used to serve any demand, 0 otherwise. J d,e,w : binary; 1 if the wavelength w W is used to serve the demand d D over the edge e E with OCS, 0 otherwise. J d : binary; 1 if the demand d D is being served by OCS, 0 otherwise. K d,e,w : binary; 1 if the wavelength w W is used to serve the demand d D over the edge e E with OPS, 0 otherwise. K d : binary; 1 if the demand d D is being served by OPS, 0 otherwise. X d,p,w : binary; 1 if the path p P and the wavelength w W are used to serve the demand d D with OCS, 0 otherwise. Y d,p,w : real; amount of bandwidth of the demand d D which is being served by the path p P d and the wavelength w W with OPS. The ILP formulation is stated below: min Z w,s.t. (3.12) w W J d + K d = 1, d D (3.13) J d,e,w + K d,e,w 1, d D,d D,e E,w W (3.14)

42 34 Design and Evaluation of Architectures for Intra Data Center Connectivity Services k D d D p P d w W X d,p,w = J d, d D (3.15) p P d,e X d,p,w 1, e E,w W (3.16) p P d w W Y d,p,w = B d K d, d D (3.17) p P k,e Y k,p,w Bm d K d,e,w + (1 K d,e,w ), d D,e E,w W (3.18) Z w J d,e,w, d D,e E,w W (3.19) J d,e,w = p P d,e X d,p,w, d D,e E,w W (3.20) Z w K d,e,w, d D,e E,w W (3.21) K d,e,w Y d,p,w, d D,e E, p P d,e,w W (3.22) M p P d,e Y d,p,w K d,e,w, d D,e E,w W (3.23) Objective function 3.12 aims at minimizing the number of wavelengths required per physical link. Constraints 3.13, 3.15 and 3.17 are the traffic constraints which ensure that all the demands are served. Constraints 3.13 are also the demand mutual exclusion constraints which guarantee that demands are not served by both OCS and OPS simultaneously. Constraints 3.14 are the wavelength mutual exclusion constraints which assure that each wavelength of each link cannot serve OCS and OPS demands simultaneously. Constraints 3.15 are the OCS non-bifurcated flow constraints which ensure that the bandwidth of a demand served with OCS cannot be split. Constraints 3.16 are the OCS wavelength clashing constraints which guarantee that every wavelength of every link can only serve one OCS demand. Constraints 3.18 are the OPS QoS constraints which ensure that the used bandwidth of a wavelength in a link cannot exceed the maximum bandwidth of any of the demands served with OPS which undergo it. The wavelength continuity constraint is enforced due to the use of a link-path formulation. Constraints from 3.19 to 3.23 are auxiliary constraints which correspond to constraints 3.4 and 3.5 from the OCS RWA ILP model and constraints 3.9, 3.10 and 3.11 from the OPS RWA ILP model.

43 Performance Evaluation Complexity Analysis The proposed ILP Model for the Hybrid RWA problem requires W + 2 D + 2 D E W +2 D P d W variables and 3 D + E W + D E P d,e W +5 D E W + D 2 E W constraints (where P d is the average path number for the demands of the demand set). The number of variables can be approximated to O(2 D E W ) and the number of constraints to O( D 2 E W ) assuming that the network is composed by a small number of clusters and that the number of demands is large enough ( D >> 6). By comparing the complexity of the proposed ILP model for the Hybrid RWA problem with the one of the OPS RWA ILP model it can be concluded that its complexity is hugely larger (about twice the number of variables and D times the number of constraints). Our computational experiences have shown us that, for a considerable amount of demands ( D 100), the mean time required to solve an instance of the Hybrid RWA ILP is higher than 10 hours on a high-end computer (8 cores running at 2,5 GHz and 16 Gigabytes of RAM). Because of this, it is necessary to develop an heuristic method in order to obtain a suboptimal solution of the problem in a reasonable amount of time. In the next section a less complex variation of the Hybrid RWA problem is discussed Hybrid With Preliminary Mapping The Hybrid With Preliminary Mapping RWA problem is a less complex variation of the Hybrid RWA problem at which the decision of which technology (OCS or OPS) should be used to serve each one of the demands is taken before the ILP problem is solved. Therefore, the Hybrid With Preliminary Mapping RWA problem receives as inputs a set of OCS demands and a set of OPS demands instead of a single demand set. Both OCS RWA and OPS RWA problems can be though of as particular cases of this problem when one of the two demands sets is empty. The goal of this problem is to find the optimal combination of OCS and OPS ligthpaths which allow to serve all the demands of both sets while also minimizing the number of required wavelengths per link Problem Formulation The Hybrid With Preliminary Mapping RWA problem can be stated as Given: 1. a transparent optical network characterized by a graph G = (N, E), where N denotes the set of nodes and E = {(i, j),( j,i) : i, j N,i j} the set of physical links; 2. a set of wavelengths per physical link denoted as W ; 3. a set of OCS demands D OCS to be served, each one of them characterized by a source node R d, a destination node S d ;

44 36 Design and Evaluation of Architectures for Intra Data Center Connectivity Services 4. a set of OPS demands D OPS to be served, each one of them characterized by a source node R d, a destination node S d, a required bandwidth B d and a maximum bandwidth per lightpath (due to its packet loss ratio requirements) Bm d ; 5. a real number M close to infinity. Find a lightpath for each demand of the OCS demand set and a set of lightpaths for each demand of the OPS demand set subject to the following constraints: 1. traffic constraint: all the demands of both sets must be served; 2. wavelength mutual exclusion: each wavelength of each link cannot serve OCS and OPS demands simultaneously; 3. OCS non-bifurcated flows: each demand which is served with OCS must use one single lighpath (the bandwidth of a demand cannot be split); 4. OCS wavelength clashing: each wavelength of each link can serve only one demand served with OCS; 5. OPS QoS constraint: the used bandwidth of a wavelength in a link cannot exceed the maximum bandwidth of any of the demands served with OPS which undergo it; 6. wavelength continuity: the same wavelength must be used in all the links of a ligthpath between the source and the destination nodes of a demand with the objective to minimize the number of required wavelengths per physical link of the network. Next is provided a link-path based ILP model for the Hybrid With Preliminary Mapping RWA problem ILP Model Let us define P as the set of paths in the physical network, P d as the set of p P associated with the demand d D OCS DOPS, and P d,e as the set of p P d which traverse the edge e E. The decision variables of the ILP model are: Z w : binary; 1 if the wavelength w W is used to serve any demand, 0 otherwise. J d,e,w : binary; 1 if the wavelength w W is used to serve the demand d D OCS over the edge e E, 0 otherwise. K d,e,w : binary; 1 if the wavelength w W is used to serve the demand d D OPS over the edge e E, 0 otherwise. X d,p,w : binary; 1 if the path p P and the wavelength w W are used to serve the demand d D OCS, 0 otherwise. Y d,p,w : real; amount of bandwidth of the demand d D OPS which is being served by the path p P d and the wavelength w W.

45 Performance Evaluation 37 The ILP formulation is stated below: min Z w,s.t. (3.24) w W J d,e,w + K d,e,w 1, d D OCS,d D OPS,e E,w W (3.25) p P d w W X d,p,w = 1, d D OCS (3.26) d D OCS p P d,e X d,p,w 1, e E,w W (3.27) p P d w W Y d,p,w = B d, d D OPS (3.28) d D OPS p P d,e Y d,p,w Bm d K d,e,w + (1 K d,e,w ), d D OPS,e E,w W (3.29) Z w J d,e,w, d D OCS,e E,w W (3.30) J d,e,w = p P d,e X d,p,w, d D OCS,e E,w W (3.31) Z w K d,e,w, d D OPS,e E,w W (3.32) K d,e,w Y d,p,w, d D OPS,e E, p P d,e,w W (3.33) M p P d,e Y d,p,w K d,e,w, d D OPS,e E,w W (3.34) Objective function 3.24 aims at minimizing the number of wavelengths required per physical link. Constraints 3.25 are the wavelength mutual exclusion, they ensure that if a wavelength of a link is serving an OCS demand it cannot also serve an OPS demand and vice-versa. Constraints 3.26 and 3.28 are the traffic constraints which ensure that all the demands must be served. Constraints 3.26 are also the OCS non-bifurcated flows constraints which ensure that the bandwidth of an OCS demand cannot be split. Constraints 3.27 are the OCS wavelength clashing constraints which ensure that every wavelength of every link can only serve one OCS demand. The wavelength continuity constraint is enforced due to the use of a link-path formulation. Constraints from 3.30 to 3.34 are auxiliary.

46 38 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Constraints 3.30 and 3.31 are the same as constraints 3.4 and 3.5 of the OCS RWA model. Constraints from 3.32 to 3.34 are the same as constraints from 3.9 to 3.11 from the OPS RWA model Complexity Analysis The proposed ILP Model for the Hybrid With Preliminary Mapping RWA problem requires W + D OCS E W + D OPS E W + D OCS P d W + D OPS P d W variables and D OCS + D OPS + E W +2 D OCS E W +3 D OPS E W + D OPS E P d W + D OCS D OPS E W constraints (where P d is the average path number for the demands of the demand set). The number of variables can be approximated to O(( D OCS + D OPS ) E W ) and the number of constraints to O(( D OCS D OPS + 2 D OCS + 4 D OPS ) E W ) under the assumption that the network is composed by a small number of clusters. It can be observed that the number of variables and constraints of an instance of this problem highly depends on the number of elements of D OCS and D OPS. Therefore, the higher the size of the demand sets the higher the complexity of the problem. It can also be observed that the number of variables of this reduced problem is approximately the same as in the original problem but the number of required constraints depends on the ratio between D OCS and D OPS. Next, the values of D OCS and D OPS which maximize the number of required constraints are obtained. The Equation 3.35 defines the number of required constraints as a function of D OCS and D OPS. n = ( D OCS D OPS + 2 D OCS + 4 D OPS ) E W (3.35) Let us define D = D OCS DOPS. Therefore the Equation 3.36 can be introduced. D = D OCS + D OPS (3.36) The Equation 3.37 can be obtained by substituting the Equation 3.36 into Equation n = ( D OCS ( D D OCS ) + 2 D OCS + 4 ( D D OCS )) E W (3.37) The Equation 3.38 can be obtained by differentiating the Equation 3.37 respect to D OCS, equalling the result to zero and isolating D OCS. D OCS = D 2 2 (3.38) The Equation 3.39 can be obtained by substituting the result into the Equation 3.36.

47 Performance Evaluation 39 D OPS = D (3.39) It is now clear that given a certain size for the superset of demands D the number of required constraints is maximum when the size of both demand sets D OCS and D OPS are approximately half of the size of D. In such worst case scenario the limit when the size of the superset of demands tends to infinity of the ratio between the number of constraints required by the Hybrid With Preliminary Mapping ILP model and this simplified ILP model (Equation 3.40) can be obtained. lim D D 2 E W ( D 2 2 D D D +2 2 ) E W = 1 4 (3.40) Therefore it can be concluded that this simplified version of the problem requires at least 1/4 less constraints than the original problem, resulting in a reduction of the complexity. Also, due to the preliminary mapping of the demands the number of nodes of the branch and bound tree which need to be explored is significantly reduced, decreasing the complexity of the problem in several orders or magnitude. However, this comes at the cost of obtaining suboptimal solutions which highly depend on the criteria used during the preliminary mapping process. Our computational experiences have shown us that, when selecting a proper threshold, the solution provided by this simplified version of the problem is very close (10% gap) to the one obtained by solving the original problem. However, the mean execution time for an instance of this ILP problem is below 5 hours, providing a substantial decrease with respect to the original problem (more than 10 hours). In the next section several results for different instances of this ILP model are shown Benchmark Results In order to evaluate the performance of the Hybrid With Preliminary Mapping ILP model several instances of the problem have been solved. The characteristics of the performed tests are the following: Topology of the network consisting on 1 cluster of 16 ToR switches. Load offered to the network consisting on 100 demands. Bandwidth required by each demand B d randomly selected between 0.01 and 1 (1% and 100% of the capacity of a wavelength respectively) both included. Maximum bandwidth per OPS lightpath (due to packet loss ratio requirements) of each demand Bm d randomly selected from a set of predefined values Bm. Source and destination nodes R d and S d of each demand are randomly picked such that R d S d.

48 40 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Hardware setup consisting on a PC with a 8 core 64 bit CPU running at 2,5 GHz and 16 GB of RAM. ILP solver software IBM ILOG CPLEX Three different tests have been performed, each one of them with a different set of Bm values. For the first test Bm = {0.2, 0.3, 0.4}, for the second test Bm = {0.35, 0.4, 0.44} and for the third test Bm = {0.5,0.6,0.7}. First, the set of demands is generated. Then the required bandwidth of each demand B d of the set is compared to a threshold value T. If B d < T the demand is mapped to the OPS set, if not (B d T ) the demand is mapped to the OCS set. The threshold values which have been considered are from 0 to 1 (both included) in steps of 0.1. For the same generated demand set, several instances of the ILP problem are solved with different threshold values and the value of the objective function (the minimum number of required wavelengths per physical link) is obtained. Each test has been repeated 25 times (with 25 different generated demand sets) in order to obtain an average value of the objective function for each threshold value. The purpose of these tests is to find the threshold value which provides the best efficiency in the use of the network resources (a lower value of the objective function) in order to take the maximum benefit of the hybrid capabilities of the network. Table 3.1 shows the average of the obtained values of the objective function and the confidence interval for each one of the three scenarios. By inspecting the obtained results it can be concluded that if a proper threshold value for the hybrid architecture (thresholds from 0.1 to 0.9 both included) is selected the hybrid architecture provides a better performance than the OCS architecture (threshold 0) and the OPS architecture (threshold 1). Also, it can be appreciated that, for each scenario, the optimal threshold value is always near to the most restrictive of the Bm values. This is so because, on average, demands which require a higher bit-rate than that value would need to split their bandwidth and to use several wavelengths (due to the QoS constraint) if they are served with OPS while a single wavelength would be required with OCS. For such optimal threshold value, demands with low bit-rate and packet loss ratio requirements are served with OPS (providing a high multiplexing gain) while high bit-rate demands are served with OCS. The average execution time for instances with thresholds from 0 to 0.2 is in the order of few seconds. Instances with 0.3 and 0.4 thresholds present a mean execution time of 40 minutes. Instances with thresholds from 0.5 to 1 require an execution time around 5 hours. In the next section the details of the on-line scenario are discussed.

49 Performance Evaluation 41 Bm Threshold Obj. Value Conf 95% ± ± ± ± ± 0.58 {0.2, 0.3, 0.4} ± ± ± ± ± ± ± ± ± ± ±0.50 {0.35, 0.4, 0.44} ± ± ± ± ± ± ,72 ± 0, ± 0, ± ± ± 0.55 {0.5, 0.6, 0.7} ± ± ± ± ± ± 0.65 Table 3.1: Optimal Thresholds for the Hybrid With Preliminary Mapping ILP Model.

50 42 Design and Evaluation of Architectures for Intra Data Center Connectivity Services 3.2. On-line Scenario Once the network operator has estimated the amount of resources required to withstand a certain traffic load the network is put into operation. The on-line scenario reproduces the conditions encountered during such phase. The main difference between the on-line and the off-line scenarios relates to the degree of knowledge about the demands characteristics such as arrival time, holding time, bandwidth and packet loss ratio requirements. On the off-line scenario a set of demands whose characteristics are all known in advance is considered. Therefore, by knowing all these data, it is possible to obtain an optimal solution (the minimum number of required resources to serve all the demands). However, during the network operation stage it is not possible to certainly know all the characteristics of the demands which are yet to come. Therefore, it is not possible to obtain an optimal solution without knowing all the information. On the on-line scenario demands are served and teared down one by one based on their arrival and holding times and the best decision is taken by taking into account the current state of the network and the characteristics of the demand. In the next sections, the details of a simulator software which allows us to evaluate the performance of the proposed architecture are discussed. Next, several benchmark results are presented in order to match the proposed hybrid architecture against an only OCS architecture and an only OPS architecture Simulator Design In order to evaluate the performance the proposed architecture a simulator software has been developed. This simulator receives as input parameters the characteristics of the desired topology such as the number of clusters, ToR switches per cluster and wavelengths per link. Also, a number of demands, a threshold value, a load interval and a load step value need to be specified. The simulator performs a load sweep between the specified interval. It starts by offering to the network a load equal to startload and then increases the load in steps of stepload until it finally reaches endload. Therefore, a total of (stopload startload)/stepload simulations are performed. For each one of the simulations the blocking probability and the mean occupation of all the links are obtained. The specified threshold value is used to determine which switching technology (OCS or OPS) should be used to serve a demand. The simulator has been designed following an event-driven approach. Each time a demand starts or finishes being served an event is triggered. This events modify the state of the network (by trying to use or to free several resources). Therefore the behaviour of the network can be emulated by running this events in the same order which they would happen. The number of events to be generated is also an input parameter of the simulator.

51 Performance Evaluation 43 The overall simulator flow diagram is shown in the Figure 3.1. First, the network topology is generated according to the clusters, ToR switches per cluster and wavelengths per link parameters. Then, for each load value from startload to endload in steps of stepload the events are generated and executed and finally the statistics are collected. Figure 3.1: Overall Simulator Flow Diagram. Next, the details of the algorithms used during the demand and event generation and the event execution are detailed Demand Generation The demand generation process follows the Algorithm 1. For the desired number of demands to be generated M a bandwidth B d between 0.01 and 1 (1% and 100% of the capacity of a wavelength respectively) is randomly chosen. Next, a maximum bandwidth per wavelength (due to packet loss ratio requirements) Bm d is randomly chosen from a set of predefined values Bm. After that, the source node R d and the destination node S d are randomly chosen from the set of ToR switches N such that R d S d. Next, the decision of which switching technology (OCS or OPS) will be used to serve the demand is taken by comparing the selected B d value with the threshold value T. If B d < T the demand will be served with OPS technology, if not (B d T ) the demand will be served with OCS

52 44 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Input: m, N, Bm, T Output: D for from m = 0 to d = M do B d = rand{[1,100]}; Bm d = rand{bm}; R d, S d = rand{n}, R d S d ; if B d < T then d = new opsdemand(r d,s d,b d,bm d ); end else d = new ocsdemand(r d,s d,b d,bm d ); end add d to D; end return D; Algorithm 1: Demand Generation. technology. Finally the new demand is added to the demand list D. Once all the demands have been generated the demand list D is returned Event Generation An event corresponds to a change in the state of the resources of the network which is produced when a demand is started or stopped being served. Two types of events have been considered: The up event and the down event. An up event consists on trying to allocate and use network resources in order to serve a demand. A down event consists on freeing the resources which have been used by a demand which does no longer need them. Each up event has associated a down event. Since the events modify the state of the network, the order at which they are executed is of a vital importance. In order to fully reproduce the behaviour of a network which is offered a certain load the events need to be run in the same order as they would happen. This is related with the inter-arrival time and holding time distributions of the demands. Both inter-arrival time and holding time of the demands are considered to follow a decreasing exponential distribution with mean values IAT and HT respectively. Therefore, the load can be expressed as the quotient between HT and IAT (Equation 3.41). Load = HT IAT (3.41)

53 Performance Evaluation 45 Input: D, interarrivaldistribution, holdingdistribution Output: F uptimestamp = 0; for each d D do uptimestamp = interarrivaldistribution.sample() + uptimestamp; e = new upevent(d, uptimestamp); add e to F; downtimestamp = holdingdistribution.sample() + uptimestamp; e = new downevent(d, downtimestamp); add e to F; end sort F by increasing time stamp order; return F; Algorithm 2: Event Generation. The event generation process follows the Algorithm 2. This algorithm generates an up an a down event for each one of the demands of the demand set D. Therefore given a set of demands D, a total of 2 D events will need to be generated. First, the variable uptimestamp which will store the time of the last demand arrival is declared and initialized to zero. Then the time stamp of the new up event uptimestamp is obtained by taking a sample of the inter-arrival time distribution interarrivaldistribution.sample() and then adding the obtained value to the time stamp of the last up event uptimestamp. Finally, a new up event e associated to the demand d with a time stamp uptimestamp is generated and added to the event list F. In order to obtain the time stamp of the down event downtimestamp, a sample of the holding time distribution holdingdistribution.sample() is took and the obtained value is added to the time stamp of the associated up event uptimestamp. Finally, a new down event e associated to the demand d with a time stamp downtimestamp is generated and added to the event list F. Once all the up and down events have been generated and added to the event list F they are sorted by increasing time stamp order Event Execution One all the events have been generated they are executed one by one by increasing time stamp order. If the event is an up event then the Algorithm 3 is executed. This algorithm tries to find free a lighpath (for OCS demands) or a set of free lighpaths (for OPS demands) between the source and destination racks. In order to do so, it iterates over a set of paths P d which contains all the paths between the source and destination racks of the demand looking for a suitable combination of wavelengths which provide enough free bandwidth to establish the demand.

54 46 Design and Evaluation of Architectures for Intra Data Center Connectivity Services First, a variable which contains the amount of bandwidth of the demand which is yet to be allocated le ftusage is declared and initialized to B d. Then, for each lightpath the usage of the most loaded link of the path peakusage is found. The usage of a wavelength w on a link e is computed as the sum of the usages of all the demands of the set of established demands H which undergo that wavelength on that link. Next, the maximum allowed usage per OPS lightpath (due to packet loss ratio requirements) maxusage is found. It can be computed as the minimum of the Bm d values of all the demands which undergo any of the links of the path on that wavelength. Then, the available bandwidth over the lighpath margin is computed as the difference between peakusage and the minimum between the maxusage and the maximum allowed load of the current demand Bm d. If margin == 100 then the full bandwidth of the lighpath is available, allowing both new OCS and OPS connections to be established. If 0 < margin < 100 then some bandwidth is available in the lighpath to be used by OPS connections (but not by OCS). If the up event is associated to an OCS demand the algorithm will explore all the lighpaths until it finds a free wavelength with margin == 100. Then the lighpath (w, p) will be added to a list of lighpaths G and le ftusage will be set to zero (since an OCS lighpath provides the bandwidth of a full wavelength). If the up event is associated to an OPS demand the algorithm will explore all the lighpaths trying to allocate as much bandwidth as possible in each one of them until the full bandwidth of the demand is successfully allocated over a set of lighpaths. Each time a lighpath with margin > 0 is found, an amount of bandwidth usage equal to min(margin,le ftusage) will be allocated over that lighpath and le ftusage will be decreased by the same amount. The lightpath (w, p,usage) will then be added to the list of lighpaths G. After a new lightpath is added to the lightpath list G the value of le ftusage is checked. If le ftusage reaches zero this means that all the bandwidth requested by the demand has been successfully allocated. Finally, all the lighpaths of the lighpath list G are established. If after exploring all the available lighpaths there is still bandwidth pending to be allocated (le ftusage > 0) the demand is blocked. If the event is a down event and its associated up event was not blocked (did successfully establish a ligthpath or a set of lighpaths) then Algorithm 4 is executed. All the lightpaths of the lightpath list G are freed. If the associated up event was blocked then there are no resources which need to be freed and therefore the algorithm does nothing.

55 Performance Evaluation 47 Input: e le ftusage = B d ; for each p P d do for each w W do peakusage = max(u 1,U 2,...,U e ), U e = d H w,e B d e p; maxusage = min(um 1,Um 2,...,Um e ), Um e = min(b 1,B 2,...,B d ) d H w,e,e p; margin = min(maxusage,bm d ) peakusage; if (d is to be served with OCS) && (margin == 100) then l = new ocsligthpath (w, p); add l to G; le ftusage = 0; end else if (d is to be served with OPS) && (margin > 0) then usage = min(margin, le f tusage); l = new opslightpath (w, p,usage); add l to G; le ftusage = le ftusage usage; end if le ftusage == 0 then for each l G do connect l; end return; end end end demand is blocked; Algorithm 3: Up Event Execution.

56 48 Design and Evaluation of Architectures for Intra Data Center Connectivity Services Data: e, G if e associated up event was not blocked then for each l in G do disconnect l; end end Algorithm 4: Down Event Execution Benchmark Results In order to evaluate the performance of the proposed hybrid architecture several simulations have been executed. The characteristics of the performed tests are the following: Topology of the network consisting on 1 cluster of 16 ToR switches. Load offered to the network consisting on demands. Inter-arrival time of the demands follows an exponentially decreasing distribution with a mean value of IAT. Holding time of the demands follows an exponentially decreasing distribution with a mean value of HT. Bandwidth required by each demand B d randomly selected between 0.01 and 1 (1% and 100% of the capacity of a wavelength respectively) both included. Maximum bandwidth per OPS lightpath (due to packet loss ratio requirements) of each demand Bm d randomly selected from a set of predefined values Bm. Source and destination nodes R d and S d of each demand are randomly picked such that R d S d. Hardware setup consisting on a PC with a 8 core 64 bit CPU running at 2,5 GHz and 16 GB of RAM. Three different tests have been performed, each one of them with a different set of Bm values. For the first test Bm = {0.2, 0.3, 0.4}, for the second test Bm = {0.35, 0.4, 0.44} and for the third test Bm = {0.5,0.6,0.7}. For each test a load sweep is performed by setting the IAT value to 1 and by increasing the HT value from 400 to 1000 in steps of 5. The purpose of these tests is to find the threshold value which provides the best efficiency in the use of the network resources (a lower BP value) in order to take the maximum benefit of the hybrid capabilities of the network. Figures 3.2, 3.3 and 3.3 show the resulting BP for each load value when Bm = {0.2,0.3,0.4}, Bm = {0.35, 0.4, 0.44} and Bm = {0.5, 0.6, 0.7} respectively.

57 Performance Evaluation 49 Figure 3.2: Blocking Probability for Bm = {0.2, 0.3, 0.4}. Figure 3.3: Blocking Probability for Bm = {0.35, 0.4, 0.44}.