Informatica Universiteit van Amsterdam. Open-source network operating systems: feature evaluation. Erik Puijk,

Transcription

1 Bachelor Informatica Informatica Universiteit van Amsterdam Open-source network operating systems: feature evaluation of SONiC Erik Puijk, June 7, 2018 Supervisor(s): Dr. Paola Grosso, Lukasz Makowski MSc Signed:

2 2

3 Abstract Open network switches are increasing in popularity and allow the deployment of different open-source network operating systems (NOS). In contrast to locked-in switches, open switches with an open-source NOS have been tested less extensively as they are quite new phenomena. This thesis examines whether open switches with an open-source network operating system, namely SONiC, can be deployed successfully to perform fundamental networking operations. Furthermore, it examines for what use cases SONiC is suitable and beneficial. We experiment with open switches in various topologies to examine the deployment of fundamental OSI Layer 2 and Layer 3 networking features. We conclude that all SONiC-supported features we tested can be deployed successfully. Moreover, we examine several use cases of open switches with SONiC and conclude that SONiC is most suitable for use cases in large-scale data centers and enterprise networks. 3

4 4

5 Contents 1 Introduction 7 2 Open networking background Open switches Network operating systems ASIC communication ASIC control module SONiC Quagga Networking features Layer 2 features Layer 3 features Experiments Preparatory phase Mellanox SN2100 and ONIE Arista 7050QX-32S and Aboot Feature tests Layer 2 features Layer 3 features Result overview Discussion Ease of use Use case scenarios for open switches Current use cases Example use cases Conclusions Future research Acknowledgements 37 9 Bibliography Appendix 41 5

6 6

7 CHAPTER 1 Introduction Network switches are essential in computer networks, to connect devices and forward frames between them on the OSI data link layer (Layer 2)[1]. Some switches are also capable of OSI Layer 3 features, such as the routing of segments among networks using IP. In this thesis, we discuss exactly this category of devices. Traditionally, switches are sold with locked-in hardware with a pre-installed network operating system (NOS) on them, without the possibility for a network administrator to install third-party NOS s or other software on them. An open switch, in contrast, does allow the user to install another operating system on the device. Open switches thus give network administrators more possibilities to customize the switch to their own needs, possibly explaining their rising popularity. Another advantage of this category of switches is the reduction of expenses, due to the possibility to install low-cost software. In the past, this cost reduction would be compensated by an increase in operational costs, because using the switches would require hiring external Linux-expertise to configure the switches [2]. However, as Linux expertise has grown over the past few years, this barrier for using open switches has become much thinner. Also, large manufacturers such as Dell and HP have been developing open switches on which NOS s like Cumulus Networks or Pica8 are already installed [3]. This removed another obstacle from companies using the switches because no manual installation is required. The increasing popularity raises questions about the suitability for use in real networks, the ease of use and how their feature sets compare to those of traditional switches. Considering that open switches are relatively new compared to locked-in switches, there is still need for testing and evaluation to assess whether open switches can replace traditional switches without loss of functionality, performance or ease of use, which the reduction in costs might not weigh up to. In this context, we examine the functioning of open switches running an open-source network operating system, namely SONiC, in a network. We study whether open switches with SONiC are able to deploy several fundamental networking features. In addition, we examine which use cases benefit from the open and flexible nature of open switches with SONiC. We therefore set out to answer the following two research questions: 1. Which networking features can be successfully deployed on open switches with SONiC? 2. Which use cases are (more) easily supported by open switches with SONiC? Chapter 2 provides background information about open networking, open switches, network operating systems (SONiC specifically) and routing suites. Chapter 3 will briefly discuss several networking features that will later be used in our experiments. Chapter 4 contains the experiments we performed to answer our first research question. In chapter 5, we discuss the experimental methods and results and other findings obtained during this research. In chapter 6, we examine several use cases of open switches with SONiC. Lastly, in chapter 7 we return to our research questions, formulate a conclusion and suggest possibilities for future research. 7

8 8

9 CHAPTER 2 Open networking background 2.1 Open switches Generally, a switch can be represented as a stack of four layered components. Figure 2.1 shows these components. Control and management plane Network operating system Hardware Silicon/ASIC Figure 2.1: Layered component stack of a switch. The silicon, or ASIC (application-specific integrated circuit), is a specified hardware element designed for a specific task. In the case of switches, this task is to quickly send packets through the network [4]. The hardware-layer includes all other physical components of the switch, like the interfaces, the input/output-ports, the LEDs and the power supply [5]. The network operating system (NOS) controls the hardware and the underlying ASIC for networking purposes and allows control and management plane applications to use the hardware. Control plane and management plane applications provide particular features to the user of the switch, in addition to those of the underlying operating system [6]. To understand the difference between open switches and traditional switches, one needs to consider the manner in which the above components interact with each other. Switches in which the NOS and the underlying hardware are disintegrated, meaning that they can be changed independently of each other, are called open switches. In traditional (locked-in) switches, this is not possible, for the switch is delivered with pre-installed software that cannot be changed. Open switches thus give the user more choice in what NOS to run. Open switches can be separated into subcategories. Bare metal switches provide the hardware required to run, and allow the user to load the NOS of choice. The manufacturers of bare metal switches are original design manufacturers (ODMs) for well-known switch merchants, which means that the ODMs products are re-branded and sold by other companies. A boot-loader allows the user to boot an NOS of choice on the device [7]. 9

10 2.2 Network operating systems A network operating system is a key component in the aforementioned component stack of open switches. The NOS controls the hardware of the device and provides applications on the switch with the hardware and software resources they need. These resources might for example be memory allocation or input and output resources ASIC communication The aforementioned application-specific integrated circuits (ASICs) are designed to handle a specific task. In networking switches, the ASIC is designed and optimized for quickly processing incoming packets according to the routing table. In order for an NOS to be able to program the ASIC, several APIs have been developed to communicate with the ASIC. The Switch Abstraction Interface 1 (SAI) is a well-known method. It is an open source framework that aims to abstract away from the ASIC, which differs for each vendor, so software can be programmed for use in multiple different switches without any changes. This allows for more freedom in the use of software independently from the hardware choice [8]. Another, less adopted method for ASIC communication is the use of SDKs developed by the ASIC vendor. In practice, this approach is not included in open-source software for open switches considering changes in the SDKs would need the application to be modified. Examples of these SDKs are SwitchX SDK by Mellanox [9] and OpenNSL by Broadcom [10] ASIC control module On top of the ASIC API, the ASIC control module provides an interface for control plane applications to communicate with the hardware. It also presents the current state of the hardware to the user of the switch. Control plane apps can use the ASIC control module to read from or write to data stored in the hardware. These applications therefore are independent from the hardware in the machine they are running on. Figure 2.2 shows the role of the ASIC control module and the ASIC API in the component stack illustrated before. Control and management plane Network operating system ASIC control module ASIC API Hardware Silicon/ASIC Figure 2.2: ASIC control module and ASIC API in the layered component stack

11 2.3 SONiC SONiC 2 (Software for Open Networking in the Cloud) is an open-source network operating system that claims to include all features to have a fully functional Layer 3 network device. It is under constant development by Microsoft. The latest release, SONiC , supports features including BGP, LLDP, link aggregation/lacp and VLAN (trunking) [11]. SONiC is Linux-based and runs on Debian Jessie. The SONiC architecture is depicted in figure 2.3. Figure 2.3: An overview of SONiC components [12]. SONiC uses SAI for programming the ASIC, which makes SONiC compatible with all ASICs that are supported by SAI. The SONiC Object Library 3 allows external applications to interact with each other and with SONiC applications. Switch State Service The Switch State Service (SwSS) acts as the ASIC control module for SONiC, using a database to interface between the network applications running on the switch and the switch hardware. Figure 2.4 shows the SwSS architecture. Network applications use a database named APP DB for reading and writing. Orchestration agents are responsible for synchronization between APP DB and another database, namely ASIC DB. To provide universality, the databases are set up as keyvalue storage. SwSS allows network applications running on SONiC to be completely independent from the hardware they are running on

12 Figure 2.4: The design of Switch State Service, showing the relation between APP DB, ASIC DB and the orchestration agents [12]. Since SONiC is open-sourced, the networking community can contribute to the improvement of SONiC. In addition, SONiC can be extended with other open-source software, be it third-party or proprietary. Figure 2.5 shows how SONiC is a part of the Open Compute Project, which is an organization that has as its main goal to design and deploy the most efficient and scalable hardware for use in data centers. OCP certified switches can use SONiC through SAI to run a Layer 2 or Layer 3 switch, and in addition allows the user to extend the NOS with third-party or OCP software components [13]. Figure 2.5: SONiC as an open switching platform for OCP users [13] Quagga Routing suites are control plane software collections that offer a variety of routing protocols to run on top of a network operating system. A routing suite exchanges routing information with other routers and updates routing information in the kernel. Quagga 4 is an example of a routing suite and provides BGP routing functionality for SONiC. It is open-source and currently supports GNU/Linux and BSD. There are two main Quagga processes present on a SONiC device: bgpd and zebra. Figure 2.6 shows how Quagga and SONiC interact when a BGP route advertisement is received. First, this advertisement is passed to bgpd, short for BGP daemon. The BGP daemon processes this route and determines whether it should be placed in the kernel routing table. If so, the route is passed to zebra, which after additional filtering updates the routing table in APP DB. This is

13 done with fpmsyncd, which is a Forwarding Plane Manager and programs the forwarding plane. Fpmsyncd uses the SONiC Object Library API. SONiC handles updating the kernel routing table. Zebra is also capable of selecting the best route across different routing protocols, but as SONiC supports only BGP this is not applicable to a SONiC device. Figure 2.6: The interaction between Quagga and SONiC when a new BGP route is received [12]. Quagga determines whether a new route should be placed in the routing table, after which SONiC takes care of updating the kernel routing table. 13

14 14

15 CHAPTER 3 Networking features This chapter briefly covers several networking features or protocols that are relevant to this research. We selected both OSI Layer 2 and 3 features that we consider fundamental to a data center switch. 3.1 Layer 2 features Layer 2 of the OSI model is the data link layer. It provides mechanisms to allow communication between devices within the same local area network (LAN). Data elements transferred on the data link layer are called (link-layer) frames. A frame is transferred between two devices on a physical link, and MAC addresses are used to address link-layer frames. The Ethernet protocol is a well-known example of a Layer 2 protocol and is used often in local area networks and wide area networks. WiFi is an alternative technology, although it is used only in local area networks [14]. LLDP The Link Layer Discovery Protocol (LLDP) allows networking devices to advertise information about their identity and capabilities. LLDP is useful for managing network resources and provides a vendor-independent mechanism for devices to exchange device information within a network. Each interface of a device sends out a frame consisting of an LLDP Data Unit (LLDPDU). An LLDPDU consists of a sequence of type-length-value (TLV) structures, each containing specific information. Each LLDPDU contains the following mandatory TLVs: Chassis ID, Port ID and Time To Live. Furthermore, optional TLVs can be included as well, such as system name, system capabilities and port description. LLDPDU frames are sent at a fixed interval, to keep the information up-to-date [15]. LACP The Link Aggregation Control Protocol (LACP) can be used to combine multiple physical ports and form a single logical channel. This process is also called link aggregation. To higher layer protocols, the aggregated links form a single channel. Link aggregation provides increased throughput due to the use of multiple physical links. It also adds redundancy because even with a physical link-failure, traffic is still capable of flowing through the logical link by making use of the other (operative) physical links [16]. STP The Spanning Tree Protocol (STP) aims at constructing and managing a loop-free topology in a bridged network. STP is supposed to prevent loops in a topology and guarantees that there are unique paths to all destinations within a topology. This is critical in local area networks, for 15

16 loops can cause Ethernet frames to be switched around endlessly. For instance, when a switch receives an ARP packet, it broadcasts this packet through all of its interfaces (except the one the ARP packet was received on). If a loop exists in the network of the switch, it will eventually receive its own broadcast ARP packet, which will be broadcast again. This endless process is known as a broadcast storm and it is one of the phenomena STP should prevent [17]. It does so by selecting a Root Bridge, and then creates a loop-free topology by blocking particular interfaces such that all other bridges have a shortest path to the Root Bridge [18]. Figure 3.1: Example of how STP creates a loop-free topology with shortest paths to the Root Bridge [19] Figure 3.1 shows an example of a topology with a loop. Switch A is selected as Root Bridge, thus all other switches should have a shortest path to switch A. By blocking the link between switch B and switch C, the loop in the topology is removed and all switches have a shortest path to switch A. VLAN (trunking) A Virtual Local Area Network (VLAN) can be described as an isolated and partitioned broadcast domain at OSI Layer 2. A Local Area Network (LAN) can be partitioned into several Virtual LANs to create multiple (logically) separate networks despite the (possible) presence of a physical connection to other devices that are not in the VLAN. This is achieved by configuring VLANs on a switch and specifying which switch interface belongs to which VLAN. The ability to define isolated broadcast domains using software makes VLAN a well-known networking feature. It allows network administrators to create multiple separated broadcast domains without having to change the physical wiring of the network [20]. A trunk link can be used to transport frames from multiple VLANs, for instance to interconnect two switches that have configured the same VLANs. This is achieved by using tags in link-layer frames to specify what VLAN a frame belongs to. These trunk links are therefore called tagged links. Links that can only transport frames from one specific VLAN are called untagged links and thus the frames that are transported on these links do not contain a VLAN tag. Figure 3.2 shows an example. The entirely blue or entirely red links are untagged links for VLAN 100 and 200 respectively, and the frames transported on these links do not contain a VLAN tag. The link between switch 1 and 2 is a tagged link, for it must transport frames from both VLAN 100 and 200 and thus a tag must be used to indicate the corresponding VLAN for a particular packet. 16

17 Figure 3.2: Example of a VLAN configuration with two switches and two VLANs [21]. trunk link between switch 1 and 2 transports traffic from both VLAN 100 and VLAN 200. The 3.2 Layer 3 features Layer 3 of the OSI model, the network layer, is responsible for the routing of data between different local area networks (LANs). Network-layer packets are addressed using unique and hierarchical IP addresses [14]. Routers are devices that forward packets to their destination by determining the route to be taken by the packet to reach its destination. In a network, routers communicate with each other to exchange information about the availability of subnets. This is done by routing protocols, which in addition define policies or algorithms that determine the best route to a destination when multiple routes are available. Inter-VLAN routing Inter-VLAN routing is used when traffic must flow between different VLANs, which in some cases may be necessary. Typically, this is accomplished by specifying a VLAN interface for each relevant VLAN, allowing the switch to route the traffic between VLANs through these VLAN interfaces. Figure 3.3 shows an example of a router (R1) that routes between VLANs. When PC A sends traffic destined for PC B, the traffic is first tagged with VLAN 20. When this traffic reaches router R1, it routes this traffic to VLAN 30 by using the configured VLAN interfaces. Then, this traffic can proceed to PC B with the VLAN 30 tag. Figure 3.3: Example of a topology in which inter-vlan routing is needed [22]. Router R1 uses the VLAN interface addresses to route the traffic from VLAN 20 to VLAN

18 BGP An autonomous system (AS) is a selection of IP prefixes that is managed by a certain network administrator. Commonly, such autonomous systems are managed by internet providers or large companies. The Border Gateway Protocol (BGP) is a well-known routing protocol used for routing between different autonomous systems, but can also be used to route within an AS. Routers running BGP can set up sessions with each other using TCP to exchange routing information, for instance subnet availability. BGP provides several mechanisms to control the propagation of routes, such as route-maps. Route maps allow the network administrator to define rules for certain prefixes [23]. For instance, there may be rules to drop an incoming route or change it before placing it in the routing table. BGP is used commonly in large-scale data centers because it scales well compared to other routing protocols [24]. 18

19 CHAPTER 4 Experiments 4.1 Preparatory phase To examine whether open switches with SONiC can be deployed successfully in a network, we decided to perform experiments on the deployment of the fundamental networking features that were described in chapter 3. The SNE OpenLab has provided two open switches for this research. Mellanox SN Arista 7050QX-32S 2 The ASICs in these switches are manufactured by different vendors: Mellanox and Broadcom, respectively. This allows us to test two different ASIC types at once, and shows whether SONiC can run on two different ASICs. Both switches are capable of operating on OSI Layer 2 and Layer 3. Figures 4.1 and 4.2 show photos of Mellanox SN2100 and Arista 7050QX-32S, respectively. More details of the switches can be found in Appendix A. Figure 4.1: Mellanox SN2100. Figure 4.2: Arista 7050QX-32S

20 The open-source NOS used in the experiments is SONiC. Section 2.3 has briefly explained the architecture of SONiC and what features it supports. SONiC is under constant development and new features are added regularly. In addition, the absence of other network operating systems that support a wide variety of open switches made SONiC an obvious NOS to experiment with. Aside from SONiC, there are various open-source network operating systems, such as OPX. SONiC, however, supports a wide variety of switches and thus allows for the straightforward extension of this research to other supported devices. Details about the version of SONiC used in this research are provided in Appendix B Mellanox SN2100 and ONIE Mellanox SN2100 comes with Mellanox Onyx as NOS. Other operating systems can be installed over ONIE 3 (Open Network Install Environment). It is an open-source project by Cumulus Networks that allows creating an open networking environment and is present by default on various open switches [25]. Figure 4.3 shows how ONIE can be used to boot a network operating system. When a device boots for the first time, a low-level boot loader boots ONIE from flash Figure 4.3: Process of first time boot using ONIE [25]. memory. ONIE then fetches the image of the NOS supplied by the switch vendor and installs this image. The next time the device boots, ONIE is not used by default and the device goes straight in the NOS. ONIE can, however, still be used to uninstall the vendor NOS and install another (open source) NOS from an image provided over the network or via USB for instance [25] Arista 7050QX-32S and Aboot Arista 7050QX-32S runs Arista EOS by default. A component of this operating system is Aboot 4, which can be used to boot image files to run other operating systems. Boot parameters can be configured in files which are stored in flash memory. In addition, Aboot provides a shell which can be used to modify boot configuration [26]

21 4.2 Feature tests In the last chapter, we discussed several networking features we consider crucial to a network switch. The objective was to test whether each of the features worked correctly on Mellanox and Arista running SONiC. The results of the experiments possibly provide an insight into whether the switches can be deployed in a data center network as a replacement of traditional switches. For most experiments, we used SONiC s config db.json configuration file to set up our testing environment. Appendix C contains the most relevant configurations. Complete configuration files can be found in the GitHub repository Layer 2 features LLDP Methods The first feature to be tested was LLDP. To verify whether LLDP correctly exchanged device information, we considered the topology depicted in figure 4.4. We wanted to verify whether all mandatory LLDP TLVs were exchanged between Mellanox and Arista. This mandatory information is as follows [15]: Chassis ID Port ID Time To Live Furthermore, we examined whether any additional information, such as system information and device capabilities, is exchanged through LLDP as well. Mellanox SN2100 Ethernet32 Ethernet36 Ethernet16 Ethernet20 Arista 7050QX-32S Figure 4.4: Topology to analyze LLDP with Mellanox and Arista. Results SONiC provides the command show lldp neighbors to view the LLDP neighbor information registered by SONiC. Appendix D1 presents the complete output of this command. Both devices presented the mandatory LLDP information in their LLDP tables. SONiC runs LLDP so that the Chassis ID that was included is taken from the MAC address of the eth0 interfaces on both switches. In addition, LLDP exchanged correct system name and system description information, and also included device capabilities and which of these capabilities were currently in use. LACP Methods In addition, we tested LACP/link aggregation behaviour. With the topology depicted in figure 4.5, we tried to configure link aggregation combining the double links between Mellanox and Arista. We considered both Layer 2 link aggregation and Layer 3 link aggregation. The latter has an IP address configured for each link aggregation interface, while the former does not. A working port channel should provide redundancy in the sense that when one of the links fail, communication is still possible through the port channel by making use of the other link. Thus, to test an aggregated link, one can simulate a physical breakdown of one of the two links and examine whether communication is still possible between the two devices

22 PortChannel1 Mellanox SN2100 Ethernet32 Ethernet36 Ethernet16 Ethernet20 Arista 7050QX-32S Figure 4.5: Topology to analyze LACP with a port channel consisting of two physical links between Mellanox and Arista. Results Appendix C1.1 shows the configuration we set for a Layer 2 port channel between Mellanox and Arista. We were unable to configure a successful Layer 2 port channel. That is, SONiC could not successfully set up a port channel with no interface addresses. PortChannel1 failed to go UP on both devices. Listing 4.1 shows that PortChannel1 was stuck in DOWN state. admin@sonic - mellanox :~$ ip a show dev PortChannel1 4: PortChannel1 : <NO - CARRIER, BROADCAST, MULTICAST, UP > mtu 9100 qdisc noqueue state DOWN group default link / ether ec :0d:9a:8d:f1:c0 brd ff:ff:ff:ff:ff:ff Listing 4.1: PortChannel1 failed to go UP. Moreover, with teamdctl we were able to view the state of PortChannel1 within the SONiC teamd docker, which is responsible for link aggregation. On both Mellanox and Arista, no ports were present as slave of PortChannel1. Listing 4.2 shows this for Mellanox. It was confirmed by a SONiC collaborator that port channel functionality is currently aimed at Layer 3 port channels and not at Layer 2 port channels 6. admin@sonic - mellanox :~$ docker exec - it 7 e504bdc03e0 bash root@sonic - mellanox :/# teamdctl PortChannel1 state view setup : runner : lacp runner : active : yes fast rate : no Listing 4.2: No slave ports present for PortChannel1. Appendix C1.2 shows the configuration of the Layer 3 port channel. Layer 3 link aggregation did work correctly. In this case, PortChannel1 did go UP and allowed us to communicate between the interface IP addresses configured on PortChannel1 ( /24 on the Mellanox side and /24 on the Arista side). Also, teamdctl now showed us the correct slave ports for PortChannel1. In addition, Layer 3 link aggregation passed our redundancy tests. Listing 4.3 shows an example of this (some output has been omitted for brevity). Appendix D2 shows more complete output of the example. In listing 4.3, it can be seen on lines 4-14 that when both links are operational, communication through the port channel is possible. In lines 20-21, one can see that one of the links is not operational anymore. Line 27 shows that despite the fact that one link broke down, PortChannel1 is still operational. Next, lines show that communication is possible through the port channel even when one of the two links is not operational. This indicates that Layer 3 link aggregation provides a redundant connection between Mellanox and Arista

23 1 # ( both links up) 2 3 # (...) 4 Ethernet32 32,33,34,35 N/ A 9100 Ethernet32 up up 5 Ethernet36 36,37,38,39 N/ A 9100 Ethernet36 up up 6 # (...) 7 8 # ( communication is fine ) 9 10 admin@sonic - mellanox :~$ sudo ping PING ( ) 56(84) bytes of data bytes from : icmp_seq =1 ttl =64 time =0.362 ms bytes from : icmp_seq =2 ttl =64 time =0.220 ms bytes from : icmp_seq =3 ttl =64 time =0.320 ms 15 # (...) # ( link failure Ethernet32!) # (...) 20 Ethernet32 32,33,34,35 N/ A 9100 Ethernet32 down up 21 Ethernet36 36,37,38,39 N/ A 9100 Ethernet36 up up 22 # (...) # ( PortChannel1 still UP) # (...) 27 4: PortChannel1 : <BROADCAST, MULTICAST, UP, LOWER_UP > mtu 9100 qdisc noqueue state UP group default 28 # (...) # ( communication still working ) admin@sonic - mellanox :~$ sudo ping PING ( ) 56(84) bytes of data bytes from : icmp_seq =1 ttl =64 time =0.294 ms bytes from : icmp_seq =2 ttl =64 time =0.310 ms bytes from : icmp_seq =3 ttl =64 time =0.296 ms 37 # (...) Listing 4.3: Behaviour of PortChannel1 when link Ethernet32 (Mellanox)-Ethernet16 (Arista) fails (some output has been omitted for brevity). STP Methods We also examined the Spanning Tree Protocol. Considering the topology in figure 4.6, if no STP is configured at all, a broadcast storm may occur because there is a network loop. Broadcast messages sent from Mellanox to Arista out of both Ethernet32 and Ethernet36 will be circulated because Arista will broadcast these back to Mellanox through Ethernet20 and Ethernet16, respectively, leading to an endless broadcast storm. STP must prevent this by selecting a Root Bridge and blocking one of the links between Mellanox and Arista. Mellanox SN2100 Ethernet32 Ethernet36 VLAN 100 Ethernet16 Ethernet20 Arista 7050QX-32S Figure 4.6: Topology to analyze STP. The current version of SONiC does not have support for STP, but we decided to try to configure STP nevertheless using brctl. We placed the relevant ports in VLAN 100 and we first set the priority of the Mellanox bridge to 100 and of the Arista bridge to 200, meaning that Mellanox should be selected as Root Bridge and thus one of the Arista ports should be set in blocking state (lower priority indicates higher chance to be selected as Root Bridge). Also, we 23

24 reversed the priorities so that Arista should be selected as Root Bridge and one of the Mellanox ports should be in blocking state. The relevant SONiC configurations (of the VLANs) can be found in Appendix C2. Results In both our tests the devices selected themselves as Root Bridge, meaning that all ports were set in forwarding state and the loop in our topology remained existent. Listings 4.4 and 4.5 show the results of the STP status on Mellanox and Arista (some output has been omitted for brevity). The full output can be found in Appendix D3. One can notice on lines 8 and 12 of both listings that all four ports are in forwarding state and in both cases the devices selected themselves as Root Bridge, which can be concluded from lines 3 and 4 in both listings. 1 admin@sonic - mellanox :~$ sudo brctl showstp Bridge 2 Bridge 3 bridge id ec0d9a8df1c0 4 designated root ec0d9a8df1c0 5 # (...) 6 7 Ethernet32 (1) 8 port id 8001 state forwarding 9 # (...) Ethernet36 (2) 12 port id 8002 state forwarding 13 # (...) Listing 4.4: Mellanox selected itself as Root Bridge (some output was omitted for brevity). 1 admin@sonic - arista :~$ sudo brctl showstp Bridge 2 Bridge 3 bridge id 00 c8.001 c737bf75c 4 designated root 00 c8.001 c737bf75c 5 # (...) 6 7 Ethernet16 (1) 8 port id 8001 state forwarding 9 # (...) Ethernet20 (2) 12 port id 8002 state forwarding 13 # (...) Listing 4.5: Arista selected itself as Root Bridge (some output was omitted for brevity). To investigate this behaviour, we used tcpdump to capture STP messages between Mellanox and Arista. We found that both devices were sending STP messages to each other, but no receiving STP messages were passed to the control plane. It was confirmed by a SONiC contributor that in the current version of SONiC, no interface trap is configured for STP messages, meaning that incoming STP messages are not passed to the control plane and thus STP is unable to operate correctly on SONiC 7. VLAN (trunking) Methods Additionally, we examined VLAN (trunking). We decided that for this experiment, and the other experiments following this one, it would be interesting to use several hosts in our testing topologies. In order to do so, we attached two physical servers to our topology and set up two virtual machines (VMs) to run on each host server. We used the same operating system on all four VMs (Linux deb9u3). We used Vagrant 8 to build and manage our virtual machine environment, and VirtualBox 9 for the virtual machine itself. Appendix B2 and B3 specify the Vagrant and VirtualBox versions

25 used in the experiments, respectively. To allow the VMs to participate in the network, we configured a bridge between the physical interfaces of the servers and the virtual interfaces of the VMs using the Vagrantfiles. The virtual interfaces could then be configured as if they were physical interfaces on four separate machines. The Vagrantfiles used in this experiment can also be found in the previously mentioned GitHub repository. To verify whether the switches are able to perform correct VLAN (trunking) functionality, we used the topology depicted in figure 4.7. For the SONiC configuration, Appendix C3 can be consulted. The link between Mellanox and Arista should be configured as a tagged (trunk) link, for it must carry frames that can belong to either VLAN 100 or VLAN 200. We examined whether the open switches allowed us to configure such tagged ports and whether packets can be delivered correctly within the same VLANs using the trunk link. For instance, VM A1 must be able to communicate with VM B1, because they are configured to be in the same VLAN. VM A1 should not be able to communicate with VM B2, because they are in different VLANs and should therefore be completely isolated from each other. VLAN interfaces VLAN 100: /24 VLAN 200: /24 VLAN interfaces VLAN 100: /24 VLAN 200: /24 Mellanox SN2100 Ethernet32 VLAN trunk Ethernet16 Arista 7050QX-32S Ethernet56 Ethernet60 Ethernet40 Ethernet44 VLAN 100 VLAN 200 VLAN 100 VLAN / / / /24 VM A1 VM A2 VM B1 VM B2 Phys. server A Phys. server B Figure 4.7: Topology to analyze VLAN (trunking) and inter-vlan routing. Two VLANs are configured with two virtual machines in each VLAN. A trunk link is present between Mellanox and Arista. Results We were able to communicate within VLANs, but not between VLANs, which is the correct and expected behaviour. As an example, listing 4.6 shows with ping that communication is possible from VM A1 to VM B1, which are in the same VLAN. vagrant@ vma1 :~ $ ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_ seq =1 ttl =64 time =0.602 ms 64 bytes from : icmp_ seq =2 ttl =64 time =0.580 ms 64 bytes from : icmp_ seq =3 ttl =64 time =0.591 ms Listing 4.6: ping from VM A1 to VM B1. In contrast, communication between devices configured in different VLANs is not possible. Listing 4.7 displays this behaviour. The example shows that we cannot communicate between VLAN 25

26 100 to VLAN 200. vma1 :~ $ ping PING ( ) 56(84) bytes of data. From icmp_ seq =1 Destination Host Unreachable From icmp_ seq =2 Destination Host Unreachable From icmp_ seq =3 Destination Host Unreachable Listing 4.7: ping from VM A1 to VM B2. This indicates that the VLAN trunk succeeded in tagging frames with the correct VLAN number, allowing for complete VLAN isolation shown in the above listings Layer 3 features Inter-VLAN routing Methods Besides VLAN trunking functionality, we also tested inter-vlan routing using the same topology as depicted in figure 4.7. Figure 4.7 shows the VLAN interface addresses we used for both Mellanox and Arista. For the relevant sections of the SONiC configuration, Appendix C4 can be consulted. If inter-vlan routing works correctly, the switches will route packets from one VLAN to another and thus we should be able to communicate between VLANs using the VLAN interfaces. For instance, VM A1 should be able to ping VM B2 and vice versa. Results Using inter-vlan routing we are able to communicate between different VLANs. Listing 4.8 shows an example of a traceroute from VM A1 to VM B2. It shows on line 3 that the VLAN interface of VLAN 100 on Mellanox ( ) is used to route the packet to VLAN 200, after which Arista delivers the packet to VM B2. 1 vagrant@ vma1 :~ $ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms Listing 4.8: traceroute from VM A1 to VM B2. BGP Methods To verify whether BGP operates correctly on the switches, we set up the topology in figure 4.8. SONiCs BGP configuration can be found in Appendix C5. We have configured the switches to be in different autonomous systems, so as to create a BGP session between Mellanox and Arista. We expect this BGP session to exchange routes between Mellanox and Arista. Specifically, Mellanox has knowledge of subnets /24 and /24 and Arista does not. BGP must propagate the routes to these subnets to Arista. Similarly, BGP must propagate routes to the subnets /24 and /24 to Mellanox. In the end, we want to be able to communicate between the virtual machines in hosts A and B using these propagated routes. 26

27 AS Mellanox SN subnet /24 subnet /24 AS Arista 7050QX-32S subnet /24 subnet /24 subnet /24 subnet / VM A1 VM A2 VM B1 VM B2 Phys. server A Phys. server B Figure 4.8: Topology to analyze BGP. Mellanox and Arista are placed in two separate autonomous systems, each containing two subnets. Results SONiC succesfully set up the two BGP sessions we configured. For example, listing 4.9 shows the current BGP sessions Mellanox has with Arista on lines 8 and 9. 1 admin@sonic - mellanox :~$ show ip bgp summary 2 Command : sudo vtysh -c " show ip bgp summary " 3 BGP router identifier , local AS number RIB entries 21, using 2352 bytes of memory 5 Peers 2, using 9312 bytes of memory 6 7 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/ Down State / PfxRcd :30: :30: Total number of neighbors 2 Listing 4.9: BGP neighbors on Mellanox, showing the two sessions with Arista. In addition, Mellanox and Arista successfully exchanged routing information regarding the subnets that were directly connected to them. Mellanox propagated the subnets /24 and /24 to Arista and Arista propagated the subnets /24 and /24 to Mellanox. For instance, listing 4.10 shows that subnets /24 (line 11) and /24 (line 13) are present in the routing table on Mellanox, and that these subnets can be reached through both (via interface Ethernet32) and (via interface Ethernet36) and that this reachability was sourced from BGP. Similarly, in the Arista routing table, subnets /24 and /24 are present and can be reached through (via interface Ethernet16) and (via interface Ethernet20). Again, these routes were learned through BGP. 27

28 1 - mellanox :~$ show ip route 2 Command : sudo vtysh -c " show ip route " 3 Codes : K - kernel route, C - connected, S - static, R - RIP, 4 O - OSPF, I - IS - IS, B - BGP, P - PIM, A - Babel, 5 > - selected route, * - FIB route 6 7 C >* /24 is directly connected, Ethernet56 8 C >* /24 is directly connected, Ethernet60 9 C >* /24 is directly connected, Ethernet32 10 C >* /24 is directly connected, Ethernet36 11 B >* /24 [20/0] via , Ethernet32, src , 00:30:05 12 * via , Ethernet36, src , 00:30:05 13 B >* /24 [20/0] via , Ethernet32, src , 00:30:05 14 * via , Ethernet36, src , 00:30:05 15 C >* /32 is directly connected, lo 16 B >* /32 [20/0] via , Ethernet32, src , 00:30:05 17 * via , Ethernet36, src , 00:30:05 18 # (...) Listing 4.10: Routing table on Mellanox, showing the routes to the subnets we configured before (some output was omitted for brevity). Ultimately, to complete the verification of BGP, we used traceroute between the virtual machines to determine whether communication is possible and if yes, what routes are taken. Listing 4.11 shows that VM A1 and VM B2 are able to communicate through Mellanox and Arista. First, a packet goes through the connected interface on the Mellanox device ( ) (line 3). Then, it is routed to the Arista device ( ) (line 4), which routes it via its interface in subnet /24. SONiC uses the loopback address configured in config db.json as Router ID in the BGP sessions, which for Mellanox is /32 and for Arista /32. 1 vagrant@ vma1 :~ $ sudo traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms ( ) ms ms ms Listing 4.11: traceroute from VM A1 to VM B Result overview Table 4.1 presents an overview of the obtained results, including several remarks. Feature Results Comments LLDP Pass - LACP Pass L2 link aggregation not working STP Fail Not supported by SONiC; packets dropped VLAN (trunk) Pass - Inter-VLAN routing Pass - BGP Pass - Table 4.1: An overview of the obtained results. 28

29 CHAPTER 5 Discussion The performed experiments provided insights into the shortcomings of several fundamental networking features when deployed on open switches with SONiC. Aside from commenting on some of the obtained results, this chapter will provide general insights into the ease of use of open switches with SONiC. LACP The first notable observation in our LACP experiments was that none of our port channels were showing up when issuing the command show interfaces portchannel. This command should show all configured port channels in SONiC. It turns out SONiC takes the configuration of minigraph.xml, which is a deprecated configuration method of SONiC, for the command show interfaces portchannel 1 instead of the main configuration file config db.json. The Layer 3 port channel we set up in the experiments has shown that it is possible to deploy a Layer 3 port channel that provides redundant communication between the devices running SONiC. In order to get this working, aside from configuring the port channel in SONiCs configuration file, we had to edit SONiCs teamd template 2. By default, SONiC sets the field min ports, which is the minimum amount of active ports in a port channel for the port channel to be operational, to ceil(0.75 number of port channel members), which was 2 in our topology. In order to test the redundancy of our port channel, we set min ports to 1. BGP Similarly, successful BGP functionality was implied by the experiments we performed. Configuring BGP neighbors and AS numbers was done in config db.json. Nonetheless, we had to edit the BGP configuration template 3 of SONiC to specify that BGP should be configured such that it also advertises directly connected networks (which was the case in our topology) and not only networks obtained from other BGP neighbors. 5.1 Ease of use SONiC allowed for straightforward configuration in the config db.json configuration file. Several examples of configuration have been shown in this thesis already. Moreover, SONiC provided some CLI configuration commands but not for all features we tested. For instance, configuration commands could be used to shut down interfaces and BGP sessions, but not to set interface addresses or configure BGP peers. Command-line configuration also allowed the user to set up VLANs and add or remove interfaces. In general, the command-line configuration provided a small subset of configuration possibilities compared to configuration in config db.json. For

30 users that prefer command-line configuration this might make SONiC less straightforward to use. SONiC did provide a wide variety of commands to view current configuration or device status. Several examples have been provided in this research. For all supported (thus not STP) features we tested there existed one or more corresponding show commands to view configuration or state information. One exception, as mentioned before, was that show interfaces portchannel takes its information from a deprecated configuration method to show the configured port channels. In short, the ease of use of SONiC depends on the preferences of the user. Users that prefer working in a configuration file will probably experience SONiC as a user-friendly NOS, while users that prefer CLI configuration might be less comfortable with SONiCs configuration. 30

31 CHAPTER 6 Use case scenarios for open switches The use of open switches with SONiC can be beneficial in various use cases. For example, SONiC allows the user to look into the source code and then use or extend it for a certain objective. In this chapter, we briefly examine a current use case that exploits the properties of open switches with SONiC, and we provide two example use cases ourselves. 6.1 Current use cases Microsoft Global Cloud Microsoft developed SONiC for use in their Microsoft Global Cloud. Since Microsoft runs one of the largest networks in the world, Microsoft sets out strict requirements for the technology that it deploys. One of these requirements is that new features can be implemented without having an impact on the end users. Since SONiC consists of several containers, each containing the resources to deploy a certain networking feature (e.g. BGP, LAG), only one container needs to be updated when there is an update or a bug fix for a certain feature, instead of replacing the whole switch image (which will result in data plane downtime). This makes SONiC suitable for use cases in which no downtime is permitted while deploying updates [13]. Another requirement is that SONiC can be used on the newest and most innovated hardware platforms. Because SONiC uses SAI, a data center can constantly innovate with newer switch hardware without having to change the software stack. Regardless of what switch hardware is used, as long as SAI is supported, all switches can be configured the same way, for their software stacks can be identical. Operators are thus able to preserve their software investments while keeping up with hardware-wise innovation. Thus, SONiC is suitable for use cases in which there may be regular changes in hardware [13]. In addition, Microsoft states the requirement that cloud-scale deep telemetry and fully automated failure mitigation has to be utilized in their cloud. Since SONiC has innovations such as NetBouncer and Everflow available, SONiC meets these requirements. NetBouncer can be deployed to automatically detect faulty devices or links within a large data center accurately [27]. Everflow debugs several network faults such as packet drops or loops. Furthermore, it can quickly identify devices that cause high latency in a network [28]. In short, SONiC is suitable for use cases in which large networks have to be monitored constantly to automatically respond to potential problems. 6.2 Example use cases In addition to the above use case of SONiC in the Microsoft Global Cloud, we provide two example use cases in which the open nature of SONiC can be deployed for useful applications. 31

32 Plug-and-play VLAN The first use case relates to companies that want to provide flexible working spots to their employees, who in turn own a laptop provided by the company. On different days the same employee might want to work on different physical locations within the company office. To gain access to the company s network, the employees can use Ethernet cables to connect their laptops with the network. Since generally a company consists of several departments with partitioned and isolated computer networks (VLANs) and an employee from any department can sit and connect anywhere, how does the network decide to which VLAN the newly connected laptop should be added? Employees want to start working right away after connecting to the network, so it is preferred their laptops are added to the correct VLAN without intervention by the network administrator. If the company network consists of open switches running SONiC, the network administrator could decide to develop an application that automatically places newly connected hosts in a particular VLAN, depending on the MAC address of the host, so that the VLAN functionality on the switch is essentially plug-and-play. That is, no manual configuration is needed every time a new host connects. This can be implemented as follows. Via SONiCs logs or LLDP entries, newly connected hosts (including their MAC address) can be discovered by the application. The network administrator can define a policy that states the relation between the host MAC address and the VLAN that host will be placed in, if at all. This is illustrated in the right side of figure 6.1. For instance, the application can decide to block hosts with unknown MAC addresses, or place hosts manufactured by the same vendor in the same VLAN (MAC addresses are vendor-specific). Next, the application can simply edit SONiCs config db.json configuration file to change the VLAN settings. We suppose that the network administrator has a list of all company laptops in use by the employees, including their MAC addresses. The network administrator can easily define a policy that maps each laptop to a certain department (VLAN). This way, no matter where an employee connects its laptop to the network, the SONiC switch will detect the MAC address of this laptop and determine in which VLAN the switch interface this new host is connected to should be placed. Similarly, if a laptop not belonging to one of the employees uses one of the Ethernet cables, its MAC address is not recognized and can be blocked by the switch. Office Other laptops Laptop with MAC x Switch SONiC Policy table Host detection on interface y y, MAC x Add interface y to VLAN z y, VLAN z Block Block Figure 6.1: An overview of the plug-and-play VLAN application. The switch detects a new host with MAC address x on switch interface y. The policy table maps from MAC address x to VLAN number z, after which the application places interface y in VLAN z. 32

33 Automatic port channel Our second use case focuses on data centers that need to provide a redundant network, because they handle important data that must reach its destination. The network administrator can decide to aggregate all connections between each switch in the data center, to provide redundancy. Such a data center may have a large number of switches, and configuring link aggregation for the connections between these switches can by time-consuming. LACP provides a way to do this, because it is able to automatically negotiate between two devices to set up a port channel. However, before this can happen, the network administrator needs to specify which switch interfaces belong to which port channel, if at all. The application in the next paragraph proposes a more flexible alternative for this LACP feature by allowing the network administrator to connect neighbors to any interface on a switch without the need of specifying that these interfaces will be used to create a port channel. The application will dynamically determine whether a port channel should be set up, and if so, with which interfaces. If the data center contains open switches with SONiC, the network administrator can develop an application that automatically sets up a port channel if it discovers that the switch has multiple physical links with the same neighbor (for instance by comparing MAC addresses or Chassis IDs of all connected devices). This way, when the network administrator connects two switches with multiple physical links, no further configuration is necessary to set up the port channel. Datacenter Other switches Switch with ID x Switch SONiC Policy table Neighbor detection on interface y y, ID x Create new port channel z y, PC z Add to existing port channel z No port channel modifications Figure 6.2: An overview of the automatic port channel application. The switch detects a new neighbor with Chassis ID x on switch interface y. Using neighbor and port channel information, the application decides whether 1) there should be no port channel modifications because the new link is the only link with the neighbor device, or 2) a new port channel z should be created with interface y and the other interface already connected to the same neighbor, or 3) the new interface y should be added to port channel z because there already is a port channel with the neighbor. When the switch detects a newly connected neighbor, the application uses neighbor and port channel information to determine whether there is already a port channel with this neighbor and if not, whether there is already a single link with this neighbor, for instance by comparing the Chassis ID of all connected neighbors. Figure 6.2 shows how this determination process. There are three possible cases: 1. There is no other link with the neighbor device. 2. There is one other link with the neighbor device. 33