On the Performance Impact of Virtual Link Types to 5G Networking

Transcription

1 On the Performance Impact of Virtual Link Types to 5G Networking Wen-Ping Lai *, Yong-Hsiang Wang Department of Communications Engineering Yuan Ze University Taoyuan, Taiwan, ROC Abstract Virtual links are important building blocks which connect virtual hosts to form virtual networking within a physical host, and thus their link types can potentially impact the performance of virtual networking. This article studies such a performance impact due to the adopted virtual link type, defined by the associated virtual ports at both ends of the link. Two virtual link types have been investigated, including the native virtual ethernet ports (i.e., veth-pair links) for Linux network namespaces and bridging, and Open vswitch-based internal ports (i.e., internal links). Several design principles are proposed based on our performance benchmarking results, which show substantial performance gaps among these two virtual link types, particularly for a large number of traffic connections, where the CPU capability could play an important role to the amplified level of performance gaps. Further outlooks into the convergence of Cloud/SDN/NFV and 5G networking are projected, where virtual networking plays an important role. Keywords-virtual port type; virtual link type; virtual networking; network namespace; container I. INTRODUCTION The global trend of next-generation networking is rapidly moving towards Cloud/SDN/NFV [1] [2], and has been generating many innovations into our current and future daily life, and network virtualization is becoming the common underlying technology. However, the complications of laying virtual networks over physical networks also impose challenging issues in many aspects. In terms of virtualization types, hypervisor virtualization is a mature technology and has been widely deployed for commercial use, such as many Cloud-based services. Famous hypervisor-virtualization platforms include VMware, VirtualBox and Xen etc. In spite of deployment easiness and wide popularity, this type of virtualization inflexibly divides and thus heavily consumes system resources. The main reason comes from the need of installing a guest operation system (Guest-OS) for generating a virtual machine (VM), not only taking a long provision time for one virtual host, but also needing a pre-determined allocation of system resources. As an emerging technology, container virtualization is remarkable for its fast provision of network deployment and light-weight use of system resources. These advantages come from the fact that a container is no longer a box confining a Guest-OS system, but just a box confining the target applications or processes, along with their associated *Corresponding author. wpl@saturn.yzu.edu.tw libraries. Thus, such a concept can also be perceived as processes containment. Linux container (LXC) and Docker container are two well-known container technologies [3], where network namespaces play the common trick behind both. A container does not only offer system resource isolation for contained processes, but also provides control and management. Isolation of system resources is realized through six different types of Linux kernel namespaces, classified as user, mount, process identity (PID), inter process communication (IPC), Unix time-sharing (UTS), and network. Another trick called control groups (cgroups) is in charge of control and management of system resources. In terms of virtual networking, virtual links are important building blocks which connect virtual hosts to form virtual networks within a single physical host. Furthermore, virtual networks on multiple physical hosts can be physically networked together to form a large cloud overlaying virtual networks. Hence, understanding the performance issue of virtual networking is of great importance, and the performances due to the adopted virtual link types provide the major key to resolve the issue. Typically, a virtual link is characteristic of its associated virtual port(s), and thus its link type is fully determined by its associated port types. The article studies how the virtual port type affects the performance of the virtual link type, and thus affects that of virtual networks. The rest of this article is organized as follow. Section II explains the concepts of namespaces, virtual bridges, virtual link/port types, and finally proposes the design principles for establishing three basic virtual linking topologies of virtual networks. These principles are based on the performance measurements presented in Section III, including analyses and discussions on how the performance gaps among different virtual links are shaped by the number of CPU cores. Finally, Section IV draws the concluding remarks and outlooks into their potential impacts to 5G-based virtual networking. II. PROPOSED DESIGN PRINCIPLES FOR ENHANCING VIRTUAL NEWORK PERFORMANCES The goal of this study is to propose useful design principles for enhancing virtual network performances. As aforementioned, the adopted virtual link type is the key factor affecting the performance of a virtual network. Hence, this study proposes the design principles on how to select an appropriate link type for each of the basic linking topologies of virtual networks which dominates the most frequently applied topologies in virtual networking. These principles are reasoned and verified by our measurements presented in the

2 next section. On the other hand, to pave a good foundation, the following concepts should be understood first before justifying the proposed design principles of using an appropriate virtual link type. A. Network namespace-based virtual hosts Network namespace is the common underlying mechanism for the container technology such as LXC and Docker for next-generation network virtualization. A virtual host can easily and rapidly be created/operated/deleted in terms of a network namespace to let each virtual host obtain its own virtual devices such as virtual network interface cards (vnics), and its own network applications binding to the same well-known application port number, such as port 80 for each per-name-space web server running on different virtual hosts networked within the same physical host. Note that five other kernel namespace types are also involved in isolating system resources: (1) user namespaces for isolation among different user/group ID number spaces, (2) mount namespaces for isolation among different filesystem mount points and thus achieving their own filesystem hierarchy, (3) PID namespaces for isolation among different process ID number spaces to let the same PID be allowable for use on different virtual hosts such as the init process with PID=1, (4) IPC namespaces for isolation among different IPC objects and POSIX message queues to let each virtual host have its own IPC identifiers and message queue filesystem, and (5) UTS namespace for isolation among different system identifiers (both hostname and hostdomain). In addition, cgroups is in charge of controlling and managing system resource allocation among different virtual hosts. As for LXC and Docker [4], more userland APIs and other features are added to enhance system resources' control and management. LXC adds secure Linux (SELinux) profiles, secure computing mode (SECOMP) policies, kernel capabilities, and chroot. Stemming from LXC, Docker introduces more advanced designs, including generating multiple containers from a single image template, which is extractable from a public or private repository for a wide variety of downloadable images. In addition, it has rich APIs for operating containers' life cycles (e.g., creation, attachment, detachment and deletion) and their networking interactions (e.g., port-mapping, direct-linking and bridgedlinking). However, the details of these are beyond the scope of this article. In this study, since network namespace plays the common trick behind both LXC and Docker as aforementioned, only the Linux network namespace (netns) manipulation tool, realized by the native command line interface (CLI) such as ip netns... in the iproute2 toolset [5], is adopted for generating virtual hosts. Note that iproute2, intending for replacing the legacy toolset called net-tools (such as the well-known ifconfig and route commands for configuring NICs and routing tables respectively), has become the default toolset to cope with the advanced kernelspace networking core designs such as NETLINK for traffic engineering and NETFILTER for packet filtering and firewalling. As for virtual networking among virtual hosts, two types of virtual links are discussed below for various virtual linking topologies. B. Virtual Link Types for Various Linking Topologies Direct linking and bridged linking are two fundamental virtual linking topologies that can expand to form more complicated ones. As shown in Fig. 1, direct linking is composed of 2 virtual hosts (e.g., h1 and h2), connected by 1 virtual link, each end of which has an associated virtual ethernet NIC, either called h1-veth or h2-veth. Such a link type is called veth-pair. Note that, when the veth-pair link is initially created in the root namespace (Root-NS) of the physical host, neither of h1-veth or h2-veth is visible to the guest namespaces (h1-ns and h2-ns) of the two virtual hosts, and only the physical host sees them. After they are respectively inserted into h1-ns and h2-ns, the visibility reverses. It is then optional to rename h1-veth and h2-veth as h1-eth0 and h2-eth0 respectively, depending on if such an intuition to dummy users is needed. Bridged linking is composed of two virtual hosts (e.g., h1 and h2) connected by a virtual bridge with two virtual links. Depending on the adopted virtual bridge type and virtual link type, Figs. 2~4 demonstrate three designs of bridged linking: (1) Linux-based bridged linking with 2 vethpair links, (2) OVS-based bridged linking with 2 veth-pair links, and (3) OVS-based bridged linking with 2 internal links, natively stemming from the OVS bridge itself and only equipped with 1 vnic per link to be inserted to a network namespace such as h1-ns or h2-ns. Note that OVS stands for Open vswitch [6], and is not only designed for virtual networking among conventional VMs, but also suitable for SDN realization among physical hosts. This study aims at finding the highest-performance design out of these three linking topologies. Thus, it is interesting to study the following two performance issues: (1) whether the virtual bridge type or the virtual link type casts an stronger impact to the performance, and (2) whether the veth-pair link or the internal link can achieve a higher performance. Fig. 1 Direct linking Fig. 2 Linux-based bridged linking with two veth-pair links Fig. 3 OVS-based bridged linking with two veth-pair links Fig. 4 OVS-based bridged linking with two internal links

3 C. Proposed Design Principles of Using Virtual Link Types This study proposes the following design principles which can resolve the performance issues 1 and 2, as mentioned above and verified by performance measurements and analyses in Section III. Principle 1: Use veth-pair links for direct linking between network namespace-based virtual hosts if direct linking is needed or desired. Principle 2: Use veth-pair links for bridged linking among multiple network namespace-based virtual hosts if only the Linux bridge is available. Principle 3: Instead of veth-pair links, select internal links for bridged linking among multiple network namespace-based virtual hosts whenever the OVS bridge is also available. Note that Principle 1 could be useful to use cases of direct linking, such as customized service chaining in NFV provided by a series of virtual hosts with specific services, where each virtual host pair is directly linked. However, be aware of the performance limit issue with veth-pair links. Also note that there is no significant performance difference between the Linux and OVS bridges if they both use vethpair links, and the major performance improvement is from using internal links, only equipped by the OVS bridge. III. PERFORMANCE MEASUREMENTS AND ANALYSES In order to verify the proposed design principles, this section presents the performance measurements and analyses, in particular for the three bridged linking topologies described by Figs. 2~4. As confirmed by our results, direct linking is natively faster than bridged linking in term of the same link type, and thus only the latter becomes the focus of this article since it is more flexible for network topology forming. In addition, these measurements also offer clear answers to the aforementioned performance issues 1 and 2. The experimental setup of performance benchmarking is described, followed by the performance comparisons among these bridged linking topologies and the in-depth messages they deliver in terms of the number of concurrent TCP traffic flows and the number of CPU cores. A. Experimental Setup of Performance Benchmarking An experimental environment was setup to benchmark the average TCP throughputs over the three bridged linking topologies. As shown in Figs. 2~4, each topology consists of two network namespace-based virtual hosts (i.e., h1-ns and h2-ns) bridged by one Linux or OVS bridge, with two virtual links made of veth-pair or internal ports. To be concise in notations, these three topologies are respectively denoted as LBR+veth-pair, OVS+veth-pair, and OVS+internal. To benchmark the average TCP throughputs, the iperf toolset was adopted to generate concurrent TCP traffic flows, each of which was connected and flowcontrolled between h1-ns (iperf client) and h2-ns (iperf server) and implemented by one iperf thread. The number of concurrent TCP connections, realized by the same number of iperf threads, was set to be {1, 2, 4, 8, 16, 32, 64} to observe its impact on the performance. For each data point in Figs. 2~4, the duration of TCP measurements is 20 seconds long, where iperf reports its TCP throughput values every 2 seconds, and the average TCP throughput was carried out over the 10 measurements in the duration. To control and observer the performance effect due to the number of CPU cores, all the measurements were conducted in an Oracle VirtualBox VM-based Ubuntu-Linux LTS environment with Linux kernel version , running on a desktop computer with 8 Intel i7 CPU cores of 2.8 GHz and 8 GB of main memory, where 4 test environments were setup to run the performance measurements under {1, 2, 3, 4} CPU core(s) and 5 GB of main memory. B. Effect of Concurrent TCP Connections Fig. 5 demonstrates the average TCP throughput variations of the three bridged linking topologies, represented by Figs. 2~4, with the number of concurrent TCP connections between the two virtual hosts. Note that the performance results of Fig. 5 were measured with 4 CPU cores within a VM. Three major messages can be extracted as follow. Each topology shows a clear performance variation with the number of concurrent TCP connections. As the number of iperf connection threads increases, the performance of each topology climbs up initially, and then decays later. Thus, there exists a highest performance peak for each topology. Such a common performance variation behavior is contributed by both the characteristics of TCP in flow and congestion controls. The superiority of the OVS+internal topology over the other two is expressed in terms of three aspects: (1) it can accept 8 concurrent TCP connections for the performance peak, i.e., 2 times those for the performance peaks of the other two; (2) it has a much smoother decay rate than the other two; (3) it has around 2~6 times of performance gains than the other two. The superiority of the OVS+internal topology in performance gains over the other two is so obvious, and it recalls our earlier question raised by performance issue 1: whether the virtual bridge type or the virtual link type casts a stronger impact to the performance. The answer is clear now, based on the following two observations: (1) when using veth-pair links with both the two virtual bridge types, namely LBR+veth-pair and OVS+veth-pair, there is no substantial performance difference between the Linux and OVS bridges; (2) it is only when using internal links with the OVS bridge that the topology's performance gets highly boosted. Namely, it is the virtual link type that casts a much stronger performance impact, instead of the virtual bridge type. In addition, these observations can also answer performance issue 2 as follows: the internal link can achieve a substantially a higher performance than the veth-pair link. C. Effect of CPU Cores Figs. 6~8, together with Fig. 5, explore the performance impact of CPU cores to the performance gaps among the three bridged linking topologies, where the results jointly

4 show a strong effect from the number of CPU cores. Three main messages from these figures can be delivered as follow. In spite of a strong effect of CPU cores on the performance gains, the leading place of the OVS+internal topology still stands firmly over the other two. When the number of CPU cores is less than 4, the performance variations of LBR+veth-pair and OVS+veth-pair behave almost the same. This reconfirms that the virtual bridge type basically make no substantial performance difference. Instead, it is again the virtual link type (i.e., the internal link, stemming from the OVS bridge) that actually achieves substantial performance gains. The performance gains of OVS+internal are strongly tuned by the number of CPU cores. Its performance gaps over the other two topologies are dramatically amplified as the number of CPU cores increases from 1 to 4. IV. CONCLUSIONS AND OUTLOOKS This article studied the performance impacts of virtual bridge types and virtual link types to three bridged linking topologies, serving as build blocks for establishing virtual networking, where the virtual bridge type was selected between the Linux native bridge and the OVS bridge, and the virtual link type was chosen between the veth-pair link and the internal link. Our results showed that the selection of virtual link type has been demonstrated to be much more crucial to optimal performance than that of virtual bridge type. Three design principles were thus proposed, which equivalently states that using internal links is more preferred than using veth-pair links. Due to the fact that internal links actually stem from the OVS bridge, the preference on internal links is equivalent to the preference on the OVS bridge. As an outlook into next-generation networking such as the convergence of Cloud/SDN/NFV, the proposed design Fig. 5 Performance Comparison with 4 CPU cores of VM Fig. 7 Performance Comparison with 2 CPU cores of VM Fig. 6 Performance Comparison with 3 CPU cores of VM Fig. 8 Performance Comparison with 1 CPU core of VM

5 for optimal virtual networking performance by selecting an appropriate virtual link type, such as internal links from the OVS bridge, is of great significance and applicable. As aforementioned, the Docker container technology is emerging as a promising design toolset for next-generation virtual networking, despite that it is still based on the Linux native namespaces for system resources isolation and the Linux cgroups for system resources control and management. However, the default virtual bridge for connecting Docker containers, called docker0, is still made of the Linux bridge, and the default virtual links are still based on the veth-pair link type. Hence, it leaves much room for performance enhancement of Docker-based virtual networking by replacing such a docker0+veth-pair bridged linking topology by the proposed OVS+internal topology, whenever possible. Indeed, the OVS bridge also supports another link type called patch-ports, particularly designed for connection between two OVS bridges. This could be useful if a series of OVS bridges are needed to be linked together for a specific virtual networking design, where the performance benchmarking could also be interesting for future work. In addition, in terms of Cloud and SDN, the OVS bridge is also a natural and better choice. For instance, the support for OpenFlow in the OVS bridge makes it unique and suitable for functioning as a SDN switch. In terms of NFV, which also heavily relies on the Docker technology for network virtualization, the OVS+internal topology is also recommended for performance boost. In fact, an open-source LTE framework called OpenAirInterface (OAI), for both 4G and 5G, is emerging as promising candidates for both research and future commercial uses [7]. For instance, we have observed that advanced concepts of creating virtual entities in the evolved packet core (EPC) [8] of OAI such as vhss, vmme and vsgw-vpgw have been implemented based on the Linux kernel namespaces. In fact, a realization of Docker container-based virtualization of the whole EPC is also publicly available from Eurecom, the creator of OAI. We have also observed a similar virtual-networking framework. Indeed, closer investigations into these two cases studies for virtual networking performances, including the adopted virtual link types and integrating physical networking with the OAI-based enodeb or CloudRAN small cells [9] [10], are currently in progress. [2] Y. Li and M. Chen, Software-defined network function virtualization: a survey, IEEE Access, vol. 3, no. 1, pp , [3] D. Bernstein, Containers and cloud: from LXC to Docker to Kubernetes, IEEE Cloud Computing, vol. 1, no. 3, pp , [4] I. Miell and A. H. Sayers, Docker in Practice. New York: Manning, [5] Linux Foundation Wiki. iproute2 [Online]. Available: [6] Linux Foundation Collaboration Projects. Open vswtich [Online]. Available: [7] N. Nikaein, M. K. Marina, S. Manickam et al., "OpenAirInterface: A flexible platform for 5G research, ACM SIGCOMM Computer Communication Review, vol. 44, no. 5, pp , [8] The journey to packet core virtualization, Alcatel-Lucent White paper, [9] Commscope. What are C-RAN small cells [Online]. Available: [10] Y. Shi, J. Zhang, and K. B. Letaief, "Group sparse beamforming for green Cloud-RAN," IEEE Trans. Wireless Commun., vol. 13, no. 5, pp , May ACKNOWLEDGMENT This work was partly supported by Taiwan's Ministry of Science and Technology under grant E and Ministry of Education for the advanced-technology course promotion project for mobile broadband under grant REFERENCES [1] D. Kreutz, P. E. Verissimo, and S. Azodolmolky, Software-defined networking: a comprehensive survey, Proceedings of the IEEE, vol. 103, no. 1, pp , 2015.