A Comparative Review: Accurate OpenFlow Simulation Tools for Prototyping

Transcription

1 322 JOURNAL OF NETWORKS, VOL. 10, NO. 5, MAY 2015 A Comparative Review: Accurate OpenFlow Simulation Tools for Prototyping Pakawat Pupatwibul, Ameen Banjar, Abdallah AL Sabbagh, and Robin Braun Centre for Real-Time Information Networks (CRIN), University of Technology, Sydney (UTS), Sydney, Australia {Pakawat.Pupatwibul, Ameen.R.Banjar}@student.uts.edu.au, {Abdallah.AlSabbagh,Robin.Braun}@uts.edu.au Abstract Several Simulation and emulation tools such as the OMNeT++ INET Framework and Mininet have been developed to evaluate the performance of Software-Defined Networking (SDN). A major challenge is how to analyse the obtained performance results of data transmission for these tools. These challenges include scaling to large networks, testing the correctness and evaluating the performance with the ability to easily migrate to a real system with minimal changes for deployment. Current methods for testing the functions and evaluating the performances of SDN include new programming languages, static analysis and debugging capability, and innovative frameworks of simulation tools. In this paper, we describe the implementation model of OpenFlow system in the INET framework for OMNeT++. The simulation approaches were designed for prototyping and evaluating new SDN-based applications accurately at low cost, while being flexible, scalable, controllable, and accessible to many users. We describe the design and the use of the simulation modules, and demonstrate its capability by carrying out a series of experiments. In addition, we compare the OpenFlow functions of OMNeT++ simulator with the popular Mininet emulator in identical traffic configurations. The measurement results derived from both tools show that they are correct, accurate, and repeatable regarding their capabilities, performance, and functionalities. Index Terms Network Performance; Mininet; Software- Defined Networks; OMNeT++ I. INTRODUCTION Over the past few years, Software Defined Networking (SDN) hasemerged as a programmable network thatoffers a wide range of functionalities for conventional networks [1]. SDN can be programmed for different needs and purposes. Network operators and developers are able to create many collections of software, which can be run as applications over a network operating system to manage networks and to control networks behaviours [2]. Therefore, the main challenge is how to develop and test new protocols for the network to behave as expectedand scales when applications are deployed in a production network [3]. As a result, the ultimate success of SDN depends on having effective ways to test and evaluate performance, and validate functionality to achieve high reliability, correctness and accuracy. Testing the functions, evaluating performance and accuracy of the networks can be performed by a number of approaches such as experimental test bed, simulation and emulation. Open Network Foundation (ONF) had presented a protocol called OpenFlow which regard as a first standardisation communication interface of SDN approaches [4]. However, there are some procedures to regulate data transmission between network nodes, efficient level of performance, high consistency, more connectivity and data protection [1]. Therefore, a new protocol has to be tested in terms of acceptability, performance and functionality which could happen by following previous approaches experimental test bed, simulation and emulation. One of the approaches to evaluate OpenFlow is to run experimental test beds built from real network devices, real operating systems and applications [5]. Although experimental test beds could provide realistic results, it is costly to conduct a large test bed environment. Another common approach is running OpenFlow tests by using simulation as a virtual network, which is based on a software model to present network devices, link connectivity, operating systems and applications [6]. A virtual network simulation has a low cost to build, is flexible to any topology, and is scalable, repeatable, and accessible to many users. On the other hand, simulation has some limitations depending on accurately building each virtual model such as host, link and switch, which can provide many similar results from different simulation runs [7]. Consequently, researchers and developers are motivated to run their tests over practical test beds and emulations rather than simulations. Emulation is similar to an experimental test bed which runs in real time, because it connects real devices with virtual devices or real applications [6]. The physical devices have real operating systems and applications, which can interact with the simulated virtual devices in an emulation environment, whereas simulators do not have real operating systems or applications. A number of SDN approaches have been developed. However, there is only limited information on testing of each one. Moreover, evaluation of performance and verification of functionality comparisons are not widely available. Simulation tools such as the OMNeT++ INET Framework and emulation tools like Mininet are suitable for the task of designing, building, and testing network architectures, and provide practical feedback when developing real world systems. Furthermore, emulation gives the ability to migrate codes to a production network. Such simulation and emulation tools allow system doi: /jnw

2 JOURNAL OF NETWORKS, VOL. 10, NO. 5, MAY TABLE I. A COMPARISON OF NS-3, OMNET++, MININET AND ESTINET [7] NS-3 OMNeT++ Mininet EstiNet OpenFlow version /1.1.0 Programming language C++ C++ Python C/C++ Operating support system GNU/Linux, FreeBSD, OS X OS X, Windows, Linux distributions OS X, Windows, Linux distributions (VM image) Supporting simulation No Supporting emulation No No Ability to use real controller No No Result repeatable No Scalability Performance correctness GUI support result High By single Process No Spanning Tree Protocol and no real controller -Monitoring Only -Configuration by C++ Middle By single Process No real controller - Configuration - Monitoring Middle By multiple Processes Performance depend on resources -Monitoring Only -Configuration by Python Linux Fedora bit High By single Process - Configuration - Monitoring designers to determine the correctness and efficiency of a design before a system is deployed. Also, tools enable evaluation of effects of various network metrics, and providemechanisms to obtain results that are not experimentally measurable on larger and distributed architectures. However, very few performance evaluations of OpenFlow architectures using simulation tools such as OMNeT++ INET Framework have been published[8][9]. Therefore, performing experiments over emulation tools like Mininet can prove the functionality of OpenFlow module in OMNeT++ INET Framework as well asthe availability of performanceevaluation and the correctness of results. Mininet results have been used to evaluate the correctness and the accuracy of network performance and functionality for OpenFlow model in OMNeT++ [10]. The measurement results collected from the mean Round- Trip-Time (RTT) in Mininet and OMNeT++ are nearly identical. While Mininet can provide realistic results and a good tool for functional testing of OpenFlow protocols and applications, OMNeT++ also offers useful facilities for prototyping and simulating SDN-based applications and scenarios. The remainder of this paper is organised as follows. Section II presents part A for background of the INET simulation package in OMNeT++ and part B for related works on OpenFlow simulation and emulation tools. In Section III, we describe a simple network topology setup for our experiments, evaluating the methodology and providing an overview of the network metrics. In Section IV, we describe the Mininet constraints and some of limitation of emulation tools. In Section V, we evaluate the accuracy of the performance results from OMNeT++ and Mininet. Then, lastly, we conclude the paper and provide directions for future work in Section VI. II. BACKGROUND Currently, very few network simulators support the OpenFlow protocols. In this section, we present the overview of an OpenFlow extension for the OMNeT++ INET framework. Other early studies of simulating and emulating OpenFlow networks are also described. A. An Overview of the OpenFlow Extension for OMNeT++ Simulation tools can provide more suitable task of designing, building, and testing for users with practical feedback when developing real world systems. This will allow system designers to determine the correctness and efficiency of a design before the system is actually deployed. It is also useful to explore the behaviour of these protocol models, capabilities and shortcomings further, by making use of simulations. OMNeT++is a discrete event network simulator based on C++. The primary goal of OMNeT++ is simplifying the integrating new modules as well as changing those already implemented [11]. The OMNeT++ environment has the INET framework, which is an open-source communication networks simulation package. This framework contains many models for wired and wireless networking protocols such as UDP, TCP, SCTP, IP, IPv6, Ethernet, and several application models [12], [13]. A variety of new routing protocols focused particularly atdistributed environment have been developed, but little performance information and no realistic performance comparisons between them are available. Until now, very few performance evaluations of OpenFlow architectures using OMNeT++ exist. Recently, the INET framework has a new extension for the OpenFlow model, which is a new toolbox for the SDN simulation environment. However, this OpenFlow packet-level simulator is still in the early development forthe INET framework, currently based on switch specification version 1.2 [10]. The implementation model uses the openflow.h header file to develop the protocol and defined messages as close as possible. The nodes implemented include the OpenFlow switch, OpenFlow controller and the most important messages use to communicate between switch and controller via secure OpenFlow channel. In addition, utility modules were also implemented to enhance required functions like controller placement and spanning tree modules.

3 324 JOURNAL OF NETWORKS, VOL. 10, NO. 5, MAY 2015 B. Related Works There are severalrecent simulation tools for developing and testing the OpenFlow protocol and applications such as OpenFlowVMS, Ns-3, Mininet and EstiNet. This subsection explains the efforts related to these tools. A prior study that aims to simulate OpenFlow networks was OpenVMSstarted in 2009 [14]. This work was designed to emulate OpenFlow enabled devices based on using virtual machines. However, virtual machines have significant limitations in regards to large resource overhead and thus do not scale very well. Moreover, it was developed to test real time functionalities of OpenFlow rather than simulating and evaluating arbitrary scenarios [10]. This also applies to Mininet, in the sense that Mininet is an emulation platform for the functional testing of OpenFlow protocols and applications. It uses lightweight OS containers to separate and emulate hosts and switches in a network, and therefore reduces overhead. The widely used NS-3 simulator has a project for OpenFlow protocol simulation as well. However, this approach only supports OpenFlow specification version 0.89, which is quite old compared to the latest version 1.4 [7], [15]. Furthermore, the NS-3 simulator is a user-level program, the same as real OpenFlow controller like NOX/POX. Therefore, the real OpenFlow controller program cannot be compiled and linked together with the NS-3 program to form a single executable program [7]. For example, it is compulsory to create C++ code from scratch to build new modules of OpenFlow switches or controller. Consequently, a real OpenFlow controller cannot be readily run in NS-3 simulations without code modification. According to the NS-3 official website, developers found it too difficult to upgrade NS-3 for OpenFlow version 1.0, and NS-3 still cannot support a real external OpenFlow controller. Other efforts like EstiNet combine the advantages of both thenetwork simulator and emulator without their respective limitations. As for emulator, EstiNet uses a real controller run on real devices and applications that can control simulated OpenFlow switches without any modification. EstiNet uses a new method of kernel reentering to support multiple hosts in a single kernel [7]. In terms of scalability, the EstiNet simulation engine has the capability to simulate large number of OpenFlow switches. Moreover, as EstiNet simulation engine can generate time-related OpenFlow performance measurements accurately, the results are repeatable. The EstiNet GUI can also show the playback of OpenFlow control packets once the simulation is finished. Table 1 shows the capabilities of NS-3, OMNeT++, Mininet and EstiNet with respect to their latest developments. Mininet needs to run a shell process to emulate virtual hosts and start up a user-space or kernel-space (OpenVswitch) to emulate each OpenFlow switch. Therefore, Mininet is less scalable compared to EstiNet, NS-3 and OMNeT++. Mininet can only be used to study the behaviour of virtual hosts, but cannot be used to study the time of network/application performance. Mininet s GUI can be usedfor observation purposes such as observing the packet playback of a simulation run, and users need to write Python scripts to set up and configure the emulation case. In contrast, OMNeT++ has a GUI which can be used for observation of results, where users need to write C++ code to set up and configure the simulation case. Overall, it is better to use OMNeT++ even if it takes time and effort to create simulations, though once modules are created, it ismuch easier to create new ones [6], [7], [10], [15]. III. ENVIRONMENT SETUP AND EVALUATION METHODOLOGY This section presents the simulation setup of OpenFlow topology for both OMNeT++ and Mininet. Comparing accuracy of OMNeT++ and Mininet is considered as the primary goal. In terms of analysing the performance results correctly, we have defined the parameters and configurations such as links, bandwidth, delay, and network size for the testing topologies. We also set the traffic configurations toidentical values for a fair comparison. There are a few reasons taken in to account when comparing OMNeT++ to Mininet. Firstly, Mininet is open and accessible for researchers and developers for SDN-based experiments. Secondly, Mininet provides a realistic setting for SDN-based environment [16]. Figure 1. OpenFlow Mesh topology with Spanning Tree Protocol In this evaluation, we used a mesh network for both tools shown in Fig. 1. We created the topologies and configurations, and use a fixed traffic profile with 4 switches, 2 hosts, and 1controller for simplicity, and in order to ensure that the controller calculates flow entries andinstalls them along the path from source to destination. For each environment, we configure the traffic control of links between nodes as 100Mbps for bandwidth and 5 ms for propagation delay. Each experiment was run for 370 seconds to study the idle_timeout and hard_timeoutof flow removal.all setups were run using Intel Core 2 CPU GHz 2.99 GHz 3.50 GB of RAM. In addition, Mininet was installed with Ubuntu LTS and a Linux generic kernel. Flow tableentries have an idle_timeout and a hard_timeout associated with them. For idle_timeout while matching packets, the flow entry will not be removed unless hard_timeout is reached, then the flow entry will force to beremoved. In Mininet and OMNeT++, the idle_timeout are set to 5 seconds by default, but

4 JOURNAL OF NETWORKS, VOL. 10, NO. 5, MAY hard_timeout is not set in Mininet, whereas in OMNeT++ is set to 120 seconds. To calculate the accuracy for both Mininet and OMNeT++, the ping process generated 370 packets with interval time of 1 second with increasing size of packet payloads 56B (8B header = 64B), 4,000B, 8,000B, 16,000B, 32,000B, 65,000B respectively. We compare the measurements of the mean RTT values collected from each tool. In Mininet, we use Floodlight remote controller. Whereas in OMNeT++ the controller is simulated with forwarding behaviour, which has complete knowledge about the networkand hence knows which other OpenFlow switches are located on the path to the destination host. As the performance metric for our evaluation is the mean RTT for the nodes in the investigated networks. We measure the RTT for the received echo reply messages. We repeated each run 6 times with different random number seeds to exclude simulation artefacts, which results in 1,850 RTT measurements.according to these measurements, we convert the mean RTT into milliseconds in order to show how well each host is connected to others. The experimental results will need recording a large number of samples. To describe the measurement of the numbers, the standard deviation and mean value of RTT are computed. The mean of the RTT is assumed to be the average value and the standard deviation is approximated using following equation [17]: T = N/R (1) where T is the packet transmission time between segments sent and receives an acknowledgement arrival measured in millisecond (round trip transmission), N is the packet size (bits), and R is the data rate bandwidth (bit/second). Then we need to calculate the round trip time RTT by the following equation: = α + 1 α T (2) where is the round trip time RTT (ms), α is the smoothing factor, is the old round trip time RTT, and T is the packet s new round trip transmission time. We can measure RTT values from the ping tool by α (value between 0 and 1) which is equal to the value multiplying with the old RTT as the top equation and then multiplying with T which is the new RTT between the segments sent and the acknowledgement arrival found from equation (1). Finally the mean RTT values are calculated after equation (2). IV. MININET CONSTRAINTS Insufficient CPU cycles and main memory consumptions in Mininetcould significantly reflect on the results of our experiments as number of emulated switches increase. Anothermajor restriction is that experiments are not able to run faster than real-time. We discuss the capabilities of OpenFlow controllers in Mininet. Moreover, the mesh topology was used to test the functionality of spanning tree protocol (STP). Mininet can use many types of controller including reference controller (ref), OpenVswitch controller (ovsc), and remote controller (NOX/POX/Floodlight/OpenDayLight). For example, you can simply decide which OpenFlow controller to use when you invoke the mn command: $ sudo mn --controller ref, ovsc, remote The reference controller is supported by Mininet package, whereas the ovsc controller is embedded with the OpenVswitch package [18]. Both controllers are only able to control up to 16 virtual switches. On the other hand, for larger networks we must use the remote controller. For the STP in Mininet, only the NOX classic [19], POX, or other topology aware SDN controllers can be performed, however the STP forref and ovsc controllers are not supported. For example, to run STP in Mininet: $./nox_core - v - v - i ptcp:6633 spanning_tree V. PERFORMANCE RESULTS This section presents a brief evaluation with respect to mean round-trip-time network metric of running Mininet and OMNeT++. Fig. 2 includes 6 measurement results of the mean RTT in both environments. Figure 2. Mean RTT of OMNeT++ and Mininet with StdDev Fig. 2 shows the mean RTT values for the increasing packet sizes from default size of 56 bytes up to 65,000 bytes. The table attached to Fig. 2 shows the values ofmean RTT in milliseconds. At the begging, the performance of OMNeT++ takes slightly longer time to process the ping than Mininet. While the packet size are increasing, the RTT values in OMNeT++ are performing nearly similar to Mininet as shown in packet size 65,000 bytes. To show the accuracy and correctness of both tools, the mean RTT of each packet size is nearly 2 ms different. Furthermore, the lines over the bar indicate the amount of variation from the mean standard deviation (σ). A low σ reflects on smaller packet sizes, whereasa high σare shown as size of packetincreases due to RTT rate are spread out over a large range of values. For example,the σin Mininet is higher than OMNeT++ of packetsizes 32,000 and 65,000 bytes. In overall,theσfor both

5 326 JOURNAL OF NETWORKS, VOL. 10, NO. 5, MAY 2015 OMNeT++ and Mininet are gradually increasing in accordance to the packet sizes. ACKNOWLEDGMENT This work is sponsored by the Centre for Real-Time Information Networks (CRIN) in the Faculty of Engineering & Information Technology at the University of Technology, Sydney (UTS). REFERENCES Figure 3. Flow entries removal by hard_timeout in OMNeT++ Fig. 3 illustrates the reason of longer mean RTT generated by OMNeT++ than Mininet usingpacket size of 56 bytes. There are 4 sudden high peaks due to hard_timeout of flow removal configured at 120 seconds, except the first high peak because of initialisation process between OpenFlow switch and controller. This is also true for Mininet,whenever the switch receives unmatched packet, it sends this packet to the controller [4]. This can result in longer initialisation time while the packet sizes are increasing (see Table II). TABLE II. INITIALISATION TIME FOR OMNET++ AND MININET (MS) The differences of initialisation time in Table II is because of packets in OMNeT++ are stored in a buffer at the switch model and sends the first packet to controller using only the buffer ID [10]. Therefore, OMNeT++ takes less initialisation time (ms) than Mininet, which sends the first packet containing the encapsulation of the complete packet [6]. VI. CONCLUSION AND FUTURE WORKS This paper has described OpenFlow extension toolbox of INET framework. A thorough overview of related works has been presented by discussing their capabilities of current developments. Some of the key constraints of Mininet were also described. The paper has presented network metrics and the topology of our study, and has evaluated the correctness and accuracy by setting up OMNeT++ and the popular Mininet platform with the same traffic configurations. It has been evident that the measured RTT values are closely identical from the mesh topology even when the size of the packet increases. We conclude that Mininet provides good facilities for OpenFlow functional testing of SDN controllers and applications, whereas OMNeT++ could also be a useful complement to existing tools for developing new SDN applications. Future studies would extend OpenFlow to a more distributed architecture. We intend to run more tests on other metrics such as link utilization, jitter rate, packet loss and we aim to scale the network topology up to hundreds of nodes or similar size of a modern data center to evaluate scalability issues. [1] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker and J. Turner, OpenFlow: Enabling Innovation in Campus Networks, SIGCOMM Computer Communication Rev., Vol. 38, March 2008, pp [2] A. AL Sabbagh, P. Pupatwibul, A. Banjar and R. Braun, "Optimization of the OpenFlow Controller in Wireless Environments for Enhancing Mobility, in 13th IEEE International Workshop on Wireless Local Networks, in the 38th IEEE Conference on Local Computer Networks (LCN), Sydney, Australia, October (2013). [3] P. Pupatwibul, A. Banjar and R. Braun, Using DAIM as a reactive interpreter for openflow networks to enable autonomic functionality, In Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM, pp [4] ONF White Paper. Software-Defined Networking: The New Norm for Networks. April [5] Open Networking Foundation, Openflow Switch Specification, version (Wire Protocol 0x04), September [6] B. Lantz, et al., "A network in a laptop: rapid prototyping for software-defined networks, in Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks, 2010, p. 19. [7] S. Y. Wang, C. L. Chou and C. M. Yang, Estinet open flow network simulator and emulator, IEEE Communications Magazine, vol. 51, no. 9, pp , [8] A. Varga, "The OMNeT++ discrete event simulation system, in Proceedings of the European Simulation Multiconference (ESM 2001), 2001, p [9] A. Varga and R. Hornig, "An overview of the OMNeT++ simulation environment, in Proceedings of the 1st international conference on Simulation tools and techniques for communications, networks and systems & workshops, 2008, p. 60. [10] D. Klein and M. Jarschel, "An OpenFlow Extension for the OMNeT++ INET Framework, OMNeT , Cannes France, March [11] A. Varga, Omnet++ user manual. OMNeT++ Discrete Event Simulation System. Available at: omnetpp. org/doc/manual/usman. html, [12] Chamberlain, T., Learning OMNeT : Packt Publishing Ltd. [13] [13]A. Varga. INET Framework for the OMNeT++ Discrete Event Simulator. com/inetframework/inet, [14] K. -K. Yap, OpenFlowVMS Simulating OpenFlow Networks, openflow. org/wk/index. php/openflowvms, March [15] [15]NS-3 version OpenFlow switch support. nsnam. org/docs/release/3. 16/models/html/openflow-switch. html, Dec [16] M. Gupta, J. Sommers and P. Barford, Fast, accurate simulation for SDN prototyping, In Proceedings of the second ACM SIGCOMM workshop on Hot topics in software defined networking, pp , August 2013.

6 JOURNAL OF NETWORKS, VOL. 10, NO. 5, MAY [17] D. Sünnen, Performance Evaluation of OpenFlow Switches, Semester Thesis at the Department of Information Technology and Electrical Engineering, [18] S. Azodolmolky. Software Defined Networking with OpenFlow, Packt Publishing Ltd, (2013). [19] N. Gude, T. Koponen, J. Pettit, B. Pfaff, M. Casado, N. McKeown, and S. Shenker, NOX: towards an operating system for networks, SIGCOMM Comput. Commun. Rev., 38(3):pp , Jul PakawatPupatwibul is currently working towards the Ph.D. degree in Information Systems, faculty of Engineering and IT from University of Technology Sydney, Australia, having graduated from Naresuan University with a B.Sc. in Computer Science, and Master s of Information Technology from UTS. He has worked attentively as a network administrator for SuanDusitRajabhat University, a government sponsored university in Thailand, for 7 years. His research interests include next generation networks, data center network, QoS and network management, especially in the area of intelligent agent-based network management systems. AmeenBanjar received his B.Sc from Taibah University (Saudi Arabia) and M.I.T advanced from University of Wollongong (Australia). He is currently working towards Ph.D. degree in Computing and Communications, at University of Technology Sydney UTS, Faculty of Engineering and Information Technology. He began his working career as a database designer and programmer at Taibah University, Information Technology Centre (ITC) in Saudi Arabia, for two years. He has a research interest in network management, especially in the area of intelligent agent-based network management systems. Abdallah AL Sabbagh received his B.Sc (Hons) on 2006 from The Open University (UK), and his MES and Ph.D. on 2010 and2013 respectively from the University of Technology, Sydney (UTS), Australia. He is currently a Lecturer at the School of Computing and Communications and a Research Engineer at the Centre for Real-time Information Networks (CRIN) within the faculty of Engineering and Information Technology (FEIT) at UTS. His research interests lie in the area of networking and distributed computing systems including: next generation networks, heterogeneous wireless networks, Software Defined Networking (SDN), Information-Centric Networking (ICN) and mobile Internet Protocol (IP) networks. Robin Braun received his B.Sc (Hons) from Brighton University (UK), and his M.Sc and Ph.D from the University of Cape Town. He holds the Chair of Telecommunications Engineering in the Faculty of Engineering and Information Technology of the University of Technology, Sydney, Australia. He is an executive member of the Centre for Real Time Information Networks (CRIN) at the University of Technology, Sydney (UTS). Prof. Braun was a member of staff of the Department of Electrical Engineering of the University of Cape Town from 1986 to He was the founder, and Director of the Digital Radio Research Group at the University of Cape Town, which supervised over 50 research degree candidates in the years that he was attached to it. Prof. Braun is currently a Senior Member of the Institute of Electrical and Electronic Engineers of the United States (IEEE).