ONOS Controller Performance Test Report

Transcription

1 ONOS Controller Performance Test Report Global SDN Certified Testing Center,SDNCTC w w w. s d n c t c. c o m ONOS Controller Performance Test Report

2 CONTENTS 1. INTRODUCTION TEST ENVIRONMENT CONFIGURATION CONTROLLERS UNDER TEST SERVER CONFIGURATION TEST TOOL TEST ITEM AND TEST RESULT CONTROL CHANNEL CAPACITY TEST TOPOLOGY DISCOVERY TIME TEST PACKET_OUT SENDING RATE TEST FLOW MOD SENDING RATE TEST END-END LINK ESTABLISHMENT TIME TEST ROLE ELECTION TIME TEST GLOBAL SDN CERTIFIED TESTING CENTER ONOS Controller Performance Test Report

3 1. INTRODUCTION As Software Defined Network (SDN) gradually becomes the new normal form in network world, the separation of transition and control makes the data plane become the simple data engine, while control plane assumes all logic and operation tasks. As the core component of control plane, the performance of SDN controller relates to the performance of the whole SDN network. With the increase of SDN business deploy speed, the performance of SDN controller will become the concerned focus of network users increasingly. OFsuite_Performance is one of the test tools of OFsuite series independently developed by Global SDN Certified Testing Center (SDNCTC), and devotes to the performance test of OpenFlow controller, it can simulate plenty of OpenFlow 1.3 switches on the general purpose Linux server, and various network topologies and all OpenFlow events. It has precise record and statistical functions, so it can effectively measure the processing capacity of controllers to OpenFlow information. This test tool is brief, efficient and easy to use, and keeps updating in order to provide more plentiful performance test cases and test scenarios for users. The results of performance test of SDN controller can provide reliable data support for users to do SDN network performance evaluation, model selection and business deploy. Global SDN Certified Testing Center leads the development and compiling of test standards in ONF and other standard structures in a long run, the detailed test Methodologies of all test cases in this report refer to the coming performance test specification of ONF, IETF and other standard organizations. This report takes ONOS controller as the controller under test, uses OFsuite_Performance to execute the test, and summarizes the results to get the performance test report. All test cases are automatically completed by OFsuite_Performance. In the large-scale network deploy, all controllers have the demand of clustered running, so except the performance of single-point running of controller, the performance test results of ONOS controller in cluster mode are also contained in this report. 2. TEST ENVIRONMENT CONFIGURATION 2.1 CONTROLLERS UNDER TEST Controller under test are ONOS controllers, and edition is the edition published on April. 2, 216. ONOS Controller Performance Test Report 1

4 2.2 SERVER CONFIGURATION The ONOS controller under test solely runs on the server, and the server configurations are as the following: Server model: Dell PowerEdge R72 CPU: Intel(R) Xeon(R) CPU E GHz quad-core RAM: 16GB 16MHz Operating system: Ubuntu server LTS 64bit Network card: 1Gbps Controller cluster: The controller cluster environment of all tests referring to cluster mode in this test report is composed of three above-mentioned servers with same configuration. 2.3 TEST TOOL The test tool is OFsuite_Performance, when executing test, OFsuite_Performance runs on a server, which is directly connected to an ONOS controller server (Standalone Mode), or respectively uses three network cards to directly connect to three ONOS controller cluster servers (cluster mode). The server configurations of this test tool are as the following: CPU: Intel(R) Xeon(R) 3.2GHz quad-core RAM: 8GB 1333MHz Operating system: Ubuntu server 12.4 LTS 64bit Network card: 1Gbps 3. TEST ITEM AND TEST RESULT 3.1 CONTROL CHANNEL CAPACITY TEST Test Purpose Test the memory usage of controller under test when maintaining a certain amount of OpenFlow 1.3 control channel connections. Test Methodology In this test, the test tool simulates a certain amount of OpenFlow 1.3 switches and connects to controller under test, and record the memory usage of the controller under test, the specific test procedures are as the following: 1. Turn on ONOS controller under test and install drivers, openflow, proxyarp and fwd modules; ONOS Controller Performance Test Report 2

5 2. Turn on test tool OFsuite_Performance, use --switches=[n] option to set a certain amount of OpenFlow 1.3 switches and connect them to the controller under test; 3. Record the memory usage conditions of controller after the connection is stable; 4. Use add-sw command to increase the quantity of switches and record the memory usage conditions of controller after the connection is stable; 5. Execute test with various switch quantities to get the final result. Test tool requirements 1. Test tool can simulate plenty of OpenFlow 1.3 switches and connect with controllers; 2. Test tool can respond to all related OpenFlow 1.3 protocol messages (Hello, Echo, feature_request, etc); 3. Test tool can monitor control channel activity at any time; 4. Test tool can adjust the quantity of connected switches neatly; 5. Test tool has thorough Log function. Test Results Standalone Mode Standalone Mode means that only one ONOS controller entity is turned on to execute test, the same below. Results can be seen in the following figure: Physical Memory Usage linear ring tree Usage(MB) Number of Switches In this test case, the ONOS is set up to start applications, i.e. drivers, openflow, proxyarp and fwd. Different numbers of switches and switch topology structures are set up to test the memory usage of the controllers that are maintaining the connection of all the switches, as shown in the graph above. In the graph above, the memory usage of the ONOS controllers with different switch nodes. The more switches the ONOS connects, the more memory it will take. Cluster Mode ONOS Controller Performance Test Report 3

6 Physical Memory Usage controller1 controller2 controller Usage(MB) Number of Switches In the cluster mode, three ONOS controllers can simultaneously connect 1 switches. When the number of switches is over 3, if the topological structure is set, the control channel may be disconnected, therefore, no topological structure is set in the cluster mode. The memory usage of every controller with different numbers of nodes in the cluster, as shown in the graph above. From the results of the test, it can be seen that there is little difference between the controllers in terms of memory usage in the cluster mode. Compared to the single point mode, the ONOS controllers in the cluster mode take more memory space. 3.2 TOPOLOGY DISCOVERY TIME TEST Test Purpose Test the discovery time of controllers for different quantities of switches and topological structures with different types. Test Methodology In this test, test tool will simulate a certain amount of switches, and the switches connect to each other to form a certain network topological structure, test the time of the used by the controller to completely discovery this topological structure, and the specific test procedures are the following: 1. Turn on the controller under test; 2. Turn on test tool, only set the quantity of switch, and topological structure is not set when turning on; 3. When the connection of switch and controller is stable, use set-topo command to set topology (predefined linear/ring/full-mesh/leaf-spine structure is supported, topological structure can also be set by users); 4. Test tool records the time when the controller sends the first LLDP message, and responds to the information according to the contents of flow table; 5. Test tool records the time when the last LLDP message is sent to the controller; 6. Use show-result command to view test result; 7. Alter the quantity of switches and set same topological structure to proceed iteration test; 8. Alter topological structure while maintain the quantity of switches to proceed iteration test; ONOS Controller Performance Test Report 4

7 9. Proceed five times of tests to get the average test result. Test Tool Requirements 1 Test tool can simulate plenty of OpenFlow 1.3 switches, and connect to controllers; 2 Test tool can respond to all related OpenFlow 1.3 protocol messages (Hello, Echo, feature_request etc); 3 Test tool can monitor control channel activity at any time; 4 Test tool can create various network topology structure; 5 Test tool can analyze Packet_out message in order to extract the LLDP message in it; 6 Test tool can analyze LLDP topology discovery message in order to give the relevant response; 7 Test tool has through Log function. Test Result Standalone Mode Topology Discovery Time linear ring tree Time(ms) Number of Switches The more the switches are, the longer it will take to find the topology of the tested controllers. On condition of the same number of switches, the more the links of topology are, the longer it will take to find the correspondent topology. Generally, the time spent on finding the topology is in a positive linear correlation with the complexity of the links. Cluster Mode ONOS Controller Performance Test Report 5

8 Topology Discovery Time linear ring tree Time(ms) Number of Switches In the cluster mode, with the same number of switches and the same topological structures, the time spent on finding the topology is longer than that in the single point mode. In addition, when the number of switches is larger than 3, the control channel is disconnected, when it cannot be statistically considered as effective results. 3.3 PACKET_OUT SENDING RATE TEST Test Purpose Test the delay and maximum rate that controller under test sends Packet_out message. Test Methodology In the test, test tool will simulate aswitch which is connected to the controller, test tool will send Packet_in message with a certain constant rate, and ARP_requests are carried in the message. Record the time used by sending Packet_in message. When receiving this Packet_in, the controller will send Packet_out message down, and test tool will record the time of sending Packet_out down after receiving Packet_out message. Calculate the response speed (latency) of controller under test for Packet_in message and sending rate (throughput) of Packet_out through all Packet_in and Packet_out message records. OFsuite_Performance test tool can reach Packet_in sending rate (TCP protocol itself is the limit, such as the size of receiving window and others) with constant speed of 5K level per second under the hardware configuration that listed in Chapter 2.3 of this text. The specific test procedures are as the following: 1. Turn on the controller under test; 2. Turn on test tool, set aswitch, do not set the topological structure, and wait the switch connects to the controller; 3. Use set-arp-rate command to set the rate of switch to send Packet_in on the command line port of test tool; 4. Wait the test tool to complete the test; 5. Use how-result command to view test result; 6. Alter the rate to send Packet_in, and test it again; 7. Do the five iteration tests and gain the average test result. ONOS Controller Performance Test Report 6

9 Test Tool Requirements 1. Test tool can simulate OpenFlow 1.3 switch, and connect to controller; 2. Test tool can respond all related OpenFlow1.3 protocol message(hello, Echo, feature_request etc); 3. Test tool can send Packet_in message containing ARP_request in order to trigger the controller under test to send Packet_out message; 4. Test tool can define the contents of Data field of Packet_in message by users; 5. Test tool can send Packet_in message in constant speed; 6. Test tool can analyze Packet_out message in order to confirm that this message is triggered by relevant Packet_in message; 7. Test tool has thorough Log function. Test Result Standalone Mode Packet_out Response Rate Packet_out rate(standalone mode) Packet_out Rate : Packets/s Packet_in Rate : Packets/s It can be seen from the test results of the above figure that when the Packet_in rate set by test tool is not more than 8 packets/s, ONOS controller can treat the received Packet_in message better, Packet_out sending rate and Packet_in rate of test tool can be the same; when the Packet_in rate set by test tool is more than 8 packets/s, it can be seen from the test results that Packet_out sending rate cannot be the same as Packet_in rate, meanwhile the delay increases accordingly (see figure below). By this token, the test has reached maximum Packet_out sending rate of the controller under test. ONOS Controller Performance Test Report 7

10 Packet_out Response Latency minimun latency (ms) maximum latency (ms) average latency (ms) Latency(ms) Packet_in rate : Packets/s What the figure above shows the delay that the controller under test sends Packet_out message, which is the delay between test tool sending Packet_in and controller under test sending relevant Packet_out back. It can be seen from the figure that with the increase of sending speed of Packet_in message, the delay of Packet_out message will be bigger and bigger. The detailed test result data are shown in the following table. A B C D E 1 packet_in rate minimum latency maximum average latency (ms) latency (ms) (ms) packet_out rate The packet_out rate in the above table means the when there is no packet loss, keep a record of the received packet_out messages and calculate the average speed instead of the number of the actually received packet_out messages. The results of the tests in the cluster mode are calculated in the same method. Cluster Mode In this test, three servers are respectively used to run the ONOS controller to form the controller cluster. In cluster mode, each Packet_in message is sent, and then a plurality of the same Packet_out messages will be sent down, test tool only records the first Packet_out message sent back by controller in order to calculate to get sending rate and delay of Packet_out, which is shown as the following figure. ONOS Controller Performance Test Report 8

11 Packet_out Response Rate Packet_out rate(cluster mode) 12 Packet_out Rate : Packets/s Packet_in Rate : Packets/s The figure above shows the test result to the rate that ONOS controller sending Packet_out in cluster mode. It can be seen that when Packet_in rate set by test tool is more than 1 packets/s, the Packet_out rate is lower than the Packet_in rate, and the processing delay will rapidly increase. Comparing to the test result of Standalone Mode above, it can be seen that the performance to send Packet_out in cluster is worse than that in Standalone Mode. Latency(ms) Packet_out Response Latency minimun latency (ms) maximum latency (ms) average latency (ms) Packet_in rate : Packets/s What the figure above shown is the latency that ONOS controller sends the Packet_out back in cluster mode. The latency performance in cluster mode is inferior to the latency performance in Standalone Mode, which is the same as Packet_out rate performance. Specific latency test result is listed in the following table. ONOS Controller Performance Test Report 9

12 A B C D E 1 packet_in rate minimum latency maximum average latency (ms) latency (ms) (ms) packet_out rate FLOW MOD SENDING RATE TEST Test Purpose Test the latency and max rate when the controller under test sends the flow entry down. Test Methodology In this test, test tool will simulate aswitch that is connected to the controller, test tool will send Packet_in message containing ARP_request up with a certain rate, test tool sends Packet_in message up containing relevant ARP_reply after receving Packet_out sent down by the controller to be tested, and the controller that will be tested will sending Flow_mod information down, then calculate the delay and rate that the controller under test sends the flow table down according to the records of Packet_in message and Flow_mod information, the specific test procedures are as the following: 1. Turn on the controller under test 2. Turn on test tool, set a switch, and do not set the topological structure, wait switch to connect to the controller; 3. Use set-arp-rate command to set the rate that switch sends Packet_in message up on test tool; 4. Wait test tool to complete the test; 5. Use show-result command to view test result; 6. Alert the rate to send Packet_in up and test it again; 7. Do five iteration tests to gain the average test result. Test Tool Requirements 1. Test tool can simulate OpenFlow 1.3 switch, and connect to the controller; 2. Test tool can respond to all related OpenFlow 1.3 protocol information(hello,echo,feature_request etc); 3. Test tool can send Packet_in message containing ARP_request up in order to trigger the controller under test to send Packet_out down; 4. Test tool can send Packet_in message containing ARP_reply up in order to trigger the controller under test to send Flow_mod addition flow table down; 5. Test tool can analyze Flow_mod information, which makes sure this information is triggered by test tool; 6. Test tool can send Packet_in message up with constant speed defined by users; 7. Test tool has thorough Log function. ONOS Controller Performance Test Report 1

13 Test Result Standalone Mode Flow Setup Rate flow_mod rate(standalone mode) Flow_mod Rate : Packets/s Packet_in rate : Packets/s It can be seen from the test result in the figure above that when Packet_in sending rate of test tool is lower than 5packets/s, ONOS controller can better process Packet_in message, the Flow_mod rate and Packet_in rate have little difference; when Packet_in rate is more than 5packets/s, the Flow_mod sending rate and Packet_in rate of ONOS controller have larger differences, by this taken, this rate is the max flow table installing rate of ONOS controller in this hardware configuration. Flow Setup Latency minimun latency (ms) maximum latency (ms) average latency (ms) 6 5 Latency(ms) Packet_in Rate : Packets/s The figure above shows the latency that ONOS controller sends the flow entry down. It can be seen from the figure, when the rate that test tool sends Packet_in is lower than 4packets/s, the latency that ONOS controller sends flow table down is lower relatively; when the rate that test tool sends Packet_in up is more than 4packets/s, the latency that ONOS controller sends flow entry down increases accordingly. The detailed test results are listed in the following table. ONOS Controller Performance Test Report 11

14 A B C D E 1 packet_in rate minimum latency maximum average latency (ms) latency (ms) (ms) packet_out rate The flow_mod rate results in the above table means when there is no loss of packets, count the received flow_mod messages and calculate the average speed instead of the number of the actually received flow_mod messages. The results of the tests in the cluster mode are calculated in the same method. Cluster Mode In this test, three servers respectively run ONOS controller to form controller cluster for the test. Test results are shown in the following figure and table. Flow Setup Rate flow_mod rate(cluster mode) Flow_mod Rate : Packets/s Packet_in rate : Packets/s In can be seen from the figure above, comparing to Standalone Mode, the rate that ONOS controller sends flow table down in cluster mode decreases obviously, and the max rate is 8 packets/s, when Packet_in rate is more than 8 packets/s, the Flw_mod rate got by test has been already not the valid result, the latency of sending flow table down also increases immensely (see the following figure). ONOS Controller Performance Test Report 12

15 Flow Setup Latency minimun latency (ms) maximum latency (ms) average latency (ms) Latency(ms) Packet_in rate The figure above shows the latency that ONOS controller sends flow entry down in cluster mode, specific test result is shown in the following table. A B C D E 1 packet_in rate minimum latency maximum average latency (ms) latency (ms) (ms) packet_out rate END-END LINK ESTABLISHMENT TIME TEST Test Purpose Test the time of setting end to end link of controller under different network topology. Test Methodology In this test, the test tool will simulate a certain amount of switches which are connected to controller by specific topological structure. In addition, the test tool will simulate two hosts, which are connected to two switches respectively (position of hosts can be user-defined). By simulating ARP request/reply process of hosts, setting time from end to end path of controller which will be measured under the mutual connection among hosts. Specific test steps are as follows: 1. Start the controller under test; 2. Start test tools, set the number of switches; ONOS Controller Performance Test Report 13

16 3. Set the network topology structure well, after the connection between switch and controller become stable; 4. Wait for the controller to complete the topology discovery process; 5. Use the set-host command to set the host to connect to a particular switch; 6. Wait for test tools to complete the test; 7. Use the show-result command to view the test results; 8. Change the number of switches or topology structure and do test again; 9. Do five iteration tests to get average test results. Test Tool Requirements 1. Test tools can simulate OpenFlow 1.3 switch, and establish a connection with the controller; 2. Test tools can respond to all relevant OpenFlow 1.3 protocol information (Hello, Echo, feature_request etc); 3. Test tools can send Packet_in message containing ARP_request to touch off the measured controller to send Packet_out; 4. Test tools can send Packet_in message containing ARP_request to touch off the measured controller to send flow mod addition flow table; 5. Test tools can resolve flow mod message to confirm whether the link is established successfully; 6. Test Tool can simulate a host connected to the switch; 7. The test tool can simulate communication between hosts. Test Results Standalone Mode Reactive Path Setup Time linear ring tree Path Setup Time(ms) Number of Switches The test results of End-end link establishment time under Standalone Mode is shown in figure. In this test, the two hosts are connected to two different switches respectively to test the link establishment time under the biggest network hop count. For Linear topology, two hosts are connected to two switches at the head and the tail of the topology of switch. The link establishment time of Linear topology is longer than that of Ring topology. Cluster Mode ONOS Controller Performance Test Report 14

17 Reactive Path Setup Time linear ring tree Path Setup Time(ms) Number of Switches The test results of End-end link establishment time under Cluster Mode is shown in figure. In this test, when the number of switches is equal to or greater than 3, ONOS controller cannot complete acquisition of Leaf-spine topological structure, the corresponding test from end to end link establishment cannot be completed, so N/A is used to show 3.6 ROLE ELECTION TIME TEST Test Purpose Test time of certainty of master slave relation of each controller of cluster of controllers Test Methodology When the controller operates in a cluster mode, the master and slave roles allocation of each controller shall be decided by internal mechanism. This process belongs to contents of east-west direction interface of SDN controller, does not belongs to OpenFlow. But its effects must be considered, if the time consuming of this process is too long. In this test, the test tool connects each simulative switch to each controller of controller cluster respectively, calculates the time difference between each TCP message and the last Role_request information, in order to obtain schedule of master slave relation of each controller of controller cluster. Specific test steps are as follows: 1. Start the controller cluster; 2. Start test tools, set the number of switch well; 3. Do packet capture at an end of the test tool; 4. Wait long enough period of time to determine the controller does not send Role_request message anymore; 5. Test tool stops the packet capture; 6. Calculate the schedule of master slave relation of each controller of controller cluster; 7. Change the number of the switch and do test again; 8. Do five iteration tests to get average test results. Test Tool Requirement 1. Test tools can simulate OpenFlow 1.3 switch, and establish a connection with the controller cluster; ONOS Controller Performance Test Report 15

18 2. Test tool can respond to Role_request message in OpenFlow 1.3 protocol; 3. Test tool can simulate OpenFlow 1.3 switch with different quantity. Test Result Role Intimation Time Time(s) Time(s) NUmber of Switches It can be seen from the test results in the above figure that the more the number of switches is, the longer the time needed by the controller cluster to complete assignment of master-slave roles is. 4. GLOBAL SDN CERTIFIED TESTING CENTER As a third-party, neutral SDN testing and certification lab, Global SDN Certified Testing Center is committed to the research and the popularization of SDN (Software Defined Network) technologies. Beyond the United States, it is the first SDN certified testing center authorized by the ONF across the world. The center enthusiastically assists the work group of ONF Interoperability Testing in making the rules for the test and establishing a testing and certification system, whereupon enhancing SDN technical standards and commercial deployment. As the third-party independent neutral evaluating organization, Global SDN Certified Testing Center supplies services to various network / terminal devices, cloud computing data centers, operators, and enterprises. Based on actual customer needs, formulate a detailed test program and certification program, and do rigorous independent testing, and ultimately deliver detailed and authoritative test report. ONOS Controller Performance Test Report 16