OpenFlow Controllers over EstiNet Network Simulator and Emulator: Functional Validation and Performance Evaluation

Transcription

1 OpenFlow Controllers over EstiNet Network Simulator and Emulator: Functional Validation and Performance Evaluation 1 Shie-YuanWang Chih-LiangChou andchun-mingyang DepartmentofComputerScience,NationalChiaoTungUniversity,Taiwan shieyuan@cs.nctu.edu.tw EstiNetTechnologies,Inc. clchou@estinet.com Abstract In this article, we use the EstiNet OpenFlow network simulator and emulator to perform functional validation and performance evaluation of the widely-used NOX OpenFlow controller. EstiNet uses an unique kernel reentering simulation methodology to enable real applications to run on nodes in its simulated network. As a result, without any modification, the real NOX OpenFlow controller readily runs on a host in an EstiNet simulated network to control thousands of simulated OpenFlow switches. Using EstiNet as the testing and evaluation platform, we studied how NOX implements the learning bridge protocol(lbp) and the spanning tree protocol(stp) based on the OpenFlow 1.0 protocol. Our simulation results show that these protocols, which are implemented as loadable components in NOX, do not synchronize their gathered information well and thus NOX may give wrong forwarding instructions toanopenflowswitchafteralinkfailure.wealsofoundthatwhennox sstpdetectsalinkfailure, itdoesnotsendamessagetoanaffectedopenflowswitchtodeleteobsoleteflowentries.asaresult, becausetheobsoleteflowentryexpiresonlyafteranidleperiodof5seconds,itmaybematchedand used endlessly causing the OpenFlow switch to continue to forward incoming matched packets onto a brokenlink.ourresultsrevealthatthelbpandstpcomponentsprovidedinnoxserveonlyasbasic implementations and lack information synchronization, and there is much room left to further enhance them.

2 2 I. INTRODUCTION Software-Defined Networks(SDN)[1] is a new type of network that can be programmed by a software controller according to various needs and purposes. The goal of SDN is to facilitate innovations in network architecture and protocol designs. To achieve this goal, the OpenFlow protocol[2] has been proposed to define the internal architecture of an OpenFlow switch and the messages exchanged between an OpenFlow controller and OpenFlow switches. In an OpenFlow network, because the operation and intelligence of the network are fully controlled by an OpenFlow controller, the correctness and efficiency of the functions implemented by the controller must be fully tested before its uses in a production network. Testing the correctness and evaluating the performance of a network protocol can be performed in several approaches. One approach is performing these tests over an experimental testbed(e.g., Emulab[3] and PlanetLab[4]). Although this approach uses real devices running real operating systems and applications and can generate more realistic testing results, the cost of building a large experimental testbed is huge and generally the testbed is not easily accessible to many users. Traditionally, modeling checking and symbolic execution have long been used to automate testing a system[5][6]. However, the scalability is the main challenge for this approach because thetestspaceisverylarge. Another common approach is via simulation, in which the operations of real devices and their interactions are modeled and executed in a software program. The simulation approach has many advantages such as lost cost, flexible, controllable, scalable, repeatable, accessible to manyusers,andfastthanrealtimeinmanycases.however,ifthemodelingofrealdevices is not accurate enough, the simulation results may differ from the experimental results. To overcome this problem, the emulation experiment approach may be used. Emulation differs from simulationinthatanemulationislikeanexperimentandthusmustbeexecutedinrealtimewhile simulation speed can be faster or slower than the real time. Furthermore, in an emulation some real devices running real operation systems and application programs will interact with some simulated devices. In contrast, in a simulation generally no real operation systems or applications are involved. In this article, we introduce the EstiNet OpenFlow network simulator and emulator[7]. EstiNet uses an unique approach to testing the functions and performances of OpenFlow controllers. By

3 3 using an innovative simulation methodology, which is called the kernel reentering methodology, EstiNet combines the advantages of both the simulation and the emulation approaches. In a network simulated by EstiNet, a simulated device can run the real Linux operating system and any UNIX-based real application program can readily run on a simulated device without any modification. With these unique capabilities, EstiNet s simulation results are as accurate as those obtained from an emulation while still preserving the many advantages of the simulation approach. In testing OpenFlow controllers, since the first and most widely-used NOX OpenFlow controller[8]isarealapplicationprogramrunnableonlinux,noxcanreadilyrunonahostin an EstiNet simulated network to control thousands of simulated OpenFlow switches. In EstiNet, because these real OpenFlow controllers are tested and evaluated in simulations rather than in emulations, the tests and evaluations can be performed much faster than real time. In addition, in EstiNet, the performance results of a simulated OpenFlow network managed by these OpenFlow controllers are correct, accurate, and repeatable. These performance results can be correctly explained based on the parameter settings(e.g., link bandwidth, delay, downtime, etc.) and configurations(e.g., network size, mobility pattern or speed) of the simulated OpenFlow network. We have used EstiNet to perform functional validation and performance evaluation of several protocols that are implemented by NOX as components. In this article, we choose the learning bridge protocol(lbp) and the spanning tree protocol(stp) as illustration examples. We studied the complicated interactions among these OpenFlow-implemented protocols and the address resolution protocol(arp) when the network traffic is TCP traffic. Our simulation results and detailed logs reveal their behavior, efficiency, and implementation flaws under the tested network settings. II. SIMULATION ARCHITECTURE OF ESTINET To implement the kernel reentering methodology, EstiNet uses tunnel network interfaces to automatically intercept the packets exchanged by two real applications and redirect them into the EstiNet simulation engine. As shown in Figure 1(a), inside the EstiNet simulation engine, a protocol stack composed of the MAC/Phy layers along with other layers below the IP layer are createdforeachsimulatedhost.packetstobesentoutonhost1aresentouttotheoutputqueue of tunnel interface 1 where the simulation engine will fetch them later. After fetching a packet

4 4 from tunnel interface 1, the simulation engine processes the packet through the protocol stack created for host 1 to simulate the MAC/Phy and many other mechanisms of the network interface usedbyhost1.forexample,theeffectsofthelinkdelay,linkbandwidth,linkdowntime,and linkbit-error-rate(ber)areallsimulatedinthephymodule.thephymoduleofhost1will deliverthepackettothephymoduleofhost2afterthelinkdelayplusthetransmissiontimeof thepacketonthislinkbasedonthesimulationclock.then,thepacketwillbeprocessedfrom thephymoduleuptotheinterfacemodule,whereitiswrittenbackintothekernelviatunnel interface 2. The packet will then go through the IP/TCP/Socket layers and finally be received by the application running on host 2, which ends its journey. By this methodology, all Linux-based real applications can run on a simulated network in EstiNet without any modification and they all use the real TCP/IP protocol stack in the Linux kernel to create their TCP connections. Figure1(b)showshowweextendthismethodologytosupportrunningarealOpenFlow controller on an EstiNet simulated network. Since real OpenFlow controllers such as NOX are normal application programs, they readily run on a simulated host in EstiNet without any modification. However, because a real OpenFlow switch needs to set up a TCP connection to the OpenFlow controller to receive its messages, we simulate the operations of each OpenFlow switchinsidethesimulationengineandletitcreateanopenflowtcpsocketboundtoanetwork interface(in this example, the used network interface is tunnel interface 2). With this design, inestinetasimulatedopenflowswitchcansetuparealtcpconnectiontoarealopenflow controller to receive its messages. All messages exchanged between a real OpenFlow controller and a simulated OpenFlow switch are accurately scheduled based on the simulation clock. Therefore, the results of functional validation and performance evaluation of a real OpenFlow controller are correct and repeatable over EstiNet. III. COMPARISON WITH RELATED TOOLS Currently, very few network simulators support the OpenFlow protocol and the most notable oneisns-3[9].ns-3isthemostwidelyusednetworksimulatorintheworld.thereisaprojectof ns-3 about supporting the OpenFlow protocol and the version of the OpenFlow protocol supported is0.89,whichisoldasthelatestversionofopenflowasofthiswritingisalready1.3.1.ns-3 simulates the operations of an OpenFlow switch by compiling and linking an OpenFlow switch C++ module with its simulation engine code. To simulate a real OpenFlow controller, ns-3 also

5 5 User Space Host 1 s Application (1) (4) Program Socket Simulation Engine Host 1 Host 2 Interface Interface ARP FIFO MAC8023 TCPDUMP Phy (5) TCP/UDP ARP FIFO MAC8023 TCPDUMP Phy (2) IP (7) (3) Tunnel 1 Tunnel 2 Kernel Space (a) The host to host case (6) Host 2 s Application Program Socket (9) (8) Simulation Engine OpenFlow Switch Socket User Space Interface Interface FIFO ARP ARP MAC8023 FIFO FIFO Phy TCPDUMP TCPDUMP MAC8023 MAC8023 Controller Socket Phy Phy TCP IP Tunnel 1 Tunnel 2 (b) The controller to OpenFlowSwitch case Kernel Space Fig. 1: Simulation Architecture of EstiNet implementsitasac++moduleandcompilesandlinksitwithitssimulationenginecode.in fact, all devices/objects simulated in ns-3 are implemented as C++ modules compiled and linked together with its simulation engine code to form a user-level executable program(i.e., the ns-3). Because the ns-3 program is a user-level program and a real OpenFlow controller such as NOX is also a user-level program, a real OpenFlow controller program cannot be compiled and

6 6 linkedtogetherwiththens-3programtoformasingleexecutableprogram.asaresult,areal OpenFlow controller cannot readily run without modification on a node in a network simulated byns-3.itisthisreasonwhyns-3hastoimplementitsownopenflowcontrollerfromscratch as a C++ module. This approach wastes much time and effort to re-implement widely-used real OpenFlow controllers. In addition, the running behavior of a re-implemented OpenFlow controllermoduleinns-3maynotbethesameasthebehaviorofarealopenflowcontroller because the former is a much-simplified abstraction of the latter. For example, as documented in ns-3, the spanning tree protocol and the MPLS function are not supported. As another example, in ns-3 there is no TCP connection between a simulated OpenFlow switch and its simulated OpenFlow controller. Since this usage is not the usage in the real world, the simulation results will differ from the real results when the TCP connection in the real world experiences packet losses or congestion. Regarding network emulators that support the OpenFlow protocol, currently there are very few such tools and the most notable one is Mininet[10]. Mininet uses the virtualization approach to create emulated hosts and uses the Open vswitch[11] to create software OpenFlow switches on a physical server. The links connecting a software OpenFlow switch to emulated hosts or to other software OpenFlow switches are implemented by using the virtual Ethernet pair mechanism provided by the Linux kernel. Because an emulated host in Mininet is like a virtual machine, real applications can readily run on it to exchange information. A real OpenFlow controller, which is alsoarealapplication,canalsorunonanemulatedhosttosetuptcpconnectionstosoftware OpenFlow switches to control them. With this approach, emulated hosts and software OpenFlow switches can be connected together to form a desired network topology and be controlled by a real OpenFlow controller. Although Mininet can be used to as a rapid prototyping for software-defined networks, it has several limitations. As stated in[10], the most significant limitation of Mininet is its lack of performance fidelity because it provides no guarantee that an emulated host in Mininet that is readytosendapacketwillbescheduledpromptlybytheoperatingsystemtosendthepacketand it provides no guarantee that all software OpenFlow switches in Mininet will forward packets atthesamerate.thepacketforwardingrateofasoftwareopenflowswitchinmininetis unpredictable and varies in every experimental run as it depends on the CPU speed, the main memory bandwidth, the numbers of emulated hosts and software OpenFlow switches that must

7 7 bemultiplexedoveracpuinmininet,andthecurrentsystemactivitiesandload.asaresult, MininetcanonlybeusedtostudythebehaviorofanOpenFlowcontrollerbutcannotbeused to study any time-related network/application performance. In contrast, EstiNet combines the advantages of both the simulation and the emulation approaches without their respective shortcomings. Like in an emulation, in EstiNet a real OpenFlow controller can readily run without modification to control simulated OpenFlow switches and real applications can readily run on hosts running a real operating system to generate realistic network traffic. However, the operations and interactions among these real applications, the real OpenFlow controller, the OpenFlow switches, hosts and links in a studied network are all scheduled by the EstiNet simulation engine based on its simulation clock, rather than be multiplexed and executed in an unpredictable way by the operating system. For this reason, differing from Mininet, EstiNet generates time-related OpenFlow performance results correctly and the results are repeatable. In Figure 2, we compare EstiNet, ns-3, and Mininet according to their latest developments. Most comparison results are self-explanatory and thus we only explain the scalability and GUI comparison results. EstiNet uses the kernel reentering methodology to use a single kernel to support multiple hosts and its simulation engine process can support multiple OpenFlow switches. As a result, it is highly scalable. Ns-3 is also highly scalable as its simulated hosts, OpenFlow switches, and controller are all implemented as C++ modules and linked together as a single process. In contrast, Mininet needs to run up a shell process(e.g.,/bin/bash) to emulate each hostandneedstorunupauser-spaceopenvswitchprocess(orakernel-spaceopenvswitch)to simulate each OpenFlow switch. As a result, it is less scalable than EstiNet and ns-3. Regarding theguisupports,whichareveryimportantfortheuser,estinet sguicanbeusedtoeasilyset upandconfigureasimulationcaseandbeusedtoobservethepacketplaybackofasimulation run.theguiofns-3,ontheotherhand,canonlybeusedforobservationoftheresultsand theuserneedstowritec++orscriptstosetupandconfigurethesimulationcase.formininet, itsguicanbeusedforobservationpurposesonlyandtheuserneedstowritepythonscriptsto set up and configure the simulation case. IV. STUDY TARGETS AND SIMULATION SETTINGS WeusedthenetworktopologyshowninFigure3tostudyhowNOXimplementsitsLBP and STP. These protocols are implemented as the switch and the spanning tree components

8 8 Fig. 2: A comparison of EstiNet, ns-3, and Mininet innox.thereasonwhywechosetostudythemisbecausetheseprotocolsareimportantand fundamental to the operations of a network and they are relatively the most complicated ones among those provided in NOX. During the simulations, these components are loaded into and reside in the core of NOX simultaneously. Nodes3,4,5and11aresimulatedhostsrunningtherealLinuxoperatingsystemwherereal applications can run without modification. Nodes 6, 7, 8, 9, and 10 are simulated OpenFlow switches supporting the OpenFlow 1.0 protocol. Node 1 is the host where NOX will be running duringsimulation.(inthefollowing,wecallitthe controllernode forbrevity.)node2isa simulated legacy(normal) switch that connects all simulated OpenFlow switches together with the controller node. It forms a management network over which the TCP connection between each simulated OpenFlow switch and the controller node will be set up. All OpenFlow messages between NOX and simulated OpenFlow switches are exchanged over this management network. In contrast, the network formed by simulated OpenFlow switches, simulated hosts, and the links connecting them together is the data network over which real applications running on simulated

9 9 hostswillexchangetheirinformation.wesetthebandwidthanddelayofeachlinkinboththe management and data networks to be 10 Mbps and 10 ms, respectively. To test the path-finding and network convergence performance of NOX after a link failure, each simulation run starts at 0 thsecandendsat100 thsecinthesimulatednetworkandthelinkbetweennodes6and7is purposely turned down between 40 th sec and 65 th sec. Fig.3:ThetopologyofthetestedOpenFlownetworkandtheoriginalpathofaTCPtraffic flow We tested the behaviorof a greedy TCP flow on the data network controlled by NOX. Because in EstiNet real applications can directly run on simulated hosts, we chose node 11 asthedestinationhostandranupthe rtcp applicationonit.wechosenode3asthesource hostandranupthe stcp applicationonit.oncethestcpsuccessfullysetsuparealtcp connectionwiththertcp,itgeneratesgreedytcptraffictothertcpsubjecttothetcperror, flow, and congestion control algorithms. All of these real applications are set to start at 30 th sec ratherthan0 thsec.thisisbecausewewantthenox sstptohaveformedastablespanning treeoverthedatanetworkbeforeanypacketentersintoit. We also tested the effects of the ARP protocol on the path-finding and network convergence speed of the simulated OpenFlow network. Normally, on a real network the ARP protocol is

10 10 enabledoneveryhostandtriggeredondemandtofindoutthe(macaddress,ipaddress) mapping relationship. However, under some circumstances. this mapping table can be pre-built toavoidthearprequest/replylatencyandinthiscasethearpprotocolisdisabledonhosts. Our simulation results show that enabling or disabling the ARP protocol can have a significant impact on the path-finding capability/speed of an OpenFlow network. V. FUNCTIONAL VALIDATION Before presenting the functions of NOX s LBP and STP over the tested network, we briefly explain the main OpenFlow messages exchanged between NOX and OpenFlow switches to implement LBP and STP. InOpenFlow1.0,aPacketInmessageisissuedbyanOpenFlowswitch(tosavespace,inthe followingwewilluse switch torefertoanopenflowswitchwhenthereisnoambiguity) tothecontrollertoaskithowtoprocessareceivedpacket.itcancontainthefullcontentof thepacketorjusttheheadersofthepacketwithafewbytesofthedatapayload.apacketout messageisissuedbythecontrollertoaswitchtoinstructithowtoprocessandsendoutapacket. ThepacketmaybecarriedinthePacketOutmessageorisapacketthatisalreadybufferedina switch waiting for a PacketOut message. A FlowModify message is issued by the controller to aswitchtoadd/modify/removeaflowentryintheflowtable.whenapacketentersaswitch, itismatchedagainstallflowentriesintheflowtable.ifitmatchesoneormoreentries,the entrywiththehighestprioritywillbeusedtoprocessthepacket.eachentrycanbeassociated withsomeactions,whichmaybedrop,forward,etc.andtheactionswillbeappliedtoa matchedpacket.ifthereisnomatchintheflowtable(whichiscalledatablemiss),aswitch candecidetodropthepacketorissueapacketinmessagetothecontrolleraskingithowto process the packet. Initially, the flow table in every switch is empty. The PortModify message isissuedbythecontrollertoaswitchtochangethestatusofoneofitsports.forexample,the statusofaportcanbesettofloodornonflood,whichmeansthatwhentheswitchneedsto sendapacketoutofallofitsports(excludingtheingressport),whetherthisportshouldbe included or not for a flood operation.

11 11 A.LBPinNOX HereweuseFigure3toexplainhowNOX slbpworksonthetestednetwork.suppose thatthesourcehost(node3)sendsitsfirsttcpdatapackettothedestinationhost(node 11).Whenthispacketentersnode6,atablemisseventisgeneratedbecausethereisnoflow entryinthetablethatcanbematchedtoit.thiseventcausesnode6toissueapacketin message to NOX asking it how to process this packet. After receiving the message, NOX learns the(sourcenode,ingressport)=(node3,leftport)mappinginformationfornode6.itthen issuesapacketoutmessagetoinstructnode6tofloodthispacketbecauseitdoesnothaveany forwarding information for the destination host(node 11) so far. After receiving the PacketOut message,node6floodsthepackettonodes9and7.tosimplifythediscussion,wenowfocus onthefloodingpathalongnodes6,9,and10. Afternode9receivesthepacket,likewhathasoccurredinnode6,itissuesaPacketInmessage tonoxandnoxlearnsthatonnode9(sourcenode,ingressport)=(node3,topport).not knowingtowhichoutputporttoforwardthispacket,noxalsosendsoutapacketouttonode9 instructing it to flood the packet. When node 10 receives the packet, the same scenario happens andnoxlearnsthatonnode10(sourcenode,ingressport)=(node3,leftport).finally,when node11(thedestinationhost)receivesthepacket,itsendsbackatcpackpackettothe sourcehost(node3).whenthispacketentersnode10,becausethereisnoentryinitsflow table for this reverse flow, node 10 issues a PacketIn message to NOX asking for instructions. Now with the mapping information learned previously, NOX issues a FlowModify message to node10instructingittoaddaflowentryof(destinationnode,outputport)=(node3,leftport) intoitsflowtableandforwardthepacketoutofitsleftport.whennode9receivesthepacket, thesamescenariooccurs.itissuesapacketintonoxandnoxinstructsittoaddanentryof (destinationnode,outputport)=(node3,topport)andforwardthepacketoutofitstopport. Thesamescenarioappliestonode6andfinallytheTCPACKpacketreachesnode3,finishing a round trip. WhentheTCPACKpacketiscomingbackonthereturningpath,becausenodes10,9,and 6eachissuesaPacketInmessagetoNOX,NOXalsolearned(sourcenode,ingressport)= (node11,bottomport),(node11,rightport),and(node11,bottomport)fornodes10,9,and6, respectively. However, we found that although NOX learns a mapping from a PacketIn message

12 12 issuedbyaswitch,itdoesnotimmediatelyaddthismappingasaflowentrytothatswitch using the FlowModify message. Instead, it silently keeps the learned mapping information until themappingisusedinthefuture.forthisreason,whenthesecondtcpdatapackettraverses nodes6,9,and10toreachnode11,eachofthesenodeswillgenerateatablemisseventand issueapacketinmessagetonoxagain.however,atthistimenoxalreadyknowshowto forward the packet to node 11 on these nodes. Therefore, it issues FlowModify messages to add theselearnedmappinginformationintotheflowtablesofnodes6,9,and10andinstructthem toforwardthepacketoutofaportaccordingtothenewlyaddedentries.sofar,allrequired flowentriesforboththeforwardandreversedirectionsofthistcpflowhavebeeninstalledin theflowtablesofnodes6,9,and10.startingfromthethirdtcpdatapacket,whenatcp DATAorACKpacketentersthesenodes,nomorePacketInmessageswillneedtobesentto NOX from these nodes. Intheabovecase,however,ifthesourcehostsendsoutUDPpacketstothedestinationhost andthedestinationhostdoesnotreplyanypackettothesourcehost,wefoundthatthebehavior oftheopenflownetworkisverydifferentfromthetcptrafficcase.whenthefirstudppacket traverses nodes 6, 9, 10 to reach the destination host, the interactions between these nodes and NOXarethesameasthoseintheTCPtrafficcase.However,becausethereisnoreturningpacket fromthedestinationhosttothesourcehost,noxhasnochancetolearn(sourcenode,ingress port)=(node11,bottomport),(node11,rightport),and(node11,bottomport)fornodes10,9, and6,respectively.asaresult,whenthesecondandalloffollowingudppacketscontinueto traversenodes6,9,and10,eachofthemwilltriggeratablemisseventoneachofthesenodes, causing an excessive number of PacketIn messages to be sent to NOX continuously. Conceivably, whenthenumberofnodesonthepathislarge,whenthesendingrateofanunidirectionaludp flowishigh,orwhentherearemanyunidirectionaludpflowsinthenetwork,noxwillbe burdened by a high rate of PacketIn messages, reducing its capability to manage a large network. B.STPinNOX NOX s STP uses the LLDP(Link Layer Discovery Protocol)[12] packets to discover the topologyofanopenflownetwork.foreachswitch,afteritispoweredonandhasestablisheda TCPconnectiontoNOX,NOXimmediatelysendsitaFlowModifymessagetoaddanentryinto its flow table. This flow entry will match future received LLDP packets and its associated action

13 13 is SendthereceivedLLDPpackettothecontroller. Foreachportofaswitch,every5seconds (the LLDP transmission interval) NOX sends a PacketOut message to the switch asking it to send the LLDP packet carried in the PacketOut message out of the specified port. Since every switch has already had a flow entry matching received LLDP packets, when a switch receives anlldppacketfromoneofitsneighboringswitches,itwillsendthereceivedlldppacketto NOX. With these received LLDP packets from all switches, NOX builds the complete network topologyandcomputesaspanningtreeoverit.foralinkthatisincluded/notincludedonthe computed spanning tree, NOX sends a PortModify message to each of the two switches that areatthetwoendsofthelink.thismessageenables/disablesthefloodingstatusoftheport connectedtothelink.foreachlinkdetectedinnox,a10-second(whichistwotimesofthe LLDP transmission interval) timer is set up in NOX to monitor its connectivity. If a link s timer expires, NOX views that this link is currently down and recomputes a new spanning tree. Then, it uses PortModify messages to change the flooding status of the affected ports. We found that the PacketIn and PacketOut messages triggered by the exchanges of LLDP packetsonalargenetworkcancauseaheavyprocessingburdenfornox.forexample,ifthere are 100 switches in the network and each has 24 ports connecting to neighboring switches, becausethereare2,400portsintotalinthenetwork,noxwillhavetoprocess2,400packetout messages plus 2,400 PacketIn messages every 5 seconds just for LLDP packets alone. VI. PERFORMANCE EVALUATION WestudyhowquicklyaTCPflowcanchangeitspathtoanewpathafteralinkfailure.In Section VI-A, we first study the details of two cases when the LLDP transmission interval is theoriginalvalueof5seconds,withthefirstcasecarriedoutwhenarpisenabledandthe secondcasecarriedoutwhenarpisdisabled.then,insectionvi-b,westudytheeffectsof thelldptransmissionintervalonthenewpathfindingtimeofthetcpflowwhenarpis enabled. A. Detailed Case Studies OnthetestednetworkdepictedinFigure3,beforethelinkbetweennodes6and7breaksat 40 thsec,nox sstpdisablesthelinkbetweennodes9and10andthelinkbetweennodes10 and8.therefore,thetcpflowfromthesourcehosttothedestinationhosttraversesnodes3,6,

14 14 Fig.4:ThenewpathofaTCPtrafficflowafterthelinkbetweennodes6and7becomes down under the ARP-enabled condition 7,10,and11.Afterthelinkbreakage,NOX sstpenablesbackthesetwolinksbutdisablesthe linkbetweennodes7and8tomaintainalooplessandconnectedtopology.inthefollowing,we showthat(1)whenarpisenabled,thenewpathtakenbythetcpflowischangedtothepath traversingnodes3,6,9,10,and11at62 thsec,asshowninfigure4;however,thetcpflow neverchangesbacktoitsoriginalpathafterthelinkdowntime;and(2)whenarpisdisabled, thetcpflowneverchangesitspathtothenewpathduringthelinkdowntimebetween40 th secand65 thsecanditbecomesactiveovertheoriginalpathafterthelinkdowntimeat84 th sec.theseresultsarecausedbytheflowidletimersusedinopenflowswitches,nox slbp and STP implementations, and their interactions with ARP. Figure5(a)showsthetimelineoftheimportanteventsoftheTCPflowwhenARPisenabled on hosts. Since the link becomes down at 40 th sec, as discussed previously, because NOX s STP usesaseparate10-secondtimertodetectthefailureofeachlink,onewouldexpectthatthenew spanningtreewouldbeformedveryquicklyaround50+ thsecandthetcpflowwouldchange tothenewpathquicklyafterthenewspanningtreeisformed(i.e.,alsoaround50+ thsec). However,thisisnotthecaseandtheTCPflowactuallychangestothenewpathandbecomes

15 15 activeataround62 thsec(atthefevent). Fig.5:ThetimelineforaTCPflowtochange/keepitspathafterthelinkbetweennodes6and 7breaksat40 thsec TheeventsA,B,C,D,EandFrepresentthetimestampswhentheTCPflowcontinues toresendapacketlostonthebrokenlink.thetimestampofeventais40.3 thsecandthe intervals between two successive retransmission events starting from event A are 0.7, 1.38, 2.72, 5.44, and seconds, respectively. These exponentially-growing intervals are caused by the TCP congestion control algorithm on the source host. The F event represents the successful retransmission of the lost packet over the new path. After that, the TCP flow becomes active in transmitting packets over the new path. The X event represents the source host sending an ARP request while the Y event represents the destination host returning the ARP reply back tothesourcehost.thexeventistriggeredbythetcpflowretransmittingthelostpacketat theftimestamp.thisisbecauseonthesourcehostthearpentryforthedestinationhosthad expired during the long TCP retransmission interval and an ARP request must be broadcast to thenetworktorebuildtheentry.(note:anarpentrymayexpireafteranidleperiodofbetween

16 16 2 and 4 seconds in the simulations, depending on the relative timing between the installation of the entry and the 2-second periodic flush operations.) In the following, we explain why the TCP flow experiences unexpected delays before it successfully changes to the new path when ARP is enabled. The reason why the retransmission attemptsateventsa,b,c,dandefailisthesame.fromtheestinetlog,wefoundthatnox s STPdoesnotdetectthelinkfailurebetween40 thsecand50 thsecandthenewspanningtree isformedat52 thsec,whichisafterthetimestampofevente.therefore,alloftheseresent TCPpacketsareforwardedoverthebrokenlinkandgetlost.(NotethatNOXusesa5-second interval to periodically update the Flood/NonFlood status of the ports of a switch and our log showsthattheupdateoccursat52 thsec.)asfortheretransmissionateventf,itsucceedsand the reason is explained below. As discussed before, the broadcast ARP request at the timestamp ofxistriggeredbytheresenttcppacketateventf.becausetheflowentriesaddedinall switches for the previous ARP request/reply transmitted by the source and destination hosts had expired during the long TCP retransmission timeout, when an ARP request enters a switch, the switch will issue a PacketIn message to NOX asking for forwarding instructions. In return, NOX sends back a PacketOut message instructing the switch to flood the ARP request(which isabroadcastpacket)outofallofitsports.althoughtherightportofnode6connectstothe brokenlink,thebottomportconnectingtonode9isfunctioning.asaresult,thearprequest cantraversenodes6,9,10toreachthedestinationhostandthedestinationhostcansendback theunicastarpreplybacktothesourcehost,likethescenariodescribedinsectionv-a. TheeffectsoftheseARPrequestandreplypacketsnotonlyinstallARPflowentriesin switches but also let NOX learn the most updated flow forwarding information for each switch. Later,whentheresentTCPpacketentersnode6,becausethereisnoentryforthisTCPflow (notethatthereisanarpentryjustinstalled,however,becauseitstypeisnottcp,thisflow entrydoesnotmatchtheincomingtcppacket),node6issuesapacketinmessagetonox asking for instructions. For this TCP packet, NOX now returns a PacketOut message instructing theswitchtoaddanupdatedandcorrectflowentryforthistcpflowandforwardthetcp packetoutofthebottomportofnode6,whichiscorrect.afterthat,thistcppacketanditsack packet follow the scenario described in Section V-A to(1) update NOX of correct forwarding informationforthistcpflowonallswitchesand(2)installcorrectflowentriesforthistcp flow in all switches. With all correct entries installed in the switches on the new path, starting

17 17 fromthesecondtcppacketafterthetimeout,thetcpflowbecomesactiveonthenewpathat 62 th sec. IncontrasttothefactthattheTCPflowcanchangetoanewpathwhenARPisenabled, whenarpisdisabledthetcpflowneverchangestothenewpathbutinsteadbecomesactive againontheoriginalpathat84 thsec.inthefollowing,weexplainthereasons. WhenARPisdisabledonhosts,asshowninFigure5(b),oneseesthattheretransmission attempt at event P fails. This failure doubles the TCP retransmission interval from seconds to22.60secondsandcausesthetcpflowtoresendthelostpacketatthetimestampofpplus 22.60seconds,whichisat84 thsec(attheqtimestamp).aftereventq,becausethelink downtimehaspassedandnoxneverfoundanewpathforthetcpflowduringthelink downtime, the TCP flow still uses its old path to successfully transmit its packets. AsforthereasonwhywhenARPisdisabledtheretransmissionattemptateventPstillfails, wefoundthatitiscausedbyabugofnox sstp.foreachentryaddedtotheflowtableofa switch,theswitchsetsupanidletimerforitandwillremovetheentryifithasnotbeenused formorethan5seconds.asaresult,theflowentryforthetcpflowexpiresandisremoved atthetimestampofeplus5seconds(whichis56 thsec).fromtheestinetlog,weobserved thatnode6doessendaflowremovedmessagetonoxtoinformitoftheremovalofthis flowentryat56 thsec.laterat62 thsecwhenaresenttcppacketentersnode6,becausethe flowentryforthistcpflowhadexpiredandbeenremovedbefore,theswitchissuesapacketin message to NOX asking for its forwarding instruction. Surprisingly, we found that NOX issues back a PacketOut message instructing the switch toaddaflowentryforthisflowandforwardtheresentpacketoutoftheportconnectedto the broken link. That is, even though NOX has received the FlowRemoved notification from theswitchat56 thsec,itstillkeepstheoldandobsoleteforwardinginformationforthisflow and gives the wrong forwarding entry and instruction to the switch at 62 th sec. Since this resentpacketislostagainonthebrokenlink,thetcpflowhastoresendthepacketatthe next retransmission timeout, which occurs at 84 th sec. We note that the EstiNet log shows thatnox sstphasdetectedthebrokenlinkandre-formedanewspanningtreeat52 thsec. However, the above results show that detecting a link failure and accordingly disabling that link does not cause NOX to automatically remove all obsolete forwarding information related to that link. These results indicate that in NOX the STP and LBP components do not synchronize their

18 18 gathered information well and thus, as shown in this study, they may result in wrong operations ofanopenflownetwork.notethatthisbugdoesnothappenwhenarpisenabledandthe ARP request/reply transmissions happen before the transmission of the resent TCP packet, as shown in Figure 5(a). Since the ARP request/reply trigger PacketIn messages sent to NOX, we conjecture that these PacketIn messages replace the wrong forwarding information stored in NOX with correct information. AnotherimportantfindingfromFigure5(a)isthattheTCPflowdoesnotchangeitspath fromthenewpathbacktoitsoriginalpathafterthelinkdowntime,eventhoughthespanning treehasbeenrestoredtotheoriginalone.thisproblemiscausedbytheflowidletimersusedin theswitchesonthenewpath.becausethetcpflowisactiveinsendingpacketsafterchanging tothenewpath,theflowentriesforthetcpflowintheseswitches,whichwerecreatedover the spanning tree during the link downtime, will never expire. Since every incoming TCP packet willmatchtheseentriesandwillnotgenerateatablemissevent,theseswitcheswillnotsend any PacketIn messages to NOX. Therefore, NOX has no chances to install new entries into the flowtablesoftheseswitchestochangethepathofthetcpflowbacktoitsoriginalone,which maybebetterthanthenewpathintermsofhopcountoravailablebandwidth. B. Effects of the LLDP Interval To study the effects of the LLDP transmission interval and the link failure detection timeout value(whichistwotimesofthelldptransmissioninterval)onthenewpathfindingtimeof thetcpflowwhenarpisenabled,wevariedthevalueofthelldpintervalfrom1,2,3,...,to10secondsandobservedatwhattimethetcpflowcanswitchtothenewpath.(note: WhenARPisdisabled,wehaveshownthattheTCPflowwillneverchangetoanewpathdue to the implementation of NOX.) Figure 6 shows the results. After careful investigations into the causes, we found that this phenomenon is caused by the complicated interactions among several timersusedinanopenflowswitchandnox.whenthelldpintervalisreducedto1second, alinkfailurecanbedetectedquicklyafter2*1=2seconds.however,becausetheflowentry forthetcpflowinnode6isusedandmatchedoneachresenttcppacket,thisobsoleteflow entry(whoseoutputportstillpointstothebrokenlink)continuestoresideintheflowtableand beusedfortheresenttcppacketsateventa,b,c,anddinfigure5(a).asaresult,allof these retransmitted TCP packets are lost on the broken link.(note: This finding shows a design

19 19 flawinnox sstpasitshouldsendaflowmodifymessagetonode6todeletetheobsolete TCPflowentryassoonasitdetectsthelinkfailure.) 100 ARP enabled New path finding time (sec) LLDP transmission interval (sec) Fig.6:ThenewpathfindingtimeofaTCPflowunderdifferentLLDPtransmissionintervals On the next retransmission attempt at event E, because the TCP retransmission timeout between eventdandeis5.44seconds(whichislargerthanthe5-secondflowentryidletimeoutvalue), theflowentryforthistcpflowhadexpiredandatablemisseventwillbegenerated.however, becausethearpentryonthesourcehosthadexpiredaswellduringthislongtcpretransmission interval,atthetimestampofevente,anarprequestisinsteadgeneratedandentersintonode 6. Starting from this moment, the scenario that we described in Section VI-A when explaining eventsx,y,andfoccursagain.therefore,thetcpflowcannowswitchtothenewpathat 52 thsecwhenthelldpintervalisreducedto1second. AlloftheresultsshowninFigure6canbeexplainedaccuratelybasedontheOpenFlow protocolandtheimplementationsofarp,tcp,andnox sstpandlbp.duetothepaper length limitation, however, we can only pick one result to explain in details. Nevertheless, this study already shows the performance fidelity of EstiNet when used to evaluate a real OpenFlow controller.

20 20 VII. CONCLUSION In this article, we present the EstiNet OpenFlow network simulator and emulator and use it as a platform to perform functional validation and performance evaluation of the NOX OpenFlow controller. EstiNet uses an unique kernel reentering simulation methodology to combine the advantages of both the simulation approach and the emulation approach. By this methodology. a real OpenFlow controller can run without modification to control thousands of simulated OpenFlow switches. In addition, real applications can run without modification on simulated hosts that run the real Linux operating system to generate realistic network traffic. Our simulation study provides important insights into how NOX implements the functions of the learning bridge protocol(lbp) and the spanning tree protocol(stp) based on the OpenFlow 1.0 protocol. NOX implements these protocols as separate components that can be loaded into the core of NOX simultaneously. Our detailed logs reveal that the LBP and STP components in NOX do not synchronize their gathered information well and thus NOX may give wrong forwarding instructions to an OpenFlow switch after a link failure. Our another finding is that whennox sstpdetectsalinkfailure,itdoesnotsendamessagetoanaffectedswitchto delete obsolete flow entries. As a result, because the obsolete flow entry expires only after an idleperiodof5seconds,itmaybematchedandusedendlesslycausingtheopenflowswitch to continue to forward incoming packets onto a broken link. Insummary,ourresultsshowthattheLBPandSTPcomponentsprovidedinNOXonly implement basic functions and lack information synchronization. As revealed in this paper, there ismuchroomlefttofurtherimprovethem. REFERENCES [1] Software-Defined Networking: The New Norm for Networks, a white paper of Open Networking Foundation, April 13, [2] Nick Mckeown, Tom Anderson, Hari BalaKrishnan, Guru Parulkar, Larry Peterson, Jennifer Rexford, Scott Shenker, and Jonathan Turner, OpenFlow: Enabling Innovation in Campus Networks, ACM SIGCOMM Computer Communication Review, Volume 38 Issue 2, April [3] B.White,J.Lepreau,L.Stoller,R.Ricci,S.Guruprasad,M.Newbold,M.Hibler,C.Barb,andA.Joglekar. AnIntegrated Experimental Environment for Distributed Systems and Networks, In Proc. of the Fifth Symposium on Operating Systems Design and Implementation, pages , Boston, MA, Dec [4] B. Chun, D. Culler, T. Roscoe, A. Bravier, L, Peterson, M. Wawrzoniak, and M. Bowman, PlanetLab: an Overlay Testbed for Broad-Coverage Services, ACM SIGCOMM Computer Communication Review, Volume 33 Issue 3, July 2003.

21 21 [5] N. Foster, R. Harrison, M.J. Freedman, C. Monsanto, and J. Rexford, A. Story, and D. Walker, Frenetic: A Network Programming Language, in Proc. of ICFP [6] Macro Canini, Daniele Venzano, Peter Peresini, Dejan Kostic, and Jennifer Rexford, A NICE Way to Test OpenFlow Applications, in Proc. of Networked System Design and Implementation, April [7] EstiNet 8.0 OpenFlow Network Simulator and Emulator, EstiNet Technologies Inc., available at [8] Natasha Gude, Teemu Koponen, Justin Pettit, Ben Pfaff, Martn Casado, Nick McKeown, Scott Shenker, NOX: towards an Operating System for Networks, ACM SIGCOMM Computer Communication Review, Volume 38 Issue 3, July 2008 [9] T. R. Henderson, M. Lacage, and G. F. Riley, Network Simulations with the ns-3 Simulator, ACM SIGCOMM 08, August 17-22, 2008, Seattle, USA. [10] Bob Lantz, Brandon Heller, and Nick McKeown, A Network in a Laptop: Rapid Prototyping for Software-Defined Networks ACM Hotnets 2010, October 20-21, 2010, Monterey, CA, USA. [11] B. Pfaff, J. Pettit, T. Koponen, K. Amidon, M. Casado, and S. Shenker, Extending Networking into the Virtualization Layer, in Prof. of HOTNETS [12] Link Layer Discovery Protocol, IEEE 802.1AB standards.