ATYPICAL SDN consists of a logical controller in the

Transcription

1 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER Mnmzng Flow Statstcs Collecton Cost Usng Wldcard-Based Requests n SDNs Hongl Xu, Member, IEEE, Zhuolong Yu, Chen Qan, Member, IEEE, ACM, Xang-Yang L, Fellow, IEEE, Zchun Lu, and Lusheng Huang, Member, IEEE Abstract In a software-defned network (SDN), the control plane needs to frequently collect flow statstcs measured at the data plane swtches for dfferent applcatons, such as traffc engneerng, QoS routng, and attack detecton. However, exstng solutons for flow statstcs collecton may result n large bandwdth cost n the control channel and long processng delay on swtches, whch sgnfcantly nterfere wth the basc functons, such as packet forwardng and route update. To address ths challenge, we propose a cost-optmzed flow statstcs collecton (CO-FSC) scheme and a cost-optmzed partal flow statstcs collecton (CO-PFSC) scheme usng wldcardbased requests, and prove that both the CO-FSC and CO-PFSC problems are NP-hard. For CO-FSC, we present a roundngbased algorthm wth an approxmaton factor f, where f s the maxmum number of swtches vsted by each flow. For CO-PFSC, we present an approxmaton algorthm based on randomzed roundng for collectng statstcs nformaton of a part of flows n a network. Some practcal ssues are dscussed to enhance our algorthms, for example, the applcablty of our algorthms. Moreover, we extend CO-FSC to acheve the control lnk cost optmzaton FSC problem, and also desgn an algorthm wth an approxmaton factor f for ths problem. We mplement our desgned flow statstcs collecton algorthms on the open vrtual swtch-based SDN platform. The testng and extensve smulaton results show that the proposed algorthms can reduce the bandwdth overhead by over 39% and swtch processng delay by over 45% compared wth the exstng solutons. Index Terms Flow statstcs collecton, cost, delay, wldcard, roundng. Manuscrpt receved March 19, 2017; revsed June 5, 2017 and August 5, 2017; accepted August 29, 2017; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Edtor S. Uhlg. Date of publcaton September 21, 2017; date of current verson December 15, Ths work was supported n part by the NSFC under Grant , Grant U , and Grant , and n part by the Natural Scence Foundaton of Jangsu Provnce n Chna under Grant BK The work of C. Qan was supported by the NSF under Grant CNS The work of X.-Y. L was supported n part by the NSF under Grant ECCS , Grant CMMI , and Grant CNS , and n part by the Natonal Scence Foundaton of Chna under Grant Some prelmnary results of ths paper were publshed n the Proceedngs of IEEE INFOCOM 2017 [1]. (Correspondng author: Hongl Xu.) H. Xu, Z. Yu, Z. Lu, and L. Huang are wth the School of Computer Scence and Technology, Unversty of Scence and Technology of Chna, Hefe , Chna, and also wth the Suzhou Insttute for Advanced Study, Unversty of Scence and Technology of Chna, Suzhou , Chna (e-mal: xuhongl@ustc.edu.cn; yzl123@mal.ustc.edu.cn; lzc223@mal.ustc.edu.cn; lshuang@ustc.edu.cn). C. Qan s wth the Department of Computer Engneerng, Unversty of Calforna at Santa Cruz, Santa Cruz, CA USA (e-mal: cqan12@ucsc.edu). X.-Y. L s wth the School of Computer Scence and Technology, Unversty of Scence and Technology of Chna, Hefe , Chna, and also wth the Department of Computer Scence, Illnos Insttute of Technology, Chcago, IL USA (e-mal: xangyang.l@gmal.com). Dgtal Object Identfer /TNET I. INTRODUCTION ATYPICAL SDN conssts of a logcal controller n the control plane and a set of swtches n the data plane [2]. The controller montors the network and determnes the forwardng path of each flow. The swtches perform packet forwardng and traffc measurement for flows based on the rules nstalled by the controller. Snce the controller s able to provde centralzed route/management control for each flow, an SDN can help to mplement fne-graned management and mprove the network resource utlzaton compared wth tradtonal networks [3]. To explore full advantages of centralzed control, an accurate global vew of flow traffc s nstrumental to varous applcatons, such as traffc engneerng, QoS routng, network attack detecton, and network management. For example, f the controller performs flow routng wthout accurate flow traffc knowledge, t wll often result n lower throughput or load mbalance. Wth the help of an accurate global vew of flow traffc, t can mprove the route QoS [3] [5], such as maxmum throughput, low latency, and hgh relablty. As another applcaton example, some securty attacks, e.g., DDoS [6], [7], are often detected by analyzng the changes of flow traffc. Accurate flow statstcs also help to detect the attacks and protect the network. Therefore, t s of vtal mportance to collect accurate flow statstcs. In an SDN, swtches are able to measure dfferent per-flow traffc statstcs, ncludng packets, bytes or duraton, through flow entres. To allow the controller to obtan traffc statstcs nformaton, OpenFlow [8] specfes two dfferent approaches for flow statstcs collecton (FSC) from the swtches. 1 One s the push-based mechansm. The controller learns actve flows and ther statstcs by passvely recevng reports from the swtches. However, several factors lmt ts applcaton n practce. Frst, the exstng push-based mechansm does not nform the controller about the behavor of a flow before the entry tmes out, as a result, push-based statstcs are not currently useful for flow schedulng. Second, to acheve effcent flow schedulng, DevoFlow [13] needs some addtonal requrements on both hardware and software for the pushbased FSC, such as counters and comparators to support report trggers. But most commodty swtches do not support the statstcs report trggers. Thrd, even though the push-based 1 Note that FSC s dfferent from the flow traffc measurement problem n SDNs [9] [12], whch studes how swtches derve flow statstcs. FSC focuses on how swtches report the collected flow statstcs to the controller IEEE. Personal use s permtted, but republcaton/redstrbuton requres IEEE permsson. See for more nformaton.

2 3588 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER 2017 collecton s feasble on SDN swtches, when the traffc vares dynamcally, the swtches wll be frequently trggered and send massve number of measurement reports to the controller, causng large cost of control channel bandwdth [14] and the controller s CPU resource [15]. The other s the pull-based method: the controller just sends a Read-State message (also called FSC request) to retreve the flow statstcs from a swtch. Snce ths mechansm needs no addtonal requrement on both hardware and software for swtches and can control the number of measurement reports, t has been wdely used n SDN applcatons [15] [17]. In ths work, we focus on the pull-based FSC method. Due to flow dynamcs, tmely collecton of flow statstcs n an SDN s requred for varous applcatons such as traffc engneerng [18]. There are two man schemes of pull-based FSC, per-flow collecton [16], [19] and per-swtch collecton [15], [17]. However, we demonstrate that both two schemes may lead to massve cost of the control channel bandwdth and long processng delay on the swtches, especally n dynamc networks. Ths cost of the control channel and swtches wll sgnfcantly nterfere wth basc functons such as packet forwardng and route update [20]. For per-flow collecton, the controller sends a request to a swtch for collectng the traffc statstc of exactly one flow. In some stuatons, the wldcard flow entres may be nstalled, whch wll be dscussed n Secton III-C. When the controller requres to collect statstcs of many flows, t may generate a large number of FSC requests on each swtch. These requests wll compete wth control messages, e.g., rule setup and update, for the downlnk bandwdth of the control channel. 2 Hence wth more FSC requests, the rule setup and update messages, servng the fundamental SDN tasks, may be delayed, lost, or dsordered [13], resultng n data plane faults or nconsstency. For per-swtch collecton, the controller sends a request to collect the traffc statstcs of all flow entres from a swtch. Ths scheme may lead to unacceptable collecton delay. For example, from the experments on the HP ProCurve 5406zl swtch, t takes about one second to collect traffc statstcs of about 5600 flows, even when there s no traffc load on the swtch [13]. Ordnary packet forwardng experences sgnfcantly lower throughput durng ths perod. Moreover, one second s stll too long for many flow schedulers such as Hedera [18] to conduct accurate routng optmzaton. For the cheap and less capable network equpment, t may lead to uplnk congeston of the control channel because many flow statstcs that have no need to collect wll also be frequently reported to the controller. Consder the case that only a small part of flows have vared transmsson rate, frequent FSC of all flows s obvously a sgnfcant waste of uplnk bandwdth. For example, t takes 96 bytes for statstcs of each flow entry specfed by OpenFlow 1.3 [8]. Then the statstcs for 16K exact-matched rules on a 5406zl swtch would cost 1.54MB. Settng the FSC rate as twce per second would requre 24.6Mbps bandwdth on a control lnk for only one swtch. 2 We use downlnk to denote the control channel from the controller to swtches and uplnk to denote that from swtches to the controller. The massve traffc on control lnks may ncrease the delay and loss rato of control commands. Therefore, t s an urgent need to desgn a new soluton of FSC wth lower control channel and swtch cost, sothatbasc functons on swtches wll be less nterfered. Our soluton s motvated by the followng consderatons. To avod long-delay collecton on some swtches and massve traffc load on control lnks, we expect that on each swtch only the traffc statstcs of a subset of flows (not all flows n the per-swtch collecton) wll be collected. We mplement the fast and selectve FSC on a swtch usng the wldcard-based FSC requests, whch can be successfully mplemented usng OpenFlow 1.3. The controller wll dstrbute FSC requests, each of whch contans one wldcard rule, to swtches. On the swtch only flows matchng the wldcard rule wll be collected and reported to the controller. We desgn algorthms to mnmze the maxmum bandwdth/delay cost among all swtches and extend our solutons to partal FSC, where only a subset of flows (not all flows) n the network are collected. The man contrbutons of ths paper are: 1) We propose the cost-optmzed flow statstcs collecton (CO-FSC) and cost-optmzed partal flow statstcs collecton (CO-PFSC) problems, and prove the NP-hardness. 2) For CO-FSC, we present a roundng-based algorthm, called R-FSC. The R-FSC algorthm acheves the approxmaton factor of f, where f s the maxmum number of swtches vsted by each flow n a network. Moreover, a prmal-dual-based algorthm wth lower tme complexty s also presented. 3) For CO-PFSC, we desgn a roundng-based algorthm for ths problem, and the approxmaton factor of the proposed algorthm s also analyzed. 4) To study the control lnk cost for flow statstcs collecton, we extend the above swtch cost optmzaton problems, and defne the lnk cost-optmzed flow statstcs collecton (LCO-FSC) problem, and present two approxmaton algorthms for the LCO-FSC problem. 5) We mplement the proposed flow statstcs collecton methods on an SDN platform usng Open vswtch [21]. The testng and smulaton results show that our algorthms help to reduce the bandwdth overhead by over 39% and swtch processng delay by over 45% compared wth the exstng FSC solutons. Moreover, our partal FSC algorthm reduces the cost by 52% compared wth FSC whle preservng almost the smlar applcaton performance. II. PRELIMINARIES In ths secton, we frst ntroduce the network and flow models n an SDN. Then, we defne the cost-optmzed flow statstcs collecton (CO-FSC) and cost-optmzed partal flow statstcs collecton (CO-PFSC) problems, respectvely. A. Network and Flow Models An SDN typcally conssts of a logcally-centralzed controller and a set of swtches, V = {v 1,...,v n }, wth n = V.

3 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3589 Fg. 1. Illustraton of a standard flow entry n a flow table. These swtches comprse the data plane of an SDN. Thus, the network topology from a vew of the date plane can be modeled by G = (V,E), where E s the set of lnks connectng swtches. Besdes the data plane lnks, there s a set of lnks servng the control channel connectng the swtches and the controller. Note that the controller may be a cluster of dstrbuted controllers [22], [23], whch help to balance the overhead among these controllers. Snce the metrc we frst evaluate s per-swtch bandwdt/delay cost, we assume that there s only one controller for smplcty of presentaton. It should be noted that we wll extend the per-swtch cost optmzaton to the lnk cost optmzaton n Secton IV. Each swtch has a flow table that performs packet forwardng and traffc measurement. A flow table conssts of a certan number of flow entres (also called rules). A standard flow entry, specfed by OpenFlow 1.3 [8], s llustrated n Fg. 1. The match felds and prorty together dentfy a unque entry n the flow table, and the swtch measures traffc n the counters feld. When a packet arrves at a swtch, the header packet wll be examned. If there s at lease one flow entry that matches the packet, ths swtch pcks the entry wth the hghest prorty and performs the acton specfed by the nstructon feld of the entry. Otherwse, the swtch reports the header packet to the controller, whch shall determne the forwardng path for ths flow, and setup a sequence of rules to the swtches on the path. B. Advantages of FSC Usng Wldcard Requests Comparng wth the prevous pull-based FSC methods, wldcard-based FSC has two man advantages. Frst, usng wldcard requests, t s feasble to collect statstcs of a subset of flows from a swtch. Thus, the wldcard soluton s able to dstrbute the statstcs collecton overhead of all flows among all swtches, whch helps to reduce the swtch cost compared wth the per-swtch method. Second, usng wldcard, the controller can collect statstcs of many flows, not just one flow, per request. Thus, the wldcard method can sgnfcantly reduce the number of requests and swtch overhead compared wth per-flow collecton method. C. Cost-Optmzed Flow Statstcs Collecton (CO-FSC) Under the general SDN framework, swtches report the header packet of each new-arrval flow to the controller. Thus, t s reasonable to assume that the controller knows the exstng flows n a network, denoted by Γ={γ 1,...,γ m }, wth m = Γ. Snce the forwardng path for each flow s determned by the controller, we also know the flow set, denoted by Γ, that passes through each swtch v.foraflow, f ts traffc statstc s gathered, the controller knows ts actual number of packets (or traffc ntensty). We say that ths flow s covered. As the traffc statstc of some flow does not change, ths flow has termnated, and the controller wll delete the correspondng entry of ths flow, and update the current flow set Γ. For smplcty, we assume that each swtch s drectly connected to a controller. We wll extend ths assumpton to a more general scenaro, n whch some swtches may not be drectly connected wth the controller, n Secton IV. Assume that there s a set of wldcards, denoted by R = {r 1,r 2,...r q }, wth q = R. For example, a natural way for settng wldcards s as follows: each wldcard r j only specfes the destnaton v j, and can match all the sources. When the controller sends a Read-State command wth wldcard r j to swtch v, or we say that wldcard r j s appled on swtch v, the swtch assembles the flow entres matchng wth ths wldcard nto a reply packet, and sends to the controller. Under ths case, assume that the covered flow set s denoted by Γ j. From ths example, the wldcard rules often satsfy the followng two features: (1) The completeness feature, that s, r Γj =Γ, v V. (2) The mutual excluson feature,.e., Γ j1 Γ j2 =Φ, j 1 j 2, v V. When wldcard r j s appled on swtch v, a flow set Γ j wll be covered/collected, and the cost on swtch v s denoted by c(γ j ). The cost functon c(γj ) s usually defned as c 1 Γ j + c 2,wherec 1 and c 2 are constant and determned by dfferent performance metrcs, such as bandwdth or delay costs. We consder the bandwdth cost as the total bandwdth for FSC of a flow set Γ j. As specfed n Openflow v1.3 [8], the bandwdth cost for statstcs collecton of a flow set Γ j conssts of two parts: (1) the request packet, whose length s 114 bytes; (2) the reply packet, whose length depends on the number of covered flows n Γ j. It s expressed as l h + l f Γ j,wherel h s the length of packet header, and l f s the length for each flow entry, respectvely. Accordng to [8], l h and l f are 74 bytes and 96 bytes, respectvely. More specfcally, the length of packet header ncludes 16 bytes (multpart request header) and 58 bytes (Ethernet+IP+TCP headers). The length for each flow entry ncludes 40 bytes (match felds for each entry) and 56 bytes (statstcs nformaton for each entry). These parameters have been valdated through our open vrtual swtch (OVS) platform. Thus, the bandwdth cost s modeled as c(γ j )=96 Γj wth unt byte. We consder the cost c(γ j ) as the delay for statstcs collecton of a flow set Γ j. By testng on the HP swtch [13], wth the ncreasng number of flows, the delay for flow statstcs collecton s almost lnearly ncreasng, and the ncrease rate s about =0.18ms/flow. When we test on our H3C swtch, t takes about 1.4ms and 21ms to collect statstcs of one flow and 100 flows usng the per-flow and per-swtch collecton nterfaces, respectvely. The ncrease rate s about 0.198ms/flow. Combnng the above testng results, the delay cost for Γ j can be approxmately modeled as c(γ j ) = 0.19 Γj wth unt ms. It should be noted that, the values of two constant parameters for delay cost may vary wth swtch traffc load [13], whch wll be dscussed n Secton III-C. When an FSC event s trggered, the controller wll send Read-State commands, each of whch contans a wldcard rule, to dfferent swtches, so that all the flows can be covered. The controller may send several collecton commands to one swtch per FSC event. As a result, the cost on swtch v s denoted

4 3590 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER 2017 by c(v ). Our objectve s to mnmze the maxmum cost among all the swtches, that s, mn max{c(v ), 1 n}, so that no performance bottleneck wll happen n SDNs. Accordngly we formalze the CO-FSC problem as follows: mn λ x j γ k Γ j 1, γ k Γ S.t. c(v )= r xj c(γj ) λ, v V (1) {0, 1}, r j R x j where x j denotes whether the controller wll send a Read- State command wth wldcard r j to swtch v or not. The frst set of nequaltes denotes that each flow wll be covered. The second set of nequaltes means that the total cost on each swtch should not exceed λ. The objectve s to mnmze the maxmum cost on all the swtches (or to acheve the cost balancng on swtches), that s, mn λ. Theorem 1: The CO-FSC problem s NP-hard. The proof of theorem 1 has been relegated to the appendx VII. D. Cost-Optmzed Partal Flow Statstcs Collecton The CO-FSC problem wll collect statstcs nformaton of all flows n a network. In fact, the statstcs nformaton of partal flows s also helpful for some applcatons. For example, the controller can know the traffc of each flow through drect measurement, e.g., by collectng statstcs of all flows n a network. Another way for traffc estmaton s to combne the drect measurement (e.g., statstcs collecton) and nference [24]. The controller can nfer the traffc of all flows through statstcs nformaton of partal flows (not all flows) and lnk load n a network. Thus, statstcs collecton of partal flows also benefts for buldng a global traffc vew, wth less cost on swtches compared wth statstcs collecton of all flows. In ths secton, we present the partal FSC problem, n whch the flow recall rato s at least a gven value β (0, 1]. The CO-PFSC(β) problem s defned as follows. Smlar to CO-FSC, let Γ, V,andR denote a flow set, a swtch set and a wldcard set n an SDN, respectvely. When we apply the wldcard r j Ron swtch v, the controller can obtan the traffc statstcs of a flow set Γ j, and ts cost s c(γj ), whch s defned n Secton II-C. The controller sends Read- State commands wth wldcards to dfferent swtches, so that at least β m flows wll be covered, where β s the flow recall rato requrement, and m s the number of flows n an SDN. The cost on each swtch v s denoted by c(v ),andweam to mnmze the maxmum cost of all the swtches. We gve the formulaton of CO-PFSC(β) as follows. mn λ z k x j γ k Γ j, γ k Γ z k β m, γ k Γ S.t. c(v )= r xj c(γj ) λ, v V (2) x j {0, 1}, r z k {0, 1}, j R γ k Γ where z k denotes whether the statstcs nformaton of flow γ k s collected (z k =1) or not. The frst set of nequaltes means that, one flow wll be covered, f at least one set Γ j contanng ths flow s collected by the controller. The second set of constrants means that the number of covered flows exceeds β m. The thrd set of nequaltes means that the cost on each swtch should not exceed λ. The objectve s to mnmze the maxmum cost on all the swtches, that s, mn λ. Note that CO-FSC s a specal case of the CO-PFSC(β) problem, wth β =1. By theorem 1, t follows Theorem 2: The CO-PFSC(β) problem s NP-hard. III. ALGORITHMS FOR CO-FSC AND CO-PFSC A. Algorthm Desgn for CO-FSC Due to the hardness of CO-FSC, we frst desgn an approxmaton algorthm usng the roundng method for ths problem (Secton III-A.1), and gve performance analyss (Secton III-A.2). Then, we present an algorthm based on prmal-dual wth lower tme complexty (Secton III-A.3). 1) A Roundng-Based Algorthm for CO-FSC: Ths secton develops a roundng-based algorthm, called R-FSC, to solve the CO-FSC problem. The algorthm conssts of two man steps. As CO-FSC s an NP-Hard problem, the frst step wll relax the nteger program formulaton to a lnear program as Eq. (3). mn λ x j γ k Γ j 1, S.t. c(v )= r xj c(γj ) λ, x j 0, γ k Γ v V r j R We can solve Eq. (3) n polynomal tme, and obtan the fractonal soluton, denoted by x. In the second step, the fractonal soluton { x} wll be rounded to 0-1 soluton { x} for flow statstcs collecton. The set of uncovered flows s denoted by Γ u, whch s ntalzed as all flows n Γ. We arbtrarly choose an uncovered flow, denoted by γ, from set Γ u. Then, the algorthm chooses a flow set Γ j, whose xj s maxmum among all these sets contanng flow γ,andset x j =1.Thats, the controller wll send a Read-State command wth wldcard r j to swtch v. Moreover, we update Γ u =Γ u Γ j.the algorthm wll termnate untl all flows are covered. The R- FSC algorthm s descrbed n Alg. 1. 2) Performance Analyss: We analyze the approxmate performance of the proposed algorthm. In the second step, (3) we arbtrarly choose an uncovered flow γ. LetΓ γ denote all the sets that contan ths flow. By the second feature of wldcard rules, Γ γ f, wheref s the maxmum number of swtches vsted by each flow. Snce each flow γ wll be covered, Γ j x j Γγ 1. Assume that a flow set Γj s chosen n some teraton, for x j s maxmum among all these sets contanng flow γ. It follows x j 1 f or f xj 1. After solvng the lnear program n the frst step of the R-FSC algorthm, we derve a fractonal soluton x and an optmal result λ for the relaxed CO-FSC problem. Accordng

5 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3591 Algorthm 1 R-FSC: Roundng-Based FSC 1: Step 1: Solvng the Relaxed CO-FSC Problem 2: Construct the relaxed problem as Eq. (3) 3: Obtan a fracton soluton x 4: Step 2: Roundng to 0-1 Soluton 5: Γ u =Γ 6: whle Γ u Φdo 7: Arbtrarly choose an uncovered flow γ 8: Choose a flow set Γ j, whose xj s maxmum among all these sets contanng flow γ, andset x j =1 9: Γ u =Γ u Γ j to the algorthm descrpton, the fnal cost of swtch v s: c(v )= r xj c(γj ) r f xj c(γj ) f λ (4) Thus, we conclude that Theorem 3: The R-FSC algorthm can acheve the f-approxmaton for the CO-FSC problem. 3) A Lower-Complexty Algorthm Usng Prmal-Dual: When a flow statstcs collecton event s trggered, we expect that the controller can mmedately determne the soluton for FSC. t s an mportant step to solve the lnear program Eq. (3) n the R-FSC algorthm. As the number of varables n Eq. (3) manly depends on the number of flows n an SDN, t may contan a large number of varables for a large-scale network, and t s rather costly n practce to solve such a lnear program. Thus, ths secton presents a lower-complexty algorthm usng prmal-dual for CO-FSC. The prmal-dual verson of Eq. (3) s gven n Eq (5), n whch two sets of varables χ and μ denote the frst and second sets of constrants n Eq. (3). max m χ k k=1 χ γ k Γ j k μ c(γ j ) 0,, j S.t. n μ 1, (5) =1 0 μ 1, We desgn the FSC-PD algorthm for the CO-FSC problem. Accordng to Eq. (5), we should ncrease the values of varables χ wth constrants so as to maxmze the objectve functon. For each flow γ k, there s a correspondng varable χ k. At the begnnng, the algorthm ntalzes a varable, Γ u, whch denotes the uncovered flow set. For the ease of algorthm desgn, we expect that the varable χ k for each uncovered flow wll almost grow n lne to a value, denoted by δ, whch wll be updated n algorthm executon. By the frst set of nequaltes of Eq. (5), we have χ k = χ γ k Γ r γ k Γ j k μ c(γ j r ) (6) For smplcty, let tc(v ) be r c(γj ). As we consder the crtcal case, t follows that δ μ tc(v) n,wheren s the number of flows through swtch v. Combnng wth the second nequalty of Eq. (5), we set each varable μ as μ = n tc(v ) v V n tc(v ) For ease descrpton, f there s an uncovered flow n set Γ j, ths set s accordngly uncovered. The algorthm manly comprses a group of teratons. In each teraton, for each uncovered flow set Γ j, the algorthm ncreases χ k for each uncovered flow γ k to a value, denoted by δ j,sothat γ k Γ j χ k = μ c(γ j ). Under ths stuaton, the varable χ k for the covered flow γ k remans unchanged. We then choose a flow set Γ j wth the mnmum value δ j among all the uncovered flow sets. Accordng to the Prmal-Dual s property, the correspondng varable x j n Eq. (1) s set to 1, whch means that Γj wll be chosen for flow statstcs collecton, or wldcard r j wll be appled on swtch v. We update the varable χ k for each uncovered flow γ k Γ j Γu as χ k = δ j. If all the flows are covered, the algorthm termnates. Otherwse, we contnue a new teraton. The FSC-PD algorthm s descrbed n Alg. 2. Algorthm 2 FSC-PD: FSC Based on Prmal-Dual 1: Γ u =Γ 2: for each swtch v V do 3: Compute μ by Eq. (7) 4: whle Γ u > 0 do 5: for each uncovered flow set Γ j do 6: Increase χ k for each uncovered flow γ k to value δ j, so that γ k Γ j χ k = μ c(γ j ) 7: Choose an uncovered flow set Γ j wth the mnmum value δ j among all the uncovered flow sets 8: for each uncovered flow γ k Γ j Γu do 9: χ k = δ j 10: Γ u =Γ u Γ j Theorem 4: The tme complexty of FSC-PD s O(mnqf), where m, n, q and f are the number of flows, the number of swtches, the number of wldcard rules, and the maxmum number of swtches vsted by each flow n a network, respectvely. Proof: The FSC-PD algorthm frst takes O(mf) to ntalze each varable μ, for all flows on each swtch wll be processed. Snce there are n swtches and q wldcard rules, the algorthm conssts of O(nq) teratons at most. In each teraton, the algorthm computes the value δ j for each uncovered flow set Γ j, whch takes a tme complexty of O(mf). Moreover, the algorthm wll cost a tme complexty of O(m) to update varable Γ u. Then, the total tme complexty of the FSC-PD algorthm s O(mnqf). B. Algorthm Desgn for CO-PFSC Ths secton studes a more general case, called partal flow statstcs collecton. We desgn an approxmaton algorthm usng the roundng method for the CO-PFSC problem (Secton III-B.1), and gve approxmaton performance analyss (Secton III-B.2). (7)

6 3592 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER ) An Approxmaton Algorthm for CO-PFSC(β): We descrbe a roundng-based algorthm, called R-PFSC, for partal flow statstcs collecton. Due to NP-Hardness, the algorthm constructs a lnear program as a relaxaton of CO- PFSC(β). We formulate the followng lnear program LP 2. mn λ z k x j γ k Γ j, γ k Γ z S.t. γ k β m, k Γ r xj c(γj ) λ, v V x j 0, v V, r j R 0 z k 1, γ k Γ By solvng the lnear program LP 2, we assume that the optmal soluton for LP 2 s denoted by x, and the optmal result s denoted by λ. AsLP 2 s a relaxaton of the CO-PFSC(β) problem, λ s a lower-bound result for CO-PFSC(β). In the second step, the controller wll determne whch wldcard rules wll be sent to each swtch for partal flow statstcs collecton. By solvng LP 2,varable z k denotes the probablty that the statstcs nformaton of flow γ k wll be collected. Then, we adapt the second step of R-FSC to determne the flow statstcs collecton n our algorthm. The set of covered flows s denoted by Γ c, whch s ntalzed as Φ. We choose an uncovered flow, denoted by γ k, wth maxmum z k.if z k s less than β, the algorthm termnates. Otherwse, the algorthm chooses a flow set Γ j, whose xj s maxmum among all these sets contanng flow γ k,andsets x j =1. Ths means that the controller wll send the Read- State command wth wldcard r j to swtch v. The algorthm wll termnate untl there have β m covered flows n set Γ c. The R-PFSC algorthm s descrbed n Alg. 3. Algorthm 3 R-PFSC: Roundng-Based Partal FSC 1: Step 1: Solvng the Relaxed PFSC Problem 2: Construct a lnear program n Eq. (8) 3: Obtan the optmal soluton x and z 4: Step 2: Determnng Flow Statstcs Collecton 5: Γ c =Φ; z =1 6: whle Γ c <β m and z β do 7: Choose an uncovered flow γ k, wth maxmum z 8: z = z k 9: f z k β then 10: Choose a set Γ j, whose xj s maxmum among all these sets contanng the flow γ k,andset x j =1 11: Γ c =Γ c +Γ j 2) Performance Analyss: We frst gve a famous lemma for performance analyss. Lemma 5 (Chernoff Bound): Gven n ndependent varables: x 1,x 2,...,x n,where x [0, 1].Letμ = E[ n ] =1 x ]. [ n Then, Pr x (1 ɛ)μ =1 (8) e ɛ2 μ 2,whereɛs an arbtrarly postve value. Number of Covered Flows Constrants: The R-PFSC algorthm may not fully guarantee that the number of covered flows exceeds β m. Under ths case, one flow γ k, wth z k β, should be covered. By the second constrant n Eq. (8), γ k Γ z k β m, so we expect that more flows (e.g., β m) wll be collected. We wll further observe the rato of covered flows through extensve smulatons n Secton V-C. In the followng, we analyze the expected number of covered flows for a specal case, where z k s a random varable wth unform dstrbuton from 0 to 1. Let ϕ denote the event of solvng Eq. (8). Varable ẑ k denotes whether flow γ k s covered or not. Each flow γ k wll be covered wth the probablty of θ k = Pr(z k β ϕ). We compute E(z k ϕ) as follows: E(z k ϕ) = E(z k β) Pr(z k β ϕ)+e(z k <β) Pr(z k <β ϕ) = 1+β θ k + β 2 2 (1 θ k)= β + θ k (9) 2 Combnng the second nequalty of Eq. (8), t follows γ k Γ E(z k ϕ) = β+θ k γ k Γ 2 β m. Thus, γ k Γ θ k β m. Then, the expected number of covered flows s: E[ γ k Γ ẑk] = γ k Γ E[ẑ k]= γ k Γ Pr(z k β ϕ) = γ k Γ θ k β m (10) Accordng to the Chernoff bound n Lemma 5, we have [ ] Pr γ k Γ ẑk (1 ρ) βm e ρ2 β m 2 (11) where ρ s an arbtrarly constant wth 0 ρ<1. Wemake the followng assumpton: [ ] Pr γ k Γ ẑk (1 ρ) βm e ρ2 β m 2 1 n 2 (12) As a result, we obtan ρ 2 log n/βm (13) We have the followng lemma Theorem 6: The R-PFSC algorthm wll cover (1 ρ)βm flows at least for statstcs collecton, wth ρ 2 log n/βm under some specal stuatons. By our analyss, the number of covered flows wll hardly be volated by a factor of 1 2 log n/βm. For example, let n, m and β be 10 3, 10 5 and 0.5, respectvely. Obvously, log n =10,andwesetρ =0.09. In other words, our R-PFSC algorthm wll collect statstcs nformaton of at least 0.91 βm flows. Bandwdth/Delay Cost Performance: After the frst step of the R-PFSC algorthm, we derve a fractonal soluton x and an optmal result λ for the relaxed CO-PFSC(β) problem. In each teraton of the second step, assume that the selected uncovered flow s denoted by γ k. It follows that z k β. If one flow set Γ j s chosen, we know that xj β f. The cost of swtch v s: c(v )= r xj c(γj ) f r β xj c(γj ) f β λ (14) Thus, t follows Theorem 7: The cost on each swtch wll not exceed f β tmes of the fractonal soluton by the R-PFSC algorthm.

7 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3593 C. Dscusson In ths secton, we gve some dscussons, ncludng the mpact of swtch CPU utlzaton on the collecton cost, the case study of the FSC, and the applcablty of our algorthms. The mpact of swtch CPU utlzaton. As specfed n the OpenFlow standard, each SDN swtch has a software mplemented OpenFlow agent (OFA) that communcates wth the controller over a secure TCP connecton. After an FSC request s sent from the controller to the swtch, the OFA encapsulates the counter feld combned wth other felds (e.g., match felds) of all matched flow entres nto a reply packet, and sends back the reply packet to the controller. The OFA performance manly depends on the swtch CPU capacty. Due to the lmted CPU capacty on the swtch, the processng delay for flow statstcs collecton wll be ncreased wth CPU utlzaton [13]. That s, the delay cost of FSC depends on the current swtch CPU state. To deal wth ths case, each swtch v s assgned wth a weght ω, whch denotes the cost growth coeffcent wth CPU utlzaton. Let p(v ) denote the current CPU utlzaton. By applyng the Lttle s law [22], 1 the weght can be approxmately set as ω = 1 p(v. ) Dfferent from CO-FSC, the objectve of the weghted CO-FSC problem s to mnmze the maxmum weghted cost of all swtches, that s, mn max{ω c(v ), v V }. Smlar to Eq. (1), we can formalze ths problem as follows: mn λ x j γ k Γ j 1, S.t. c(v )= r xj ω c(γ j ) λ, {0, 1}, r x j γ k Γ v V j R (15) From Eq. (15), ths s the weghted verson of the CO- FSC problem. We can modfy the R-FSC and FSC-PD algorthms a lttle for the weghted verson. It s smlar to consder the mpact of the swtch CPU utlzaton on the swtch cost for the CO-PFSC problem. The case study of the FSC. Many works [13], have mentoned the fact that a small percentage of large (or elephant) flows typcally accounts for a large percentage of total traffc, and plays an mportant role for route performance. After collectng the flow statstcs nformaton from swtches, the controller determnes a set of elephant flows, denoted by Γ e, and ranks all these elephant flows n the decreasng order of traffc ntensty. For each flow γ Γ e, the controller fnds the least congested path between the flow s endponts as ts new route. After determnng new routes for all the elephant flows n Γ e, we execute the route reconfguraton [20], [25], whch can reduce the route reconfguraton delay compared wth that for all flows n a network. Note that our proposed flow statstcs collecton method can also be combned wth other routng methods n an SDN. The applcablty of our algorthms. Ths paper assumes that all the flows n the network are known. We note that our proposed algorthms can also be appled n dfferent stuatons. 1) For the proactve manner, the controller pre-nstalls rules for flows so as to mprove the network scalablty. One may say that though the rules for some flows are pre-nstalled, there may be no traffc for these flows. That s, the controller may not exactly know all the flows runnng through the network. To deal wth ths case, we try to collect the statstcs nformaton of all flows wth nstalled rules for the followng two reasons. On one hand, when the controller nstalls a rule wth proactve manner, the controller can not judge whether ths flow s runnng or not. To guarantee the completeness of FSC, we regard that all the flows may be runnng. On the other hand, as descrbed n B4 [26], the controller wll perodcally update the flow tables accordng to the traffc amount n the last perod. Thus, t s reasonable to regard that most flows wth pre-nstalled rules are actve n ths perod. 2) In some envronments, the wldcard rules, whch match IP prefxes or arbtrary header ranges, may be nstalled n a flow table for hgh scalablty. When a wldcard rule s nstalled on a flow table, the statstcs of all flows matchng ths wldcard rule wll be aggregated, and we can not dstngush the statstcs nformaton of each ndvdual flow. In ths stuaton, all flows matchng a wldcard rule can be regarded as a flow, also called macroflow [27]. In fact, our flow statstcs collecton s desgned based on nstalled rules n the flow tables. That s, we collect the values of the counter feld n flow entres. Thus, when the wldcard rules are appled, we can not collect the statstcs nformaton of ndvdual flows, but the aggregated flows specfed by the wldcard rules. The applcablty for multple controllers. In a largescale SDN wth more swtches, t s an effcent way to deploy multple controllers for the sngle-controller congeston avodance. However, the dstrbuted processng on dfferent controllers may lead to nconsstency of the flow statstcs nformaton [28]. Though some applcatons, such as traffc engneerng, may not requre hghly consstent statstcs, but most of FCAPS management applcatons [29] requre consstency of managed nformaton. In the future, we wll study the effcent mechansm for consstency mantenance of flow statstcs nformaton. For wldcard-based flow statstcs collecton, when the controller sends a wldcard rule command, the swtch wll compare wth the flow entres wth ths wldcard rule, encapsulate the statstcs nformaton of all matched flow entres nto a reply packet, and send back to the controller. Compared wth the per-flow and perswtch FSC method, the wldcard-based method needs to match all the entres n the whole flow table. IV. LINK COST-OPTIMIZED FSC There are two knds of connecton schemes between the controller and swtches, out-band and n-band, respectvely. The above sectons focus on the swtch cost optmzaton for

8 3594 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER 2017 Fg. 2. Illustraton of dfferent swtch-controller connecton schemes. For example, swtch v 1 s drectly connected wth the controller. But, swtch v 2 s connected wth the controller through a control path v 2 v 1 controller. flow statstcs collecton n an SDN,.e., for the out-band scheme. In ths secton, we consder the n-band connecton scheme, whch has also been appled n many applcatons. For example, to provde dynamc controller provsonng n an SDN, many works [22] adopt the n-band controller connecton, n whch the control messages are transferred through a data plane path. Under ths connecton scheme, snce flow statstcs nformaton wll be forwarded to the controller through some lnks, ths secton wll extend the CO-FSC problem to acheve the (control) lnk cost optmzaton. A. Lnk Cost-Optmzed Flow Statstcs Collecton (LCO-FSC) Problem In ths secton, we defne the lnk cost-optmzed flow statstcs collecton (LCO-FSC) problem, whch s also the extended verson of the CO-FSC problem. In an SDN, we assume that there s a cluster of controllers n the control plane. To better control/manage all the swtches, each swtch wll connect/assocate wth a controller through TCP long-term connecton. Physcally, there are two cases for the swtchcontroller connecton. One s that a swtch s drectly connected wth a controller. In ths case, the LCO-FSC problem s equvalent to the CO-FSC problem, whch has been consdered n Secton III-A. The other s that the physcal control connecton may pass through one or several swtches. As shown n Fg. 2, swtch v 1 s drectly connected wth the controller. But, swtch v 2 s connected wth the controller through a control path v 2 v 1 controller. For smplcty, we assume that each swtch v has bult a path p(v) to the assocated controller. When an FSC event s trggered, the controller wll send Read-State commands, each of whch contans a wldcard rule, to dfferent swtches, so that all swtches wll report the correspondng statstcs nformaton to the controller and all the flows can be covered. As descrbed n Secton II-C, the bandwdth cost on swtch v s denoted by c(v ). For each control lnk e, ts traffc load s the total amount of all statstcs nformaton through ths lnk, whch s descrbed as l(e) = v V, e p(v) c(v ). In Fg. 2, both the data traffc and control traffc wll path through some lnks connectng swtches, e.g., v 1 v 2 and v 1 v 4. Logcally, we dvde such a lnk nto two sub-lnks, data sub-lnk and control sub-lnk, respectvely. Wthout confuson, these sub-lnks are also called as lnks. To better guarantee the successful forwardng of FSC and control commands, each control lnk wll be allocated a bandwdth (also called ts capacty), denoted by c(e). Inthe scenaro of flow statstcs collecton, we expect that each control lnk should reman some bandwdth for control commands. Otherwse, due to congeston on some control lnks, the control commands can not be successfully sent to some swtches. Thus, our objectve s to acheve the load balancng among all the control lnks, that s, mn max{ l(e) c(e),e Ec },where E c s a set of control lnks. Accordngly we formalze the LCO-FSC problem as follows: mn η S.t. γ k Γ j x j 1, c(v )= r xj c(γj ), l(e) = c(v ) η c(e), v V, e p(v) {0, 1}, r x j γ k Γ v V e E c j R (16) where x j denotes whether the controller wll send a Read- State command wth wldcard r j to swtch v or not. The frst set of nequaltes denotes that each flow wll be covered. The second set of nequaltes denotes the total bandwdth cost on each swtch. The thrd set of nequaltes means that the total traffc load on each control lnk, where η s the control lnk load factor. The objectve s to acheve the load balancng among all the control lnks, that s, mn η. Theorem 8: The LCO-FSC problem s NP-hard. Proof: Consder a specal case of the LCO-FSC problem, n whch each swtch wll drectly connect wth a controller. Ths becomes the CO-FSC problem. Combnng wth Theorem 1, t follows that the LCO-FSC problem s NP-hard. B. An Approxmaton Algorthm for LCO-FSC In ths secton, we wll modfy the R-FSC algorthm a lttle so as to solve the LCO-FSC problem. We call ths algorthm as MR-FSC. The proposed MR-FSC algorthm conssts of two man steps. In the frst step, we relax the nteger program formulaton n Eq. (16) to a lnear program as Eq. (17). mn η x j γ k Γ j 1, c(v )= S.t. r xj c(γj ), l(e) = v V, e p(v) c(v ) η c(e), x j 0, γ k Γ v V e E c r j R (17) We can solve Eq. (17) n polynomal tme, and obtan the fractonal soluton, denoted by x. In the second step, the fractonal soluton wll be rounded to 0-1 soluton for flow statstcs collecton. The set of uncovered flows s denoted by Γ u, whch s ntalzed as all flows n Γ. We arbtrarly choose an uncovered flow, denoted by γ, from set Γ u. Then, the algorthm chooses a flow set Γ j, whose xj s maxmum among all these sets contanng flow γ, and set x j = 1. That s, the controller wll send a Read-State command wth wldcard r j to swtch v. Moreover, we update Γ u =Γ u Γ j.

9 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3595 The algorthm wll termnate untl all flows are covered. The MR-FSC algorthm s descrbed n Alg. 4. Algorthm 4 MR-FSC: Modfed Roundng-Based FSC 1: Step 1: Solvng the Relaxed LCO-FSC Problem 2: Construct the relaxed formulaton as Eq. (17) 3: Obtan a fracton soluton x 4: Step 2: Roundng to 0-1 Soluton 5: Γ u =Γ 6: whle Γ u Φdo 7: Arbtrarly choose an uncovered flow γ 8: Choose a flow set Γ j, whose xj s maxmum among all these sets contanng flow γ, andset x j =1 9: Γ u =Γ u Γ j C. Performance Analyss We analyze the approxmate performance of the proposed algorthm. In the second step, we arbtrarly choose an uncovered flow γ. LetΓ γ denote all the sets that contan ths flow. By the second feature of wldcard rules, Γ γ f, wheref s the maxmum number of swtches vsted by each flow. Snce each flow γ wll be covered, Γ j x j Γγ 1. Assume that a flow set Γ j s chosen n some teraton. It follows xj 1 f. After solvng the lnear program n the frst step of the MR-FSC algorthm, we derve a fractonal soluton x and an optmal result η for the relaxed LCO-FSC problem. Accordng to the algorthm descrpton, the fnal cost of each control lnk e s: l(e) = v V, e p(v) r xj c(γj ) f v V, e p(v) r xj c(γj ) f η c(e) (18) Thus, we can conclude that Theorem 9: The MR-FSC algorthm can acheve the f- approxmaton for the LCO-FSC problem. D. A Greedy Algorthm for LCO-FSC To determne the FSC soluton mmedately, ths secton desgns a greedy algorthm, called G-FSC, wth lower tme complexty for LCO-FSC. In each teraton, we randomly choose an uncovered flow, denoted by γ. LetΓ γ denote all the sets that contan ths flow. Then, we wll choose a flow set from Γ γ, so that the load factor of all control lnks s mnmzed. Then, the algorthm wll update the uncovered flow set, and wll be termnated untl all flows are covered. The greedy algorthm s descrbed n Alg. 5. Theorem 10: The tme complexty of G-FSC s O(mfh), where m, f and h are the number of flows, the maxmum number of swtches vsted by each flow, and the maxmum hop number of all control lnks n a network, respectvely. Proof: Snce there are m flows n a network, the algorthm conssts of O(m) teratons at most. In each teraton, we wll randomly choose an uncovered flow. Then, we determne the Algorthm 5 G-FSC: Greedy Method for FSC 1: Γ u =Γ 2: whle Γ u > 0 do 3: Randomly choose an uncovered flow γ 4: Determne the set Γ γ 5: Choose a flow set from Γ γ so that the load factor of all control lnks s mnmzed 6: Update the uncovered flow set Γ u swtch and ts wldcard wth the less control traffc load factor, whch takes a tme complexty of O(fh). Then, the total tme complexty of the G-FSC algorthm s O(mfh). We should note that our proposed G-FSC algorthm can be modfed to solve the partal flow statstcs collecton problem. For PFSC, we just modfy the teraton condton as Γ u > (1 β) m n Lne 2. After the algorthm termnates, there are at least β m covered flows n a network. V. PERFORMANCE EVALUATION Ths secton frst ntroduces the metrcs and benchmarks for performance comparson (Secton V-A). Then, we mplement our algorthms on the SDN platform, and gve the testng results (Secton V-B). Fnally, we evaluate our proposed algorthms by comparng wth the prevous methods through smulatons (Secton V-C). Our smulatons are run on Mnnet [30], whch s a wdely-used smulator for SDN. A. Performance Metrcs and Benchmarks Ths paper manly cares for swtch (bandwdth/delay) cost and lnk cost of FSC for dfferent problems, ncludng CO-FSC, CO-PFSC and LCO-FSC. When a swtch s wth a heavy FSC load, t may heavly nterfere wth ts basc functons, such as data forwardng and updates. Thus, we expect to mnmze the maxmum bandwdth cost on all swtches. Moreover, we expect to fnsh the flow statstcs collecton n a fast manner. Thus, t s requred to mnmze the maxmum delay on all swtches. When a control lnk s wth a heavy traffc control load, t may be congested, whch may ncrease the delay and loss rato of control commands. Thus, we expect to mnmze the maxmum bandwdth cost on all control lnks. We use the followng performance metrcs n our numercal evaluaton. 1) The maxmum bandwdth cost on any swtch adurng a run of testng/smulaton. As descrbed n Secton II-C, for a flow set Γ j, ts bandwdth cost s defned c(γ j )=96 Γj wth unt byte. 2) The maxmum delay cost on any swtch durng a run of smulaton. For a flow set Γ j, ts delay cost s defned c(γ j )=0.19 Γj wth unt ms. 3) The total number of requests per FSC. We compare the total number of requests necessary to collect statstcs nformaton of all flows. 4) The algorthm runnng tme. We manly compare the runnng tme of R-FSC and FSC-PD, both desgned for the CO-FSC problem, by changng the number of flows.

10 3596 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER ) The rato of covered flows. Gven a flow recall rato β, the R-PFSC algorthm may not fully guarantee that the number of covered flows exceeds β m. The rato of covered flow s the number of covered flow by our R-PFSC algorthm dvded by the number of all flows n a network. 6) The control lnk load factor. We measure the traffc amount of flow statstcs collecton on each control lnk, and compte the load factor of all control lnks n a network. 7) The load balancng factor. As an applcaton example, we regard that the (partal) flow statstcs knowledge wll beneft to the effcent routng. To measure the route effcency, the load factor of a data lnk s the traffc load dvded by the data lnk capacty. The load balancng factor s the maxmum load factor among all data lnks. We compare the proposed FSC algorthms wth the mostrelated, state-of-the-art work of OpenTM [19] and Cemon [17] by both testng and smulatons. OpenTM and Cemon are typcal studes for per-flow and per-swtch statstcs collecton mechansms, respectvely. OpenTM s a flow-based FSC method, n whch the controller wll collect the statstcs nformaton of each flow from a swtch along the route path randomly, so that the bandwdth/delay cost can be reduced. The objectve of Cemon s to reduce the total bandwdth cost on the controller n an SDN. The algorthm chooses the most cost-effectve swtches and removes the covered flows, untl all flows are covered. B. Test-Bed Evaluaton 1) Implementaton on the Platform: There are two dfferent ways for buldng the SDN platform. One s based on the physcal swtches, such as H3C S SC-HI swtches. The current verson of ths physcal swtch only supports the per-flow and per-swtch statstcs collecton, and we have mplemented two methods on the platform usng the RESTful APIs specfed by the Opendaylght controller. For example, to obtan the statstc of a flow, we should specfy the values of some parameters, such as swtch-d, table-d and flow-d, n the mplementaton. Moreover, we have tested the delay for per-flow and per-swtch statstcs collecton on the H3C swtch. Unfortunately, snce our H3C SDN swtches are developed based on the tradtonal swtchng framework, they do not support the wldcard-based collecton currently. In the future verson, provded that the hardware can support the wldcardbased flow statstcs collecton, we can easly mplement our proposed algorthms usng RESTful API nterfaces by adaptng our per-flow and per-swtch mplementatons. The other s based on the vrtual swtches. Snce these vrtual swtches are mplemented by the software, they usually support all three schemes of flow statstcs collecton. We mplement the per-flow, per-swtch and wldcard-based FSC algorthms on a real OVS-based test-bed. Our SDN platform s manly composed of three parts: a server nstalled wth the controller s software, a set of vrtual swtches, and some vrtual machnes (also called termnals). Specfcally, we choose Ryu [31], whch s an open source project, as the Fg. 3. Topology of the SDN platform. Our platform s composed of three parts: a controller, sx OpenFlow enabled vrtual swtches {v 1,v 2,v 3,v 4,v 5,v 6 } and four vrtual machnes {u 1,u 2,u 3,u 4 }. The controller s drectly connected wth each swtch. For smplcty, we omt the controller n ths fgure. controller s software. The Ryu controller s runnng on a server wth a core processor and 4GB of RAM. The vrtual swtch s mplemented usng the OVS [21]. The topology of our SDN platform s llustrated n Fg. 3. The forwardng plane of an SDN comprses of 6 vrtual swtches, whch support the OpenFlow v1.3 standard. In the system testng, each flow s dentfed by three elements, source IP, destnaton IP and TCP port, so that each termnal s able to generate dfferent numbers of flows to other termnals. To mplement the wldcard-based flow statstcs collecton on the vrtual swtch, the controller sends the request nterface, whch s defned as the OFPFlowStatsRequest class, to the target swtch, and the match feld n the OFPFlowStatsRequest class s descrbed by the OFPMatch class. For example, when we try to collect the statstcs nformaton of flows whose destnaton s , we create a OFPFlowStatsRequest object, n whch we set two parameters n the OFPMatch object as: eth_type=0x800, and pv4_dst = ( , ), and send t to the target swtch. 2) Testng Results: We generate 600 and 1200 flows n the network, respectvely, and observe the maxmum bandwdth cost of all swtches by dfferent FSC algorthms. There are four termnals and accordngly 12 termnal pars n the system. In the testng, each termnal par wll generate the same number of flows, and the controller wll choose a path randomly for each flow. Moreover, we allocate the sze for each flow accordng to the classcal 2-8 rule [13]. The left plot of Fg. 4 shows that our proposed R-FSC and FSC-PD algorthms can perform better than OpenTM and Cemon, respectvely. More specfcally, the maxmum bandwdth cost of OpenTM s about 0.342Mb and 0.689Mb for 600 flows and 1200 flows, respectvely. Gven 1200 flows, the maxmum bandwdth costs of the Cemon, FSC-PD and R-FSC algorthms are 0.665Mb, 0.434Mb, and 0.274Mb, respectvely. In other words, our R-FSC and FSC-PD algorthms can reduce the maxmum bandwdth cost about 60.2% and 37.0%, respectvely. The rght plot of Fg. 4 shows the maxmum bandwdth cost by changng the value of flow recall rato β. Our testng shows that, when the flow recall rato s smaller, the controller needs to collect statstcs nformaton of a smaller number of flows, whch results n a smaller bandwdth cost. For the case of 1200 flows,

11 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3597 Fg. 4. Maxmum bandwdth cost. Left plot: Dfferent algorthms; rght plot: Dfferent flow recall ratos β. Fg. 6. Maxmum delay cost vs. number of flows for CO-FSC. Left plot: Topology (a); rght plot: Topology (b). Fg. 5. Maxmum bandwdth cost vs. number of flows for CO-FSC. Left plot: Topology (a); rght plot: Topology (b). Fg. 7. Number of total requests vs. number of flows for CO-FSC. Left plot: Topology (a); rght plot: Topology (b). the maxmum bandwdth cost s 0.079Mb, 0.084Mb, 0.173Mb, and 0.274Mb, respectvely. That s, the partal flow statstcs collecton mechansm (e.g., β =0.4) can reduce the maxmum bandwdth cost 69.2% compared wth the flow statstcs collecton mechansm (.e., β = 1.0). These testng results show hgher effcency of our proposed wldcard-based flow statstcs collecton methods compared wth per-flow and perswtch FSC methods. C. Smulaton Evaluaton 1) Smulaton Settng: In the smulatons, as runnng examples, we select two practcal and typcal topologes, one for campus networks and the other for datacenter networks. The frst topology, denoted by (a), contans 100 swtches, 200 servers and 397 lnks from [32]. The second one s a fat-tree topology [33], whch has been wdely used n many datacenter networks. The fat-tree topology has totally 80 swtches (ncludng 16 core swtches, 32 aggregaton swtches, and 32 edge swtches) and 192 servers. Due to capacty constrant of our smulaton platform, the capacty of each physcal lnk s set as 100Mbps on both topologes. We execute each smulaton 100 tmes, and average the numercal results. For the flow sze, the authors of [13] have shown that less than 20% of the top-ranked flows may be responsble for more than 80% of the total traffc. Thus, we allocate the sze for each flow accordng to ths rule. 2) Smulaton Results: We run four groups of smulatons to check the effectveness of our proposed FSC algorthms. The frst set of smulatons observes how the number of flows affects the swtch cost performance, ncludng maxmum bandwdth cost and maxmum delay cost, of dfferent algorthms on two topologes. From Fg. 5, our proposed algorthms, both R-FSC and FSC-PD, can sgnfcantly reduce the maxmum bandwdth cost compared wth both two methods, especally the per-swtch method. That s because, Cemon collects the statstcs nformaton of all flows on some swtches, whch leads to hgher bandwdth cost on control lnks between these swtch and the controller. For example, when there are 1000 flows per server on average, our FSC-PD algorthm can reduce the maxmum bandwdth cost from 19Mb by Cemon to only 6Mb. Fg. 5 shows that FSC-PD reduces the maxmum bandwdth cost by about 39% and 53% compared wth the OpenTM and Cemon methods, respectvely. From Fg. 6, our proposed algorthms can sgnfcantly reduce the maxmum delay cost compared wth both per-swtch and perflow collecton methods. For the per-flow method, frequent collecton requests lead to serous collecton delay. Fg. 6 shows that FSC-PD can reduce the delay cost of all swtches by about 52% and 45% compared wth OpenTM and Cemon, respectvely. From Fgs. 5 and 6, we fnd that our roundngbased R-FSC algorthm performs better than the FSC-PD algorthm based on prmal-dual. Fg. 7 shows the total number of requests for per FSC event. Obvously, snce Cemon s a per-swtch FSC method, t needs the least number of requests for FSC among four algorthms. Our proposed R-FSC and FSC-PD algorthms requre almost the smlar number of requests wth vared number of flows, and only need 1/30-1/50 FSC requests as OpenTM. Moreover, FSC-PD needs almost the same number of requests as R-FSC n topology (b) due to ts structured topology. However, Fg. 8 shows that FSC-PD can save much more runnng tme compared wth R- FSC. More specfcally, the runnng tme of FSC-PD s only about 1/4-1/10 as that of R-FSC, and the ncreasng rato of runnng tme of FSC-PD s less than that of number of flows n the network. Though both Cemon and OpenTM algorthms need less runnng tme than FSC-PD, runnng tme of all three algorthms s acceptable. These results show that our proposed FSC-PD algorthm s much scalable and ft for the large-scale networks. The second set of sx smulatons observes the dfferent performance metrcs of our R-PFSC algorthm by changng the flow recall rato β on two topologes. In these smulatons, there are 40K flows (.e., about 200 flows per server) by default

12 3598 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER 2017 Fg. 8. Comparson of runnng tme. Left plot: Topology (a); rght plot: Topology (b). Fg. 12. Control lnk load factor vs. number of flows for LCO-FSC. Left plot: Topology (a); rght plot: Topology (b). Fg. 9. Rato of covered flows vs. β by the R-PFSC algorthm. Left plot: Topology (a); rght plot: Topology (b). Fg. 10. Maxmum bandwdth cost vs. number of flows for CO-PFSC. Left plot: Topology (a); rght plot: Topology (b). Fg. 11. Maxmum delay cost vs. number of flows for CO-PFSC. Left plot: Topology (a); rght plot: Topology (b). n the network. From Fg. 9, we fnd that, for a gven parameter β, the rato of covered flows by our R-PFSC algorthm mostly exceeds β, whle t s very close to β n some cases, e.g., β = 0.6. Ths fgure shows that our R-PFSC algorthm can satsfy the flow recall rato n most stuatons. From Fgs. 10 and 11, the maxmum bandwdth/delay cost of FSC ncreases when the flow recall rato β ncreases. That s because, wth ncrease of flow recall rato, statstcs of more flows wll be collected, whch results n a hgher cost, ncludng bandwdth cost and delay cost. Both two fgures show that the R-PFSC algorthm wth β =0.8 can decrease the maxmum bandwdth/delay cost by about 52% compared wth that wth β =1.0. The thrd set of smulatons observes how the number of flows affects the lnk cost performance, ncludng control lnk load factor and maxmum bandwdth cost on lnks, of dfferent Fg. 13. Control lnk load factor vs. number of flows by varyng β for LCO-FSC. Left plot: Topology (a); rght plot: Topology (b). algorthms. For both two topologes, we assume that a thrd of all swtches are drectly connected wth the controller. Then, we wll construct a wdth-frst tree for other swtches so that each swtch can buld connecton wth the controller. We set the capacty of each (logcal) control lnk as 6Mbps for topologes (a) and (b), respectvely. Specfcally, accordng to the smulaton settng, the capacty of each lnk connectng two swtches s 100Mbps. We dvde ths lnk nto two logcal lnks. One s the data lnk wth 94Mbps, and the other s the control lnk wth 6Mbps. From Fg. 12, our proposed algorthms, both MR-FSC and G-FSC, can sgnfcantly reduce the control lnk load factor compared wth both per-swtch and per-flow collecton methods, especally the per-swtch method. That s because, Cemon collects the statstcs nformaton of all flows just from some swtches, whch leads to hgher cost on some control lnks between these swtch and the controller. For example, when there are totally 50K flows n a network, and the collecton perod s 1s, our MR-FSC and G-FSC algorthms reduce the control lnk load factor from 0.79 by Cemon to only Fg. 14 shows that G-FSC reduces the control lnk load factor by about 52% and 65% compared wth the OpenTM and Cemon methods, respectvely. From Fg. 13, the control lnk load factor of FSC ncreases when the flow recall rato β ncreases, whch s smlar to Fg. 10. Fg. 13 shows that the G-FSC algorthm wth β =0.4 can decrease the maxmum bandwdth/delay cost by about 48% compared wth that wth β =1.0. Fgs.14and15plotthe maxmum bandwdth cost of all control lnks for dfferent algorthms. There fgures have the smlar curves as those n Fgs. 5 and 10, respectvely. We can obtan two conclusons from these fgures. 1) For FSC, both MR-FSC and G-FSC algorthms sgnfcantly reduce the maxmum bandwdth cost compared wth both Cemon and OpenTM, respectvely. 2) Wth ncreasng of the parameter β, the maxmum bandwdth cost s ncreasng accordngly. As an applcaton example, after (partal) flow statstcs collecton, we can re-route flows usng the routng method, as descrbed n Secton III-C, for better network performance,

13 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3599 Fg. 14. Maxmum bandwdth cost of control lnks vs. number of flows. Left plot: Topology (a); rght plot: Topology (b). Fg. 15. Maxmum bandwdth cost of control lnks vs. number of flows. Left plot: Topology (a); rght plot: Topology (b). Fg. 16. Load-balancng factor vs. β by flow statstcs collecton and re-routng. Left plot: Topology (a); rght plot: Topology (b). such as load balancng. In the smulaton, there are 40K flows n a network. The fourth group of smulatons observe the route performance by changng the parameter β. Fg.16showsthat the load balancng factor wll be reduced wth a larger flow recall rato β (or wth statstcs knowledge of more flows). For example, R-PFSC at β =0.8 can reduce the cost 52% compared wth that at β =1.0, wth ncreased load-balancng factor only about 5%. VI. RELATED WORKS Recently, SDN [2] has become an emergng technology for future networks. Google desgned and mplemented B4, whch took a software defned networkng archtecture to connect ther data centers across the planet [26]. The authors of [3] presented SWAN, whch boosted the utlzaton of nter-datacenter networks by centrally controllng when and how much traffc each servce sent. Most prevous works [3], assume that the controller knows traffc ntensty of each flow to provde effcent route selecton n a network. However, the flow traffc ntensty s often unknown n advance n many applcatons, and dynamcally changed durng flow forwardng. A related problem wth our statstcs collecton s the flow traffc measurement, and the comprehensve survey can be found n [34]. The prevous traffc measurement solutons are mostly mplemented through the samplng technque. Open- Sample [10] leveraged sflow packets [35] to provde nearreal-tme measurements of both network load and ndvdual flows. Yu et al. [9] used a sketch-based measurement lbrary to automatcally confgure and manage resources for measurement actvtes. The smlar sketch-based traffc montorng method was also studed n [36]. The authors of [12] allocated resources for sketch-based measurement tasks to ensure a user-specfed mnmum accuracy. Some works studed the rule placement and traffc measurement for an SDN. STAMP [37] used (de)aggregaton measurement mechansm, whch dynamcally parttoned the flow entres to allow fnegraned or course measurement tasks of ncomng flows. The authors of [24] combned the drect measurement (e.g., flow statstcs collecton) and nference technques based on network tomography to derve a hybrd network montorng scheme, whch could strke a balance between measurement overhead and accuracy. All the above methods often estmate the flow sze wth less overhead, whch s dfferent from our statstcs collecton. Note that, our FSC solutons can be combned wth traffc measurement methods for dfferent applcatons. In a general SDN, each swtch counts the traffc of each flow through the counter feld n the flow entry. OpenFlow [8] specfed two dfferent approaches, push-based and pull-based, for flow statstcs collecton. The frst one s the push-based collecton. FlowSense [38] utlzed the PacketIn and FlowRemoved messages, whch were sent by swtches to the controller when a new flow come n or upon the expraton of a flow entry. Devoflow [13] extended OpenFlow wth a new push-based statstcs collecton mechansm for dentfyng the elephant flows and re-routng them. However, the push-based mechansm requred addtonal hardware support on swtches, or some modfcaton on the packet head (such as sflow [35]). These requrements mght not be fully supported by most commodty swtches, whch lmted the applcaton of the push-based mechansm. The second one s the pull-based collecton, whch s smple and has been wdely used n many SDN applcatons. OpenTM [19] was desgned for traffc matrx estmaton usng smple logc for queryng flow table counters. The logc was based on keepng statstcs for each actve flow n the network. The nformaton about actve flows was pulled from the swtches perodcally. OpenNetMon [16] presented an approach and open source software mplementaton to montor end-to-end QoS metrcs of per-flow, especally throughput, delay and packet loss, n OpenFlow networks. The authors used an adaptve fetchng mechansm to pull data from swtches where the rate of the queres ncreased when flow rates dffer between samples and decreased when flows stablzed. PayLess [39] focused on the tradeoff between accuracy and network overhead. It provded a flexble RESTful API for flow statstcs collecton at dfferent aggregaton levels. The most related works wth ours were FlowCover [15] and CeMon [17], whch proposed a low-cost per-swtch montorng scheme to support varous network management tasks. As a collecton event was trggered, the controller collected the statstcs nformaton of all the flows n a network. Some applcatons, e.g., flow re-routng [13], requre that the pull-based statstcs should be collected frequently enough,

14 3600 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 25, NO. 6, DECEMBER 2017 whch may result n more serous per-swtch cost, preventng from packet forwardng on swtches. VII. CONCLUSION In ths paper, we have studed the effcent FSC mechansms to reduce the bandwdth cost and processng delay n an SDN. We have proposed to use wldcard-based FSC to avod the dsadvantages of both per-flow and per-swtch FSC, and presented several approxmaton algorthms for both FSC and partal FSC problems. The testng and extensve smulaton results show hgh effcency of our proposed algorthms. Snce the delay on the swtch may depend on ts traffc load, and not be fully lnear wth the traffc amount of FSC, n the future, we wll study more practcal delay model for flow statstcs collecton. APPENDIX PROOF OF THEOREM 1 Proof: We prove the NP-hardness by showng that the unrelated processor schedulng (UPS) problem [40] s a specal case of CO-FSC. Consder a specal case, n whch one wldcard can only match one flow on each swtch. We regard the swtches and flows n the CO-FSC problem as the processors and tasks n the UPS problem. Assume that a flow γ passes through a set of swtches, denoted by V γ = {v 1,...,v s }. It means that the statstcs nformaton of flow γ wll be collected from one of swtches n V γ. It can be seen as that ataskγ wll only be schedulng on one of the processors v 1,...,v s. Thus, the cost of task γ on swtch v s c 1 + c 2, f v V γ ; otherwse, ts cost s. As a result, the cost of each swtch becomes the cost of each processor. Then, CO-FSC s equvalent to the followng problem: how to schedule these tasks, so that the makespan of all the processors s mnmzed. Thus, ths s a typcal UPS problem, whch s NP-Hard [40]. Snce UPS s a specal case of the CO-FSC problem, CO-FSC s an NP-Hard problem too. REFERENCES [1] H. Xu, Z. Yu, C. Qan, X.-Y. L, and Z. Lu, Mnmzng flow statstcs collecton cost of SDN usng wldcard requests, n Proc. IEEE INFOCOM, [2] N. Gude et al., NOX: Towards an operatng system for networks, ACM SIGCOMM Comput. Commun. Rev., vol. 38, no. 3, pp , [3] C.-Y. Hong et al., Achevng hgh utlzaton wth software-drven WAN, n Proc. ACM SIGCOMM, 2013, pp [4] M. F. Bar, S. R. Chowdhury, R. Ahmed, and R. Boutaba, PolcyCop: An autonomc QoS polcy enforcement framework for software defned networks, n Proc. IEEE SDN Future Netw. Servces (SDN4FNS), Nov. 2013, pp [5] H. Xu, X. L, L. Huang, J. Wang, and B. Leng, Hgh-throughput anycast routng and congeston-free reconfguraton for SDNs, n Proc. IEEE/ACM 24th Int. Symp. Qualty Servce (IWQoS), Jun. 2016, pp [6] B. Wang, Y. Zheng, W. Lou, and Y. T. Hou, DDoS attack protecton n the era of cloud computng and software-defned networkng, Comput. Netw., vol. 81, pp , Apr [7] Q. Yan, F. R. Yu, Q. Gong, and J. L, Software-defned networkng (SDN) and dstrbuted denal of servce (DDoS) attacks n cloud computng envronments: A survey, some research ssues, and challenges, IEEE Commun. Surveys Tuts., vol. 18, no. 1, pp , 1st Quart., [8] B. Pfaff et al., OpenFlow swtch specfcaton v1.3.0, Tech. Rep., [9] M. Yu, L. Jose, and R. Mao, Software defned traffc measurement wth opensketch, n Proc. 10th USENIX Symp. Netw. Syst. Desgn Implement. (NSDI), 2013, pp [10] J. Suh, T. T. Kwon, C. Dxon, W. Felter, and J. Carter, OpenSample: A low-latency, samplng-based measurement platform for commodty SDN, n Proc. 34th Int. Conf. Dstrb. Comput. Syst. (ICDCS), Jun./Jul. 2014, pp [11] Y. Afek, A. Bremler-Barr, S. L. Febsh, and L. Schff, Samplng and large flow detecton n SDN, ACM SIGCOMM Comput. Commun. Rev., vol. 45, no. 4, pp , [12] M. Moshref, M. Yu, R. Govndan, and A. Vahdat, Scream: Sketch resource allocaton for software-defned measurement, n Proc. CoNEXT, Hedelberg, Germany, 2015, pp [13] A. R. Curts et al., DevoFlow: Scalng flow management for hghperformance networks, ACM SIGCOMM Comput. Commun. Rev., vol. 41, no. 4, pp , Aug [14] M. Aslan and A. Matrawy, On the mpact of network state collecton on the performance of SDN applcatons, IEEE Commun. Lett., vol. 20, no. 1, pp. 5 8, Jan [15] Z. Su, T. Wang, Y. Xa, and M. Hamd, FlowCover: Low-cost flow montorng scheme n software defned networks, n Proc. IEEE Global Commun. Conf. (GLOBECOM), Dec. 2014, pp [16] N. L. M. van Adrchem, C. Doerr, and F. A. Kupers, OpenNetMon: Network montorng n OpenFlow software-defned networks, n Proc. IEEE Netw. Oper. Manage. Symp., May 2014, pp [17] S. Zhyang, T. Wang, Y. Xa, and M. Hamd, CeMon: A cost-effectve flow montorng system n software defned networks, Comput. Netw., vol. 92, pp , Dec [18] M. Al-Fares, S. Radhakrshnan, B. Raghavan, N. Huang, and A. Vahdat, Hedera: Dynamc flow schedulng for data center networks, n Proc. NSDI, 2010, p. 19. [19] A. Tootoonchan, M. Ghobad, and Y. Ganjal, OpenTM: Traffc matrx estmator for OpenFlow networks, n Proc. Int. Conf. Passve Actve Netw. Meas., 2010, pp [20] H. Xu et al., Real-tme update wth jont optmzaton of route selecton and update schedulng for SDNs, n Proc. IEEE ICNP, Nov. 2016, pp [21] B. Pfaff et al., The desgn and mplementaton of open vswtch, n Proc. USENIX NSDI, 2015, pp [22] T. Wang, F. Lu, J. Guo, and H. Xu, Dynamc SDN controller assgnment n data center networks: Stable matchng wth transfers, n Proc. INFOCOM, Apr. 2016, pp [23] T. Koponen et al., Network vrtualzaton n mult-tenant datacenters, n Proc. 11th USENIX Symp. Netw. Syst. Desgn Implement. (NSDI), 2014, pp [24] Z. Hu and J. Luo, Crackng network montorng n DCNs wth SDN, n Proc. IEEE INFOCOM, Apr./May 2015, pp [25] X. Jn et al., Dynamc schedulng of network updates, n Proc. ACM Conf. SIGCOMM, 2014, pp [26] S. Jan et al., B4: Experence wth a globally-deployed software defned WAN, n Proc. ACM SIGCOMM, 2013, pp [27] R. Narayanan et al., Macroflows and mcroflows: Enablng rapd network nnovaton through a splt SDN data plane, n Proc. Eur. Workshop Softw. Defned Netw. (EWSDN), Oct. 2012, pp [28] D. Levn, A. Wundsam, B. Heller, N. Handgol, and A. Feldmann, Logcally centralzed? State dstrbuton trade-offs n software defned networks, n Proc. 1st Workshop Hot Topcs Softw. Defned Netw., 2012, pp [29] A. Clemm, Network Management Fundamentals. Indanapols, IN, USA: Csco Press, [30] The Mnnet Platform. Accessed: Feb. 20, [Onlne]. Avalable: [31] S. Ryu, Framework communty: Ryu SDN controller, Tech. Rep., [32] The Network Topology From The Monash Unversty. Accessed: Feb. 20, [Onlne]. Avalable: twk/bn/vew/infocus/largepacket-swtchngnetworktopologes [33] M. Al-Fares, A. Loukssas, and A. Vahdat, A scalable, commodty data center network archtecture, ACM SIGCOMM Comput. Commun. Rev., vol. 38, no. 4, pp , [34] A. Yassne, H. Rahm, and S. Shrmohammad, Software defned network traffc measurement: Current trends and challenges, IEEE Instrum. Meas. Mag., vol. 18, no. 2, pp , Apr [35] P. Phaal and M. Lavne. (Jul. 2004). SFlow Verson 5. [Onlne]. Avalable:

15 XU et al.: MINIMIZING FLOW STATISTICS COLLECTION COST USING WILDCARD-BASED REQUESTS IN SDNs 3601 [36] T. Wellem, Y.-K. La, and W.-Y. Chung, A software defned sketch system for traffc montorng, n Proc. 11th ACM/IEEE Symp. Archt. Netw. Commun. Syst., May 2015, pp [37] M. Malboub, L. Wang, C.-N. Chuah, and P. Sharma, Intellgent SDN based traffc (de)aggregaton and measurement paradgm (STAMP), n Proc. IEEE INFOCOM, Apr./May 2014, pp [38] C. Yu et al., FlowSense: Montorng network utlzaton wth zero measurement cost, n Passve and Actve Measurement. Berln,Germany: Sprnger, 2013, pp [39] S. R. Chowdhury, M. F. Bar, R. Ahmed, and R. Boutaba, PayLess: A low cost network montorng framework for software defned networks, n Proc. IEEE Netw. Oper. Manage. Symp. (NOMS), May 2014, pp [40] J. K. Lenstra, D. B. Shmoys, and É. Tardos, Approxmaton algorthms for schedulng unrelated parallel machnes, Math. Program., vol. 46, nos. 1 3, pp , Hongl Xu (M 08) receved the B.S. degree n computer scence and the Ph.D. degree n computer software and theory from the Unversty of Scence and Technology of Chna n 2002 and 2007, respectvely. He s currently an Assocate Professor wth the School of Computer Scence and Technology, Unversty of Scence and Technology of Chna. He has authored or coauthored over 70 papers, and held about 30 patents. Hs man research nterest s software-defned networks, cooperatve communcaton, and vehcular ad hoc network. Zhuolong Yu s currently pursung the M.S. degree n computer scence wth the Unversty of Scence and Technology of Chna. Hs research nterests nclude software-defned networks and moble computng. Xang-Yang L (F 15) receved the bachelor s degrees from the Department of Computer Scence and the Department of Busness Management, Tsnghua Unversty, n 1995, and the M.S. and Ph.D. degrees from the Department of Computer Scence, Unversty of Illnos at Urbana Champagn, n 2000 and 2001, respectvely. He was a Professor wth the Illnos Insttute of Technology. He held an EMC-Endowed Vstng Char Professorshp wth Tsnghua Unversty. He s currently a Professor and the Dean wth the School of Computer Scence and Technology, Unversty of Scence and Technology of Chna. He has authored a monograph Wreless Ad Hoc and Sensor Networks: Theory and Applcatons. Hs research nterests nclude wreless networkng, moble computng, securty and prvacy, cyber physcal systems, socal networkng, and algorthms. He and hs students receved several best paper awards and a best demo award. He has been an ACM Dstngushed Scentst snce He was a recpent of Chna NSF Outstandng Overseas Young Researcher (B). Zchun Lu s currently pursung the Ph.D. degree n computer scence wth the Unversty of Scence and Technology of Chna. Hs research nterests nclude software-defned networks and data securty. Chen Qan (M 08) receved the B.S. degree from Nanjng Unversty n 2006, the M.Phl. degree from The Hong Kong Unversty of Scence and Technology n 2008, and the Ph.D. degree from The Unversty of Texas at Austn n 2013, all n computer scence. He s currently an Assstant Professor wth the Department of Computer Engneerng, Unversty of Calforna at Santa Cruz. He has authored or coauthored over 60 research papers n hghly compettve conferences and journals. Hs research nterests nclude computer networkng, network securty, and Internet of Thngs. He s a member of the ACM. Lusheng Huang receved the M.S. degree n computer scence from the Unversty of Scence and Technology of Chna n He s currently a Senor Professor and a Ph.D. Supervsor wth the School of Computer Scence and Technology, Unversty of Scence and Technology of Chna. He has authored or coauthored sx books and over 300 journal/conference papers. Hs research nterests are n the areas of Internet of Thngs, vehcular ad hoc network, nformaton securty, and dstrbuted computng.