NETWORK Function Virtualization (NFV) has been an

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 1 Resource Aware Routng for Servce Functon Chans n SDN and NFV-Enabled Network Janng Pe, Peln Hong, Kapng Xue, Senor Member, IEEE, and Defang L, Student Member, IEEE Abstract Owng to the Network Functon Vrtualzaton (NFV) and Software-Defned Networks (SDN), Servce Functon Chan (SFC) has become a popular servce n SDN and NFV-enabled network. However, as the Vrtual Network Functon (VNF) of each type s generally mult-nstance and flows wth SFC requests must traverse a seres of specfed VNFs n predefned orders, t s a challenge for dynamc SFC formaton to optmally select VNF nstances and construct paths. Moreover, the load balancng and end-to-end delay need to be pad attenton to, when routng flows wth SFC requests. Addtonally, fne-graned schedulng for traffc at flow level needs dfferentated routng whch should take flow features nto consderaton. Unfortunately, tradtonal algorthms cannot fulfll all these requrements. In ths paper, we study the Dfferentated Routng Problem consderng SFC (DRP-SFC) n SDN and NFV-enabled network. We formulate the DRP-SFC as a Bnary Integer Programmng (BIP) model amng to mnmze the resource consumpton costs of flows wth SFC requests. Then a novel routng algorthm, Resource Aware Routng Algorthm (RA-RA), s proposed to solve the DRP-SFC. Performance evaluaton shows that RA-RA can effcently solve the DRP-SFC and surpass the performance of other exstng algorthms n acceptance rate, throughput, hop count and load balancng. Index Terms Servce Functon Chan, Software-Defned Networks, Network Functon Vrtualzaton, Dfferentated Routng, Flow Feature. 1 INTRODUCTION NETWORK Functon Vrtualzaton (NFV) has been an arsng technology decouplng the software from hardware devces recently. It has the potental to sgnfcantly reduce the Operatng Expenses (OPEX) and Captal Expenses (CAPEX) and facltate the flexblty of new servces deployment wth ncreased aglty and faster tme-to-value [1], [2]. Software-Defned Networks (SDN) s a new network paradgm whch decouples the control plane and data plane. Accordng to the central control and flexble management, SDN controller can effcently control the network forwardng among Network Functons (NFs) based on the acqured nformaton about the network [3]. Owng to the technologes of NFV and SDN, many NFs such as Frewall (FW), Deep Package Inspecton (DPI), Intruson Detecton System (IDS), Intruson Preventon System (IPS) and Wde Area Network (WAN) optmzers can be software-orented, programmed and deployed flexbly on the Commercal-Of-The-Shelf (COTS) devces [4], [5], [6], whch are known as the Vrtual Network Functons (VNFs). Benefttng from NFV and SDN, Servce Functon Chan (SFC) has been proposed as a popular servce paradgm. An SFC defnes an ordered or partally ordered set of VNFs and orderng constrants that must be appled to packets, frames and/or flows selected as a result of classfcaton [7], [8]. SFC provdes the means so that the traffc can naturally pass through a set of specfed VNF nstances sequentally wthout the nterventon mposed by dfferent servces resdng at J. Pe, P. Hong, K. Xue and D. Lee are all wth the Key Laboratory of Wreless-Optcal Communcatons, Chnese Academy of Scences, School of Informaton Scence and Technology, Unversty of Scence and Technology of Chna, Hefe, 230027, Chna. E-mal: janngp@mal.ustc.edu.cn, plhong@ustc.edu.cn, kpxue@ustc.edu.cn, ldf911@mal.ustc.edu.cn. dfferent physcal devces [9], [10]. Wth the applcaton of SFC, hgh acceleraton of traffc performance wll be provded by more ntellgent traffc routng strateges n today s Internet Servce Provder (ISP) networks. Even though SFC s hopeful to enhance the flexblty and cost effcency n the ISP networks [9], [11], however, a set of new challenges come correspondngly, whch should be well addressed. Frstly, the VNF of each type s generally multnstance and flows wth SFC requests must pass through a seres of VNF nstances n predefned orders to satsfy the requrements of users. For example, n Fg. 1, there are four types of VNFs deployed n the network and each of them contans multple nstances. VNF11, VNF12 and VNF13 ndcate the frst, second and thrd nstances of VNF1, respectvely. The rest of VNF nstances satsfy the same rule. Supposng that a flow wth SFC request starts from node A and needs to traverse the nstances of VNF1, VNF2, VNF3 and VNF4 before arrvng at node J. Nevertheless, n the network, there exst many paths (such as the dotted lnes marked wth dfferent colors) that traverse dfferent VNF nstances and can satsfy the requrement. Therefore, t s a challenge for dynamc SFC formaton to make an optmal strategy selectng VNF nstances from mult-nstance NFV envronment and routng flows wth SFC requests to traverse these selected VNF nstances n predefned orders. Secondly, as the bandwdth on lnks should not be the only resource to be consdered n SDN and NFV-enabled network, flow table entres on SDN swtch nodes whch are resolved n Ternary Content Addressable Memory (TCAM) [12], [13], the CPU on functon nodes whch hold VNF nstances and end-to-end delay cannot be neglected as well. Hence, a mechansm should be desgned to make a trade off among multple knds of resources and pay attenton to the end-to-end delay, whch can reduce the network congeston and keep the network wth hgh performance. Thrdly, to deal

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 2 SDN Controller A Control all the swtches A D VNF1 VNF2 VNF3 VNF4 An example of a flow wth SFC request C(VNF12/VNF22) E(VNF13/VNF31/VNF41) F(VNF32/VNF42) G H(VNF23/VNF33/VNF43) Fg. 1. VNF nstance selecton and path constructon for flows wth SFC requests n SDN and NFV-enabled network wth the ncreasngly heterogeneous traffc n the network, a dfferentated routng strategy wth the consderaton of flow features s hoped by network operators to acheve fnegraned schedulng for traffc at flow level [9], [14]. Gven the challenges ntroduced by flows wth SFC requests n SDN and NFV-enabled network, there are two problems n the followng to be solved: () how to make dfferentated routng strategy for dfferent knds of flows wth SFC requests to optmally select VNF nstances and construct the paths wthout volatng the predefned orders. () how to acheve load balancng among multple knds of resources, when routng flows wth SFC requests. These problems are denoted as the Dfferentated Routng Problem consderng SFC (DRP-SFC). The contrbutons of ths paper are lsted as follows. We make a detaled analyss of the DRP-SFC and formulate t as a Bnary Lnear Programmng (BIP) model amng to mnmze the resource consumpton costs of flows wth SFC requests. We separate the flows nto dfferent knds based on resource preferences, and defne the relatve cost to balance the resource consumpton and route the heterogeneous traffc at flow level dfferentatedly n SDN and NFV-enabled network. Consderng mult-resource constrants (bandwdth, flow table entres on swtch nodes, CPU on functon nodes and end-to-end delay) and flow features comprehensvely, we propose a novel routng algorthm, Resource Aware Routng Algorthm (RA-RA), to solve the DRP-SFC n SDN and NFV-enabled network. Detaled smulaton results show that, comparng wth other algorthms n exstng lteratures, RA- RA can effcently solve the DRP-SFC and obtan hgher performance n acceptance rate, throughput, hop count and load balancng. The rest of the paper s organzed as follows: we revew the related works n Secton 2. The system model s presented n Secton 3. In Secton 4, we gve the defnton of the relatve cost and formulate the DRP-SFC as a BIP model. Secton 5 descrbes the RA-RA algorthm. In Secton 6, we valdate the effectveness of RA-RA and compare t wth some exstng algorthms. Fnally, Secton 7 concludes the paper. I J J 2 RELATED WORKS Recently, advancements n the feld of NFV and SDN make SFC drawn sgnfcant attenton n both the standardzaton organzatons and research communtes. The Servce Channg Workng Group n Internet Engneerng Task Force (IETF) completed a set of related SFC use-cases drafts referred to the SFC archtecture [7], moble networks [15] and datacenters [16]. The VNF Forwardng Graph (VNFFG) was proposed by European Telecommuncatons Standards Insttute (ETSI) to descrbe the connectvty between VNFs [17]. Based on NFV and SDN, new SFC archtectures are proposed to nterconnect dfferent VNF nstances n specfed orders [18], [19]. Moreover, wth an ncreasng number of tenants launchng ther applcatons n clouds, t s also advocated by Cloud Servce Provders (CSPs) to construct SFC archtectures n clouds to meet the demands of tenants, promote the cloud performance and reduce the OPEX/CAPEX [1], [20]. Medhat et al. [21] came up wth a servce functon selecton algorthm for the servce functon nstance selecton and servce functon path creaton problem. In order to realze load balancng among VNF nstances, the authors selected the specfed VNF nstances by tradng off the delay feature of flows, load condtons of VNF nstances and dstance n the network. And the Shortest Path (SP) algorthm s used to produce a complete path. Based on the Eucldean dstance and SP algorthm, Oh et al. proposed a Vrtual Machne (VM) selecton algorthm to create optmal servce-chan paths for flows wth SFC requests [22]. When creatng the optmal path, ths algorthm frst constructs a 3D vector space based on the requrement of flow and the statements of VMs n the network. Then the rght VMs are selected by calculatng the Eucldean dstance from the requrement pont, and the SP algorthm s used to concatenate these selected VMs. Mechtr et al. [23] proposed a novel egendecomposton based approach to cope wth the VNF placement and channg problem across dstrbuted cloud envronments. Ths algorthm frst needs to extend the adjacent matrx of a VNFFG to the same dmenson of physcal network s and then execute the Umeyamas egendecomposton approach to select the VNF nstances and construct paths. All the papers n [21], [22], [23] route the flows wth SFC requests by two-stage algorthms whch need to complete the selecton of VNF nstances at the frst stage, then construct the paths concatenatng the selected VNF nstances as the predefned order n the second stage. When routng the flows wth SFC requests n the network, both the selecton of VNF nstances and the constructon of paths have nfluence on the network performance. As splttng the relatonshp between the selecton of VNF nstances and path constructon, these two-stage algorthms wll lead to sub-optmal soluton for the routng of flows wth SFC requests. Dstnct from the two-stage algorthms, RA-RA s a one-stage algorthm whch fnshes these two processes meanwhle. And RA-RA can effcently make strateges for flows wth SFC requests by elmnatng sub-optmal paths. Consderng flows wth SFC requests onlne routng problem n SDN framework, Cao et al. [24] proposed a novel algorthm named Compettve Onlne Algorthm for Traffc

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 3 Steerng (COATS). In COATS, the authors teratvely updated the costs on lnks and routed flows wth SFC requests based on a layered graph. Bar et al. [25] defned the VNF deployment and onlne routng problem as VNF Orchestraton Problem (VNF-OP). In the paper, all the nodes n the network can support VNF nstances, and the costs of VNF deployment, energy, data forwardng, resource fragments and Servce Level Objectve (SLO) volaton are consdered. Then based on Vterb algorthm [26], the ProvsonTraffc s proposed to select the path wth the lowest cost to route the flow wth SFC request n the network. Facng the node-constraned servce chan routng problem n SDN framework, Dwarak et al. [27] proposed an adaptve servce routng algorthm to solve ths problem. In the algorthm, the network graph s transformed to a layered graph consderng process steps. Then the path wth the mnmum end-to-end delay s obtaned to route the flow wth SFC request by executng the conventonal SP algorthm on layered graph. The papers n [28] explctly states that the largest open source SDN controller, OpenDaylght, has or wll support the random selecton algorthm, round robn algorthm, SP algorthm and Load Balancng algorthm for the selecton of VNF nstances n SDN framework. To solve the DRP-SFC, the tradeoff among multple knds of resources and end-to-end delay need to be consdered comprehensvely. In SDN and NFV-enabled network, all knds of resources ncludng the bandwdth, flow table entres and CPU and end-to-end delay have nfluence on the network performance. Unbalanced utlzaton of resources wll lead to low network performance due to network congeston. And long end-to-end delay also results n low Qualty of Servce (QoS). In [24], the authors only pad attenton to the bandwdth on lnks, when routng flows wth SFC requests. In [25], the authors routed the flows wth SFC requests by mnmzng the OPEX cost and resource fragments. And only the end-to-end delay and CPU resource are taken nto account, respectvely, n [27] and [28]. In RA-RA, all the resources ncludng bandwdth, flow table entres and CPU and end-to-end delay are consdered n the meantme, when solvng DRP-SFC. The off-lne optmal algorthms to route flows wth SFC requests are studed n [25] and [29]. Wth the help of CPLEX, the off-lne optmal soluton to route flows wth SFC requests s realzed n [25]. Guo et al. [29] studed a jont optmzaton of MddleBox Selecton and Routng (MBSR) problem. In order to solve MBSR, the authors formulated ths problem as an nteger programmng model to maxmze the throughput for a specfed set of sessons wth SFC requests n SDN network. Then a polynomal algorthm usng the Markov approxmaton technque s proposed, whch adjusts the selected mddleboxes for sessons wth SFC requests randomly and terates to fnd the best result. Nevertheless, most of the mentoned works neglect to acheve the dfferentated routng for flows wth SFC requests. In the paper, we take the flow features nto consderaton and formulate the DRP-SFC as a BIP model wth the objectve to mnmze the resource consumpton costs for flows wth SFC requests. Then, the routng algorthm named RA-RA s proposed to solve the DRP-SFC n SDN and NFV-enabled network. To the best of our knowledge, ths work s the frst effort that not only manages to make effcent dfferentated routng strateges for flows wth SFC requests, but also acheves load balancng among multple knds of resources n SDN and NFV-enabled network. 3 SYSTEM MODEL 3.1 Physcal Network wth VNF Instances We consder the physcal network as an undrected graph G = (V, L), where V and L ndcate the node set and lnk set, respectvely. u, v V are physcal nodes. uv L stands for the physcal lnk connectng the physcal nodes u and v. There exst two knds of nodes n the network. One s swtch node that s responsble to forward data to neghbor nodes based on the control sgnals from SDN controller. And the other knd s functon node whch not only takes charge of nformaton forwardng but also holds VNF nstances to process flows wth SFC requests. We defne V fn V as the set of functon nodes and V sn V as the set of swtch nodes. M represents the set of all the VNF nstances and m M represents the m th VNF nstance deployed n the network. In the paper, SFCR s used to represent the SFC request of flow. We use Cu ft to symbolze the flow table capacty on node u. The rato of remanng flow table entres on node u, when routng SFCR, s represented by r ft,u. The CPU capacty on node u s Cu cpu, and the bandwdth capacty on lnk uv s Cuv bw. When routng SFCR, the r cpu,u represents the rato of remanng CPU on node u, and r,uv bw stands for the rato of remanng bandwdth on lnk uv. It s worth notng that, VNF nstances are only allowed to be deployed on functon nodes, and swtch nodes do not need to process the flows wth SFC requests, so we neglect the CPU consumpton on swtch nodes. Moreover, as the mcro datacenters and cloud datacenters can serve as functon nodes [17], [30], comparng wth swtch nodes, we do not consder the flow table consumpton on functon nodes as well. 3.2 Flows wth SFC Requests In an SDN and NFV-enabled network, the flows orgnatng from users should always traverse a set of VNF nstances concatenated n predefned orders to satsfy ther demands. In the paper, we assume that all the flows are wth SFC requests and each SFC request conssts of an ngress node, an egress node and a seres of VNF requests. We use a sx-tuple to represent the SFC request of a flow. For SFCR n Eq. (1), S represents the ngress node and T represents the egress node. The sequence of VNF requests of SFCR s defned as Ω. l = Ω represents the length of SFCR, whch ndcates the total number of VNF requests of an SFC request. Ω (j) stands for the j th VNF request, j = 1, 2,..., l. For SFCR, the bandwdth and CPU consumptons and the maxmum tolerated delay are represented by F bw, F cpu respectvely. However, for an SFC request, the assumpton that the bandwdth and CPU consumptons on lnks and VNF nstances are set as fxed and the same values, respectvely, s just for concse, and t s easy to extend the followng formulaton to support dstnctve ones. SFCR = {S, T, Ω, F bw, F cpu, F delay }, Ω = {Ω (1), Ω (2),..., Ω (l)}, l = Ω and F delay, (1)

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 4 In the system model, the servce functon graph Ḡ = ( V, L ) s used to depct Ω. The servce functon graph s a dgraph, where V and L represent the set of nodes and lnks, respectvely. ū, v V represent two nodes and ū v L ndcates the lnk connectng nodes ū and v on Ḡ. Fg. 2 shows a servce functon graph for SFCR, where S, T, Ω (1) and Ω (2) are nodes and SFCR must traverse S, Ω (1) and Ω (2) n order before arrvng at T. sparse elephant flow, computatonally dense mce flow, computatonally dense dog flow and computatonally dense elephant flow. 4 PROBLEM FORMULATION In ths secton, we frst gve the defnton of relatve cost, then formulate the DRP-SFC as a BIP model. S Ω (1) Ω (2) T Fg. 2. An example of servce functon graph 3.3 Classfcaton of Flows based on Flow Features For the purpose of cost-effcent Traffc Engneerng (TE) n the network, t s mportant for ISPs to mprove network performance and acheve load balancng wth dfferentated routng strategy based on flow features [14], [31], [32]. In the network, elephant flows carry the most of the traffc volume, whle ther number s small. Though mce flows are short-lved and carry a small number of packets, there exst large number of them, whch also have mportant mpact on the network performance and cannot be neglected [31]. Moreover, consderng the computaton consumpton, the computatonally ntensve workload s defned n [32] whch consumes lots of computaton resources on servers but requres small bandwdth durng data transmsson. Accordng to these flow features, several values of thresholds have been proposed to dfferentate flows n the network [33]. However, as the data plan and control plan are coupled and ntegrated n today s network archtecture, exstng TE technologes are prevented to acheve truly dfferentated servces to adapt to uneven and hgh varable traffc patterns [34]. On the control, n SDN, the controller can acheve central network montorng and management. Based on the technques of packet-based samplng, flow statstcs, hardware/software modfcaton and data stream mnng [35], [36], t s hopeful to effcently obtan flow features n SDN and NFV-enabled network to deal wth the ncreasngly heterogeneous traffc wth fne-graned schedulng at flow level. In the paper, we take the resource preferences of flows as flow features and classfy the flows wth SFC requests nto dfferent knds. On the perspectve of bandwdth preference, all the flows wth SFC requests can be dvded nto three knds whch are mce flow, dog flow, and elephant flow. Mce flows are such flows that are short-lved and consume lttle bandwdth. Elephant flows are on the contrary, whch are long-lved and consume large amount of bandwdth. And dog flows are defned between mce flows and elephant flows. On the perspectve of CPU preference, all the flows are dvded nto two knds whch are computatonally sparse flow and computatonally dense flow. Therefore, based on the consumptons of bandwdth and CPU, there are sx knds of flows n the network, whch are computatonally sparse mce flow, computatonally sparse dog flow, computatonally 4.1 Defnton of Relatve Cost In our work, the relatve costs are defned to ndcate the resource condtons n the network. v,uv bw, vft,u and vcpu,u represent the relatve costs of bandwdth, flow table and CPU, when routng SFCR, respectvely. The relatve costs have recprocal relatonshps to the remanng resources. For example, n Eq. (2a), the numerator represents the maxmum bandwdth capacty n the network, and the denomnator represents the dfference between the remanng bandwdth on lnk uv and the bandwdth consumpton of SFCR. The value range of the relatve cost on each lnk s unform between (1, + ). Accordng to the form of Eq. (2a), the less bandwdth remans on a lnk, the bgger relatve cost of the lnk wll be. And the relatve cost grows very fast, f the remanng bandwdth on a lnk approaches zero. Eq. (3a) and Eq. (4a) abde by the smlar rule of Eq. (2a). Therefore, a lnk or node can be determned as a bottleneck, f ts relatve cost s bg. v bw,uv = v ft,u = v cpu,u = max uv L Cbw uv r bw,uv Cbw uv F bw F bw > µ, (2a) 0 F bw µ. (2b) max Cu ft u V sn r ft,u Cft u 1 F bw < ν, (3a) 0 F bw ν. (3b) max Cu cpu u V fn r cpu,u Ccpu u F cpu F cpu > ω, (4a) 0 F cpu ω. (4b) Addtonally, flow features are consdered n the defnton of relatve cost and the thresholds µ, ν and ω are used to dfferentate the flows wth SFC requests. The flows wth bandwdth consumpton larger than ν are set as elephant flows, and the flows wth bandwdth consumpton smaller than µ are set as mce flows (µ < ν). A flow s set as dog flow, f the bandwdth consumpton s between µ and ν. The flows are set as computatonally dense flows, f the CPU consumpton s larger than ω. And the flows wth CPU consumpton smaller than ω are set as computatonally sparse flows. Snce the bandwdth consumpton of elephant flows s huge and the number of them s small, t s better to pay more attenton on the remanng bandwdth rather than the remanng flow table entres on swtch nodes. On the contrary, due to the fact that the bandwdth consumpton of mce flows s neglgble but the number of them s large, we should pay more attenton on the remanng flow table entres on

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 5 swtch nodes, rather than the remanng bandwdth on lnks. Therefore, for elephant flows, the relatve costs of bandwdth are calculated accordng to Eq. (2a) and the relatve costs of flow table entres are set as 0 due to Eq. (3b). And, for mce flows, the relatve costs of bandwdth are set as 0 due to Eq. (2b) and the relatve costs of flow table entres are calculated accordng to Eq. (3a). For dog flows, because the bandwdth consumpton and the number of them are medum-szed, both the bandwdth and flow table entres should be taken nto account, and the relatve costs are computed accordng to Eq. (2a) and Eq. (3a), respectvely. As the CPU consumpton of computatonally dense flows s much larger than computatonally sparse flows, we should pay more attenton to the CPU consumpton of computatonally dense flows. Therefore, we compute the relatve costs of CPU consumpton for computatonally dense flows accordng to Eq. (4a), and set t as 0 due to Eq. (4b) for computatonally sparse flows. 4.2 BIP Formulaton Next, we formulate the DRP-SFC n detal and the notatons used n ths part are descrbed n TABLE 1. The bnary varable xū,m represents whether VNF request ū V s served by VNF nstance m M. 1, ū s served by VNF nstance m, xū,m = (5) 0, otherwse. We use bnary varable yu m to represent whether VNF nstance m M s hosted on functon node u V fn. 1, VNF nstance m s hosted on u, yu m = (6) 0, otherwse. The next two bnary varables represent whether ū v L traverses lnk uv L or node u V, respectvely: 1, ū v traverses lnk uv,,uv = (7) 0, otherwse. 1, ū v traverses node u,,u = (8) 0, otherwse. For lnk uv, the bandwdth consumpton of SFCR cannot exceed the remanng bandwdth on physcal lnks: F bw ū v L,uv r bw,uvc bw uv, uv L (9) The flow table consumpton of SFCR cannot exceed the remanng flow table entres on physcal nodes, so the followng constrant must be satsfed:,u r ft,u Cft u, u V sn (10) ū v L Besdes, all the CPU consumpton of SFCR cannot exceed the remanng CPU on the selected functon nodes: F cpu xū,myu m r cpu,u Ccpu u, u V fn (11) ū V m M G = (V, L) TABLE 1 Notatons Physcal Network Network graph G wth the sets of nodes V and lnks L, u, v V, uv L. V sn, V fn Sets of swtch nodes and functon nodes V = V sn V fn. Cuv bw,cu ft, Cu cpu r,uv bw, rft,u, rcpu,u v,uv bw, vft,u, vcpu,u F bw Capactes of bandwdth, flow table and CPU. Ratos of remanng bandwdth, flow table entres and CPU, when routng SFCR. Relatve costs of bandwdth, flow table and CPU, when routng SFCR. d,uv Delay on uv L, when routng SFCR. M Set of VNF nstances n the network, m M. Ḡ = ( V, L ) Servce Functon Chans Servce functon graph Ḡ wth the sets of nodes V and lnks L of SFCR, ū, v V, ū v L. S, T The ngress node and egress node of SFCR. Ω The sequence of VNF requests of SFCR ; Ω = {Ω (1), Ω (2),..., Ω (l)}, l = Ω., F cpu, F delay The bandwdth and CPU consumptons and maxmum tolerated delay of SFCR. Bnary Varables xū,m Whether ū s served by m M for SFCR. y m u Whether m M s hosted on u V fn.,uv Whether ū v L traverses uv L for SFCR.,u Whether ū v L traverses u V for SFCR. For SFCR, the end-to-end delay must be smaller than the maxmum tolerated delay: uv L d,uv,uv F delay (12) ū v L Once a lnk s selected, both the end ponts of ths lnk must be selected as well: 1,,uv = 1, u, v V, uv L, ū v L, (13) 0, otherwse.,u,v = Eq. (14) ensures that the routng path of SFCR s consecutve and cannot be splt: v V (,uv,vu) = ū v L 1, u = S, 1, u = T, 0, otherwse. (14) In order to ensure that the routng path traverses all the functon nodes whch contan the selected VNF nstances for SFCR, the next equaton must be satsfed: xū,myu m,u, u V fn, ū V, ū v L, m M (15)

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 6 Eq. (16) s used to ensure that each VNF request ū V \ {S, T } on Ḡ can only be served by one VNF nstance: m M xū,m = 1, ū s served by VNF nstance m, 0, otherwse. (16) Due to the fact that each VNF nstance can only be hosted on one functon node, then the next unqueness constrant s satsfed: u V fn y m u = 1, m M (17) In the paper, we use the relatve costs v,uv bw, vft,u and v cpu,u to evaluate whether the lnks and nodes are congested and to route flows wth SFC requests based on flow features dfferentatedly. The resource consumpton cost of the routng path for SFCR s calculated as below. R = uv L ū v L v bw,uv,uv + u V sn u V fn ū V m M v ft,u,u+ ū v L v cpu,u xū,my m u (18) As the relatve costs are used to ndcate the resource condtons on lnks and nodes, we also use the sum of relatve costs to ndcate the resource condton of a path. If the resource consumpton cost of a path s small, we can determne that there are abundant remanng resources and small number of bottleneck lnks or nodes on ths path, and routng flows wth SFC requests on ths path wll not ncur network congeston. On the contrary, f the resource consumpton cost of a path s very bg, we can determne that there exst bottleneck lnks or nodes on ths path, and another path wth smaller resource consumpton cost should be found to avod network congeston and acheve load balancng. Therefore, our objectve n Eq. (19) s to mnmze the resource consumpton costs for flows wth SFC requests under the constrants of Eq. (9)-(17). Mnmze R s.t. Eq.(9) (17) 5 PROPOSED ALGORITHM (19) In ths secton, we propose a novel routng algorthm named RA-RA. RA-RA can effcently solve the DRP-SFC by transformng the orgnal network graph to Logcal Functon Graph (LFG). When executng RA-RA, we frst need to calculate the relatve costs on the lnks, swtch nodes and functon nodes. Based on the relatve costs and the sequence of VNF requests, we search for the canddate VNF nstances n the network and arrange them n order to construct the LFG. LFG s a dgraph, where each path from the ngress node to egress node along wth the lnk drecton satsfes the predefned order. Fnally, the routng paths for flows wth SFC requests are obtaned based on a modfed k-shortest path algorthm. 5.1 Constructng LFG In the paper, we solve the DRP-SFC by constructng LFG. LFG s a dgraph whch s comprsed of the ngress node, egress node and canddate VNF nstances for a flow wth SFC request. Then, we construct the LFG for SFCR accordng to the orgnal network graph. The LFG s denoted as Ĝ = ( V, L ). û, v V represent two nodes and û v L stands for a lnk on LFG. When constructng LFG, the frst process s to fnd the ngress node, egress node and all the canddate VNF nstances and arrange them as the predefned order of SFCR. Above all, the ngress node S s selected and placed n the 1 st column. Then, due to the sequence of VNF request Ω = {Ω (1), Ω (2),..., Ω (l)}, l = Ω, the VNF nstances belongng to the same type of Ω (1) are selected and placed n the 2 nd column. Next, we execute the same operaton sequentally to the VNF nstances whch belong to the same type of Ω (2) to Ω (l). Fnally, the egress node T s found and placed n the (l + 2) th column. When fnshng the frst process, the second process s to produce the lnks on LFG. For each two adjacent columns, we connect each node n the last column to all the nodes n the next column. And the lnk drecton s from the node n the last column to the node n the next column. On the LFG, each lnk s correspondng to a path whch s obtaned by executng SP algorthm such as Djkstra on network graph. Here, the relatve costs defned n Eq. (2a)- (4b) are set as the costs on physcal lnks and nodes. For two VNF nstances hosted on dfferent functon nodes, a flow wth SFC request needs to traverse a complete path to pass through these two VNF nstances. Therefore, the lnk connectng the VNF nstances on dfferent functon nodes s correspondng to a path on network graph. Nevertheless, because an functon node s allowed to deploy multple VNF nstances, for the SFC request served by the VNF nstances on one functon node, the lnk wll be correspondng to a functon node on the network graph. Accordng to the relatonshp between network graph and LFG, the relatve costs of bandwdth, flow table and CPU on network graph are all transformed to the lnks on LFG. For SFCR, the lnk cost of û v s set as v,û v. The bnary varables, zû v,uv and zû v,u, ndcate whether û v L traverses uv L and u V, respectvely. zû v,uv = 1, f û v L traverses uv L and zû v,u = 1, f û v L traverses u V. In addton, an mappng functon Π( ) s used to obtan the functon node that a VNF nstance s deployed on and v cpu,π(û) ndcates the relatve cost of CPU on the functon node Π(û). Eq. (20) shows the calculaton of v,û v, where the three parts n Eq. (20) represent the relatve costs of bandwdth, flow table and CPU, respectvely. v,û v = v,uvzû v bw,uv + v ft,u zû v,u+ uv L u V sn v cpu,π(û) + (20) vcpu,π( v), û v 2 L, û, v V On LFG, the path wth the mnmum resource consumpton cost s calculated based on Eq. (21).,û v s a bnary varable and represents whether ū v L on Ḡ traverses

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 7 Algorthm 1: RA-RA 1:Input: Network graph: G = (V, L), Resource capactes: Cuv bw, Cft u, Ccpu u, Remanng ratos: r,uv bw, rft,u, rcpu,u, Flow wth SFC request: SFCR, Thresholds: µ, ν, ω, Iteraton tmes: K; 2:Output: Φ; 3:Intalze k = 1; 4:Set Φ and Φ as ø; 5:Remove all the lnks and nodes wth less resources to serve SFCR ; 6:Fnd max uv L Cbw Cu ft u V sn uv, max 7:Calculate v,uv bw, vft,u, vcpu,u 8:Construct LFG; Functon 1 9:whle k K do and max Cu cpu on G; u V fn based on Eq. (2a)-(4b); 10: Φ = Calculate the k th shortest path on LFG based on Eq. (21); 11: f Isempty( Φ) then 12: Φ = Transform Φ from Ĝ to G; 13: f Φ satsfes all the constrants of Eq. (9)-(17) then 14: Receve SFCR ; 15: Update r bw,uv, rft,u 16: return Φ; 17: end 18: else 19: return Routng Faled; 20: end 21: k = k + 1; 22:end 23:return Routng Faled; and rcpu,u ; Functon 1: Construct LFG 1:Input: Network graph: G = (V, L), Relatve costs: v,uv bw, vft,u, vcpu,u, Flow wth SFC request: SFCR ; 2:Output: Ĝ = ( V, L ); 3:for j = 1 : l do 4: η(j + 1) = Fnd the VNF nstances belongng to the same type of Ω (j) on G; 5:end 6:η(1) = S ; 7:η(l + 2) = T ; 8:for j = 1 : (l + 1) do 9: for û η(j) do 10: for v η(j + 1) do 11: Calculate v,û v based on Eq. (20); Djkstra 12: end 13: end 14:end 15:Construct Ĝ = ( V, L ) based on v,û v, û v L ; 16:return Ĝ = ( V, L ); the path wth the mnmum resource consumpton cost n Eq. (21) for each flow wth SFC request. Addtonally, notng that, on LFG, there s only one path for each combnaton of VNF nstances whch can satsfy the demand of the SFC request, the path set of LFG s only a subset of orgnal network graph. Therefore, LFG provdes a smplfed vew of the network topology, whch makes the path computaton effcent by elmnatng sub-optmal solutons. SFCR : A Ω (1) Ω (2) Ω (3) Ω (4) J û v L on Ĝ. And,û v = 1, only when ū v L traverses û v L. Mnmze v,û v,û v (21) ū v L û v L As for LFG, all the canddate VNF nstances are arranged n the predefned order. Then, each path from the ngress node to egress node satsfes the demand of SFC request. If the path derved from Eq. (21) satsfes all the constrants of Eq. (9)-(17), we wll get the fnal soluton to route SFCR by transformng ths path to network graph. For example, Fg. 3 shows the LFG for SFCR. In the fgure, the flow starts from the ngress node A and needs to traverse the nstances of Ω (1), Ω (2), Ω (3), and Ω (4) n order before arrvng at node J. Assumng that the path wth the mnmum resource consumpton cost calculated n Eq. (21) s A VNF11 VNF21 VNF31 VNF41 J, then due to Fg. 1, we know that SFCR wll be processed by VNF11 and VNF21 on B and VNF31 and VNF41 on E. On network graph, f the path derved from LFG s A B VNF11 VNF21 E VNF31 VNF41 G J and all the constrants of Eq. (9)-(17) are satsfed, then ths path s selected to route SFCR. On LFG, snce the resource consumpton cost s used to ndcate the resource condton of a path, we can acheve load balancng and avod bottleneck and congeston by fndng S VNF11 VNF12 VNF13 VNF21 VNF22 VNF23 Fg. 3. Solvng DRP-SFC on LFG 5.2 RA-RA Routng Algorthm VNF31 VNF32 VNF33 VNF41 VNF42 VNF43 RA-RA provdes an effcent way to acheve load balancng and dfferentated routng for flows wth SFC requests. There are two steps ncluded n RA-RA. The frst step s to construct the LFG accordng to the relatve costs, and the second step s to run a modfed k-shortest path algorthm on LFG to fnd the path that has the mnmum resource consumpton cost and satsfes all the constrants n Eq. (9)- (17) for each flow wth SFC request. The pseudocode of RA-RA s descrbed below. Frstly, we ntalze the current teraton tmes k and empty Φ and Φ whch are used to record the paths on network graph and LFG, respectvely (lnes 3-4 n Algorthm 1). Then, all T

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 8 the nodes and lnks wth less resources to serve SFCR are removed from the network graph (lne 5 n Algorthm 1). Next, we compute the maxmum capactes of bandwdth, flow table and CPU n the network (lne 6 n Algorthm 1). After that, we calculate the values of v,uv bw, vft,u and vcpu,u whch represent the relatve costs of bandwdth, flow table and CPU (lne 7 n Algorthm 1). Functon 1 presents the constructon of LFG based on the relatve costs v,uv bw, vft,u and vcpu,u. The VNF nstances belongng to the same type of Ω are recorded n η sequentally (lnes 3-5 n Functon 1). Accordng to LFG, we add the ngress node S and egress node T to η(1) and η(l + 2) (lnes 6-7 n Functon 1). Then, the SP algorthm, Djkstra, s used to produce the lnks for LFG (lnes 8-14 n Functon 1). The LFG s constructed based on the values of v,û v (lne 15 n Functon 1). And we return the LFG n lne 16 of Functon 1. We modfy the k-shortest path algorthm to get the soluton on LFG (lnes 9-22 n Algorthm 1). K represents the maxmum teraton tmes. We frst compare whether the current teraton tmes k s bgger than K (lne 9 n Algorthm 1). If k K, the path wth the frst mnmum resource consumpton cost s calculated on LFG and f found, ths canddate path wll be recorded n Φ (lne 10 n Algorthm 1). Then we check whether Φ s empty (lne 11 n Algorthm 1). If there s canddate path n Φ, ths path wll be transformed from LFG to network graph and recorded n Φ (lne 12 n Algorthm 1). Next, the path recorded n Φ wll be checked whether t satsfes all the constrants of Eq. (9)-(17) (lne 13 n Algorthm 1). If all the constrants are satsfed, ths flow wll be receved and routed by the path recorded n Φ (lne 14 n Algorthm 1). After that, we update the ratos of remanng bandwdth, flow table entres and CPU on lnks, swtch nodes and functon nodes, respectvely (lne 15 n Algorthm 1). If there s no avalable path derved from LFG, ths flow s dened to be served (lne 19 n Algorthm 1). If the path wth the frst mnmum resource consumpton cost returned from LFG cannot satsfy all the constrants of Eq. (9)-(17), then we set k = k + 1 to fnd the path wth the next mnmum resource consumpton cost on LFG (lne 21 n Algorthm 1). The RA-RA wll stop and deny ths flow untl k exceeds the maxmum teraton tmes K (lne 23 n Algorthm 1). 5.3 Complexty Analyss In ths subsecton, we gve a detaled complexty analyss of RA-RA n the worst stuaton. When executng RA-RA, frst, we need to calculate the relatve costs on lnks and nodes and the complexty s O( V + L ). In the DRP-SFC, the worst stuaton s that each functon node deploys all types of VNF nstances. Under ths crcumstance, we need to execute the SP algorthm at most 1 2 V fn 2 + 2 V fn tmes to produce all the lnks on LFG. The complexty of the SP algorthm on G = (V, L) s O( L + V log V ), then the complexty of constructng LFG results n O( V fn 2 ( L + V log V )). On LFG, there are at most l V fn +2 nodes and (l 1) V fn 2 +2 V fn lnks, then the worst stuaton s to terate K tmes to get the soluton, whch runs n O(Kl 2 V fn 2 ( V fn + logl)). The complexty of path transformaton and resource update s O(1). Because l, whch Fg. 4. CORONET CONUS Topology represents the length of SFC request, s a fnte and small value, then the complexty to solve the DRP-SFC by RA-RA at the worst stuaton s O(Kl 2 V fn 3 + V fn 2 ( L + V log V )). 6 PERFORMANCE EVALUATION Ths secton depcts the smulaton settngs and the performance comparson between the RA-RA and exstng algorthms ncludng the COATS [24], SP [28] and Egendecomposton [23] algorthms. All the algorthms are mplemented wth MATLAB 2016a and performed on a computer wth Intel(R) Core(TM) 7-4790 CPU 3.60 GHz and 32 GB RAM. 6.1 Smulaton Settngs 6.1.1 Topology Settngs The network graph we use s a US carrer networks topology named CORONET CONUS Topology (shown n Fg. 4), whch composes of 60 nodes and 79 lnks [29], [37]. In the network, we select functon nodes based on the node degree. All the nodes are sorted n descendng order accordng to the node degree and the top 30% of nodes are set as functon nodes. There are 20 types of VNF nstances deployed n the network. And we deploy 8 types of VNF nstances on each functon node. Therefore, there are 18 functon nodes and 144 VNF nstances belongng to 20 dfferent VNF types [38] n the network. The bandwdth capacty of each lnk s set as 1200 Mbps [29]. The capactes of flow table and CPU on the swtch nodes and functon nodes are set as 800 unts [39] and 8000 MIPS [38], respectvely. In the smulaton, d,uv s calculated accordng to Eq. (22), where the frst part represents the queung delay and the second part d prop uv represents the propagaton delay on lnk uv [27]. d tx uv stands for the transmsson delay on lnk uv and we set t as 10 us [40]. The propagaton delay d prop uv s calculated accordng to the length between the nodes u and v. All the network parameters used n ths paper are descrbed n TABLE 2.,uv r,uv bw d,uv = 1 rbw d tx uv + d prop uv, uv L (22)

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 9 6.1.2 SFC Request and Flow Dstrbuton Settngs For each flow, we set F bw as a constant whch randomly falls n (0, 10] Mbps. F cpu s proportonal wth F bw. The unt of F cpu s MIPS and F cpu equals to the product between the value of F bw and a constant dstrbuted n 0 and 10 randomly [32]. F delay s set between 50-100 ms [41]. We classfy all the flows nto sx knds based on the bandwdth and CPU consumptons [31], [32]. The dstrbuton of flows satsfes the law of two to eght, where elephant flows account for 20%, mce flows account for 50% and dog flows account for 30%. The thresholds to dfferentate flows wth SFC requests are set as µ = 0.1 Mbps, ν = 1 Mbps, ω = 5 MIPS. The bandwdth and CPU consumptons of these flows are set as below: computatonally sparse mce flow: The CPU consumpton s no more than 5 MIPS and the bandwdth consumpton s between 0 and 0.1 Mbps. computatonally sparse dog flow: The CPU consumpton s no more than 5 MIPS and the bandwdth consumpton s between 0.1 and 1 Mbps. computatonally sparse elephant flow: The CPU consumpton s no more than 5 MIPS and the bandwdth consumpton s between 1 and 10 Mbps. computatonally dense mce flow: The CPU consumpton s more than 5 MIPS and the bandwdth consumpton s between 0 and 0.1 Mbps. computatonally dense dog flow: The CPU consumpton s more than 5 MIPS and the bandwdth consumpton s between 0.1 and 1 Mbps. computatonally dense elephant flow: The CPU consumpton s more than 5 MIPS and the bandwdth consumpton s between 1 and 10 Mbps. Moreover, the length of SFC request n ths smulaton s 4, and the maxmum teraton tmes of RA-RA s 5. In the smulaton, each experment s repeated 20 tmes. 6.1.3 Introducton of Comparng Algorthms In the smulaton, the performance of RA-RA s compared wth the COATS, SP and Egendecomposton algorthms. SP s realzed and ntegrated n the OpenDaylght platform whch s the largest open source SDN controller to schedule flows wth SFC requests n servce functon selecton framework. It s worth notng that we use Egen to represent the Egendecomposton algorthm n the paper. Before ntroducng the smulaton results, we would lke to gve a bref descrpton to these comparng algorthms. COATS: Each lnk keeps a cost whch s calculated based on the remanng bandwdth on the lnk. Then COATS constructs a layered graph and selects the path wth the lowest cost to route a flow wth SFC request. SP: The Djkstra algorthm s used to select approprate VNF nstances and route a flow wth SFC request from the ngress node to egress node passng through the selected VNF nstances wth the shortest path sequentally. Egen: Frst, Egen constructs the adjacent matrxes of SFC request and network topology, respectvely. Then, the adjacent matrx of SFC request s extended wth TABLE 2 Smulaton Parameter Settngs Descrpton Network topology Value CORONET CONUS Topology Proporton of functon node 30% Proporton of mce, dog, and elephant flows 50%, 30% and 20% Total VNF types 20 Number of VNF types per functon node 8 Maxmum teraton tmes of RA-RA 5 Parameters Descrpton Value Cuv bw Bandwdth capacty on lnk uv 1200 Mbps Cu ft Flow table capacty on swtch node u 800 unts C cpu u CPU capacty on functon node u 8000 MIPS F bw Bandwdth consumpton (0, 10] Mbps F cpu CPU consumpton (0, 100] MIPS F delay Maxmum toleraton delay [50, 100] ms d tx uv Transmsson delay on lnk uv 10 us d prop uv Propagaton delay on lnk uv µ, ν, ω Thresholds of flow classfcaton defned on network topology 0.1 Mbps, 1 Mbps and 5 MIPS M Total VNF nstances 144 Ω Length of SFC request 4 the same dmenson of network topology s. Fnally, the Umeyama s egendecomposton approach and wdest-shortest path routng algorthm are used to compute the optmal matchng between the SFC request and network topology. 6.2 Smulaton Results 6.2.1 Comparson of Average Acceptance Rate, Throughput and Hop Count In Fg. 5, we present the comparson between the RA-RA and COATS, SP and Egen n terms of average acceptance rate, throughput and hop count. Fg. 5(a) shows the average acceptance rate of these algorthms. Average acceptance rate reflects the flows served by the network accountng for the total arrval ones. In the smulaton, RA-RA performs the best, whch gets about 10% hgher n average acceptance rate than that of COATS. And the performance of COATS s about 5% hgher than that of SP and 20% hgher than that of Egen. In RA- RA, we acheve load balancng among multple knds of resources and manage to route flows wth SFC requests dfferentatedly. Therefore, RA-RA does better than other comparng algorthms. As for COATS, on the one hand, t s a varaton of the SP algorthm whch s benefcal to reduce the consumptons of bandwdth and flow table n the network. On the other hand, COATS balances the

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 10 A v e r a g e A c c e p t a n c e R a t 1 0 0 8 0 R A - R A C O A T S S P E g e n 0 0 0 0 0 0 0 8 0 0 0 N u m b e r o f F l o w s w t h S F C R e q (a) Average acceptance rate A v e r a g e T h r o u g h p u t ( M b 0 0 5 0 0 0 0 0 3 0 0 0 0 0 1 0 0 0 R A - R A C O A T S S P E g e n 0 0 0 0 0 0 0 0 8 0 0 0 N u m b e r o f F l o w s w t h S F C R e q (b) Average throughput C D F ( % ) 1 0 0 8 0 R A - R A C O A T S P E g e n 0 0 1 0 3 0 H o p C o u n t (c) Average hop count Fg. 5. The comparson of average acceptance rate, throughput and hop count C D F ( % ) 1 0 0 R A - R A _ 0 0 C O A T S _ 0 0 S P _ 0 0 E g e n _ 0 0 8 0 R A - R A _ 3 5 0 0 C O A T S _ 3 5 0 0 S P _ 3 5 0 0 C D F ( % ) 1 0 0 R A - R A _ 0 0 C O A T S _ 0 0 S P _ 0 0 E g e n _ 0 0 8 0 R A - R A _ 3 5 0 0 C O A T S _ 3 5 0 0 S P _ 3 5 0 0 C D F ( % ) 1 0 0 R A - R A _ 0 0 C O A T S _ 0 0 S P _ 0 0 E g e n _ 0 0 8 0 R A - R A _ 3 5 0 0 C O A T S _ 3 5 0 0 S P _ 3 5 0 0 0 0 0 0 0 0 8 0 0 1 0 0 0 1 0 0 0 0 8 0 0 R e m a n n g B a n d w d t h ( M b p s R e m n n g F l o w T a b l e E n t r e s (a) CDF of average remanng bandwdth (b) CDF of average remanng flow table entres 0 0 0 0 0 0 0 0 8 0 0 0 R e m a n n g C P U ( M I P S ) (c) CDF of average remanng CPU Fg. 6. CDF of average remanng bandwdth, flow table entres and CPU consumpton of bandwdth by defnng the cost due to the remanng bandwdth on lnks. Comparng wth the SP and Egen, COATS can serve more flows n the network. However, for the reason that COATS neglects the capactes of flow table and CPU, t wll result n the resource exhauston on the nodes wth fewer resources, whch leads to worse performance comparng wth RA-RA. As for Egen, the Umeyama s egendecomposton approach cannot ensure to get the optmal path and the wdest-shortest path routng algorthm results n longer paths n the channg soluton, so there are more resource consumptons, whch leads to the worst performance n the smulaton. Fg. 5(b) presents the average throughput of these four algorthms. The average throughput reflects the total bandwdth of flows receved successfully n the network. In ths part, the throughput of RA-RA s about 700 Mbps hgher than that of COATS, 1500 Mbps hgher than that of SP and 3000 Mbps hgher than that of Egen. Egen results n the lowest throughput, because the longer paths consume more bandwdth and flow table entres. The performances of COATS and the SP are lower than that of RA-RA because of unbalanced utlzaton of the resources n the network. On the contrary, RA-RA realzes the load balancng and dfferentated routng by fne-graned flow schedulng. Therefore, RA-RA can avod bottlenecks on the lnks and nodes, whch results n hgher throughput comparng wth other algorthms. Fg. 5(c) presents the average hop count of these four algorthms. The average hop count represents the number of nodes a flow wth SFC request needs to traverse before reachng the egress node. Accordng to the result, SP performs the best, the COATS surpasses RA-RA and Egen s the worst. Due to the fact that Egen tends to route flows wth SFC requests wth long paths, the paths wth hop counts dstrbutng from 15 to 35 account for about 55%, whch s only about 15% for RA-RA, 8% for COATS and 3% for SP. When routng flows wth SFC requests, RA-RA takes bandwdth, flow table, CPU and flow features nto consderaton at the same tme, so t ncurs a lttle longer paths than COATS s. As SP always fnds the paths wth the shortest hop counts for flows, the performance of SP n ths smulaton s the best among other algorthms. In addton, from Fg. 5(c), we get that the RA-RA, COATS and SP are prone to produce short paths, whch are benefcal to reduce end-to-end delay and satsfy low-delay demands. 6.2.2 Comparson of Average Remanng Bandwdth, Flow Table Entres and CPU The CDF curves n Fg. 6 present the comparson of these algorthms n terms of average remanng bandwdth, flow table entres and CPU. Fg. 6(a) presents the CDF of the average remanng bandwdth on lnks when 2000 and 3500 flows wth SFC requests are successfully receved. The performance of

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 11 A v e r a g e A c c e p t a n c e R a t 7 0 5 0 3 0 R A - R A C O A T S S P E g e n 4 6 8 1 0 1 2 N u m b e r o f V N F T y p e s p e r F u n c t (a) Average acceptance rate vs number of VNF types per functon node A v e r a g e A c c e p t a n c e R a t 8 0 7 0 5 0 3 0 R A - R A C O A T S S P E g e n 1 2 3 4 5 L e n g t h o f S F C R e q u e s t 1 0 % % 3 0 % % (b) Average acceptance rate vs length of SFC (c) Average acceptance rate vs proporton of request functon node A v e r a g e A c c e p t a n c e R a t 7 0 5 0 3 0 1 0 R A - R A C O A T S S P E g e n P r o p o r t o n o f F u n c t o n N o d e Fg. 7. Comparson of average acceptance rate n dfferent scenaros COATS s hgher than that of all the other algorthms. The performance of RA-RA outperforms that of SP, and Egen falls to balance the bandwdth consumpton on lnks. Because of longer paths and unbalanced resource utlzaton for Egen, when recevng 2000 flows, the proporton of the paths wth remanng bandwdth less than 600 Mbps s about 37%, whch s only 12% for SP, 10% for RA-RA and 3% for COATS. When recevng 3500 flows, the proporton of bottleneck lnks of SP ncreases to 10%, whle there are no bottleneck lnks for RA-RA and COATS. Ths s because SP fals to acheve load balancng on the bandwdth, whch leads to network congeston. Nevertheless, when recevng 3500 flows, for RA-RA, there are about 40% of lnks wth the remanng bandwdth between 400 Mbps and 800 Mbps, whle, for COATS, t s about 60%. Therefore, the dstrbuton of remanng bandwdth for COATS on lnks s more balanced comparng wth that of RA-RA. The reason s that, for COATS, only the bandwdth on lnks s consdered when routng flows wth SFC requests. And RA-RA needs to balance the consumptons of bandwdth, flow table and CPU on lnks and nodes, so COATS can get better performance n ths smulaton. Fg. 6(b) llustrates the CDF of average remanng flow table entres on swtch nodes. Here, the performance of RA-RA outperforms that of other algorthms and the performances of COATS and SP are both better than that of Egen. Because of more resource consumpton, when recevng 2000 flows, there are about 5% of swtch nodes becomng bottlenecks for Egen. When recevng 3500 flows, there are about 9% and 12% of swtch nodes becomng bottlenecks for COATS and SP, whle there s no swtch node runnng out of flow table entres for RA-RA. Moreover, for COATS and SP, there are about 40% of swtch nodes wth remanng flow table entres fewer than 300 and 400 unts, respectvely, whle the proportons for RA-RA are only about 22% and 35%. Ths s because the relatve costs defned n Eq. (2a)-(4b) can ndcate the resource condtons on lnks and nodes. If the resources on lnks or nodes are gong to be used up, the relatve costs wll ncrease quckly, whch can protect them from beng exhausted. So RA-RA can effcently avod congeston and acheve load balancng by mnmzng the resource consumpton costs for flows wth SFC requests n the network. Fg. 6(c) descrbes the CDF of average remanng CPU on functon nodes. In ths part, the performance of RA-RA surpasses that of other algorthms, whle Egen dose better than COATS and the SP performs the worst. When recevng 2000 flows, there are no bottleneck nodes that are short of CPU resource for these four algorthms. However, when recevng 3500 flows, the bottleneck nodes of COATS and SP ncrease obvously. Ths s because both the COATS and SP neglect to optmze the CPU on functon nodes. Furthermore, for COATS and SP, the utlzaton of CPU on functon nodes s unbalanced comparng wth RA-RA. As shown n Fg. 6(c), when recevng 3500 flows, the proporton of functon nodes of whch remanng CPU are from 2000 to 4000 MIPS s about 70% for RA-RA, whch s only about 35% for COATS and SP. Though, Egen gets balanced dstrbuton of CPU resource, the huge amounts of bandwdth and flow table consumptons lead to low network performance. 6.2.3 Comparson of Average Acceptance Rate n Dfferent Scenaros In Fg. 7, we change the number of VNF types per functon node, the length of SFC requests and the proporton of functon node to compare the average acceptance rate of RA-RA wth comparng algorthms. The average acceptance rate s evaluated under 8000 flows wth SFC requests. Fg. 7(a) shows the average acceptance rate under d- fferent number of VNF types per functon node. In the smulaton, the average acceptance rates of these tested algorthms grow quckly when ncreasng the number of VNF types per functon node. Ths s because, when the number of VNF types on functon nodes ncreases, t s more possble for SFC requests to be served by one functon node nstead of beng splt on the VNF nstances dstrbuted on several functon nodes. Therefore, t s benefcal for the reducton of bandwdth and flow table consumptons by ncreasng the number of VNF types per functon node. In addton, for RA- RA, COATS and SP, the average acceptance rates grow slowly, when ncreasng the number of VNF types per functon node from 10 to 12. The reason s that, comparng wth the bandwdth and flow table entres, CPU on functon nodes becomes scarce whch prevents the network from recevng flows wth SFC requests. Ths can be proven by the curve of Egen. Generally, comparng wth CPU on functon nodes, the

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 12 bandwdth and flow table entres are more scarce for Egen. When ncreasng the number of VNF types, the consumptons of bandwdth and flow table reduce and there are more flows wth SFC requests can be served n the network. Therefore, the curve of Egen grows quckly wth the ncreasement of VNF types per functon node. Fg. 7(b) shows the average acceptance rate under d- fferent length of SFC request. The longer the length of SFC request s, the more resource consumpton wll be n the network. Therefore, these curves drop quckly when ncreasng the length of SFC request. Due to the fact that COATS and SP neglect to balance the consumptons on bandwdth, flow table and CPU at the same tme and cannot dfferentatedly route flows wth SFC requests based on flow features, the performance gaps between these two algorthms and RA-RA become obvous, when the length of SFC request ncreases. Fg. 7(c) shows the average acceptance rate under dfferent proporton of functon nodes. More proporton of functon nodes means more flow table entres and more CPU resources n the network. When the proporton of functon node s small, there are few VNF nstances deployed n the network. Then, there are few choces for an SFC request to select VNF nstances to satsfy ts predefned order. Therefore, when the proporton of functon node stays low between 10% and 20%, the performance gaps among these tested algorthms are small. And the performance gaps become obvous, when ncreasng the proporton of functon nodes between 20% to 40%. 7 CONCLUSION In the paper, to make a dfferentated routng strategy wth the optmal dynamc SFC formaton and load balancng among multple resources for flows wth SFC requests, we study the DRP-SFC n SDN and NFV-enabled network. Ths problem s formulated as a BIP model wth the am to mnmze the resource consumpton costs for flows wth SFC requests. In order to solve the DRP-SFC, we have proposed a novel routng algorthm named RA-RA. RA- RA makes effcent selecton of VNF nstances and fnd the assocated paths for flows wth SFC requests by transformng the network graph to LFG. In RA-RA, relatve costs are used to balance the resource consumptons and avod congeston n the network. Moreover, we take the resource preference as flow features and classfy all the flows nto dfferent knds to acheve dfferentated routng for flows wth SFC requests. The performance evaluaton shows that RA-RA can effcently solve the DRP-SFC and obtan hgher network performance n terms of average acceptance rate, throughput, hop count and load balancng, comparng wth other algorthms n exstng lteratures. In the future work, we ntend to extend our approach n a number of ways. We want to extend our approach to deal wth VNF deployment problem n both ISP network and datacenter. We ntend to desgn hgh-performance dfferentated routng algorthm for flows wth SFC requests to reduce the computaton complexty and enhance the tmeeffcency of routng computaton. And we also want to acheve fast falure-reslence for flows wth SFC requests n SDN and NFV-enabled network n the future. ACKNOWLEDGMENTS Ths work s supported by the Natonal Natural Scence Foundaton of Chna (NSFC) under Grant No. 61671420, 61672484. REFERENCES [1] R. Mjumb, J. Serrat, et al., Network functon vrtualzaton: Stateof-the-art and research challenges, IEEE Commun. Surveys Tuts., vol. 18, no. 1, pp. 236 262, 2016. [2] C. Pham, N. H. Tran, and S. Ren, Traffc-aware and Energy-effcent vnf Placement for Servce Channg: Jont Samplng and Matchng Approach, IEEE Trans. Serv. Comput., 2017. [3] OpenFlow Swtch Specfcaton: Verson 1.5.1. [Onlne]. Avalable: https://www.opennetworkng.org/mages/stores/downloads/sdn-resources/onf-specfcatons/openflow/openflow-swtchv1.5.1.pdf. [4] J. Martns, M. Ahmed, C. Racu, et al., ClckOS and the art of network functon vrtualzaton, n Proc. the 11th USENIX Conference on Networked Systems Desgn and Implementaton., pp. 459 473, 2014. [5] J. Sherry, S. Hasan, C. Scott, et al., Makng mddleboxes someone else s problem: network processng as a cloud servce, SIGCOMM Comput. Commun. Rev, vol. 42, no. 4, pp. 13 24, 2012. [6] J. Lu, Y. L, and Y. Zhang, Improve Servce Channg Performance wth Optmzed Mddlebox Placement, IEEE Trans. Serv. Comput., vol. 10, no. 4, pp. 560 573, 2017. [7] J. Halpern and C. Pgnataro, Servce Functon Channg (SFC) Archtecture, n IETF Internet-Draft, draft-etf-sfc-archtecture-11, RFC 7665, 2015. [8] P. Qunn and T. Nadeau, Problem statement for servce functon channg, n Internet Requests for Comments, RFC Edtor, RFC 7498, July 2015. [Onlne]. Avalable: http://www.rfcedtor.org/rfc/rfc7498.txt. [9] W. John, K. Pentkouss, G. Agapou, et al., Research drectons n network servce channg, n Proc. IEEE SDN4FNS, pp. 1 7, 2013. [10] D. L, P. Hong, K. Xue, and J. Pe, Vrtual Network Functon Placement Consderng Resource Optmzaton and SFC Requests n Cloud Datacenter, IEEE Trans. Parallel Dstrb Syst, 2018. [11] J. G. Herrera and J. F. Botero, Resource Allocaton n NFV: A Comprehensve Survey, IEEE Trans. Netw. Servce Manag, vol. 13, no. 3, pp. 518 532, 2016. [12] H. Huang, S. Guo, P. L, et al., Cost Mnmzaton for Rule Cachng n Software Defned Networkng, IEEE Trans. Parallel Dstrb Syst, vol. 27, no. 4, pp. 1007 1016, 2016. [13] A. Dxt, F. Hao, S. Mukherjee, et al., Elastcon: an elastc dstrbuted sdn controller, n Proc. ACM ANCS, pp. 17 28, ACM, 2014. [14] T. Wood, K. K. Ramakrshnan, J. Hwang, et al., Toward a softwarebased network: ntegratng software defned networkng and network functon vrtualzaton, IEEE Network Magazne, vol. 29, no. 3, pp. 36 41, 2015. [15] W. Haeffner, J. Napper, et al., Servce functon channg use cases n moble networks, n IETF Internet Draft draft-etf-sfc-use-casemoblty-07, 2016. [16] W. Lu, H. L, O. Huang, et al., Servce functon channg (sfc) general use cases, n IETF Internet-Draft draft-lu-sfc-use-cases-08, 2014. [17] D. Bhamare, R. Jan, M. Samaka, et al., A survey on servce functon channg, J. Netw. Comput. Appl., vol. 75, pp. 138 155, 2016. [18] G. Cheng, H. Chen, H. Hu, et al., Enablng network functon combnaton va servce chan nstantaton, Computer Networks, vol. 92, no. 2, pp. 396 407, 2015. [19] I. Trajkovska, M.-A. Kourts, C. Sakkas, et al., SDN-based servce functon channg mechansm and servce prototype mplementaton n NFV scenaro, Comput. Stand. Interfaces, vol. 54, no. 4, pp. 247 265, 2017. [20] R. Yu, G. Xue, V. T. Klar, et al., Network functon vrtualzaton n the mult-tenant cloud, IEEE Network, vol. 29, no. 3, pp. 42 47, 2015. [21] A. M. Medhat, G. Carella, C. Lck, et al., Near optmal servce functon path nstantaton n a mult-datacenter envronment, n Proc. IEEE CNSM, pp. 336 341, 2015.

RESOURCE AWARE ROUTING FOR SERVICE FUNCTION CHAINS IN SDN AND NFV-ENABLED NETWORK 13 [22] H. Oh, D. Yu, Y.-H. Cho, et al., Desgn of an Effcent Method for Identfyng Vrtual Machnes Compatble wth Servce Chan n a Vrtual Network Envronment, Internatonal Journal of Multmeda and Ubqutous Engneerng, vol. 9, no. 11, pp. 197 208, 2014. [23] M. Mechtr, C. Ghrb, and D. Zeghlache, A Scalable Algorthm for the Placement of Servce Functon Chans, IEEE Trans. Netw. Servce Manag, vol. 13, no. 3, pp. 533 546, 2016. [24] Z. Cao, M. Kodalam, T. V. Lakshman, et al., Traffc steerng n software defned networks: plannng and onlne routng, ACM SIGCOMM Comput. Commun. Rev., vol. 44, no. 4, pp. 65 70, 2014. [25] M. F. Bar, S. R. Chowdhury, R. Ahmed, et al., Orchestratng vrtualzed network functons, IEEE Trans. Netw. Servce Manag., vol. 13, no. 4, pp. 725 739, 2016. [26] A. Vterb, Error bounds for convolutonal codes and an asymptotcally optmum decodng algorthm, IEEE Trans. Inform. Theory, vol. 13, no. 2, pp. 260 269, 1967. [27] A. Dwarak and T. Wolf, Adaptve servce-chan routng for vrtual network functons n software-defned networks, n Proc. ACM HotMIddlebox, pp. 32 37, ACM, 2016. [28] OpenDaylght. [Onlne]. Avalable: https://www.opendaylght.org/. [29] H. Huang, S. Guo, J. Wu, et al., Jont mddlebox selecton and routng for software-defned networkng, n Proc. IEEE ICC, pp. 1 6, 2016. [30] F. Jalal, K. Hnton, and R. Ayre, Fog computng may help to save energy n cloud computng, IEEE J. Sel. Areas Commun., vol. 34, no. 5, pp. 1728 1739, 2016. [31] T. Benson, A. Akella, and D. A. Maltz, Network traffc characterstcs of data centers n the wld, n Proc. IMC, pp. 267 280, 2010. [32] D. Klazovch, P. Bouvry, and S. U. Khan, GreenCloud: a packetlevel smulator of energy-aware cloud computng data centers, J. Supercomp, vol. 62, no. 3, pp. 1263 1283, 2012. [33] R. Trestan, K. Katrns, and G.-M. Muntean, OFLoad: An OpenFlow-based Dynamc Load Balancng Strategy for Datacenter Networks, IEEE Trans. Netw. Servce Manag, vol. 14, no. 4, pp. 792 803, 2017. [34] I. F. Akyldz, A. Lee, P. Wang, et al., Research challenges for traffc engneerng n software defned networks, IEEE Network, vol. 30, no. 3, pp. 52 58, 2016. [35] S.-C. Chao, K. C.-J. Ln, and M.-S. Chen, Flow Classfcaton for Software-Defned Data Centers Usng Stream Mnng, IEEE Trans. Serv. Comput., 2016. [36] D. Adam, G. Antch, R. G. Garroppo, et al., Towards an SDN network control applcaton for dfferentated traffc routng, n Proc. IEEE ICC, pp. 5827 5832, IEEE, 2015. [37] Optcal Network Desgn and Plannng. [Onlne]. Avalable: http://www.monarchna.com/topology.html. [38] T.-W. Kuo, B.-H. Lou, K. C.-J. Ln, et al., Deployng chans of vrtual network functons: On the relaton between lnk and server usage, n Proc. IEEE INFOCOM, pp. 1 9, IEEE, 2016. [39] K. He, J. Khaldy, A. Gember-Jacobson, et al., Measurng Control Plane Latency n SDN-enabled Swtches, n Proc. ACM SIGCOMM SOSR, pp. 1 6, ACM, 2015. [40] R. Ramaswamy, N. Weng, and T. Wolf, Characterzng network processng delay, n IEEE GLOBECOM, vol. 3, pp. 1629 1634, IEEE, 2004. [41] L. Pantel and L. C. Wolf, On the mpact of delay on real-tme multplayer games, n Proc. ACM NOSSDAV, pp. 23 29, 2002. Janng Pe was born n 1992. He receved hs B.S. degree from the Department of Informaton and Electrcal Engneerng (IEE), Chna Unversty of Mnng and Technology (CUMT), n 2015 and he s pursung for hs Ph.D n USTC wth hs advsor Peln Hong now. Hs research nterests nclude SDN, NFV, and the network resource orchestraton and management. Peln Hong was born n 1961. She receved her B.S. and M.S. degrees from the Department of Electronc Engneerng and Informaton Scence (EEIS), Unversty of Scence and Technology of Chna (USTC), n 1983 and 1986. Currently, she s a Professor and Advsor for Ph.D. canddates n the Department of EEIS, USTC. Her research nterests nclude next-generaton Internet, polcy control, IP QoS, and nformaton securty. She has publshed 2 books and over 100 academc papers n several journals and conference proceedngs. Kapng Xue (M09-SM15) receved hs B.S. degree from the Department of Informaton Securty, Unversty of Scence and Technology of Chna (USTC), n 2003 and receved hs Ph.D. degree from the Department of Electronc Engneerng and Informaton Scence (EEIS), USTC, n 2007. Currently, he s an Assocate Professor n the Department of Informaton Securty and Department of EEIS, USTC. Hs research nterests nclude next-generaton Internet, dstrbuted networks and network securty. Defang Lee was born n 1991. He receved hs B.S. degree from the Department of Electronc Engneerng and Informaton Scence (EEIS), Unversty of Scence and Technology of Chna (USTC), n 2014 and he s pursung for hs Ph.D n USTC wth hs advsor Peln Hong now. Hs research nterests nclude SDN, NFV, and the network resource orchestraton and management..