NETWORK function virtualization (NFV) [1] allows network

Transcription

1 1664 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY 2018 Vrtual Network Functon Placement Consderng Resource Optmzaton and SFC Requests n Cloud Datacenter Defang L, Student Member, IEEE, Peln Hong, Kapng Xue, Senor Member, IEEE, and Janng Pe Abstract Network functon vrtualzaton (NFV) brngs great convenences and benefts for the enterprses to outsource ther network functons to the cloud datacenter. In ths paper, we address the vrtual network functon (VNF) placement problem n cloud datacenter consderng users servce functon chan requests (SFCRs). To optmze the resource utlzaton, we take two less-consdered factors nto consderaton, whch are the tme-varyng workloads, and the basc resource consumptons (BRCs) when nstantatng VNFs n physcal machnes (PMs). Then the VNF placement problem s formulated as an nteger lnear programmng (ILP) model wth the am of mnmzng the number of used PMs. Afterwards, a Two-StAge heurstc soluton (T-SAT) s desgned to solve the ILP. T-SAT conssts of a correlaton-based greedy algorthm for SFCR mappng (frst stage) and a further adjustment algorthm for vrtual network functon requests (VNFRs) n each SFCR (second stage). Fnally, we evaluate T-SAT wth the artfcal data we compose wth Gaussan functon and trace data derved from Google s datacenters. The smulaton results demonstrate that the number of used PMs derved by T-SAT s near to the optmal results and much smaller than the benchmarks. Besdes, t mproves the network resource utlzaton sgnfcantly. Index Terms Vrtual network functon placement, resource optmzaton, servce functon chan, tme-varyng workloads, mult-tenancy, basc resource consumptons, correlaton-based algorthm Ç 1 INTRODUCTION NETWORK functon vrtualzaton (NFV) [1] allows network functons (NFs) or mddleboxes tradtonally attached on specfc hardwares to be realzed n software and dynamcally outsourced to be run on any common off-theshelf servers, whch brngs great convenences and flexbltes n programmablty, management and polcy nteractons between NFs. Nowadays, more and more network operators and enterprses leverage NFV technology to reduce the cost of nfrastructure constructon and management [2], [3]. When hundreds of users outsource ther NFs to the cloud [4], [5], vrtual network functons (VNFs) placement (also mentoned as VNF nstantaton) becomes an mportant but dffcult problem for the cloud servce provders (CSPs), whch has receved more and more attenton from academa and ndustry. In the problem, one or more postons where to nstantate a seres of VNFs should be decded wth an optmzaton objectve, such as mnmzng the resource consumptons [6], [7] and maxmzng the number of users requests that can be served [8]. A good placement soluton can mprove the utlzaton effcency of network resources The authors are wth the Key Laboratory of Wreless-Optcal Communcatons, Chnese Academy of Scences, School of Informaton Scence and Technology, Unversty of Scence and Technology of Chna. No. 96 Jnzha Road, Hefe, Anhu Provnce , P. R. Chna. E-mal: {ldf911, janngp}@mal.ustc.edu.cn, {plhong, kpxue}@ustc.edu.cn. Manuscrpt receved 14 Jan. 2017; revsed 13 Dec. 2017; accepted 29 Jan Date of publcaton 5 Feb. 2018; date of current verson 13 June (Correspondng author: Defang L.) Recommended for acceptance by B. Ravndran. For nformaton on obtanng reprnts of ths artcle, please send e-mal to: reprnts@eee.org, and reference the Dgtal Object Identfer below. Dgtal Object Identfer no /TPDS and reduce the CAPtal EXpendtures and OPeratng EXpenses (CAPEX/OPEX) greatly, whch wll produce more profts for the CSPs. Bascally, VNF placement problem s usually formulated as an nteger programmng or lnear programmng model, and lots of heurstc schemes have been proposed [6], [9], [10], [11]. Generally, a user s servce request s accomplshed by a servce functon chan (SFC), whch s constructed by a lst of VNFs wthn a specfed order [12], [13]. Thus, we treat a user s NF outsourcng request as an SFC request, smplfed as SFCR. Meanwhle, to make a dstncton, we call the elements n a specfc SFCR as VNF requests (VNFRs), correspondng to the VNFs n an SFC. Gven SFCRs proposed by dfferent users, the VNF placement problem s specfc to how to deploy a seres of SFCs, separately contanng multple VNFs, onto physcal machnes (PMs) n the cloud. Dfferent from most works n exstng, the VNFs are mult-tenant capable n our consderatons, whch means that one VNF can provde servces for multple tenants at the same tme [14]. Specfcally, the VNFRs n SFCRs can be seen as the tenants n our paper. To optmze the resource utlzaton whle placng the VNFs, we take the followng two factors nto consderaton: 1) Consoldatons of Tme-Varyng Workloads. Generally speakng, the workloads of the NFs are tme-varyng, and the peaks and valleys n one NF s workload do not necessarly concde wth the others statstcally [15], [16]. For a mult-tenant capable VNF, the capacty of the VNF s commensurate wth the workloads of all VNFRs on t, whch mples that resources are allocated to the VNFs based on workloads of all ß 2018 IEEE. Personal use s permtted, but republcaton/redstrbuton requres IEEE permsson. See ht_tp:// for more nformaton.

2 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD Fg. 1. A schematc dagram of VNFR composton wth tme-varyng workload. VNFRs on them. So based on the predcted workloads of VNFRs, we can gather together the VNFRs, demandng the same type of VNF and wth complementary workload patterns, on the same multtenant capable VNF to mprove the utlzaton effcency of network resources. Referrng to the real cases n [15], [16], [17], we use a smple schematc dagram to demonstrate the advantages. As s shown n Fg. 1, the left part shows the tme-varyng CPU demand of 4 VNFRs for VNF A, the rght part shows two combnatons of the 4 VNFRs: n combnaton 1, VNFR 1 and VNFR 3 are hosted on VNF A 1, VNFR 2 and VNFR 4 are hosted on VNF A 2 ; n combnaton 2, VNFR 1 and VNFR 2 are hosted on VNF A 1, VNFR 3 and VNFR 4 are hosted on VNF A 2. Compared wth combnaton 1, the workload of combnaton 2 s more smooth, and the peak value of combnaton 2 s 6 compared wth 10 n combnaton 1, so we just need to allocate at most 6 unts CPU resources to VNF A 1 and A 2 respectvely n combnaton 2, compared wth 10 unts n combnaton 1. Thus, we can say that combnaton 2 leads to a more effcent utlzaton of PM s resources than combnaton 1. 2) Basc Resource Consumptons and the Sharng. We consder the basc resource consumptons (BRCs) when nstantatng VNFs n a PM. BRCs are the resource consumptons for the basc functon mantanng of one software, lke the resource consumptons to mantan the OS and related lbrares of one VNF [18], [19] before t starts to provde servce. Specfcally, for the mult-tenant capable VNFs, BRCs are the resource consumptons to mantan the shared applcaton platform and database by the VNFRs on them. We concretze them as the CPU and memory consumptons n ths paper, whch are the CPU BRCs (c-brcs) and memory BRCs (m-brcs) respectvely. Also, we assume that BRCs of one VNF are fxed, and they have no relatonshp wth VNF s processng capacty. For a seres of VNFRs that are hosted on the same mult-tenant capable VNF, they share the same block of BRCs. By gatherng the VNFRs that demand the same type of VNF together, we can save more VNF nstances, then BRCs decrease subsequently, leadng to fewer node resource consumptons. As stated before, users NF requests are accomplshed by SFCs, so the flow of one user s request must go through a seres of VNFs n a specfed order to get served. If all the VNFs that an SFCR needs are n the same PM (Scenaro 1), the traffc loads between the VNFs wll be restrcted n the PM, and the correspondng bandwdth consumptons on the lnks wll not occur. However, f the VNFs needed are dstrbuted among multple PMs (Scenaro 2), the correspondng bandwdth consumptons on the lnks cannot be avoded. Compared wth Scenaro 2, Scenaro 1 needs more VNF nstances, so BRCs are more n Scenaro 1. Thus, we can fnd that there s a conflcton between bandwdth consumptons and BRCs, and we need to make a balance between them. In summary, we consder the VNF placement problem n cloud datacenter wth a seres of known SFCRs. To optmze the utlzaton of network resources, we take the tmevaryng workloads and BRCs nto consderaton. Afterwards, the VNF placement problem s formulated as an nteger lnear programmng (ILP) model, amng to mnmze the number of used PMs. The VNF placement problem has turned out to be NP-hard [6], [9], [10], [20], [21], so s the VNF placement problem n our paper. Thus we solve the ILP wth a small number of SFCRs usng Gurob [22] and propose a Two-StAge heurstc soluton (T-SAT) to solve the ILP when the number of SFCRs s large. T-SAT can be dvded nto two parts: one s the correlaton-based greedy algorthm, whch ams to map the SFCRs to the PMs accordng to the correlaton between ther predcted workloads, and the other s a further adjustment process based on VNFRs, whch ams to gather the VNFRs that demand the same type of VNF together to save more node resources, owng to BRCs sharng among VNFRs on the same multtenant capable VNF. Our major contrbutons can be summarzed as: 1) To mprove the utlzaton effcency of network resources, we consder the tme-varyng feature of NF requests workloads. Specfcally, we desgn a correlaton-based algorthm to handle them. 2) We consder BRCs and mult-tenancy of VNFs and reveal the conflcton between bandwdth consumptons and BRCs. Then we explot the fact that VNFRs on the mult-tenant capable VNFs can share BRCs to save more node resources whle keepng a balance between bandwdth consumptons and BRCs. 3) We formulate the VNF placement problem as an ILP model and propose a novel two-stage soluton, T- SAT. Then T-SAT s evaluated n detal wth the artfcal data we compose wth Gaussan functon and the trace data derved from Google s datacenters [23]. The smulaton results demonstrate that T-SAT can mprove the utlzaton of network resources sgnfcantly and acqure a smaller number of used PMs than the benchmarks. The remander of ths paper s organzed as follows. Secton 2 revews the related works. In Secton 3, we descrbe the system model and the related concepts. Secton 4 formulates the VNF placement problem and our soluton T-SAT s ntroduced n Secton 5. Afterwards, Secton 6 s the performance evaluaton of T-SAT. Fnally, we conclude the paper n Secton 7.

3 1666 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY RELATED WORK NFV and the related VNF placement problem have attracted plenty of attenton of researchers. In [6], the structures of SFCs are not known forward, and the authors proposed a model for formalzng the SFCs, then they formulated the VNF mappng problem as a Mxed Integer Quadratcally Constrants Program (MIQCP). A closed-loop wth crtcal mappng feedback algorthm to jontly optmze the topology desgn and mappng of multple SFCs s presented n [7]. In [8], they thought that the relatonshp between lnk and server usage plays a crucal role n the jont VNF placement and path selecton problem. Then the authors gave a mathematcal program analyss about t, and proposed a heurstc soluton based on that. In [9], the authors studed the VNFs placement n the datacenter where optcal technologes are employed, wth the goal of mnmzng the expensve optcal/electronc/optcal (O/E/O) converson tmes for NFV channg. Cohen et al. [11] consdered the actual placement of VNFs wthn the physcal network, and provded a near optmal soluton wth theoretcally proven performance. In [24], Rankothge et al. optmzed the resource allocaton for VNFs n cloud datacenter usng genetc algorthms. However, few of the exstng works about VNF placement problem pay attenton to the mult-tenant capable VNFs, despte the fact that the mult-tenant capable VNFs can lead to a more effcent utlzaton of network resources. It s worth notng that the authors n [10] mentoned that more VNFs mean more consumpton of the computng resources, but fewer VNFs result n more traffc loads. They ponted out the conflcton between the node resource consumptons and bandwdth consumptons, but they dd not consder the overhead when nstantatng VNFs. In [25], when calculatng the resource consumptons of one VM, the authors consdered the overhead of vrtualzaton besdes the loads of tenants, whch has some smlarty wth the BRCs we consder. However, they dd not pay attenton to the conflcton between the node resource consumptons and bandwdth consumptons. Although few works consder the tme-varyng workload n VNF placement problem, we fnd that plenty of works have been done on the VM consoldaton problem consderng tme-varyng workload. In [16], the authors thought that the peaks and valleys n one workload pattern do not necessarly concde wth the others. Thus, the unused resources of a low utlzed VM can be borrowed by the other co-located VMs wth hgh utlzaton. They proposed a VM selecton algorthm that seeks to fnd those VM combnatons wth complementary workload patterns based on Pearson s correlaton coeffcent. Zhang et al. [26] proposed to reserve some extra resources on each PM to accommodate bursty workload. And they used the two-state Markov chan to capture the burstness of workload and developed a novel server consoldaton algorthm. Zheng et al. [27] proposed PowerNets to save the energy consumpton n datacenter based on the tme-varyng workloads. We have the smlar vew wth the authors n [17] that multplexng VNFs wth dynamc workload properly wll ncrease the utlzaton effcency of network resources. Besdes, we have to consder the nteractons between the VNFs at the same tme owng to the property of SFCs, whch results n the conflcton between node resource consumptons and bandwdth consumptons. Referrng to prevous works, we formulate the VNF placement problem as an ILP model wth the am of mnmzng the number of used PMs whle consderng the tmevaryng workloads and BRCs. Then we propose a two-stage heurstc soluton to solve t. 3 SYSTEM MODEL ANDRELATED CONCEPTS In ths secton, we ntroduce the system model and make a further explanaton for the concepts referred to before. 3.1 System Model We represent the substrate network as an undrected graph G ¼ðN s ;E s Þ, where N s ndcates the set of total nodes n substrate network and E s ndcates the lnk set of the substrate network. Specfcally, we use P to ndcate the set of total PMs and R to ndcate the set of total access swtches, whch handle the servce accessng of SFCRs. We assume that there are jgj SFCRs n total, and use a 4- tuple ðng g ; eg g ; C g ;Eg nþ to ndcate SFCR g, where ng g ndcates the ngress node of SFCR g, eg g ndcates the egress node of SFCR g, C g ndcates the set of total VNFRs n SFCR g and Eg n ndcates the set of logcal lnks between the nodes of SFCR g. Generally, the ngress node and egress node of the same SFCR should be hosted on the same access swtch. 3.2 Mult-Tenancy and BRCs Mult-tenancy s a software archtecture prncple n the realm of the Software as a Servce (SaaS) busness model [14], whch allows multple tenants to share the same software nstance [28]. Compared wth the sngle-tenant archtecture, n whch each tenant gets ts own nstance of applcaton, mult-tenancy can lead to hgher resource utlzaton, lower servce prce, and more effcent management for the CSPs. So we assume that the VNFs are mult-tenant capable n ths paper. As a result of mult-tenancy, we just need to place one nstance of some specfed VNF n a PM [14]. But from the vew of the whole network, there are multple nstances of one specfed VNF. BRCs are the resource consumptons for the basc functon mantanng of one software. Tradtonally, a VNF s hosted on one VM, and t needs ndependent guest OS and lbrares [29], so BRCs are the resource consumptons to mantan the functonng of VM platform, guest OS and lbrares that one VNF needs. For a mult-tenant capable VNF, the resource consumptons mantanng the shared applcaton platform and database can be seen as BRCs. And the VNFRs on the same mult-tenant capable VNF can share the BRCs, even f they belong to dfferent SFCRs. Besdes, we assume that BRCs of one VNF are fxed and they have no relatonshp wth the processng capacty of that VNF. 3.3 Correlaton Coeffcents We need to evaluate the correlaton between the workloads of dfferent SFCRs or VNFRs, avodng that SFCRs or VNFRs whose workloads have hgh correlaton are placed together. By sayng the two workloads have hgh correlaton, we mean that ther peaks and valleys are concde wth each other frequently. To evaluate the correlaton, we

4 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD Fg. 2. A VNF placement nstance wth a 3-layer fat-tree topology. should compare the workloads wthn a perod. So every workload varable wthn tme nterval T s correspondng to a sequence of values. Mathematcally speakng, workload varables are vectors nstead of constants. Several coeffcents are often used to evaluate the correlaton between two vectors, such as Pearson s correlaton coeffcent [16], [30], Mnkowsk Dstance [31], and Cosne correlaton coeffcent [17]. We apply the above three mentoned correlaton coeffcents to T-SAT respectvely, and fnd that the Cosne correlaton coeffcent can acqure the best effect from plenty of experments. So we choose t as the fnal canddate to T-SAT. The Cosne correlaton coeffcent s as follows: P n ¼1 cos XY ¼ X Y pffffffffffffffffffffffffffffffffffff P n p ffffffffffffffffffffffffffffffffff P n ¼1 X2 ¼1 Y : (1) 2 The bgger the cos XY s, the more related vectors X and Y are. In our paper, we need to gather the SFCRs or VNFRs wth complementary workload patterns together. Hence the smaller the cos XY s, the more lkely the two SFCRs or VNFRs are mapped n the same PM. 4 PROBLEM STATEMENT In ths secton, we gve a descrpton about the VNF placement problem we consder through a smple case, and complete the formulatons about the problem. 4.1 Problem Descrpton Fg. 2 shows a non-trval example about how to place VNFs for two SFCRs n a 3-layer fat-tree topology. For the fat-tree topology, the swtches n the core layer (core swtches) are responsble for the servce accessng. In the fgure, VNF r ndcates dfferent VNFR, correspondng to VNF that ndcates dfferent types of VNFs respectvely, 2fa; b; c; d; eg; SFCR 1 enters the datacenter from swtch 1, and SFCR 2 from swtch 4. We assume that VNF a r 1 s mapped on PM 21; VNF b r 1 on PM 23; VNF c r 1 on PM 25; VNF d r 1 on PM 29; VNF a r 2 on PM 36; VNF e r 2 on PM 33; VNF c r 2 on PM 25; VNF d r 2 on PM 29; Then we need to place two nstances of VNF a on PM 21 and 36, respectvely; an nstance of VNF b on PM 23; an nstance of VNF c on PM 25; an nstance of VNF d on PM 29; an nstance of VNF e on PM 33. And the total BRCs are 6 shares. The flow path of SFCR 1 s 1! 5! 13! 21! 13! 6! 14! 23! 14! 5! 1! 7! 15! 25! 15! 7! 2! 9! 17! 29! 17! 9!1, whch s demonstrated n blue arrows. The flow path of SFCR 2 s 4! 12! 20! 36! 20! 11! 19! 33! 19! 11! 2! 7! 15! 25! 15! 7! 2! 9! 17! 29! 17! 10!4, whch s demonstrated n red arrows. If we change the mappng poston of VNF a r 2 from PM 36 to PM 21, we do not need to place the nstance of VNF a on PM 36, so PM 36 s free out and the total BRCs become 5 shares. However, the flow path of SFCR 2 becomes 4! 6! 13! 21! 13! 5! 1! 11! 19! 33! 19! 11! 2! 7! 15! 25! 15! 7! 2! 9! 17! 29! 17! 10! 4. The number of lnks that SFCR 2 passes through ncreases from 22 to 24, so the bandwdth consumptons ncrease. If we change the mappng poston of VNF c r 2 from PM 25 to PM 29, we need to nstantate another nstance of VNF c on PM 29 and the BRCs become 7 shares. However, the flow path of SFCR 2 becomes 4! 12! 20! 36! 20! 11! 19! 33! 19! 12! 3! 10! 17! 29! 17! 10!4. The number of lnks that SFCR 2 passes through decreases from 22 to 16, so the bandwdth consumptons decrease. The above smple case shows a VNF placement process and reveals the conflcton between BRCs and bandwdth consumptons. Generally speakng, we accomplsh the mappng of the SFCRs or VNFRs frst, and then based on the mappng results, we determne the placement of the related VNFs and allocate resources to them commensurate wth the workloads of VNFRs on them. It s worth notng that the servng capacty of one VNF s determned by the VNFRs t serves, whch s a constant value once the placement of VNFs s accomplshed. Moreover, as stated before, the tmevaryng workloads also have to be consdered to mprove the utlzaton effcency of the network resources. The actvatng of one PM s usually assocated wth much energy and captal cost, and fewer PMs mean fewer costs and motvate the ncreasng of the network resource utlzaton. So n ths paper, our target s to mnmze the number of used PMs. Consderng the constrants of the lmted resources, namely CPU, memory and bandwdth, we formulate the VNF placement problem as an nteger lnear programmng model. 4.2 Problem Formulaton The notatons used n the followng equatons are descrbed n Table 1. Frst, we must ensure that one node n an SFCR s mapped on one and only one node n the substrate network, so for SFCR g jp Xj 1 u¼0 jrj 1 X u¼0 x g;n n ;n s u ¼ 1;ns u 2 P; nn 2 C g nfng g ; eg g g; x g;n n ;n s u ¼ 1;ns u 2 R; nn 2fng g; eg g g; x g;ngg ;n s u ¼ x g;eg g ;n s u ;ns u 2 R; (2a) (2b) (2c) where g 2 G. Eq. (2a) restrcts that one VNFR can only be mapped on one PM. Eqs. (2b) and (2c) restrct that ng g and

5 1668 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY 2018 TABLE 1 Notatons Pramaters Descrptons SFCR related G set of total SFCRs, g 2 G s an SFCR. ng g ngress node of SFCR g. eg g egress node of SFCR g. C g set of total VNFRs n SFCR g. Eg n set of logcal lnks between the nodes of SFCR g. n n, nn j two nodes n one SFCR. ðn n ;nn j Þ logcal lnk between nn and nn j, ðnn ;nn j Þ2En g. F types number of VNFs, f 2f0; 1;...; F 1g s an element. Topology related N s set of total nodes n substrate network. P set of total PMs. R set of total access swtches. E s lnk set of the substrate network. n s u, ns v two nodes n the substrate network. ðn s u ;ns v Þ substrate lnk between ns u and ns v. Samplng related T tme wndow length of the workload we consder. t D sample nterval of the workload. L sample tmes to the workload n tme wndow T. ndcates the sequence number of the samplng, 2f0; 1;...; L 1g. Resource related cpu g;n nðt D Þ CPU consumpton by VNFR n n n SFCR g at tme t D. mem g;n n ðt D Þ memory consumpton by VNFR n n n SFCR g at tme t D. b g;n n ;n nðt DÞ bandwdth consumpton by logcal lnk j (n n nn j ) n SFCR g at tme t D. brc cpu f BRCs of CPU when placng an nstance of VNF f, f 2 F. brc mem f BRCs of memory when placng an nstance of VNF f, f 2 F. CPU capacty of PM n s u. C cpu n s u Cn mem s u C lnk memory capacty of PM n s u. ðn s u ;ns v Þ lnk capacty of substrate lnk ðn s u ;ns v Þ. u cpu ; u mem ; u lnk resource volaton threshold for CPU, memory, and lnk wthn tme nterval T, respectvely. Bnary varables x g;n n ;n s u whether n n n SFCR g s hosted on substrate node n s u, ns u 2 P [ R. l g;n n ;f whether VNFR n n n SFCR g demands VNF f. y g;n n ;n n j ;ns u ;ns whether the logcal lnk ðn n v ;nn jþ n SFCR g s mapped on substrate lnk ðn s u ;ns v Þ. z f;n s u whether an nstance of VNF f s placed on PM n s u, ns u 2 P. h n s u whether PM n s u s actvated, ns u 2 P. eg g of SFCR g can only be mapped on the same access swtch. Then we should ensure that the total resource consumptons by the VNFRs n one PM should not exceed the resource capacty of that PM at each samplng tme t D, where t D s the sample nterval of the workload. So the constrant for CPU consumptons s jgj 1 X jcx g j 1 cpu g;n vðt D Þx g;n n ;n s u ¼0 þ XF 1 f¼0 brc cpu f z f;n s u C cpu n s u ;ns u 2 P; where z f;n s u ndcates whether an nstance of VNF f s placed on PM n s u. In Eq. (3), P F 1 f¼0 brccpu f z f;n s u ndcates the total CPU BRCs n PM n s u. z f;n s u reles on x g;n n ;n s and l u g;n n ;f ( z f;n s u ¼ 1; P jgj 1 P jcg j 1 ¼0 x g;n n ;n s l u g;n n ;f 1; (4) 0 otherwse; where n s u 2 P, and l g;n n ;f ndcates whether VNFR n n n SFCR g demands VNF f, f so, l g;n n ;f ¼ 1. So Eq. (4) ndcates that we need to place an nstance of VNF f on PM n s u for the VNFRs that demand t. Lke Eq. (3) for CPU, we can formulate the memory constrants as jgj 1 X jcx g j 1 mem g;n n ðt D Þx g;n n ;n s u ¼0 (5) þ XF 1 f¼0 brc mem f z f;n s u Cn mem s ; u where n s u 2 P. In fact, for a tme perod, the tmes of resource overutlzaton can be tolerated wthn certan range. Then we defne the followng bnares: 1; Eq: 3 s volated; I cpu ðt D Þ¼ (6) 0 otherwse; 1; Eq: 5 s volated; I mem ðt D Þ¼ (7) 0 otherwse; where I cpu ðt D Þ and I mem ðt D Þ ndcate whether the CPU and memory constrants are volated at samplng tme t D, respectvely. Therefore, the CPU and memory constrants wthn tme nterval T can be X L 1 ¼0 X L 1 ¼0 (3) I cpu ðt D Þ=L u cpu ; (8) I mem ðt D Þ=L u mem ; (9) where u cpu and u mem ndcate the resource volaton threshold for CPU and memory, respectvely. Eqs. (8) and (9) ndcate that the total resource volaton tmes wthn tme nterval T must not exceed a threshold. The relatonshp between T, t D, and L s t D ¼ T=L: (10) Followng the node constrants, we ntroduce the lnk constrants subsequently.

6 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD Frst, the lnk varable s 1; condton s satsfed; y g;n n ;n n j ;ns u ;ns ¼ (11) v 0 otherwse; where the condton n Eq. (11) s that logcal lnk ðn n ;nn j Þ n SFCR g s mapped on substrate lnk ðn s u ;ns v Þ, ðns u ;ns v Þ2Es. In general, each logcal lnk s correspondng to a path n the substrate network. However, n the VNF placement problem, two dfferent nodes n one SFCR can be mapped on the same substrate node, and then the correspondng logcal lnk s lmted n one PM. Smlar to the constrants about node resources, we have the followng lnk resource constrant at samplng tme t D jgj 1 X X ðn n ;nn j Þ2En g b g;n n ;n nðt DÞy j g;n n ;n n j ;ns u ;ns v Clnk ðn s u ;ns vþ; (12) where ðn s u ;ns v Þ2Es and g 2 G. The lnk resource constrant wthn tme nterval T s 1; Eq: 12 s volated; I lnk ðt D Þ¼ (13) 0 otherwse; X L 1 ¼0 I lnk ðt D Þ=L u lnk ; (14) where u lnk ndcates the resource volaton threshold for lnk resources wthn tme nterval T. For the flow of SFCR g, the followng constrants must be satsfed X Ns n s v X Ns n s v y g;n n ;n n j ;ns u ;ns v 1; y g;n n ;n n j ;ns v ;ns u 1; X Ns XNs y g;n n ;n n n s j ;ns u ;ns y v g;n n ;n n v n s j ;ns v ;ns u v ¼ x g;n n ;n s u x g;n n j ;ns u ; (15a) (15b) (15c) where ðn s v ;ns u Þ2Es ; ðn n ;nn j Þ2En g ; g 2 G. Eq. (15a) ndcates whether logcal lnk ðn n ;nn jþ n SFCR g s mapped on one of the substrate lnks that leave out node n s u, f so, P N s n s y v g;n n ;nn j ;ns u ;ns equals 1. Eq. (15b) ndcates v whether the same logcal lnk s mapped on one of the substrate lnks that go n node n s u, f so, P N s n s y v g;n n ;nn j ;ns v ;ns u equals 1. The two equatons also ensure that one logcal lnk cannot be splt. Eq. (15c) ensures that the path n the substrate network s consecutve for a flow. Fnally, we use h n s u to ndcate whether PM n s u s actvated, n s u 2 P. h n s reles on x u g;n v ;ns u ( h n s u ¼ 1; P jgj 1 0 otherwse; P jcg j 1 ¼0 x g;n v ;n s u 1;ns u 2 P; (16) Eq. (16) ndcates that even f only one VNFR s mapped on the PM, the PM has also to be actvated. The total number of actvated PMs s jpj 1 H t ¼ X h n s u : (17) u¼0 Our target s to mnmze the number of actvated PMs mn H t ; s:t: Eq: 2 to Eq: 17: (18) The VNF placement problem s formulated as an nteger lnear programmng model, whch s NP-hard. For the ILP, we can derve the optmal results by exstng optmzaton solvers, lke Gurob [22], when the problem scale s small. However, t s nfeasble to get the optmal results n foreseeable tme when the problem scale s large. So we desgn a Two- StAge heurstc soluton to solve the ILP n polynomal tme. 5 PROPOSED ALGORITHM In ths secton, we present the two-stage heurstc soluton, T-SAT. A general dea about T-SAT s presented frst n Secton 5.1, then the detals about the algorthms are demonstrated n Sectons 5.2 and Framework of T-SAT T-SAT conssts of a correlaton-based greedy algorthm for mappng SFCRs to PMs and a further adjustment process to reduce BRCs by gatherng the VNFRs that demand the same type of VNF together. We map all the SFCRs one by one to the PMs usng the correlaton-based greedy algorthm at the frst stage. Durng the process, one SFCR s mapped on one PM as a whole unless the total resources demanded by the SFCR cannot be satsfed by the PM. For those SFCRs that cannot be mapped onto one PM as a whole, we splt each of them nto several VNFRs and map these VNFRs one by one to the PMs. If we place VNFs based on the above results, BRCs wll occupy a bg share of the resources n the PMs. Because each PM has to nstantate almost all knds of VNFs n t and dfferent VNF nstances cannot share BRCs. On the contrary, the nteractng traffc between PMs s the least n above stuaton. We can reduce BRCs by reducng the number of reduplcated VNF nstances. So we gather the VNFRs demandng the same type of VNF together nto a smaller group of PMs n the second stage, based on above SFCRs mappng results. Then we can nstantate fewer VNFs, and BRCs are reduced. Subsequently, the total resource consumptons n PMs are fewer, and the number of used PMs decreases. However, the above processes may ncrease the nteractng traffc between the PMs. So we must ensure that the lnk capacty constrants are not volated durng the adjustment process. 5.2 Correlaton-Based Greedy Algorthm Algorthm 1 s the framework of the correlaton-based greedy algorthm. Frst, we establsh the correlaton coeffcent matrx about all SFCRs. s a jgjjgj square symmetrc matrx. m ;j s one of ts elements, whch ndcates the correlaton coeffcent between the workloads of SFCR and SFCR j. The workload of one SFCR s the lnear accumulaton of all VNFRs workloads n t. We set m ;j as an nfnty value when ¼ j. Otherwse, m ;j s calculated by Eq. (20).

7 1670 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY 2018 Algorthm 1. Correlaton-Based Greedy Algorthm 1: Input: The set of all the SFCRs: G; The ntal status of datacenter: S 0 ; 2: Output: Number of used PMs: P 0 ; Status of datacenter: S 1 ; 3: Establsh the correlaton coeffcent matrx about all SFCRs. 4: whle G s not empty do 5: Start a new PM, P 0 ¼ P 0 þ 1. 6: Pck out the par of SFCRs n whose workloads have the mnmal correlaton coeffcent. 7: Select the SFCR that consumes more resources as the chosen SFCR. 8: Make a copy of current G as G c. 9: whle G c s not empty do 10: f PM can hold the chosen SFCR then 11: Put the chosen SFCR nto the PM. 12: Remove the chosen SFCR from G and G c. 13: f G c s not empty then 14: Take the SFCRs n the PM as a whole, ndcated as. 15: Calculate the workload correlaton coeffcents between and each of the rest SFCRs. 16: Pck out the SFCR whose workload has the mnmal correlaton wth the workload of to be mapped nextly. 17: else 18: break. 19: end 20: else 21: Decompose the chosen SFCR nto VNFRs and try to map them nto PMs that have VNFRs n themselves based on frst-ft. 22: f the above process s successful then 23: Remove the chosen SFCR from G c and G. 24: else 25: Remove the chosen SFCR from G c. 26: end 27: f G c s not empty then 28: Pck out the SFCR that has the mnmal workload correlaton coeffcent wth to be mapped nextly. 29: else 30: break. 31: end 32: end 33: end 34: end 35: return P 0, S 1. As stated before, we consder the constrants of three types of resources, namely CPU, memory and bandwdth. So there are three vectors for each SFCR mathematcally, and the mathematc descrpton for the workload of SFCR can be formulated as 0 M cpu 0 cpu 1... cpu L 1 mem 0 mem 1... mem L 1 band 0 band 1... band L 1 1 A; (19) where L s the sample tmes we apply to the workload. For Eq. (19), the three rows of M ndcate the vector varables of CPU, memory and bandwdth respectvely. The correlaton coeffcent m ;j between SFCR and SFCR j can be calculated as m ;j ¼ fðm 0 ; M0 j ÞþfðM1 ; M1 j ÞþfðM2 ; M2 jþ; (20) where f s the functon referred to n Secton 3.3 to calculate the correlaton coeffcent of two vectors. After establshng the correlaton coeffcent matrx, we pck out the par of SFCRs, the workloads of whch have the mnmal correlaton coeffcent. Then the SFCR n the par that consumes more resources s selected and mapped n the PM. At lne 8 n Algorthm 1, we make a copy of current SFCR set G, ndcated as G c, amng to traverse all the rest SFCRs and check out whether the chosen SFCR can be mapped n the current PM (lnes 9-33 n Algorthm 1). If the chosen SFCR can be hosted n current PM as a whole, we put t n the PM and remove t from both G and G c (lnes n Algorthm 1). After that, we choose the next SFCR to be mapped (lnes n Algorthm 1). Frst, the SFCRs already n the PM are taken as a whole, ndcated as. Then we calculate the correlaton coeffcent between the workload of and the workload of each SFCR n G c.at last, the SFCR n G c whose workload has the mnmal correlaton wth the workload of s chosen to be mapped nextly. If the chosen SFC cannot be hosted n current PM as a whole, we decompose the SFCR nto several ndvdual VNFRs and try to map these VNFRs n PMs that already have VNFRs n based on frst-ft algorthm (lnes n Algorthm 1). If these VNFRs cannot be mapped totally, t ndcates that the chosen SFCR needs an empty PM to be mapped. So we keep t n G to be mapped n future and only remove t from G c. Then f G c s not empty, the SFCR n G c whose workload has the mnmal correlaton wth the workload of s chosen to be mapped nextly. The above processes are repeated untl G c s empty. After the breakng of the nner whle loop (lnes 9-33 n Algorthm 1), we need to check f G s empty. If G s empty, t means that all the SFCRs are mapped on the PMs. Based on the above SFCR mappng results, each used PM contans almost all knds of VNFRs. If we place the related VNF nstances based on that, each of PMs has to nstantate almost all types of VNFs, whch wll results n a great volume of BRCs. However, there s lttle nteractng traffc between the PMs n ths stuaton, so the utlzaton of bandwdth resource s low relatvely. Then the next step s to reduce BRCs wth the ncreasng of the bandwdth utlzaton, tryng to reduce the number of used PMs further. 5.3 Two-Stage Adjustment Algorthm In the second stage, we desgn an adjustment algorthm to reduce BRCs. The adjustment algorthm s dvded nto two parts. The frst step tres to gather the VNFRs demandng the same type of VNF together nto a smaller number of PMs ntra the same cluster, where a cluster s a collecton of PMs that are close to each other. By gatherng the VNFRs demandng the same type of VNF together, we can reduce the number of reduplcated VNF nstances, and then reduce the volume of BRCs. After the adjustment n each cluster, we sort all the PMs n ascendng order based on the average

8 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD resource utlzaton of each PM, and desgn an algorthm movng out the VNFRs from the PMs of low resource utlzaton to the PMs of hgh resource utlzaton. If all the VNFRs are moved out from one PM, the PM s released Intra Cluster Adjustment The frst step s to gather together the VNFRs demandng the same type VNF ntra each cluster. To state our algorthm more brefly, we treat all those VNFRs n the same PM that demand the same type of VNF as a whole, ndcated as VNFR w. When we mgrate a VNFR w from one PM to another, we mean that all the correspondng VNFRs demandng the same type VNF are mgrated. Algorthm 2 gves the adjustment process ntra each of the clusters. Frst, we calculate the traffc gong through each VNFR w (lne 3 n Algorthm 2), whch s ndcated as VNFR t w. VNFR t w s the accumulaton of the traffc flowng n and out of all VNFRs belongng to the correspondng VNFR w. After the calculaton of all the VNFR t w s, we pck out the VNFR w that has the mnmal VNFR t w n each of the PMs (lne 10 n Algorthm 2) and store them n a lst, whch s ndcated as D. Nextly, the target PMs are chosen for each of these VNFR w s (lne 11 n Algorthm 2), and these VNFR w s are moved to the target PMs, respectvely. Algorthm 2. Intra Cluster Adjustment Algorthm 1: Input: Number of used PMs: P 0, Status of datacenter: S 1 ; 2: Output: Number of used PMs: P 1, Status of datacenter: S 2 ; 3: Calculate the traffc gong through each VNFR w n each PM, ndcated as VNFR t w. 4: Calculate the number of used clusters N c. 5: ¼ 0. 6: whle <N c do 7: Store the PMs n cluster z to a lst L PM. 8: Backup the status of datacenter. 9: whle L PM s not empty do 10: Fnd the VNFR w that has the mnmal VNFR t w n each of the PMs belongng to cluster z, ndcated as D ¼fD 0 ; D 1...D h 1 g, h s the sze of the cluster. 11: Choose the target PM for each VNFR w n D. Functon 1 12: Mgrate each VNFR w n D to the target PM. 13: f elmnate_exceed() s true then 14: Check all the PMs n the cluster z, remove the unqualfed PM from L PM. 15: else 16: Restore the status of datacenter. 17: Break. 18: end 19: end 20: ¼ þ 1. 21: end 22: return P 1, S 2. The detals about how to fnd the target PM for each VNFR w n D are shown n Functon 1. For a VNFR w n D, whch s ndcated as D j, the target PM must have the VNFR w that demands the same type of VNF wth D j. Then we take two factors nto consderaton, whch are the VNFR t w of the VNFR w n the target PM (lne 9 n Functon 1) and the correlaton between the workload of D j and the workload of the total VNFRs already n the target PM (lne 7 and n Functon 1). We quantfy the above two consderatons as a seres of weghted factors, v (lne 18 n Functon 1), and v s the weghted dfference between normalzed VNFR t w and normalzed correlaton coeffcent. Fnally, the PM wth the bggest v s chosen as the target PM for D j. In ths way, on the one hand, we avod the movng of the VNFR w wth larger VNFR t w, ncurrng as less nteractng traffc between PMs as possble. On the other hand, t leads to hgher utlzaton of the resources n PMs, owng to the less correlaton between the workload of D j and the workload of the total VNFRs already n the target PM. a and b are the adjustment factors n the calculaton of v. Functon 1. Choose the Target PM 1: Input: Status of datacenter: S 2, D; 2: Output: The target PMs lst: L t ; 3: j ¼ 0. 4: whle j<hdo 5: for each PM n cluster z do 6: f D j n the PM then 7: Calculate the resource demand matrx M j for D j. 8: else f the PM has the same knd VNFR w wth D j then 9: Record VNFR t w of VNFR w n the PM as t, and store t n T. 10: 11: end 12: for each PM n cluster z that has the same knd VNFR w wth D j do 13: Establsh the resource demand matrx for the VNFRs already n the PM, ndcated as M u. 14: Calculate the correlaton coeffcent between M j and M u as m, then store t n. 15: end 16: Normalze all the values n T and. 17: for each par of values for one PM n T and do 18: Calculate the weghted factor v, v ¼ a t b m : 19: end 20: Pck out the PM that has the bggest v nto L t. 21: j ¼ j þ 1. 22: end 23: return L t. By gatherng the VNFR w sdemandngthesametypeof VNF together, some PMs do not need to place the correspondng VNF nstances. Therefore BRCs n these PMs are saved. However, the above processes may cause resource over-utlzaton on some PMs. So functon elmnate_exceed() (lne 13 n Algorthm 2) s desgned to elmnate the resource over-utlzaton. The man dea of elmnate_exceed() s to mgrate part of the VNFRs from the PM wth resource over-utlzaton to other PMs that have free resources. Only f there s no resource overutlzaton n all PMs belongng to the current cluster, elmnate_exceed() returns True. If not, we abandon all the above processes, and restore the status of the datacenter (lne 16 n Algorthm 2), then start to deal wth next cluster (lnes 17 and 20 n Algorthm 2).

9 1672 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY 2018 For lne 14 n Algorthm 2, the PM s unqualfed f one of the followng condtons s satsfed: (1) Condton 1: The average resdual bandwdth of the lnk that connects the PM drectly s less than half of the orgnal value, where the orgnal value s the resdual bandwdth after the frst stage. (2) Condton 2: All the VNFRs n the PM are moved out. Condton 1 s to ensure that there s stll bandwdth left to execute the second adjustment process. Condton 2 s to free out the empty PM from the adjustment process. After mplementng Algorthm 2, almost each PM has some free CPU and memory resources. The next step s to gather these free resources together to empty more PMs Inter Clusters Adjustment Algorthm 3 along wth Procedures 1 and 2 descrbes the second adjustment process n detal. Frst the used PMs are sorted based on ther average resource utlzaton n ascendng order, then we try to move all VNFRs one by one from the PMs of low resource utlzaton to the PMs of hgh resource utlzaton. Algorthm 3. Inter Clusters Adjustment Algorthm 1: Input: Number of used PMs: P 1, Status of datacenter: S 2 ; 2: Output: Number of used PMs: P 2, Status of datacenter: S 3 ; 3: Calculate the average resource utlzaton of all used PMs. 4: Put all used PMs nto a lst P l, and sort them n ascendng order based on the average resource utlzaton. 5: Set two ponters, r b and r e. r b ponts to the source PM, ndcated as PM s, at the begnnng of P l. r e ponts to the destnaton PM, ndcated as PM d, at the end of P l, and a shft value d, d ¼ 0 ntally. 6: whle 1 do 7: Backup the current status of datacenter. 8: whle r b < r e do 9: for each VNFR n PM s do 10: f PM d can hold the VNFR then 11: Mgrate the VNFR to PM d. 12: Modfy the status of datacenter. 13: end 14: end 15: f PM s s empty then 16: P 1 ¼ P : break. 18: else 19: r e ¼ r e 1. 20: end 21: end 22: f r b ¼ r e then 23: Procedure 1. 24: else f r b < r e then 25: Procedure 2. 26: 27: end 28: P 2 ¼ P 1. 29: return P 2, S 3. In order to realze the mgraton effcently, we put all used PMs n a lst P l, then we set two ponters r b and r e n the begnnng and end poston respectvely, and the correspondng PMs are called source PM, PM s and destnaton PM, PM d (lne 5 n Algorthm 3). After that, we move the VNFRs from PM s to PM t (lne 6-27 n Algorthm 3). If all VNFRs n PM s are moved out, PM s s empty and the number of used PMs decreases (lne 16 n Algorthm 3). If not, we decrease r e (lne 19 n Algorthm 3), choosng a new PM as PM d for the VNFRs n PM s. Lnes are used to check the causes that break the nner whle loop (lnes 8-21 n Algorthm 3). Procedure 1. Modfy r b and r e when r b ¼ r e 1: Restore the precous status of datacenter. 2: d ¼ d þ 1. 3: Set r e to the end of P l. 4: f r b þ d r e then 5: break. 6: else 7: Exchange the PM correspondng to r b þ d and r b. 8: end Procedure 2. Modfy r b and r e when r b < r e 1: r b ¼ r b þ 1. 2: Set r e to the end of P l. 3: f r b ¼ r e then 4: break. 5: else 6: contnue. 7: end If r b ¼ r e, t mples that all PMs whose resource utlzaton arehgherthanthatofpm s have been traversed, and the VNFRs n PM s cannot be empted. So we need to restore the prevous status of the datacenter to the begnnng of ths round adjustment (lne 1 n Procedure 1). Then we ncrease the shft value d n order to choose the next PM s (lne 2 n Procedure 1), and roll back r e to the end of P l (lne 3 n Procedure 1). If r b þ d r e (lne 4 n Procedure 1), t means that each of the used PMs has been treated as PM s once, and we should stop the adjustment process (lne 5 n Procedure 1). Otherwse, we exchange the PMs correspondng to r b þ d and r b, makng the PM correspondng to r b þ d as the new PM s. If r b < r e, t mples that PM s s empted. In ths occason, we need to choose the next PM n lst P l as the new PM s (lnes 1 n Procedure 2). If r b ¼ r e (lne 3 n Procedure 2) after the ncreasng of r b, t means that each of the used PMs has been treated as PM s once, and we need to stop the adjustment process. When T-SAT s completed, the mappng of all VNFRs s determned. Then we can place correspondng VNFs and allocate the resources to each VNF accordng to the demands of VNFRs on t. 5.4 Complexty Analyss We gve a detaled tme complexty analyss about T-SAT n ths part. For Algorthm 1, the complexty manly focuses on the two whle loops n lnes 4-34 and lnes 9-33 respectvely. For the nner whle loop (lnes 9-33 n Algorthm 1), the tme complexty s

10 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD jg c jþðjg c j 1Þþþ1 ¼jG c jðjg c jþ1þ=2: (21) We assume that each PM can hold SFCRs n average and jgj= s an nteger for the sake of dervng the formulatons clearly. Then for the outer whle loop (lnes 4-34 n Algorthm 1), t runs jgj= tmes, and the total tme complexty of Algorthm 1 s jgjðjgjþ1þ=2þðjgj ÞððjGj Þþ1Þ=2þþ ð þ 1Þ=2 ¼ðjGjþ Þð2jGj 2 þ jgjþ3jgjþ=ð12 Þ: (22) It can be seen that the tme complexty of Algorthm 1 s at the level of OðjGj 3 = Þ. For Algorthms 2 and 3, we frst ntroduce two varables, N c and h, whch are the number of the clusters contanng used PMs and the number of PMs n one cluster respectvely. So there are N c h used PMs n total at most. The tme complexty of the whle loop (lnes 9-19 n Algorthm 2) manly results from the process of lne 10 and lne 11 n t. Each PM contans F knds of VNFRs at most, so the tme complexty of lne 10 n Algorthm 2 s h F at most. The tme complexty of lne 11 n Algorthm 2 (Functon 1) s 5h. So the tme complexty of the whle loop (lnes 9-19 n Algorthm 2) s hðhf þ 5hÞ, and the tme complexty of Algorthm 2 s N c hðhf þ 5hÞ, whch s at the level of OðN c h 2 Þ. Knowng the total number of VNFRs, whch s P jgj 1 jc gj, and the total used PMs after the frst stage, whch s N c h. Then n each PM, there are about P jgj 1 jc gj=ðn c hþ VNFRs n average. So for Algorthm 3, the teraton tmes of the nner whle (lnes 8-21 n Algorthm 3) s P jgj 1 jc gj=ðn c hþðn c hþ¼ P jgj 1 jc gj. We need to traverse all the used PMs to see f t can be empted. So the above nner whle loop runs N c h tmes, then the tme complexty upper bound of Algorthm 3 s N c h P jgj 1 jc gj. Consderng jgj= ¼ N c h, the tme complexty of Algorthm 3 s at the level of Oð P jgj 1 jc gjjgj= Þ. In general, the total complexty of correlaton-based T- SAT soluton s OðjGj 3 = þ N c h 2 þ P jgj 1 jc gjjgj= Þ. We should notce that h s constant when the datacenter topology s determned, N c s also lmted by the sze of datacenter and P jgj 1 jc gj < jgj 2. So we can see that the tme complexty of T-SAT s at the level of OðjGj 3 = Þ. 6 PERFORMANCE EVALUATION In ths secton, we evaluate T-SAT n detal and compare t wth the modfed FFD (FFD_w), PowerNets [27] and Algorthm H (AH) [9]. In FFD_w, we sort all the VNFRs n descendng order based on ther average resource demand frst, and then the VNFRs are mapped on the PMs one by one based on frst-ft (FF) algorthm. For each group of results, we use the average of the results from 10 groups of experments to reduce the accdental errors. 6.1 Smulaton Results wth Gaussan Data Smulaton Settngs In evaluaton, we use a 3-layer fat-tree of Clos topology [32], where the sze of the topology s determned by the number of ports n the swtches. For a k-ary Clos topology, there are ðk=2þ 2 k-port core swtches; k pods, each of whch contanng two layers of k=2 swtches; and k 3 =4 hosts. We set k ¼ 16 n the smulaton. Each PM has 100 unts CPU resources, 100 unts memory resources, and the lnk capacty s a varable. Both a and b are set to be 0.5 n Functon 1. u cpu ; u mem and u lnk are all set to be Workload of SFCRs The workload of one SFCR s the lnear accumulaton of all VNFRs workloads n t. For one VNFR, ts CPU, memory and bandwdth workload are all generated accordng! G x ðt; sþ ¼ Xk 1 A pffffffffff exp ðx t Þ 2 : (23) 2p s ¼0 2s 2 Eq. (23) s the accumulaton of k Gaussan functons. ffffffff p A 2p s ndcates the ampltude of th Gaussan functon. Because every Gaussan functon has only one peak, the value of whch equals the ampltude, and kt s are dfferent from each other, so k also ndcates the number of peak values of one workload. In the smulaton, we have 50 percent Elephant VNFRs whose CPU A, memory A and bandwdth A are all randomzed values obeyng unform dstrbuton from 2 to 3 unts and 50 percent Mce VNFRs whose CPU A, memory A and bandwdth A are all randomzed values obeyng unform dstrbuton from 0.2 to 0.3 unts. For both the Elephant and Mce VNFRs, the s s are randomzed values obeyng unform dstrbuton from 0.35 to 0.4. For one SFCR, the VNFRs n t belong to only one type, ether Elephant VNFR or Mce VNFR. We assume that the tme nterval T s a day. As we know, t determnes the poston of the peak value of one Gaussan functon, so we separate the SFCRs nto three types accordng to dfferent t s: Day SFCR (d-sfcr): The peak loads of ths knd of SFCR manly show up at daytme. We apply the 24- hour clock, so t s are chosen from tme ponts [8,9,10,11,14,15,16,17,20]. Nght SFCR (n-sfcr): The peak loads manly show up at nght. And t s are chosen from tme ponts [0,1,2,3,4,5,22,23]. Randomzed Tme SFCR (r-sfcr): The peak loads show up randomly at the whole day. It occupes a share of the total SFCRs, and the rest are separated evenly between the d-sfcr and the n-sfcr. Fg. 3 shows the Gaussan workloads of three types of SFCRs, n whch the numbers wth the underlne are the ndexes of dfferent knds of VNFRs, whch vary from 0 to 19, so F ¼ 20 n the smulaton. SFCRs need not to have the same number or same knds of VNFRs n our smulatons. In fact, the number of VNFRs n each SFCR s a random varable varyng from 1 to 20. In addton, n Fg. 3, n band s the traffc that flows n one VNFR, whle out band s the traffc that flows out of one VNFR. Generally, out band s dfferent from n band after the processng of one VNF. Because G x ðt; sþ s a contnuous functon, we apply the unform samplng to the workload to get the workload vectors. In the smulaton, we sample the workload every other 0.1 from 0 to 24.

11 1674 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY 2018 Fg. 3. Gaussan workload of dfferent knds of SFCRs Results wth Gaussan Data a) Performance Comparsons wth Dfferent BRCs. We expose the nfluence of BRCs on the performance of T-SAT n ths part. To demonstrate the relatonshp more brefly and clearly, we set values of c-brcs and m-brcs as the same. Fg. 4a shows the results. Error bars represent the 95 percent confdence ntervals, so are they n the followng results. From the fgure, we can see that wth the ncreasng of BRCs, T-SAT always performs better than the benchmarks. It s because we combne the VNFRs that demand the same type of VNF together, and then fewer VNF nstances are placed, so BRCs are fewer. Moreover, the bgger the volume of BRCs s, the better T-SAT performs. The fewer PMs, the less power consumpton and operaton cost consequently. b) Performance Comparsons wth Dfferent Lnk Capacty. Another factor that may nfluence the performance of our soluton s the lnk capacty. So n ths part, we take the lnk capacty as the varable and compare the performance of dfferent solutons. From Fg. 4b, we can see that wth the ncreasng of lnk capacty, the number of used PMs decreases. Because more bandwdth resources lead to more adequate adjustment of T-SAT, then the used PMs are fewer. Moreover, T-SAT performs much better than the benchmarks when the bandwdth resource s scarce. Because T-SAT s nclned to put the whole SFCR n one PM, and then there s less nteractng traffc between the PMs. Subsequently, there are fewer lnk bottlenecks n the network, and the resources of PM can be utlzed more adequately. Besdes, T-SAT consders the correlaton between the workloads of dfferent SFCRs, whch leads to a hgher utlzaton of PMs resources. c) Performance Comparsons wth Dfferent Number of SFCRs. In ths part, we evaluate the performance of dfferent solutons va the varyng number of SFCRs. As Fg. 4c shows, the number of used PMs ncreases wth more SFCRs. Meanwhle, T-SAT results n the least PMs. Ths phenomenon ndcates that T-SAT has a consstent advantage over the benchmarks wth the ncreasng of SFCRs. d) Performance Comparsons wth Dfferent Rato of r-sfcr. In ths part, we treat the rato of r-sfcr as the varable and compare the performance of dfferent solutons. From Fg. 4d, we can see that the more r-sfcrs, the more used PMs resultng from the solutons except for FFD_w. Meanwhle, T-SAT stll performs the best. For the performance declnng of T-SAT, t s because the more r-sfcr, the fewer SFCRs whose workloads are complementary, and then the resource consumptons are more. As for the performance of FFD_w, t s because FFD_w does not consder the workload correlaton between dfferent VNFRs, so t s more lkely to put VNFRs whose workloads are less related together when the share of r-sfcr ncreases, whch ncreases the utlzaton of the PMs resources. As a result, the number of used PMs decreases wth the ncreasng of r-sfcr rato for FFD_w. e) T-SAT versus Optmal Results. We reveal the gap between the performance of T-SAT and the optmal results n ths part. Wth the help of Gurob [22], the optmal results are derved when the number of SFCRs s small. In the smulaton of ths part, each SFCR contans 10 VNFRs n average. From Fg. 5, we can see that T-SAT can acqure near effect compared wth the optmal results derved by Gurob when the number of SFCRs s small. f) The Utlzaton of Network Resources. Fg. 6 shows the CDF curves of resource utlzaton resultng from dfferent solutons. The CDF of resource utlzaton s calculated usng CDFð% 0 Þ¼ Nð% % 0Þ R t ; (24) Fg. 4. Performance comparson of dfferent algorthms wth Gaussan data.

12 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD TABLE 2 Jobs Dstrbuton Based on Task Number Task Number n a Job The Correspondng Number of Jobs 1 2, to to Fg. 5. Performance of dfferent algorthms versus optmal results. where R t s the number of total used PMs or lnks, and Nð% % 0 Þ ndcates the number of PMs or lnks whose resource utlzaton % are below the threshold % 0. Fgs. 6a and 6b show the CDF curves of CPU and memory utlzaton wth BRCs. We can see that the CPU and memory utlzaton of 80 percent PMs exceeds 70 percent by applyng T-SAT, Powernets, and AH, whch s very hgh. The hgh utlzaton owes to the consderaton of tmevaryng feature of the workloads. Fg. 6c s the CDF of bandwdth utlzaton. We can see that the bandwdth utlzaton s also very hgh, and the utlzaton of about 80 percent PMs exceeds 50 percent. Fgs. 6d and 6e are the CDF curves of CPU and memory utlzaton wthout accountng for the occupaton of BRCs. Excludng BRCs, the resource consumptons are only caused by users demands. Thus for the same set of SFCRs, the hgher resource utlzaton n the PMs, the fewer PMs wll be used. From the fgures, we can see that T-SAT has the hghest resource utlzaton excludng BRCs, whch strengthens ts performance advantages over the benchmarks. 6.2 Smulaton Results wth Google Data Smulaton Settngs In ths secton, we evaluate T-SAT wth the data derved from Google Cluster Trace [23]. Google cluster trace records the resource utlzaton of the machnes n Google s datacenters n the form of jobs. A job s comprsed of one or more tasks, each of whch s accompaned by a set of resource requrements used for schedulng (packng) the tasks onto machnes. The relatonshp between job and tasks s very smlar to the relatonshp between SFCR and VNFRs. So we use jobs n the Google cluster trace to smulate the SFCRs and use tasks n the jobs to smulate the VNFRs n the SFCRs. Table 2 shows the dstrbuton of jobs derved from part of the data set based on the task number n them. We choose the jobs whose task numbers are between 2 and 20 as our smulaton data, so the number of total jobs s However, the trace only provdes the normalzed resource consumptons of CPU and memory, so we use Eq. (25) to ca lculate the normalzed resource consumptons of bandwdth U lnk ¼ 0:5 U cpu þ 0:5 U mem : (25) We use the same topology as that n the Gaussan data part. Because the CPU and memory consumptons are normalzed n Google cluster trace, we set both the capacty of CPU and memory resource to 1 unt. As for the lnk capacty, we treat t as a varable Results wth Google Data Fg. 7 shows the results wth Google cluster trace data. From the fgures, we can draw the consstent conclusons wth the Gaussan data. However, we fnd that the advantages of T- SAT are not so notceable. Ths phenomenon exposes one of the shortcomngs of correlaton-based algorthms, whch s senstve to the data set. When the workloads of dfferent SFCRs or VNFRs have less correlaton wth each other, the correlaton-based algorthms usually can acheve a better performance. However, f the workloads always concde wth each other, the correlaton based algorthms do not exhbt ther advantages sgnfcantly. Fg. 8 shows the utlzaton of CPU, memory and bandwdth wth Google data. From the fgure, we can see that both average utlzaton of CPU and bandwdth are about 50 percent. But for the memory utlzaton, t s so hgh that Fg. 6. CDF of network resource utlzaton wth Gaussan data.

13 1676 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 29, NO. 7, JULY 2018 Fg. 7. Performance comparson of dfferent algorthms wth Google data. Fg. 8. CDF of network resource utlzaton wth Google data. about 80 percent PMs exceed 90 percent, from whch we can nfer that the Google trace data we apply n the smulatons has a preference for memory resource. 7 CONCLUSION We address the VNF placement problem n cloud datacenter wth a set of known SFCRs. In order to optmze the utlzaton of network resources, the tme-varyng workloads and BRCs are taken nto consderaton. Besdes, we reveal the conflcton between BRCs and the nteractng traffc between the PMs. Then we formulate the problem as an ILP model, amng to mnmze the number of used PMs, and propose a two-stage heurstc algorthm, T-SAT. To evaluate the performance of our soluton, we make an analyss n detal through smulatons wth artfcal data we compose wth Gaussan functons and Google trace data. In the smulatons, we compare T-SAT wth the benchmarks n exstng. The smulaton results show that T-SAT can mprove the utlzaton of network resources sgnfcantly and acqure a smaller number of used PMs than the benchmarks. Moreover, the performance of T- SAT s very near to the optmal results. Meanwhle, we reveal one shortcomng of the correlaton based solutons, whch s senstve to the data set. In fact, the consderatons of tme-varyng workloads and BRCs not only apply to VNF placement problem n the datacenter networks, and t wll be our future work to study the VNF placement problem wth the two factors n a wder scenaro. ACKNOWLEDGMENTS The authors sncerely thank the anonymous referees for ther valuable suggestons that have led to the present mproved verson of the orgnal manuscrpt. Ths work s partly supported by the Natonal Natural Scence Foundaton of Chna under Grant No and No , Youth Innovaton Promoton Assocaton CAS under Grant No and the Fundamental Research Funds for the Central Unverstes. REFERENCES [1] Network functons vrtualsaton (NFV), ETSI, NFVGS, (2015). [Onlne]. Avalable: NFV/Docs/NFV_Whte_Paper3.pdf [2] H. Jeon and B. Lee, Network servce channg challenges for VNF outsourcng n network functon vrtualzaton, n Proc. Int. Conf. Inf. Commun. Technol. Convergence, 2015, pp [3] B. Han, V. Gopalakrshnan, L. J, and S. Lee, Network functon vrtualzaton: Challenges and opportuntes for nnovatons, IEEE Commun. Mag., vol. 53, no. 2, pp , Feb [4] J. Sherry, S. Hasan, C. Scott, A. Krshnamurthy, S. Ratnasamy, and V. Sekar, Makng mddleboxes someone else s problem: Network processng as a cloud servce, SIGCOMM Comput. Commun. Rev., vol. 42, no. 4, pp , [5] G. Gbb, H. Zeng, and N. McKeown, Outsourcng network functonalty, n Proc. ACM SIGCOMM Workshop Hot Topcs Softw. Defned Netw., 2012, pp [6] S. Mehraghdam, M. Keller, and H. Karl, Specfyng and placng chans of vrtual network functons, n Proc. 3rd Int. Conf. Cloud Netw., 2014, pp [7] Z. Ye, X. Cao, J. Wang, H. Yu, and C. Qao, Jont topology desgn and mappng of servce functon chans for effcent, scalable, and relable network functons vrtualzaton, IEEE Netw., vol. 30, no. 3, pp , May/Jun [8] T.-W. Kuo, B.-H. Lou, K. C.-J. Ln, and M.-J. Tsa, Deployng chans of vrtual network functons: On the relaton between lnk and server usage, n Proc. IEEE INFOCOM, 2016, pp [9] M.Xa,M.Shrazpour,Y.Zhang,H.Green,andA.Takacs, Network functon placement for NFV channg n packet/optcal datacenters, IEEE J. Lghtw. Technol., vol. 33, no. 8, pp , Apr [10] P.-W. Ch, Y.-C. Huang, and C.-L. Le, Effcent NFV deployment n data center networks, n Proc. IEEE Int. Conf. Commun., 2015, pp

14 LI ET AL.: VIRTUAL NETWORK FUNCTION PLACEMENT CONSIDERING RESOURCE OPTIMIZATION AND SFC REQUESTS IN CLOUD [11] R. Cohen, L. Lewn-Eytan, J. S. Naor, and D. Raz, Near optmal placement of vrtual network functons, n Proc. IEEE INFOCOM, 2015, pp [12] J. Halpern and C. Pgnataro, Servce functon channg (SFC) archtecture, (2015). [Onlne]. Avalable: rfc/rfc7665.txt [13] Network Functons Vrtualsaton (NFV); Management and Orchestraton, NFV-MAN, ETSI, NFVGS, vol. 1, p. v0, 2014, [Onlne]. Avalable: MAN/001_099/001/ _60/gs_NFV-MAN001v010101p.pdf [14] C.-P. Bezemer and A. Zadman, Mult-tenant SaaS applcatons: Mantenance dream or nghtmare? n Proc. Jont ERCIM Workshop Softw. Evoluton Int. Workshop Prncples Softw. Evoluton, 2010, pp [15] V. Sekar, N. Eg, S. Ratnasamy, M. K. Reter, and G. Sh, Desgn and mplementaton of a consoldated mddlebox archtecture, n Proc. 9th USENIX Conf. Netw. Syst. Des. Implementaton, 2012, pp [16] X. Meng, C. Isc, J. Kephart, L. Zhang, E. Boullet, and D. Pendaraks, Effcent resource provsonng n compute clouds va VM multplexng, n Proc. 7th Int. Conf. Autonomc Comput., 2010, pp [17] W. Ln, S. Xu, J. L, L. Xu, and Z. Peng, Desgn and theoretcal analyss of vrtual machne placement algorthm based on peak workload characterstcs, Soft Comput., vol. 21, no. 5, pp , [18] BEA Weblogc Applcaton Consoldaton Strateges, WLDJ. [Onlne]. Avalable: [19] C.-J. Guo, W. Sun, Z.-B. Jang, Y. Huang, B. Gao, and Z.-H. Wang, Study of software as a servce support platform for small and medum busnesses, n New Fronters n Informaton and Software as Servces. Berln Germany: Sprnger, 2011, pp [20] M. F. Bar, S. R. Chowdhury, R. Ahmed, and R. Boutaba, On orchestratng vrtual network functons, n Proc. 11th Int. Conf. Netw. Servce Manage., 2015, pp [21] R. Mjumb, J. Serrat, J.-L. Gorrcho, N. Bouten, F. De Turck, and R. Boutaba, Network functon vrtualzaton: State-of-the-art and research challenges, IEEE Commun. Surveys Tuts., vol. 18, no. 1, pp , Jan. Mar [22] Gurob optmzer reference manual, verson 7.0, Gurob. [Onlne]. Avalable: pdf [23] C. Ress, J. Wlkes, and J. L. Hellersten, Google cluster-usage traces: Format+ schema, Google Inc., Whte Paper, pp. 1 14, [24] W. Rankothge, F. Le, A. Russo, and J. Lobo, Optmzng resource allocaton for vrtualzed network functons n a cloud center usng genetc algorthms, IEEE Trans. Netw. Servce Manage., vol. 14, no. 2, pp , Jun [25] Z. A. Mann and A. Metzger, Optmzed cloud deployment of mult-tenant software consderng data protecton concerns, n Proc. 17th IEEE/ACM Int. Symp. Cluster Cloud Grd Comput., 2017, pp [26] S. Zhang, Z. Qan, Z. Luo, J. Wu, and S. Lu, Burstness-aware resource reservaton for server consoldaton n computng clouds, IEEE Trans. Parallel Dstrb. Syst., vol. 27, no. 4, pp , Apr [27] K. Zheng, X. Wang, L. L, and X. Wang, Jont power optmzaton of data center network and servers wth correlaton analyss, n Proc. IEEE INFOCOM, 2014, pp [28] D. Banks, J. Erckson, M. Rhodes, and J. S. Erckson, Mult-tenancy n cloud-based collaboraton servces, 2009, [Onlne]. Avalable: pdf?orgn=publca ton_detal [29] S. Natarajan, et al., An Analyss of Lghtweght Vrtualzaton Technologes for NFV, [Onlne]. Avalable: html/draft-natarajan-nfvrg-contaners-for-nfv-03 [30] X. L, A. Ventresque, J. O. Iglesas, and J. Murphy, Scalable correlaton-aware vrtual machne consoldaton usng two-phase clusterng, n Proc. Int. Conf. Hgh Perform. Comput. Smul., 2015, pp [31] V. Perlbakas, Dstance measures for PCA-based face recognton, Pattern Recognt. Lett., vol. 25, no. 6, pp , [32] M. Al-Fares, A. Loukssas, and A. Vahdat, A scalable, commodty data center network archtecture, SIGCOMM Comput. Commun. Rev., vol. 38, no. 4, pp , Defang L receved the BS degree from the Department of Electronc Engneerng and Informaton Scence (EEIS), Unversty of Scence and Technology of Chna (USTC), n He s workng toward the PhD degree at the Unversty of Scence and Technology of Chna wth hs advsor Peln Hong now. Hs research nterests nclude SDN, NFV, and the network resource orchestraton and management. He s a student member of the IEEE. Peln Hong receved the BS and MS degrees from the Department of Electronc Engneerng and Informaton Scence (EEIS), Unversty of Scence and Technology of Chna (USTC), n 1983 and 1986, respectvely. Currently, she s a professor and advsor for PhD canddates n the Department of EEIS, USTC. Her research nterests nclude next-generaton Internet, polcy control, IP QoS, and nformaton securty. She has publshed 2 books and more than 100 academc papers n several journals and conference proceedngs. Kapng Xue (M09-SM15) receved the BS degree from the Department of Informaton Securty, Unversty of Scence and Technology of Chna (USTC), n 2003 and the PhD degree from the Department of Electronc Engneerng and Informaton Scence (EEIS), USTC, n Currently, he s an assocate professor n the Department of Informaton Securty and Department of EEIS, USTC. Hs research nterests nclude nextgeneraton Internet, dstrbuted networks, and network securty. He s a senor member of the IEEE. Janng Pe receved the BS degree from the Department of Informaton and Electrcal Engneerng (IEE), Chna Unversty of Mnng and Technology (CUMT), n He s workng toward the MS degree at the Unversty of Scence and Technology of Chna (USTC) wth hs advsor Peln Hong now. Hs research nterests nclude SDN, NFV, and the network resource orchestraton and management. " For more nformaton on ths or any other computng topc, please vst our Dgtal Lbrary at