Service Overlay Forest Embedding for Software-Defined Cloud Networks

Transcription

1 Serice Oerlay Forest Embedding for Software-Defined Cloud Networks Jian-Jhih Kuo, Shan-Hsiang Shen, Ming-Hong Yang, De-Nian Yang, Ming-Jer Tsai and Wen-Tsuen Chen Inst. of Information Science, Academia Sinica, Taipei, Taiwan Dept. of Computer Science & Information Engineering, National Taiwan Uniersity of Science & Technology, Taipei, Taiwan Dept. of Computer Science & Engineering, Uniersity of Minnesota, Minneapolis MN, USA Dept. of Computer Science, National Tsing Hua Uniersity, Hsinchu, Taiwan and arxi:73.95 [cs.ni] 7 Mar 7 Abstract Network Function Virtualization (NFV) on Software- Defined Networks (SDN) can effectiely optimize the allocation of Virtual Network Functions (VNFs) and the routing of network flows simultaneously. Neertheless, most preious studies on NFV focus on unicast serice chains and thereby are not scalable to support a large number of destinations in multicast. On the other hand, the allocation of VNFs has not been supported in the current SDN multicast routing algorithms. In this paper, therefore, we make the first attempt to tackle a new challenging problem for finding a serice forest with multiple serice trees, where each tree contains multiple VNFs required by each destination. Specifically, we formulate a new optimization, named Serice Oerlay Forest (SOF), to minimize the total cost of all allocated VNFs and all multicast trees in the forest. We design a new 3ρ - approximation algorithm to sole the problem, where ρ denotes the best approximation ratio of the Steiner Tree problem, and the distributed implementation of the algorithm is also presented. Simulation results on real networks for data centers manifest that the proposed algorithm outperforms the existing ones by oer 5%. Moreoer, the implementation of an experimental SDN with HP OpenFlow switches indicates that SOF can significantly improe the QoE of the Youtube serice. I. INTRODUCTION The media industry is now experiencing a major change that alters user subscription patterns and thereby inspires the architects to rethink the design []. For example, the lie ideo streaming on Anato [] enables online ideo editing for content proiders, ad insertion for adertisers, caching, and transcoding for heterogeneous user deices. Google has acquired Anato with the aboe abundant functions and integrated its architecture into Google Cloud to deelop the nextgeneration Youtube. i Therefore, it is enisaged that the nextgeneration Youtube requires more computation functionalities and resources in the cloud. For distributed collaboratie irtual reality (VR), it is also crucial to allocate distributed computation resources for important tasks such as collision detection, geometric constraint matching, synchronization, iew consistency, concurrency and interest management [3] [5]. Network Function Virtualization (NFV) has been regarded as a promising way [], [6] that exploits Virtual Machines (VMs) to diide the required function into building blocks connected with a serice chain [7]. A serice chain passes through a set of Virtual Network Functions (VNFs) in sequence, and Netflix [8] has adopted AWS [9] to support the serice chains. Current commercial solutions usually assign an indiidual serice chain i for each end user for unicast [] [3]. Neertheless, it is expected that this approach is not scalable because duplicated VNFs and network traffic are inoled to sere all users if they require the same content, such as lie/linear content broadcast. The global consumer research [] manifests that although the unicast ideo on demand becomes more and more popular, the lie/linear content broadcast and multicast nowadays still account for oer 5% of iewing hours per week, from companies such as Sony Crackle [5] and Pluto TV [6], because it effectiely attracts the users through a shared social experience to instantly access the contents. Howeer, currently there is no effectie solution to support large-scale content distributions with abundant computation functionalities for content proiders and end users. For scalable one-to-many communications, multicast exploits a tree to replicate the packets in branching routers. Compared with unicast flows, a multicast tree can effectiely reduce the bandwidth consumption in backbone networks by oer 5% [7], especially for multimedia traffic [8]. Currently, shortest-path trees are employed by Internet standards (such as PIM-SM [9]) because they can be efficiently constructed in a distributed manner. Neertheless, the routing is not flexible since the path from the source to each destination needs to follow the corresponding shortest path. Recently, the flexible routing for traffic engineering becomes increasingly important with the emergence of Software-Defined Networks (SDNs), whereas centralized computation can be facilitated in an SDN controller to find the optimal routing, such as Steiner Tree [] in Graph Theory or its ariations [], []. Thus, multicast traffic engineering has been regarded ery promising for SDNs. Neertheless, the aboe approaches and other existing multicast routing algorithms [3], [] are not designed to support NFV because the nodes (e.g., the source and destinations) that need to be connected in a tree are specified as the problem input. On the contrary, here VNFs are also required to be spanned in a tree for NFV, and the problem is more challenging since VMs also need to be selected, instead of being assigned as the problem input. Moreoer, multicast NFV indeed is more complicated when it is necessary to employ multiple multicast trees as a forest for a group of destinations, and this feature is crucial for Content Delier Networks (CDNs) with multiple ideo source serers. In this case, the ideo source also needs to be chosen for each end user [5]. In this paper, therefore, we make the first attempt to explore the resource allocation problem (i.e., both the VM selection,

2 source selection, and the tree routing) for a serice forest inoling multiple multicast trees, where the path from a source to each destination needs to traerse a sequence of demanded serices (i.e., a serice chain) in the tree. ii We formulate a new optimization problem for multi-tree NFV in software defined cloud networks, named, Serice Oerlay Forest (SOF), to minimize the total cost of the selected VMs and trees. Gien the sources and the destinations with each destination requiring a chain of serices, the SOF problem aims at finding an oerlay forest that ) connects each destination to a source and ) isits the demanded serices in selected VMs in sequence before reaching the destinations. Fig. first compares a serice tree and a serice forest. Fig. is the input network with the cost labeled beside each node and edge to represent the link connection cost and the VM setup cost, respectiely. Assume that there are two destinations 9 and, and their demanded serice chain consists of two VNFs, f and f in order. A Steiner tree in Fig. spanning source node and both destinations incurs the total cost as 3 if VMs and 3 are employed. Note that the edge between VMs and 3 is isited twice to reach destination, and the cost of the edge is thus required to be included twice. More specifically, the edge costs from source to VM 3 (f ), from VM 3 to VM (f ), and from VM to destinations 9,, are, 3, = 8, respectiely. Thus, the edge cost is = 3 while the node cost is + =. By contrast, the cost of a serice forest with two trees and four VMs selected is in Fig., which significantly reduces the cost by about 6 %. This example manifests that consolidating the serices in few VMs may not always lead to the smallest cost because the edges to connect multiple destinations are also important. Therefore, multiple trees with multiple sources are promising to further reduce the cost. iii In this paper, we first proe that the problem is NP-hard. To inestigate the problem in depth, we will step-by-step reeal the thinking process of the algorithm design from the singlesource case to the general case, and then propose a 3ρ - approximation algorithm, i named Serice Oerlay Forest Deployment Algorithm () for the general case, where ρ denotes the best approximation ratio of the Steiner Tree problem (e.g., the current best one is.39). The single-source case is more difficult than the traditional Steiner tree problem because not only the terminal nodes (i.e., source and destinations) need to be spanned but also a set of VMs is required to be selected and spanned to install VNFs in sequence. Also, the general case is more challenging than the single-source case, because a serice tree is necessary to be created for each source, and ii In this paper, we first consider the static multicast, and then clarify the static case is already a good step forward and discuss how to adapt the proposed algorithm to the dynamic case in Sections VII-A and VII-C, respectiely. iii In this paper, we assume that the setup cost for a source node is negligible. The source with the setup cost is further discussed in Appendix D. i Compared with the traditional Steiner Tree problem, the problem considered in this paper is more difficult due to new SDN/NFV constraints inoled. Indeed, seeral recent research works [], [] on SDN multicast and NFV serice chain embedding (e.g., [], [3]) hae massie approximation ratios (e.g. O( D ), where D denotes the number of destinations, and O( C ), where C denotes the length of demanded serice chain). By contrast, the approximation ratio of this paper is 3ρ, where ρ is the best approximation ratio of the Steiner Tree problem (e.g., the current best one is.39), which is smaller than the aboe works. Moreoer, the simulation results manifest that empirically the performance is ery close to the optimal solutions obtained by the proposed Integer Programming formulation f f 6 9 f f 5 3 f 3 3 f source VM switch destination Fig.. Comparison of serice trees and serice forests. Input network. Steiner tree with predetermined VMs. Serice Oerlay Forest. the VNF conflict (i.e., a VM is allocated with too many VNFs from multiple trees) tends to occur in this case. Therefore, is designed to ) assign multiple sources for aried trees with multiple VMs, ) allocate the VMs for each tree to proide a serice chain for each destination, and 3) find the routing of each tree to span the selected source, VMs, and destinations. Simulation on real topologies manifests that can effectiely reduce the total cost for data center networks. In addition, a distributed is proposed to support the multi-controller SDNs. Implementation on an experimental SDN for Youtube traffic also indicates that the user QoE can be significantly improed for transcoded and watermarked ideo streams. The rest of this paper is organized as follows. The related works are summarized in Section II. We formulate the SOF problem in Section III and design the approximation algorithms in Sections IV and V. The distributed algorithm is presented in Section VI. Some important issues are discussed in Section VII. The simulation and implementation results are presented in Section VIII, and we conclude this paper in Section IX. II. RELATED WORK Traffic engineering for unicast serice chains in SDN has drawn increasing attention recently. Lukoszki et al. [] point out that the length of a serice chain is necessary to be bounded and present an efficient online algorithm to maximize the number of deployed serice chains, whereas the maximal number of VMs hosted on a node is also guaranteed. Xia et al. [] jointly consider the optical and electrical domains and minimize the number of domain conersions in all serice chains. Moreoer, Kuo et al. [3] strike the balance between link utilization and serer usage to maximize the total benefit. Neertheless, the aboe studies only explore unicast routing for serice chains and do not support multicast. Multicast traffic engineering for SDN is more complicated than traditional unicast traffic engineering. Huang et al. [6] first incorporate the flow table scalability in the design of the multicast tree routing in SDN. Shen et al. [] then further consider the packet loss recoery in reliable multicast routing for SDN. Recently, the routing of multiple trees in SDN [] has been studied to ensure that the routing follows both the link capacity and the TCAM size. The problem is more challenging due to the aboe two constraints, and the best approximation ratio that can be achieed is only D (i.e., the maximum number of destinations in a tree). Howeer, the dimension of serice allocation in VMs has not been explored in the aboe work. Recently, special cases on a tree [7], [8] with only one source and one VM hae been explored. Oerall, the aboe approaches are not designed to support a serice forest with multiple VNFs and multiple trees, and the problem here is more challenging because VNF conflict due to the oerlapping of trees will occur.

3 To the best knowledge of the authors, this paper is the first one that explores both routing and VM selection for multiple trees to construct a forest in SDN. As explained in Section I, the serice forest is important for many emerging and crucial multimedia applications in CDN that require intensie cloud computing. III. THE SERVICE OVERLAY FORE PROBLEM A serice oerlay forest consists of a set of serice oerlay trees. Each serice oerlay tree spans one source, a set of VMs for enabled VNFs, and a subset of destinations. Any two serice oerlay trees do not oerlap since each destination only needs to connect to a source ia a serice chain in a tree. In the following, we first formally define the problem. We are gien: ) a network G = {V = M U, E}, where each link e E is associated with a nonnegatie cost c(e) denoting the connection cost of link e to forward the demand of destinations, each irtual machine (VM) M is associated with a nonnegatie cost c() denoting the setup cost of VM to run a irtual network function (VNF), and each switch U is associated with cost, ) a set of destinations D V requesting the same demand, 3) a set of sources S V haing the demands of destinations, and ) a chain of VNFs C = (f, f,, f C ) required to process the demand of destinations. The Serice Oerlay Forest (SOF) problem is to construct a serice oerlay forest consisting of the serice oerlay trees with the roots in S, the leaes in D, and the remaining nodes in V, so that there exists a chain of VNFs from a source to each destination. A chain of VNFs is represented by a walk, which is allowed to traerse a node (i.e., a VM or a switch) multiple times. In each walk, a clone of a node and the corresponding incident links are created to foster an additional one-time pass of the node, and only one of its clones is allowed to run VNF to aoid duplicated counting of the setup cost. For example, in the second feasible forest (colored with light gray) of Fig., a walk from source to destination 8 passes VM twice without running any VNF, and there are two clones of VM on the walk. For each destination t D, SOF needs to ensure that there exists a path with clone nodes (i.e., a walk on the original G) on which f, f,, f C are performed in sequence from a source s S to t in the serice oerlay forest. The objectie of SOF is to minimize the total setup and connection cost of the serice oerlay forest, where the setup and connection costs denote the total cost of the VMs and links, respectiely. Note that the cost of a link in G is counted twice if the link is duplicated because its terminal nodes are cloned. In this paper, it is assumed that a VM can run at most one VNF in the network G. The scenario that requires a VM to support multiple VNFs can be simply addressed by first replicating the VM multiple times in the input graph G. Example. Fig. presents three examples for the serice oerlay forests. The first serice oerlay forest consists of two serice oerlay trees, where the demand of destination 8 (or 9) is routed from source (or ) along the walk (,,,, 6, 8) (or, 3,, 5, 7, 9), and the demand is processed by VNFs f and f at VMs and 6 (or 3 and 7), respectiely. The total cost of the first serice oerlay forest is 8, where the setup cost and connection cost are 5 and 3, respectiely. In 6 8 source switch VM destination f f 6 8 f f f f 7 9 f 3 6 f f Fig.. Example of serice oerlay forests. The input network G. The serice oerlay forests with C = (f, f ) constructed for G. the second serice oerlay forest (including only one tree), source first routes the demand to VM for VNF f. Subsequently, VM forwards the demand to VM 7 and VM for VNF f, respectiely. Finally, the demand is forwarded towards destinations 8 and 9, respectiely. The setup cost and connection costs of the second serice oerlay forest are 3 and 9, respectiely. In the third serice oerlay forest (tree), the demand is first routed from source to VM 3 for VNF f and then toward VM for VNF f, and finally to destinations 8 and 9, respectiely. The third serice oerlay forest is an optimal serice oerlay forest with the setup cost and connection cost as and 7, respectiely. A. Integer Programming In the following, we present the Integer Programming (IP) formulation for SOF. Our IP formulation first identifies the serice chain for each destination and then constructs the whole serice forest accordingly. To find the walk of the serice chain, we first assign the VMs corresponding to each VNF in the walk and then find the routing of the walk between eery two consecutie VMs. More specifically, SOF includes the following binary decision ariables. Let γ d,f,u denote if node u is assigned as the enabled VM for VNF f in the walk to destination d. Let π d,f,u, denote if edge e u, is located in the walk connecting the enabled VM of VNF f and the enabled VM of the next VNF f N. Note that the aboe walk will belong to a serice tree rooted at the enabled candidate node of VNF f. Therefore, to find the serice forest for f, let binary ariable τ f,u, represent if edge e u, is located in the forest. On the other hand, binary ariable σ f,u represents if node u is assigned as the enabled VM of serice f for the whole serice forest. Notice that each destination d may desire a different VNF f on the same enabled VM u according to γ d,f,u, but the constraint later in this section will ensure that only one VNF is allowed to be allocated to u by properly assigning σ f,u accordingly. The objectie function for SOF is as follows. min c(u)σ f,u + min f C u V f C e u, E c(e u, )τ f,u,, where the first term represents the total setup cost of all VMs, and the second term is the connection cost of the serice forest. The IP formulation contains the following constraints. ) Serice Chain Constraint. The following four constraints first assign the enabled VM for each serice chain. 8

4 γ d,fs,s =, d D, () s S γ d,f,u =, d D, f C, () u M γ d,fd,d =, d D, (3) γ d,fd,u =, d D, u V {d}. () Constraint () ensures that each destination chooses one source s in S as its serice source, where f S denotes the function as the source of the serice chain. Constraint () finds a node u from M as the enabled VM of each VNF f for each destination. Constraints (3) and () assign only destination d for function f D, where f D denotes the function as the destination of the serice chain. Here notations f S and f D are incorporated in our IP formulation in order to support the routing constraints described later. In other words, a serice chain traerses the nodes with f S, f,..., f C, f D sequentially. ) Serice Forest Constraint. The following two constraints assign the enabled VM for the whole serice forest. γ d,f,u σ f,u, d D, f C, u V, (5) σ f,u, u V, (6) f C Constraint (5) assigns u as the enabled VM of VNF f for the whole serice forest if u has been selected by at least one destination d for VNF f. Constraint (6) ensures that each node u is in charge of at most one VNF. 3) Chain and Forest Routing Constraints. The following two constraints find the routing of the whole serice forest. π d,f,u, π d,f,,u γ d,f,u γ d,fn,u, N u N u d D, f C {f S }, u V, (7) π d,f,u, τ f,u,, d D, f C {f S }, e u, E. (8) Constraint (7) is the most complicated one. It first finds the routing of the serice chain for each destination d. For the source u of a serice chain, γ d,fs,u = and γ d,fn,u = γ d,f,u =, where f is the first VNF in C. In this case, the constraint becomes π d,fs,u, π d,fs,,u. N u N u It ensures that at least one edge e u, incident from u is selected for the serice chain because no edge e,u incident to u is chosen (i.e., π d,fs,,u = for the source u). By contrast, N u for any intermediate switch u in the walk from the source to the enabled VM of f, γ d,fs,u = and γ d,f,u =, and the constraint becomes π d,fs,,u π d,fs,u,. N u N u When any edge e,u incident to u has been chosen in the walk, the aboe constraint states that at least one edge e u, incident from u must also be selected in order to construct the serice chain iteratiely. The aboe induction starts from the source of the walk to the preious node of the enabled VM of f. Afterward, for the enabled VM u of f, γ d,f,u = and γ d,fs,u =, and the constraint becomes π d,fs,,u π d,fs,u,. N u N u Since π d,fs,,u = for only one edge e,u in the walk incident to u, the aboe constraint is identical to π d,fs,u,. N u Therefore, π d,fs,u, is allowed to be for eery edge e u, to minimize the objectie function, implying that no data of f S will be sent from the enabled VM of f. By contrast, π d,f,u, will be for one edge e u, due to constraint (6), implying that the enabled VM of f will delier the data in one edge incident from u, and the aboe induction repeats sequentially for eery serice f in C until it reaches the destination d. Finally, constraint (8) states that any edge e u, is in the serice forest if it is in the serice chain for at least one destination d. B. The Hardness The SOF problem is NP-hard since a metric ersion of the Steiner Tree problem (see Definition ) can be reduced to the SF problem in polynomial time. The complete proof is presented in Appendix A. Definition. [] Gien a weighted graph G = {V, E} with edge costs, a root r V and a node set U V \{r}, a Steiner Tree is a minimum spanning tree that roots at s and spans all the nodes in U, where U. Theorem. The SOF problem is NP-hard. IV. SPECIAL CASE WITH SINGLE TREE In this subsection, we propose a ( + ρ )-approximation algorithm, named Serice Oerlay Forest Deployment Algorithm with Single Source (-SS) to explore the fundamental characteristics of the problem, and a more complicated algorithm for the general case with multiple sources will be presented in the next section. -SS includes the following two phases. The first phase chooses the most suitable VM to install the last VNF (i.e., called last VM in the rest of this paper) and then finds a minimum-cost serice chain between the source and the last VM. Afterward, the second phase finds a minimum-cost Steiner tree to span the VM and all the destinations. The selection of the last VM is crucial due to the following trade-offs. Choosing a VM closer to the source tends to generate a shorter serice chain, but it may create a larger serice tree if the last VM is distant from all destinations. Also, it is important to address the trade-off between the setup cost and connection cost, because a VM with a smaller setup cost will sometimes generate a larger tree. The pseudo code of -SS is presented in Appendix E (see Algorithm ). Therefore, to achiee the approximation ratio, it is necessary for -SS to carefully examine eery possible VM to derie a Steiner tree and ealuate the corresponding cost. For eery VM u, to obtain a walk W G (i.e., serice chain) from source s to u with C VMs (so that the VNFs f, f,, f C can be installed in sequence) in G, we first propose a graph transformation from G to G and then find the k-stroll [9] from s to u defined as follows. Definition. Gien a weighted graph G = {V, E} and two nodes s and u in V, the k-stroll problem is to find the shortest walk that isits at least k distinct nodes (including s and u) from s to u in G. The next section will extend the serice chain into a serice tree with multiple last VMs.

5 -SS constructs an instance G = {V, E} of the k-stroll problem from G as follows. Let V consist of s and all VMs in G (i.e., V = M {s}). Let E contain all edges between any two nodes in V (i.e., G is a complete graph). The cost of the edge between nodes and in E is defined as follows, c(, ) = c(u)+c( ) if = s, c( c((a, b))+ )+c(u) else if = s, (a,b) P c( )+c( ) otherwise, where u and P denote the last VM and the shortest path between nodes and in G, respectiely. In other words, the cost of each shortest path in G is first included in the cost of the corresponding edge in E. Afterward, since the data always enter and leae the VM running an intermediate VNF ( f C ), the setup cost of the VM is shared by the incoming and outgoing edges of the VM. Finally, the setup cost of last VM u is shared by the outgoing edge of s and the incoming edge of u. The edge costs of G are assigned in the aboe way to ensure that the shortest walk with C VMs in G is identical to the shortest path with C + nodes in G. Clearly, G can be constructed in polynomial time. Then, -SS finds a k-stroll walk W G that isits exactly C + distinct nodes from source s to the last VM u (i.e., k = C +) in G. Then, -SS finds the corresponding walk W G (i.e., a serice chain from s to u in G) that isits exactly C distinct VMs in G by concatenating each shortest path corresponding to a selected edge in C, and each path connects two consecutie nodes, u j and u j+, on walk W G, where j C. Finally, the demanded VNFs f, f,..., f C can be deployed in order on the walk with C VMs from s to u. Example. Fig. 3 presents an illustratie example for - SS. First, for VM 7, the walk W G = (u, u,, u C + ) with u = and u C + = 7 is obtained as follows. An instance G = {V, E} of the k-stroll problem is first constructed with s =, M = {, 3,, 5, 6, 7}, and u = 7, where V is set to {,, 3,, 5, 6, 7}, E is set as {(x, y) x, y V}, the cost of the edge between nodes and 6 is set to c((, )) + c((, )) + c((, 6)) + c(5)+c(6) =, and the cost of the edge between nodes and 6 is set as c((, )) + c((, 6)) + c()+c(6) = 3. Subsequently, we acquire a walk W G = W G = (,,, 3, 5, 7) in G and the corresponding walk W G = (,,,, 3, 5, 7) in G. After W G is obtained, the serice oerlay forest with the last VM (i.e., 7) is constructed, where the demand is first routed from source to VM 7 along the walk W G = (,,,, 3, 5, 7), and f, f, f 3,f, and f 5 is processed at VMs,, 3, 5, and 7, respectiely. After finding the Steiner tree rooted at VM 7, the demand is then routed to destination 8 by traersing switches and 6, and directly to destination 9. The total cost in the end of the second phase is 5. In the following, we present seeral important characteristics for graph G, which play crucial roles to derie the approximation ratio. First, the cost of a walk (u, u,, u C + ) from s = u to the last VM u = u C + without traersing a node multiple times in G is equal to the sum of the total setup cost of u, u 3,, u k, plus the total connection cost of the shortest paths between eery u j and u j+ for j =,,, k in G. Second, the edge costs in G satisfy the triangular inequality, as described in the following lemma. For readability, the detailed proof is presented in Appendix B f f f3 3 f 5 f Fig. 3. Example of serice oerlay forest by -SS. The input network G. The constructed instance of the k-stroll problem G, where the walk W G between nodes and 5 is shown in bold. The serice oerlay forest with C = (f, f, f 3, f, f 5 ) constructed for G. Lemma. The graph G satisfies triangular inequality. Let c(fm OP T OP T ) and c(fe ) denote the setup and connection costs of the optimal serice oerlay forest F OP T, respectiely. Based on the aboe two characteristics, the following theorem deries the approximation ratio of -SS. The complete proof is presented in Appendix C. Theorem. The cost of F is bounded by ( + ρ )c(f OP T ). That is, -SS is a (+ρ )-approximation algorithm for the SOF problem with one tree. Time Complexity Analysis. -SS constructs M instances of the k-stroll problem, and each of them employs the Dijkstra algorithm M times to compute the edge costs of each instance, where O(T d ) denotes the time to run the Dijkstra algorithm. Moreoer, let O(T k ) denote the time to sole a k- stroll instance [9], and let O(T s ) represent the time to append a Steiner tree by []. Therefore, the oerall time complexity is O( M (T d M + T k + T s )). V. GENERAL CASE WITH MULTIPLE TREES In this section, we propose a 3ρ -approximation algorithm, named Serice Oerlay Forest Deployment Algorithm (), for the general SOF problem with multiple sources. Different from -SS, here we select multiple sources to exploit multiple trees for further reducing the total cost, and it is necessary to choose a different subset of destinations for each source to form a forest. In other words, both the last VMs and the set of destinations are necessary to be carefully chosen for the tree corresponding to each source. To effectiely sole the aboe problem, our idea is to identify a short serice chain from each source to each destination as a candidate serice chain and then encourage more destinations to merge their serice chains into a serice tree, and those destinations will belong to the same tree in this case. More specifically, first constructs an auxiliary graph G with each candidate serice chain represented by a new irtual edge connecting the source and the last VM of the chain. Also, eery source is connected to a common irtual source. finds a Steiner tree spanning the irtual source and all destinations, and we will proe that the cost of the tree in G is no greater than 3ρ c(f OP T ). Neertheless, a new challenge arises here because the serice chains corresponding to the selected irtual edges in the aboe approach may oerlap in a few nodes in G, and the solution thereby is not feasible if any oerlapping node in this case needs to support multiple VNFs (see the definition of SOF). in Section V-B thereby reises the aboe solution into multiple feasible trees, and we proe that can still maintain the desired approximation in Section V-A. The pseudo code of is presented in Appendix E (see Algorithm ).

6 ŝ 3 7 ˆ ˆ ˆ 3ˆ 3 ˆ 5ˆ 5 6ˆ 6 8 7ˆ 7 9 Fig.. Example of instance construction of the Steiner tree problem. The input network G. The instance of the k-stroll problem G constructed, where the walk W G between nodes and 6 is shown in bold. The instance of the Steiner tree problem. A. -Bounded Steiner Tree first constructs an auxiliary graph G to effectiely extract multiple serice chains and group the destinations. Specifically, let V S consist of the duplicate ˆ of each source S, and let V M contain the duplicate ˆ of each VM M. Therefore, V = V {ŝ} V S V M, where ŝ denotes the irtual source. Also, let EŝS include the edges between ŝ and ˆ for each ˆ V S. Let E SM consist of the irtual edges (representing the candidate serice chain) between ˆ and û for each ˆ V S and û V M, and let E MM include the edges between and ˆ for each M. Then, E = E EŝS E SM E MM. Moreoer, the cost of each edge in EŝS E MM is assigned to, and the cost of the irtual edge between ˆ V S and û V M in E SM is equal to the cost of the k-stroll walk that isits C VMs between and u in G. We first present an illustratie example for the aboe graph transformation. Example 3. Fig. presents an example to construct the instance G = {V, E} of the Steiner tree problem with the graph G shown in Fig., where S = {, }, M = {, 3,, 5, 6, 7}. The output G is presented in Fig.. first replicates G in G. Subsequently, it duplicates the sources and by creating nodes ˆ and ˆ, and VMs, 3,, 5, 6, 7 by creating nodes ˆ, ˆ3, ˆ, ˆ5, ˆ6, ˆ7 in G. Then, the costs of edges (ŝ, ˆ), (ŝ, ˆ), (ˆ, ), (ˆ3, 3), (ˆ, ), (ˆ5, 5), (ˆ6, 6), and (ˆ7, 7) are all set to. To derie the cost of the irtual edge (ˆ, ˆ6), finds the walk from source to VM 6 in G as follows. First, it constructs an instance of the k-stroll problem G shown in Fig.. Then, we obtain a walk W G = W G = (,,, 3, 5, 6) in G. By combining the shortest paths with each path connecting two consecutie nodes in W G, we find the desired walk W G = (,,,, 3, 5, 6) in G. Thus, the cost of link (ˆ, ˆ6) is set to the cost of W G, which is equal to c() + c() + c(3) + c(5) + c(6) + c(, ) + c(, ) + c(, ) + c(, 3) + c(3, 5) + c(5, 6) =. The following lemma first indicates that the cost of the constructed Steiner tree in G is bounded by ρ 3c(F OP T ), by showing that there is a feasible Steiner tree T = {V T, E T } in G with the cost bounded by 3c(F OP T ). Lemma. A feasible Steiner Tree with the cost no greater than 3c(F OP T ) exists in G. Proof: We first show that there is a T-like graph, T = {V T, E T }, with a cost of at most 3c(F OP T ) in G. Afterward, we extract the desired T from T. Let D OP T denote the set of the destinations in the serice oerlay tree rooted at source in F OP T. In addition, for the serice oerlay tree rooted at source in F OP T, let m OP T be the representatie last VM chosen from all the VMs running f C on the paths from to the destinations in D OP T. Moreoer, let T be the optimal Steiner tree rooted at m OP T that spans all destinations in D OP T in G. Then, let V T consist of ) ŝ, ) the duplicate ˆ (in V S ) of each source in F OP T, 3) the duplicate m OP ˆ T (in V M ) of each m OP T in F OP T, ) each m OP T in F OP T, and 5) all VMs and switches (including all destinations in D) in all optimal Steiner trees T in G. Let E T include the edges between ) ŝ and ˆ, ) ˆ and m OP ˆ T, 3) m OP T and m OP ˆ T for each spanned source in F OP T, and ) all links in all optimal Steiner trees T in G for each used source in F OP T. Note that for each source in F OP T, the cost of the edge between ˆ and m OP ˆ T in T is bounded by twice of the cost of the shortest walk that isits C VMs between and m OP T in G. Since there is a walk between and OP T in F OP T, the total cost of the edges in E T E SM is bounded by c(f OP T ). In addition, the cost of T is restricted by the connection cost of the serice oerlay tree with root in F OP T, because the latter one not only spans m OP T and the destinations but also spans the source and other VMs (running f, f,..., f C ). Thus, the total cost of eery edge in E T E is bounded by c(f OP T ). Since the cost of each edge in E T EŝS or E T E MM is, the cost of T is bounded by 3c(F OP T ). Furthermore, there is a subgraph (more specifically, a tree) T of T that spans the irtual node and all the destinations in G. Hence, the cost of T is smaller than that of T and is bounded by 3c(F OP T ). B. -Bounded Serice Oerlay Forest After finding a Steiner tree T in G with a bounded cost of at most 3ρ c(f OP T ) by the aboe ρ -approximation algorithm, to limit the total cost of the serice oerlay forest, will deploy each serice chain with the corresponding irtual edge in T E SM and the route traffic ia the edges in T E. Specifically, first ) adds each corresponding walk of the spanned irtual edge one by one in G and then ) adds all VMs, switches, and links in T G to F. Example. Fig. 6 presents an example for the construction of the serice oerlay forest with C = (f, f, f 3, f, f 5 ) in Fig. by. First, an instance G = {V, E} of the Steiner Tree problem is constructed with the input parameters G, S = {, }, M = {, 3,, 5, 6, 7}, and C = (f, f, f 3, f, f 5 ), and a Steiner tree T in G using the ρ -approximation algorithm in [] is obtained, as shown in Fig. 6. Neertheless, multiple walks in G corresponding to the spanned irtual edges in T may oerlap in a few VMs, and the solution in this case is infeasible if any oerlapping VM in this case needs to perform different VNFs (see the definition of SOF in Section III). The situation is called VNF conflict in this paper. In the following, we present an effectie way to eliminate the conflict by tailoring the oerlapping walks without increasing the cost. To address the VNF conflict, when a walk W G = (,,, n ) in G is added to the serice oerlay forest F, it is encouraged to augment F with a modified walk W = (u, u,, u n ) based on W G. Note that a VM or switch is allowed to be passed without processing any VNF by simply forwarding the data. Moreoer, a VNF conflict happens when two walks compete for a clone to perform different VNFs. Fig. 6 presents an example of the VNF conflict, where W and W respectiely desire to run f and f on VM. Suppose that a walk W (between source s and VM ) faces the VNF conflict with another walk W (between source

7 s s W fj fi u W W fjfi u w fh Fig. 5. Resole of VNF conflicts between two walks, where the black solid (or dashed) line denotes the original (or updated) W, and the red dashed (or solid) line denotes the original (or updated) W. Fig. 5,, and show the resole of the first, second, and third kinds of VNF conflicts, respectiely. s and VM ) in F. We sole the conflict between W and W by changing the source of W from s to s (attaching W to W ), or changing the source of W from s to s (attaching W to W ) without adding new links, VMs, and switches to F and without enabling new VMs in F for VNFs. Example 5. Following Example, finds walks W G, = (,,,, 3, 5, 6) and W G, = (, 3, 5, 3,,, 7) in G, where f, f, f 3, f, f 5 are installed at VMs,, 3, 5, 6 on W G,, and also at VMs 3, 5,,, 7 on W G,, respectiely. After W G, is added to F, we hae F = {W }, where W consists of one clone for source, two clones of VM, and one clone for VMs, 3, 5, 6 due to F = in the beginning. As W G, is added to F, augments F with W, where W includes one clone for source, one clone for VMs 3, 5,, (on which f 3, f, f, and f are already running on W ), and two new clones for VMs 3 and 7, as shown in Fig. 6. Specifically, let u be the first VM, where W experiences the VNF conflict with W by backtracking W. Recall in Fig. 6, for example, that VM is the first conflict node with W by backtracking W. Let f, f,, f C denote the VNFs required to be performed in sequence on W and W. Let f i and f j be the VNFs located at u on W and W, respectiely. effectiely addresses the VNF conflict in details as follows. First, if j i, attaches W to W through u by changing W to the concatenation of the sub-walk of W from s to u (on which f, f,, f i are installed in sequence, identical to W ) and the sub-walk of W from u to (on which f i+, f i+,, f C are running in sequence, identical to W ), as shown in Fig. 5. Example 6. Following Example 5, W first experiences the VNF conflict with W at (the clone of) VM, where f and f are installed on W and W, respectiely. The sequence numbers of the VNFs at VM on W and W are and, respectiely. The condition j i is not satisfied since j = and i =. then checks the next condition. Note that one of the three conditions must be satisfied. Second, if there is another VM w such that W experiences the VNF conflict with W at w, where f h with h j is on W, attaches W to W through w by changing W to the concatenation of the sub-walk of W from s to w (on which f, f,, f h are running in sequence, identical to W ), the sub-walk of W from w to u, and the sub-walk of W from u to (on which f h+, f h+,, f C are running in sequence, identical to W ), as illustrated in Fig. 5. Example 7. Following Example 6, W experiences another VNF conflict with W at VM 5, where f and f are performed on W and W, respectiely. Since the sequence number of the s W s s W fj fi u s W ŝ ˆ ˆ ˆ 3ˆ 3 ˆ 5ˆ 5 6ˆ 6 8 7ˆ 7 9 ff ff3 f3f ff f W f5 7 W f f f3 f f f5 7 9 Fig. 6. Example of construction of the serice oerlay forest by. The Steiner tree, shown in bold line. Two walks with VNF conflict. The serice oerlay forest with C = (f, f, f 3, f, f 5 ) constructed for G in Fig.. VNF at VM 5 on W is not smaller than that of the VNF at VM on W, attaches W to W through VM as follows. first steers W along the sub-walk of W from source to VM 5 (i.e., the walk (,,,, 3, 5)) on which f, f, f 3, f are running in sequence at VMs,, 3, 5, respectiely, identical to W. Subsequently, it continues steering W along the subwalk of W from VM 5 to VM (i.e., the walk (5, 3,, )), and the sub-walk of W from VM to VM 7 (i.e., the walk (, 7)) on which f 5 is run at VM 7, identical to W. Finally, the sub-walk (5, 3,,, 7) on the reised W can be shortened to be a walk (5, 7). The constructed serice oerlay forest for G is displayed in Fig. 6. Otherwise, attaches W to W through u by changing W to the concatenation of the sub-walk of W from s to u (on which f, f,, f j are running in sequence, identical to W ) and the sub-walk of W from u to (on which f j+, f j+,, f C are run in sequence, identical to W ), as shown in Fig. 5. Moreoer, when a walk W experiences the VNF conflict with multiple walks W, W,, W l in F in sequence by backtracking W, resoles the VNF conflict between W and W, W,, W l one-by-one. The following theorem deries the approximation ratio for. Theorem 3. The cost of the constructed serice oerlay forest F is bounded by 3ρ c(f OP T ). Proof: First, the cost of Steiner tree T in G is bounded by ρ times of the optimal Steiner tree in G. Since the cost of the optimal Steiner tree in G is bounded by 3c(F OP T ) according to Lemma, the cost of T is limited by 3ρ c(f OP T ). In addition, since the cost of the edge between ˆ V S and û V M of u in T is identical to the cost of the walk that isits C VMs between and u in G, the cost of F constructed in G is equal to the cost of T and thereby bounded by 3ρ c(f OP T ) if no VNF conflict occurs in F. On the other hand, when the VNF conflict between two walks happens, one of the two walks in F is updated, and no new link, VM, and switch is added to F, and no VM in F is newly created to perform the VNF. Thus, the cost of F reised for resoling the VNF conflict is still bounded by 3ρ c(f OP T ). The theorem follows. Time Complexity Analysis. We follow the notations in the time complexity analysis of -SS. To generate the instance of the Steiner tree problem, constructs S M instances of the k-stroll problem, and each of them employs the Dijkstra algorithm M times to compute the edge costs of each instance. Then, soles the k-stroll instance by [9] to derie the costs of irtual edges (i.e., corresponding candidate serice chains). To eliminate the conflict, in the worst case, all the added walks in F are appended to the newly added walk, and the complexity is O( M 3 ). There-

8 fore, the total time complexity is dominated by constructing and soling k-stroll instance and finding a Steiner tree, i.e., O( S M ( M T d + T k ) + T s ). VI. DIRIBUTED IMPLEMENTATION For large SDNs, it is important to employ multiple SDN controllers, where each one monitors and controls a subset of the network [3] [3], and the communication protocols [33] between controllers are deeloped to facilitate scalable control of the whole network. In the following, therefore, we discuss the distributed implementation of the proposed algorithm in Section V to support multi-controller SDNs. Note the controller that receies the request is elected to be the leader, which is responsible for progress tracking and phase switching. First, shortest-path routing plays a fundamental role in to build the auxiliary graph G and the serice chain corresponding to each edge in G. To find a shortest path traersing multiple domains, it is necessary for each controller to first abstract a matrix that consists of the lengths between eery pair of border routers oer the Southbound interface [33] within its domain. Afterward, each controller propagates the matrix to the other controllers along with the Network Layer Reachability Information of SDNi Wrapper oer East-West Interface. which is used to share the connectiity information with the neighboring controllers. More specifically, let s and t denote the source and the destination, respectiely. The controller C s coering s can find the corresponding domain by the IP prefix of t. Then, controller C s informs the controller C t that coers t of the lengths of all shortest paths from s to all broader routers of C t. Afterward, controller C t can respond the best broader router to controller C s, and the length of a shortest path can be acquired accordingly. Equipped with the shortest-path computation from multiple controllers, each controller can acquire the length of each shortest path between a VM in its domain and any other VM (or source). Thus, once the forest construction is initiated, eery controller that coers a source will communicate with other controllers to collect the matrices of lengths between any two VMs and the lengths between any source and any VM. Then, the controller can find all candidate serice chains from its coered source to each VM and creates a irtual link in G representing the serice chain to connect the irtual source and the corresponding last VM. Afterward, a distributed Steiner tree algorithm [3] can be employed by multiple controllers to find the Steiner tree, where the computation load originally assigned to each switch in the distributed algorithm can be finished by its controller instead. In, it is important to address the VNF conflicts in multiple domains. To achiee this goal, each controller first remoes the useless candidate serice chains that do not connect with any destination, and then informs any other controller whose coerage is isited by any remaining serice chain. When one of the informed controllers obseres a VNF conflict of two serice chains, it notifies the other controller to collaboratiely remoe the conflict according to the conflict elimination algorithm described in Section V-B. Finally, each controller deletes the irtual source, deploys the remaining serice chains, and forwards the content to the destinations by SOF. VII. DISCUSSION A. Static Mulitcast Trees with Serice Chaining To the best knowledge of the authors, this paper is the first one that explores the notion of the serice forest, i.e., the fundamental multi-tree multicast problem with serice chaining, and proides approximation algorithms with theoretical bounds. Therefore, we first consider the fundamental problems for static SDN/NFV multicast and then extend the proposed algorithms to the dynamic case in Section VII-C. Actually, static multicast is crucial for backbone ISPs. In this situation, each terminal node of a multicast tree is usually an edge router or a local proxy serer of the ISP, instead of a dynamic user client. For example, current lie streams are sent by the source (i.e., headends or content serers) and trael through the high-speed backbone network to the access nodes and edge nodes (e.g., Digital Subscriber Line Access Multiplexer (DSLAM) [35], or a Mobile Edge Computing (MEC) serer [36]) ia static multicast trees [7], [35] [39] (e.g., Chung-Hwa Telecom MOD []), whereas the dynamic user join and leae are handled by the local access nodes and edge nodes. Static multicast trees can significantly reduce the backbone bandwidth consumption for each stream and thereby is much more scalable to support a large number of ideo channels. In this case, each access node usually seres hundreds or thousands of end users and streams one (or few) channel(s) to each user according to the aailable bandwidth between the access node and user deices (e.g., set-top boxes). Moreoer, for the massiely multi-user irtual reality (e.g., gaming) [] [5], the serers create a irtual enironment with a 3D model, player aatars, and scripts, and then transmit the data by static multicast to seeral MEC serers [36], which always need to appear in a multicast group. In the aboe life examples, our proposed algorithms can facilitate static multicast with serice chaining (i.e., multiple stages of serers) to support a large number of streams between the headend serer and the local access nodes/edge nodes. B. Model and Online Deployment In the online scenario, when a new request arries, allocates the required resources for the request by constructing a serice forest according to the current link and node costs. To balance the network resource consumption and accommodate more requests in the future, congested links and nodes are unnecessary to be assigned with higher costs for encouraging to employ the links and nodes with low loads [6] [8]. In this paper, therefore, we exploit [6], which is designed for online adaptie routing in the Internet, to assign a conex cost to each link or node. The cost will significantly increase as the load linearly grows, to aoid oerwhelming the link or node. More specifically, let l and p denote current load and capacity of the link or node, respectiely, and the cost c is set according to the utilization (i.e., l/p) as follows and illustrated in Fig. 7. c = l if l/p /3, 3l /3p else if l/p /3, l 6/3p else if l/p 9/, 7l 78/3p else if l/p, 5l 68/3p else if l/p /, 5l 38/3p otherwise.

9 Load Fig. 7. The cost function with different load l and capacity p =. The cost model properly handles the online situation by assigning a huge cost to a more congested node or link. Therefore, will aoid choosing the aboe congested node or link to minimize the total cost of the serice forest. thereby can mitigate the impact on a VM of other VMs colocated with an oerloaded node. Indeed, the cost model can be applied to both priate and public cloud networks, where resource optimization and load balancing are usually addressed. For example, Chung-Hwa Telecom MOD [] is built in its priate clouds while Netflix [8] adopts AWS [9]. Neertheless, each request has a different duration, and an approach without considering the duration of the request is inclined to incur fragmentation of the network resources and degrade the performance. Howeer, the duration of a stream (e.g., a VR multi-player game) is usually difficult to be precisely predicted, and many current approaches thereby adaptiely reroute [9] [53] and migrate the VM [8], [5] [57] to relocate the network resources when congestion occurs. Similar to the aboe approaches, when a node or link becomes congested, reroutes the serice forest by letting the users downstream to the aboe node or link re-join the forest again (explained in the reply of the first question), where the current path in the forest is remoed only after the new join path is created to aoid serice interruption [9] [53]. C. Adjustments for Various Dynamic Cases. In the following, we extend to support the dynamic join and leae of destination users and the addition and deletion of NFVs in a serice forest after a session starts. To address the dynamic case, a simple approach is to run again for the whole forest. Neertheless, this approach tends to incur massie computation loads in the SDN controller, especially when users frequently join and leae the multicast group or change the computation tasks in the serice forest. In the following, therefore, we extend to properly handle the dynamic case [], []. ) Destination Leae. When a destination leaes the serice forest, if is a leaf node, remoes and all intermediate nodes and links in the path connecting and the closest upstream branch node in the serice forest, where a branch node is a node in the forest with at least two child nodes. By contrast, if is not a leaf node, because there are other destination users in the subtree rooted at, is not allowed to remoe the path connecting to the upstream branch node. ) Destination Join. When a new destination user joins the serice forest, finds the walk from to the forest with the lowest cost. More specifically, for each node u in the forest F that can be a candidate branch node to connect, let f(u) denote the index of the last installed VNF between a source s and u in the forest. To derie the cost in the walk from u to, finds the walk with k = C f(u)+ from u to to install the ( C f(u)) new VNFs in the walk by exploiting k-stroll in the transformed graph (see Section IV). Let W G (u, ) = (u,..., u k ) denote the acquired walk, where u = u and u k =. In this case, needs to install the new VNFs f f(u)+,..., f C on the aboe walk from u to, and the cost of the forest is increased by min u F {c(w G (u, ))}. carefully examines eery possible u in the existing serice forest to effectiely minimize the increasing cost, and the node u leading to the smallest cost is selected to sere the new destination user accordingly. 3) VNF Deletion. When VNF f j is remoed from the serice forest, for each VM that installs a VNF f j, connects the VM u with the upstream VNF f j to the VM w with the downstream VNF f j+ (along the minimum-cost path from u to w in the original G) in the forest, where the source (or destination) can be regarded as the VM with the upstream (or downstream) VNF f j (or f j+ ) if f j is the first (or last) VNF. ) VNF Insertion. When VNF f j is inserted to the serice forest, for each pair of VMs u and w with VNFs f j and f j+, respectiely, installs f j on an aailable VM, and connects u to and to w in the forest such that the sum of ) the connection cost of the path between u and, ) the setup cost of, and 3) the connection cost of the path between and w is minimized. When f j is the first (or last) VNF, the source (or destination) is regarded as the VM with VNF f j (or f j+ ). In addition, if two pairs of VMs (u, w ) and (u, w ) with VNFs f j and f j+ choose the same VM to install VNF f j, remoes all intermediate nodes and links in the path connecting u and in the forest in order to reduce the total cost of the forest (i.e., aoid creating redundant paths in the forest). 5) Link Congestion. For any congested link e between the VMs with VNF f j and f j+, updates the link cost according to [6] and then re-connects the two VMs with the path associated with the lowest cost. can effectiely aoid choosing a congested link because the cost of the link will be extremely large. On the other hand, if e is between the source and a VM (or VM and a destination), the source (or the destinations) is regarded as the upstream VM (or the downstream VM) and handled in a similar manner. 6) VM Oerload. For any oerloaded VM between the VMs with VNF f j and f j+, updates the node cost according to [6] to find an aailable VM and then reconnects it to the upstream VM and downstream VM with the path haing the lowest cost. Therefore, can also aoid selecting an oerloaded VM. On the other hand, if is the first VNF (or the last VNF), the source (or the destinations) is regarded as the upstream VM (or the downstream VM) and then handled in a similar manner. A. Simulation Setup VIII. NUMERIC RESULT We conduct simulations to compare different approaches in two inter-data-center networks: IBM SoftLayer [58] and Cogent [59]. SoftLayer contains 7 access nodes with 9 links and 7 data centers, whereas Cogent has 9 access nodes with