Guaranteeing Heterogeneous Bandwidth Demand in Multitenant Data Center Networks

Transcription

1 1648 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 Guaranteeing Heterogeneous Bandwidth Demand in Multitenant Data Center Networks Dan Li, Member, IEEE, Jing Zhu, Jianping Wu, Fellow, IEEE, Junjie Guan, and Ying Zhang Abstract The ability to provide guaranteed network bandwidth for tenants is essential to the prosperity of cloud computing platforms, as it is a critical step for offering predictable performance to applications. Despite its importance, it is still an open problem for efficient network bandwidth sharing in a multitenant environment, especially when applications have diverse bandwidth requirements. More precisely, it is not only that different tenants have distinct demands, but also that one tenant may want to assign bandwidth differently across her virtual machines (VMs), i.e., the heterogeneous bandwidth requirements. In this paper, we tackle the problem of VM allocation with bandwidth guarantee in multitenant data center networks. We first propose an online VM allocation algorithm that improves on the accuracy of the existing work. Next, we develop a VM allocation algorithm under heterogeneous bandwidth demands. We conduct extensive simulations to demonstrate the efficiency of our method. Index Terms Bandwidth allocation, data center networks, network virtualization. I. INTRODUCTION T HE CLOUD computing platform [6] [8], employing the Infrastructure as a Service (IaaS) model, is becoming tremendously popular given its economies of scale, flexibility, and low operational cost. Despite the large momenta it grows, many companies are still reluctant to move their services, especially critical applications, to the cloud, due to the lack of performance guarantees and the concerns of security and privacy [9], [10]. For instance, applications with high response time requirement, like real-time financial transactions and remotesurgery,cannotberuninsuchanenvironmentwith unpredicted performance. Thus, the cloud providers are seeking mechanisms to support separation and performance isolation between tenants. However, while the host-based virtualization techniques can provide host-level isolations successfully, the applications' performance may still be severely slowed down due to the varied network bandwidth. Manuscript received November 15, 2012; revised October 14, 2013 and February 25, 2014; accepted July 06, 2014; approved by IEEE/ACM TRANSACTIONS ON NETWORKING Editor S. Sengupta. Date of publication August 04, 2014; date of current version October 13, This work was supported by the National Key Basic Research Program of China (973 program) under Grant 2014CB347800; the National Natural Science Foundation of China under Grants No , No , and No ; and the National High-tech R&D Program of China (863 program) under Grant 2013AA D. Li, J. Zhu, J. Wu, and J. Guan are with Tsinghua University, Beijing , China ( tolidan@tsinghua.edu.cn). Y. Zhang is with Ericsson Research, San Jose, CA USA. Color versions of one or more of the figures in this paper are available online at Digital Object Identifier /TNET Thus, it is critical for the cloud operators to deploy efficient network resource allocation and isolation techniques, for isolating performance among tenants, minimizing disruption, as well as preventing malicious network attacks. The network resource allocation mechanism should have the following properties. First, it should allow tenants to specify the demand of their applications in an intuitive way and support flexible updates. It means a simple and easy-to-use interface between tenants and providers. Yet, it needs to be flexible to support various types of services. Second, the provider should still be able to leverage the advantage of multiplexing to support many virtual networks on their physical network. The greater the amount of sharing is, the more revenue the cloud provider gets. Third, the allocation method needs be efficient in order to scale with the size of the tenant. It should keep up with the request rate so that a decision of accepting or rejecting a tenant should be made with minimum delay so that the user experience is not affected. Similarly, once it is accepted, the virtual machines (VMs) should be immediately allocated to the tenants for running their applications. Finally, the scheme should be easy to implement, without relying on modifications to the commercial switches and routers. Most existing performance isolation solutions for cloud data center networks, however, cannot satisfy all the above requirements. Traditional fair queuing and congestion control methods [3], [4] may cause tenant unfairness and are prone to malicious attacks. Recently proposed contention-based bandwidth-sharing mechanisms consider tenant fairness [2], [12], but do not specify the tenant's bandwidth demand, and thus may not meet the bandwidth requirement of tenants. The works closest to ours are Secondnet [1] and Oktopus [5], both of which use the tenants' prespecified demands to allocate bandwidth but using different abstractions. Secondnet abstracts the bandwidth demand as the bandwidth matrix among VMs, which is difficult to be specified accurately before application is actually executed. On the other extreme, Oktopus uses a hose model to represent the tenant's bandwidth request, where the tenant specifies the same bandwidth for all the VMs. We argue that this approach does not satisfy the flexibility requirement, as it assumes that all the VMs of one tenant require an equal amount of bandwidth. However, in reality, one tenant or one application may employ machines with different roles, e.g., one database and one front-end server, each of which has distinct network behavior. We call this phenomenon heterogeneous bandwidth requirement. Two pieces of recent work tried to improve its efficiency [12], [13] along different dimensions, but neither of them has tackled the problem of varied bandwidth demand across VMs IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See for more information.

2 LI et al.: GUARANTEEING HETEROGENEOUS BANDWIDTH DEMAND IN MULTITENANT DATA CENTER NETWORKS 1649 To the best of our knowledge, we are the first to tackle the allocation under heterogeneous bandwidth demand model. We propose the model of abstraction and an efficient algorithm for assigning heterogeneous bandwidths among VMs of a task. More specifically, in this work, we make the following contributions. We improve the existing VM allocation algorithm for homogeneous bandwidth requirements. We identify that the state-of-the-art method for homogeneous demand [5] can be significantly improved because it only considers the upper bound of available VM slots, without taking into account the fact that solutions may have more than one hole in terms of the number of accepted VMs. We improve this allocation scheme by taking the discrete solutions into consideration, which makes more accurate estimation of the acceptability for a tenant and significantly postpones the first tenant rejection. We propose a heterogeneous bandwidth demand abstraction, where the tenant can specify a diverse set of bandwidth demands for different VMs. We believe this model is much more realistic as a tenant is likely to deploy VMs with different roles or running different application components. We also prove that the VM allocation for heterogeneous bandwidth demandisannp-hardproblem. We develop a heuristic VM allocation algorithm toward finding the best VM placement strategy for any incoming tenant request in real time, which is shown to be NP-hard. Simulation results show that our algorithm performs close to the offline optimal allocation and is much better than simply applying the algorithm based on homogenous demand. The rest of this paper is organized as follows. We first formulate the problem in Section II. Sections III and IV present the VM allocation algorithms for homogeneous and heterogeneous bandwidth demands, respectively. We use simulation to evaluate our method in Section V. We also demonstrate its benefit usingatestbedinsectionvi.finally,sectionviidiscussesthe related works. We conclude in Section VIII. II. PROBLEM STATEMENT A. Virtual Network Service Model The cloud provider offers virtual network services to customers for interconnecting VMs assigned to these customers. The customers do not need to be aware of the topology and the technology details of the underlying physical infrastructure, but only be concerned if the high-level service level agreements (SLAs) can be met. In order to satisfy the application-level SLAs, the network performance should be predictable so that the overall performance can be modeled and controlled. A natural way to support predicted network performance is to allocate sufficient bandwidth for each tenant. Thus, in most existing proposals [1], [5], each tenant can specify her requirement of the network service in terms of topology connectivity, security policy, as well as the required bandwidth of each node. In this work, we focus on the bandwidth requirement, which has direct impact on application performance. There are two widely used abstraction models to capture the bandwidth demand. One is traffic matrix [1], where the tenant specifies the bandwidth needed between any pair of VMs. From the cloud provider's perspective, the bandwidth matrix model captures more exact information which may lead to more accurate resource management. However, from the tenants' perspective, it is difficult to estimate the accurate pairwise bandwidth demands before the actual execution of the application. Moreover, the tenant cannot take advantage of the multiplexing nature of communications among VMs. Thus, hose model is proposed in [5] as an alternative. Here, the tenant specifies the corresponding bandwidth demand for each VM as (,where is the total number of VMs requested by the tenant). To maintain the simplicity of this model, in [5], it is assumed that the bandwidth request from VMs is homogeneous, i.e., all 's are equal. We call this model the homogeneous model. We argue that this simplification affects the accuracy and efficiency of the bandwidth allocation in a real environment. In practice, VMs of different roles can have distinct bandwidth requirements. For example, the bandwidth consumptions for a front-end Web server and for a back-end storage server are expected to have significantly different network behaviors. Hence, in this paper we extend the homogeneous tenant requests to heterogeneous bandwidth demands. Given the bandwidth request of from a tenant, we assume that the tenant is requesting a nonblocking virtual network infrastructure to provide the network service, i.e., there should be no oversubscription. This assumption requires more network resource from cloud providers, but offers better service to the tenant as if all the VMs are connected to a nonblocking virtual switch. Note that our work can be easily extended to a virtual network service with oversubscription. B. Virtual Machine Allocation Strategy We consider the scenario of real-time request acceptance and resource allocation. The problem can be described as follows. Given the new incoming request,the cloud provider allocates the VMs from the physical servers and reserves corresponding network bandwidth that can satisfy any arbitrary traffic matrix that can be derived from the hose model. For simplicity, we start with designing the allocation under a tree structure as the cloud network topology, which is also commonly used in other related work. We plan to extend it to other topologies e.g., Fat-tree, BCube in the future. Given a tenant's request, one intuitive solution is to allocate all the VMs to a low-level subtree so that all the communications remain within the subtree, saving bandwidth in the upper level. In Oktopus [5], authors proposed a recursive VM allocation approach for homogeneous bandwidth demand. The approach takes two steps, i.e., acceptability determination and allocation enforcement. In the first step, the algorithm searches the tree in a bottom-up way to find the possible number of VMs a subtree can host, and decides which subtree can serve the entire tenant. In the second step, the algorithm assigns the exact number of VMs to the physical servers of the targeted subtree. After allocating the VMs to physical servers, the policy is enforced by the rate limiters in the hypervisors and the cloud orchestration controller. Using the network-wide information, the

3 1650 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 Fig. 2. Structure of an AR. The tenant requests for 27 VMs, and the AR contains 28 bits. Fig. 1. Example to demonstrate the holes in the feasible solution space for the number of VMs to allocate in a subtree. The switch is connected to two children servers and. has 8 available VM slots, and has 11 available VM slots. The residual bandwidth capacity for each link is labeled beside the link. controller will separately calculate the bandwidth reserved for each VM on each server, according to the bandwidth request and the ongoing flow information of the tenant. The reservation is made to guarantee that the tenant can get more bandwidth resource if there is remnant but never exceed its bandwidth request. Thus, the network performance among the tenants can be isolated. C. Problem With Existing Solutions The most relevant algorithm with our approach is proposed in Oktopus [5]. It examines the allocation by checking the number of VMs a subtree can host. This number is determined by two factors, namely, the number of available VM slots in the subtree and the residual bandwidth of the outgoing link of the subtree. More specifically, when a tenant is requesting for VMs with bandwidth for each, the subtree should be able to host VMs, and the residual bandwidth of its outbound link should be at least. The Oktopus allocation algorithm has two problems. First, it cannot handle heterogeneous bandwidth demands. Second, it ignores an important property of the search space, i.e., the solution space can be discrete, which may result in suboptimal allocation results. We take Fig. 1 as an example to explain the problem of discrete solution space. Assume a tenant requests for 12 VMs with 100 Mb/s per VM ( Mbps ). The number of VMs the subtree can host is constrained by both the available VM slots and the outbound bandwidth. Server 's outbound bandwidth is 400 Mb/s and has 8 empty VM slots. Hence, it can hold VMs with full speed at the same time. It can also host 8 VMs because in this case there will be only 4 VMs outside the server and the 400-Mb/s link can also support it. Hence, the feasible solution space for server is.asfor server, its feasible solution space is due to the limit of 100-Mb/s outbound link. As a result, the feasible solution space for the subtree under switch is, which is a discrete space with more than one hole. If in practice such a request arrives and the allocation algorithm is unaware that it cannot allocate 10 VMs into the subtree under this switch, it may allocate 2 VMs to other part of the network and end up with failing to allocate the rest of the 10 VMs to this subtree, which Oktopus might do. III. VM ALLOCATION FOR HOMOGENEOUS BANDWIDTH DEMAND In this section, we propose a new method under the homogeneous model, i.e., the required bandwidth of each VM is equal ( ), where is the total number of VMs to request. Our method includes two steps: the acceptability determination phase where we decide whether there is sufficient resource to accept the request, and the allocation enforcement phase, where we compute the best locations for hosting these VMs. A. Acceptability Determination The acceptability determination phase finds how many VMs can be hosted for each subtree in the network and whether the request should be accepted or rejected in current network state. As presented in Section II, the solution space can be discrete, i.e., there may be more than one continuous subranges in the solution space for the number of VMs a subtree can host. We cannot simply use an integer to indicate the upper bound of the number of VMs that can be put in a subtree. Instead, we introduce a new structure called allocation range (AR) to indicate the space of the possible VMs each subtree can host rooted under a certain node. With a request of VMs, every node maintains an AR with bits, denoted as. Each encodes the set of possible values that subtree can host. Precisely, if ( ) VMs can be allocated into the subtree rooted from node,the th leftmost bit of is set as 1, and it is a feasible solution for node. For example, if is , it means that the number of VMs that subtree can host is 0, 1, 2, 5, or 6. An AR can have several continuous ranges for feasible solutions. We define the first continuous feasible solution range the basic section, and other feasible solution ranges the extended sections. If we use an array of lower bounds and upper bounds to denote these sections, it will be in the form of,where is the upper bound of the basic section, and and are the lower bound and the upper bound of the th extended section, respectively. The lower bound of the basic section is 0 because allocating no VM into a subtree is always a feasible solution. Fig. 2 shows the structure of an AR of 28 bits ( ). 1) Computing ARs for Servers: The AR of a physical server is determined by both the number of available VM slots in the server and the residual bandwidth capacity of the outbound link of the server. Suppose server has available VM slots, and the residual bandwidth of its outbound link is. If we only consider the constraint of available VM slots, the feasible solution space is. Since the total bandwidth of the outbound link is, it can serve a maximum of VMs in order to reserve

4 LI et al.: GUARANTEEING HETEROGENEOUS BANDWIDTH DEMAND IN MULTITENANT DATA CENTER NETWORKS 1651 amount of bandwidth for each VM, which indicates that we cannot allocate more than into this subtree and the other part of the network at the same time. Hence, if we consider the bandwidth constraint alone, the feasible solution space is,or if. Combining and, is calculated as follows: is given by the fol- Combining the above two cases, lowing formula: Basic Section Splitting: Note that there is at most one hole for the AR of a server node, and the starting point of the extended section (if there is) is always larger than.since the 's of lower-layer nodes need to be used to further compute the 's of upper-layer nodes in the tree, we assume the upper bound of the basic section is no greater than.if the upper bound of the first continuous feasible solution space is greater than, we split it into a basic section and an extended section. For instance, if the feasible solution space is, we split it as.thissimplification does not affect the performance, but improves the AR computing efficiency. Thus, with this adjustment, we can safely say that the AR of any server has the following property. Property 1: The basic section always falls within, and every extended section falls within. 2) Computing ARs for Switches: We compute the AR of a switch using a recursive way. We first define two components of :, which is aggregated from the ARs of its children nodes, and, which is constrained by the residual bandwidth of the outbound link. Then, we take the intersection of both components. We first discuss how to calculate. Suppose switch has children nodes, the residual bandwidth of its outbound link is, and the AR of each child node is. When we allocate VMs into the subtree rooted from switch, we are actually splitting these VMs into parts, i.e.,, and allocating them into each child node while making sure their sum is. falls in the feasible solution space of if and only if for any ( ), falls within the feasible solution space of.we assume that Property 1 holds for all s of the children nodes. Next, we will show that this property still holds after the aggregation into. If every is within the basic section of,i.e.,, then we simply have. Ifthereissomechild that falls within the extended section of,i.e.,, then there cannot be another child that falls within the extended section of ; otherwise, the sum of will be greater than (considering property 1). Hence, in this case we have when 0 and where is the total number of extended sections in. The aggregated feasible value is thus constrained by. Similar to the server node, when considering the constraint of the outbound link of switch,wehave,or if. Note that all the extended sections in ARs of the children nodes will also appear in the extended sections of.ifthe upper bound of the first continuous feasible solution space in is greater than, we will split the basic section similarly to the server case, i.e., splitting the first continuous space into two parts and ending the basic section at. Therefore, Property 1 also holds for the ARs of any switch, which recursively holds true or the ARs of upper-level switches. As soon as the ARs for all the nodes in the network are computed, we can draw the acceptability conclusion by simply examining the 's. The tenant request will be rejected if the th bit in the AR of the root switch is 0, and be accepted otherwise. B. Allocation Enforcement Once the request is accepted, the next question is where to allocate, i.e., deciding the exact number of VMs to be allocated into each subtree. The allocation enforcement is performed in a top-down iterative manner, until VMs are allocated to each physical. Oktopus [5] takes a greedy method for VM allocation, i.e., allocating as many VMs as possible to the subtrees that are handled earlier. Their heuristic is to maximize the available bandwidth in the upper level. We argue that optimizing for the bandwidth utilization is not enough. Instead, we propose to jointly optimize the following two metrics. Maximizing the square sum of fragment sizes. We define the concept of fragment similar to the one used in disk management. When we allocate VMs into a subtree, we may not use up all the available VM slots, and the unused VM slots compose the fragment of this subtree. Since higher-level links are more likely to become the bottleneck in a tree network, we want to make the fragment size as large as possible, so that in the future, more VMs can be allocated into the subtree without consuming higher-level link bandwidth. Hence, we try to maximize the square sum of the fragment sizes. Minimizing the link bandwidth utilization. The links carrying aggregated traffic in the data center have been witnessed to be bottlenecks. If the allocation can constrain the inter-vm traffic within a smaller-sized subtree, we can save bandwidth on the upper layer for future allocations. The minimization is from top layers to bottom layers since higher-layer links are more likely to be the bottleneck of the data center network. In fact, which objective is more important highly depends on future requests. This is illustrated in the example shown in Fig. 3. In the initial state shown in Fig. 3(a), there are 3

5 1652 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 Algorithm 1: Assigning the number of VMs to each child node on node for homogeneous bandwidth demand. Fig. 3. (a) (c) Example to show different allocation enforcement strategies. A switch is connected to three children servers,,and. available VM slots on the left two servers and 5 available VM slots on the rightmost server. Supposing the request is, we have two ways to allocate depending on the objectives. If maximizing the square sum of fragment sizes, we will allocate the 5 requested VMs into the left two servers, and the resultant networkstateisshowninfig.3(b). However, if minimizing the link bandwidth utilization, we will allocate the 5 VMs into the rightmost server, and the resultant network state is shown in Fig. 3(c). If the next tenant request is,(b)willbeabletoaccept it while (c) will not. However, if the next tenant request is, (c) will accept it while (b) will not. Therefore, in our method, we use a mixed strategy for allocation enforcement using two steps. We take minimizing the link bandwidth utilization as the top priority. If there are more than one solution occupying the same amount of bandwidth resource, then we try to find the one with the highest square sum of fragment sizes. Consider that we allocate VMs into the subtree rooted from node,whichhas children nodes. The algorithm needs to find an allocation solution so that.thebasic idea is to preferably allocate VMs to an extended section (if any) with the highest upper bound. It is because, for any solution that there issome within the extended sections of,thelarger is, the less link bandwidth will be occupied by the solution at this level. Note that there can be at most one extended section allocated among all the children nodes (because the starting point of every extended section is larger than ). As for solutions in which all the VMs are allocated to the basic sections of 's, or there are remaining VMs to allocate to the basic sections of 's after allocating VMstoanextended section, they will occupy an equal amount of link bandwidth since. To maximize the square sum of fragment sizes, we preferably allocate VMs to basic sections with the lowest upper bounds. Algorithm 1 shows the process of allocation enforcement run on node to assign VMs to each child node.thetimecomplexity is,where is number of children nodes. This process runs iteratively from top to bottom until every VM is assigned to a physical server in the network. 1 Allocate Input: : The number of allocated VMs to. Output: : Allocation solution. 2 Sort all extended sections by in a descending order; 3 for each extended section do 4 if & then 5 ; 6 return true; 7 end 8 if then 9 ; 10 for all ; 11 return true; 12 end 13 end 14 return ; 15 Find Input: : excluded child; : total number of VMs to allocate. Output: : Allocation solution 16 Sort all, by in an ascending order ; 17 ; 18 while do 19 if then 20 for all ; 21 ; 22 return true; 23 else 24 ; 25 end 26 end 27 return false; IV. VM ALLOCATION FOR HETEROGENEOUS BANDWIDTH DEMAND In the heterogeneous bandwidth demand model, the tenant requests for different bandwidths for different VMs, in the form of,where is the total number of VMs to request. For the ease of presentation, in the following we assume the array has been sorted in an ascending order. Similarly, there are two phases for the allocation, which will be described separately. A. Acceptability Determination The first step is to search in the solution space to determine whether there is sufficient resource to accept the request, we find out computationally challenging. We first discuss its computing hardness.

6 LI et al.: GUARANTEEING HETEROGENEOUS BANDWIDTH DEMAND IN MULTITENANT DATA CENTER NETWORKS ) Problem Hardness: Theorem 1: The acceptability determination problem for heterogeneous bandwidth demand is NP-complete. Proof: We prove the theorem by reducing the triple 3-partition problem, which is a well-known NP-complete problem, into our acceptability determination problem for heterogeneous bandwidth demand. Suppose there is a set of positive integers. The sum of these integers is, and each integer is between (, ). The triple 3-partition problem is defined as to find subsets to form a partition of so that in each subset the sum of integers is equal to. This is known to be an NP-complete problem. Now we consider the scenario that there are servers linked to a switch. Each server has 3 available VM slots, and the capacity of outbound link of each server is.ifwe can determine the acceptability of the heterogeneous request in this network, there mustbeacorresponding triple 3-partition for these integers. It is easy to verify whether a given allocation strategy can be satisfied by the network in polynomial time. Therefore, the acceptability determination problem for heterogeneous bandwidth demand is NP-complete. 2) Programming-Based Optimization: The problem can be solved using a programming approach, which will generate the optimal results. The input values are as follows. is the number of physical servers in the network, ( ) is the number of available VM slots in the th server, is the number of nodes (including both servers and switches) in the network, ( ) is the residual capacity on the outbound link of the th node, is the set of of servers located within the subtree rooted at the th node, is the total number of VMs in the tenant request, and ( )isthe bandwidth requirement of the th VM in the request. We also have the following programming variables. (, )willbe1ifthe th VM in the request is allocated to the th server, and 0 otherwise. ( ) is the aggregated bandwidth requirement of all VMs allocated into the th server. ( ) is the bandwidth that should be reserved on the outbound link of the th node. ( ) is the number of available VM slots within the subtree rooted at the th node after the allocation. If we choose to minimize the utilization of link bandwidth in the allocation enforcement, the programming formulation will be s.t. (1) (2) (3) (4) Constraints (1) and (2) guarantee that the bandwidth to be reserved on the outbound link of a node should be the smaller one between the aggregated bandwidth requirement of VMs allocated to the subtree and that allocated to the rest part of the network. Constraint (3) ensures that the reserved bandwidth in the outbound link is not greater than the residual bandwidth on the link. Constraints (4) and (5) guarantee that the number of VMs allocated to each server does not exceed its number of available VM slots, and each VM should be allocated exactly once. Clearly, this programming is a nonconvex quadratic constraint programming. If we choose to maximize the square sum of fragment sizes in the allocation enforcement, we will set (where ) as the optimization goal. We can also use a multiobjective programming to optimize the twofactorsinamixedway. 3) Approximate Solution: Since the acceptability determination problem is NP-complete and the programming method cannot be computationally efficient, we need to design an approximate solution for online decision, scaling to a large size of network. We build our algorithm on top of our solution for homogeneous bandwidth demand in Section III. The key difference between heterogeneous demand and homogeneous demand is that, for a request of VMs, the number of possible VM sets is for every subtree in homogeneous demand, while in heterogeneous demand, due to the bandwidth diversity among the VMs. We propose the following heuristics to reduce the computational complexity. In the heterogeneous demand, for a request of VMs, we have VM sets, and it is possible that each set has a unique total outgoing bandwidth. However, in general cases, the sets can be divided into several groups, while in each group the differential of the total outgoing bandwidth requirement between different sets can be insignificant. We divide the VM sets into a number of groups, with each group identified by three metrics: the number of VMs in the set, ; the minimum bandwidth requested by VMs in the set, ; and the maximum bandwidth requested by VMs in the set,. Hence, a group is denoted by. We put VM sets with equal values of all the three metrics into the same group. For example, VM set and will be put into the same group. We modify Algorithm 1 to handle heterogeneous requests in the following way. Each node maintains ARs instead of one. We use to denote the AR of node with as the lower bound of the VMs allocated and as the lower bound. If all VM sets within group are allocatable in the subtree rooted at node, then the th leftmost bit of is set as 1. Similar to Algorithm 1, in our heterogeneous allocation algorithm the ARs of each node can be computed in a bottom-up manner. The ARs of servers can be computed from its available VM slots and residual bandwidth, while the ARs of switches can be computed from the residual bandwidth of the outbound link and the ARs of its children nodes. We again compute of servers and switches separately. (5)

7 1654 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 Computing ARs for Servers: We consider computing for a server with available VM slots and residual bandwidth on the outbound link. The feasible solution space constrained by VM slots is. As for the feasible solution space constrained by the outbound link capacity, if, then any VM set can be put into the server, and we have.otherwise, for similar reasons with homogeneous demands, we suppose. To take the worst cases into account, when we calculate the upper bound of the first continuous space and the lower bound of the second continuous space, we need to guarantee that the VM sets with the highest possible bandwidth requirements from the group can be allocated. As a result, we have We may not find a proper value for, which indicates that the second continuous space does not exist. Basic Section Splitting: is the intersection of and. If, we also need to conduct basic section splitting as we do for homogeneous bandwidth demand. The splitting point is. Computing ARs for Switches: of a switch,isthe intersection of, which is aggregated from the ARs of its children nodes, and, which is constrained by the residual bandwidth of the outbound link. is calculated in the same way as we do in computing ARs for servers. In what follows, we only discuss how to calculate. Generally speaking, should be aggregated from the basic section and extended sections in the AR of each child node. There can be at most one extended section used for each aggregation since the summed bandwidth of VMs from two extended sections will exceed the total bandwidth requirement of any VM set. As for the aggregation of the basic sections from ARs of the children nodes, a naive way is to simply sum up all 's from every child node. However, in this way many feasible solutions will be missed. We use an example to illustrate the problem. Assume we calculate for a switch, which has two children nodes and, which have available bandwidth of 3.5 and 4.2 Gb/s, respectively. The bandwidth requirement of VMs and the calculation result of ARs are given by Fig. 4. If we make naive aggregation of basic sections of ARs within the same group, we have. However, note that we preferably consider VMs with the highest bandwidth requirements when calculating. Hence, when we have allocated VMs to a child node, the maximum sequence of the rest of the VMs in the set is at most. Therefore, we may be able to allocate more VMs because if Fig. 4. Example showing AR aggregation with VM upper bound decreasing will get better result than naive AR aggregation. the maximum VM sequence of the set is lower, the set can be put into a group with lower, and the worst case bandwidth requirement of the group will be smaller. In this example, since child node can host 5 VMs with the highest bandwidth, we can decrease the maximum VM sequence in child node as.inthisway,wehave, which increases the solution space compared to the naive aggregation. Impact of AR Aggregation Ordering: If a switch has children nodes, there are possible orders to aggregate ARs of the children nodes. In Algorithm 1, aggregating the ARs in different orders will not affect the result since all the VMs are identical. However, in the heterogeneous case, the aggregation orders will affect the results. In the example above, if we aggregate the ARs in a different order, we can also get as. Now we investigate how the aggregation order affects the aggregation result. Suppose we calculate the AR of node by aggregating the ARs of its children nodes. The residual bandwidth of the th child's outgoing link is.wetakealook at the AR of the th child,, where the upper bound and lower bound of the AR are and, respectively. If the th bit of is set as 1, it means that we can allocate any combination of VMs between the th and the th VM to the subtree rooted at this child, and the outgoing bandwidth of child is able to hold these VMs. If we only consider the case of basic section, we have. In the aggregation step, when we aggregate the AR of the th child for a node, if we choose the th bit from, there may be some remaining network capacity in the outgoing link of child that cannot be used at all for later usage. We call this link capacity the wasted capacity for the th child, denoted as. If we only consider the basic section, we have.wedefinethe aggregation-wasted capacity for a node as the sum of all its children's wasted capacities after aggregation, denoted as. This metric indicates how much capacity of the outgoing links of the children nodes cannot used by the AR aggregation. When we allocate different numbers of VMs to the children nodes, their wasted capacities are not the same, and their summary can also be different. After aggregation, we have,inwhich is the total number of VMs that can be allocated from this aggregation. Note that sometimes may be greater than, which is the greatest bandwidth requirement of the VMs not allocated. From the

8 LI et al.: GUARANTEEING HETEROGENEOUS BANDWIDTH DEMAND IN MULTITENANT DATA CENTER NETWORKS 1655 Algorithm 2: Calculating on node. Fig. 5. Example of how the AR aggregation order affects the aggregation-wasted capacity and the upper bound of the basic section. The request is, and the residual outgoing capacities of the two children nodes are 20 and 17, respectively. Fig. 6. Example to show that greedy allocation may cause failure in allocation, while strict continuous allocation will always be applicable. expression, greater usually indicates that fewer VMs can be accepted by this AR aggregation. Fig. 5 gives an example of such difference due to different aggregation ordering. The bandwidth request is, and we now compute for a given node.forthetwo children nodes of, and, their residual outgoing capacities are 20 and 17, respectively. If we aggregate their ARs by the order of, the aggregation-wasted capacity for node is 3, and the upper bound of the basic section of is 4. However, if we aggregate the ARs in a reversed order, the aggregation-wasted capacity will be 10, and the upper bound of the basic section will be only 3. Therefore, the best AR aggregation order is the one that gives the minimum aggregation-wasted capacity. Yet finding the exact minimum aggregation-wasted capacity is very complicated. Here, we use a simple method to find an approximate solution. When we compute an AR for a node,wealso compute its wasted capacity for the upper bound of its basic section. In the aggregation step, when we choose the next child node, we preferentially choose the one whose AR has the smallest wasted capacity. Algorithm 2 illustrates the process of calculating on a switch node. In the algorithm, indicates the wasted capacity of node if we allocate into it VMs, where is the upper bound of the basic section of. After calculating by intersecting and on a switch, we conduct basic section splitting in the same way as we do for servers. 1 AggregateAR Input: : smallest VM sequence in the group; : largest VM sequence in the group. Output:. 2 ; 3 for each extended section do 4 ; 5 end 6 ; 7 return ; 8 Input: : smallest VM sequence in the group; : largest VM sequence in the group; : excluded child node. Output: : upper bound of the aggregated basic section. 9 ; 10 ; 11 while & do 12 ; 13 if then 14 ; 15 ; 16 end 17 ; 18 end 19 return ; B. Allocation Enforcement Next, we present the allocation enforcement method for heterogeneous demands. Algorithm 3 shows how to assign the VM set into several subsets when node hosts the VM set of, one for each child node. VM allocation is a reversion of AR aggregation. As aforementioned, in aggregation, if we take different orders to aggregate ARs of the children nodes, we may have different results. In allocation, we have the similar problem. On each node, we may allocate VMs to its children nodes in different orders, but only some of the orders can reach a successful allocation. Fig. 6 gives an example to show that sometimes greedy allocation seems applicable at a certain switch, but may cause allocation failure when going to lower-level switches. In the example, the outgoing link capacity of the top switch is 20, and capacities of the two children switches are 10 and 9, respectively. We have four VMs to allocate:. Though allocating seems to be a better choice than since it fully utilizes the outgoing link bandwidth of the upper-layer switch, when it goes to the children switches, we cannot find any feasible solutions to further allocate the three VMs. Due to the reason above, to ensure that we can always find a successful allocation, we assign VMs to children nodes in the

9 1656 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 Algorithm 3: Assigning the set of VMs to each child node on node for heterogeneous bandwidth demand. 1 Allocate Input: : the bandwidth of VMs in an ascending order. Output: : Allocation solution, is the set of allocated VMs for child node. 2 Sort all extended sections by in a descending order; 3 for each extended section do 4 if & then 5 ; 6 return true; 7 end 8 if then 9 ; 10 for all ; 11 return true; 12 end 13 end 14 return ; 15 Find Input: : excluded child; : the bandwidth of VMs in an ascending order. Output: : Allocation solution. 16 for any ; 17 ; 18 while & do 19 ; 20 if then 21 ; 22 else 23 ; 24 end 25 ; 26 ; 27 end 28 if then 29 return true; 30 end 31 return false; same order as that in the aggregation step. In other words, the children node with the highest upper bound of the basic section will be allocated first (line 19). Continuous Allocation: From Algorithm 3, we note that we always allocate a set of continuous VMs from the bandwidth request to each child node (line 21), though the ARs of the children nodes do not require doing so. Next, we explain why it is allocated in this way. Consider the scenario that we have allocated a group of continuous VMs to the subtree rooted from a certain node. The upper bound and lower bound for the VM indices are and, respectively. Now we split the set of VMs into subgroups and allocate them into the subtree rooted at children nodes of node. The allocating order is exactly the same as the order when we get the aggregated AR of node. Suppose we are allocating VMs to a child node now. The upper bound and lower bounds for the unallocated VMs are and, respectively. We need to look up to find how many VMs at most can be allocated into the subtree rooted at node. Since the AR only gives the number of VMs that can be allocated in but not the exact set of VMs, we can always choose to allocate the VMs with the highest bandwidth requirement preferentially. Hence in later allocation, when looking up the ARs of the other children nodes, we can take a lower VM upper bound. If we start allocating VMs with the highest bandwidth requirement from the first child to the last child in the allocation order, we will split the group of VMs into subgroups, while the VMs in each subgroup are continuous as well. Impact on the Acceptance Determination Algorithm: In Section IV-A, we showed that each node in the network maintains ARs to handle heterogeneous bandwidth demand. By observing the continuous allocation in the VM allocation process, we can simplify the acceptance determination on the following two aspects. First, when we compute the value of for the AR constrained by the outgoing bandwidth, we can preferentially allocate the VMs with the highest bandwidth demand. Thus, we have. In our original AR computation algorithm, when obtaining, we are not sure whether the VMs between the th and the th VM will be available in practical allocation. Therefore, when we compute the lower bound for the extended section, we have to consider the worst case, which starts allocating VMs with the smallest bandwidth requirements. However, based on the continuous VM allocation, we can start allocating with the highest bandwidth requirements, since all VMs at hand are allocatable, and allocating the one with the highest bandwidth requirement can utilize the bandwidth as high as possible. It benefits the acceptance determination algorithm because we can get a lower. Second, we only need to maintain ARs for each node, instead of ARs in the original algorithm. Since we can compute starting from the VM with the highest bandwidth requirement, it is no longer necessary to keep the lower bound of the VM set. is the AR when only VMs whose bandwidth requirements are no higher than can be allocated. It will also accelerate the AR computation process. V. SIMULATION RESULTS In this section, we use simulations to evaluate the performance of our VM allocation algorithms for multitenancy data center networks. A. Simulation Setup The topology we use is a three-layer -array tree structure. Specifically, the core switch (the root of the tree network) connects aggregation switches, each aggregation switch connects ToR switches, and each ToR switch connects to physical

10 LI et al.: GUARANTEEING HETEROGENEOUS BANDWIDTH DEMAND IN MULTITENANT DATA CENTER NETWORKS 1657 Fig. 7. Performance of homogeneous AR algorithm against Oktopus handling homogeneous bandwidth requests. Mb/s,. (a) Total number of tenants accepted. (b) Average VM slot utilization. (c) First rejected request. servers. Each physical server has VM slots. The link capacity between ToR switch and servers is 1 Gb/s, and that between switches is 10 Gb/s. These are realistic settings commonly used in the literature as well. For each simulation, we randomly generate 100 groups of tenant requests, and every group contains 2000 tenant requests. We then take the average of the 100 groups. Among different requests, the numbers of requested VMs are subject to the negative exponential distribution, where is the average number of VMs per tenant. As for the requested VM bandwidths, in homogeneous bandwidth requests, the bandwidths among different tenant requests follow the normal distribution of,where is the average bandwidth requested by the tenants. For heterogeneous bandwidth requests, we generate bandwidths following two different patterns, namely, Gaussian pattern and bipartite pattern. In the Gaussian bandwidth pattern, we randomly generate the bandwidth request of every VM in a request following the normal distribution of, where is called the bandwidth heterogeneity index. If is larger, the standard deviation among the requested bandwidths of VMs within the same request is higher. In the bipartite bandwidth pattern, for each request, we divide the VMs into two groups: with high bandwidth requirements and with low bandwidth requirements. We fix the total amount of bandwidth required by all the VMs and vary the ratio of high bandwidth over low bandwidth, which should be the same as the ratio of the number of low-bandwidth VMs over the number of high-bandwidth VMs. We use to represent the ratio. In every group of simulation, the 2000 tenants arrive and leave the network under M/M/S queuing model (Erlang system). In M/M/S queuing system, the basic arrival and leaving rates of each tenant are and, respectively. The actual arrival and leaving rates of tenants in the network are and,inwhich is the number of tenants currently inhabiting in the network. Hence, the more tenants there are, the more likely there is one of them to leave. In our simulation, we set the ratio. Upon tenant arrival, if the data center network cannot accept the request, the request will be rejected. When a tenant leaves the data center, resources allocated to the tenant will be released and reused in subsequent allocation. We evaluate the performance of the allocation algorithms by the following three metrics: 1) the total number of tenants accepted after all the 2000 requests are processed; 2) the average VM slot utilization across time; 3) the index of the first rejected tenant request. B. Homogeneous Bandwidth Request In this set of simulations, we compare the performance between Oktopus and our homogeneous algorithm to process homogeneous bandwidth requests. We setup a three-layer 10-array tree, and each server has 10 VM slots. We vary from 10 to 50, and set Mb/s. Fig. 7 shows the results. Fig. 7(a) and (b) demonstrates that our homogeneous allocation algorithm accepts 30% more tenants and achieves 40% higher VM utilization than Oktopus does, due to the improved VM allocation strategy. The most significant advantage of our allocation algorithm over Octopus is shown by Fig. 7(c), in which we find that the first tenant rejection by our algorithm comes much later than Oktopus. The reason is that Oktopus has false negative when determining the acceptability of a tenant request and is likely to reject a request even though the data center still has sufficient resource. On the other hand, our allocation algorithm determines the acceptability much more accurately and can accept much more tenants before the first rejection occurs. It is worth noting that the false-negative determination when rejecting a tenant does not mean that the cloud provider cannot accept subsequent tenant requests. However, the cloud provider can indeed benefit from a later rejection because it makes better usage of the data center resource and will attract more tenants. C. Heterogeneous Bandwidth Request Then, we change the request to heterogenous bandwidth demand. For comparison, we not only run our heterogeneous allocation algorithm, but also directly apply our homogeneous allocation algorithm and the Oktopus algorithm to handle the heterogeneous requests in the following way: translating the heterogeneous bandwidth request of to, so that the tenant's requirement can be satisfied. Of course, this calibration will cause considerable bandwidth waste. We set Mb/s and when generating the tenants' requests. As aforementioned, we use two different patterns to generate the heterogeneous bandwidth requests, namely, the Gaussian

11 1658 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 Fig. 8. Performance of heterogeneous and homogeneous AR algorithm against Oktopus handling heterogeneous bandwidth requests following Gaussian distribution. Mb/s,,and. (a) Total number of tenants accepted. (b) Average VM slot utilization. (c) First rejected request. Fig. 9. Performance of heterogeneous and homogeneous AR algorithm against Oktopus handling heterogeneous bandwidth requests in bipartite patterns. Mb/s,,and. (a) Total number of tenants accepted. (b) Average VM slot utilization. (c) First rejected request. pattern and the bipartite pattern. Fig. 8 shows the results for the former pattern, in which we vary the bandwidth heterogeneity index,, from 1.2 to 2.5. We find that our heterogeneous allocation algorithm outperforms the other two allocation algorithms by accepting 30% more tenants and achieving higher VM utilization. The first rejected tenant in our heterogeneous allocation algorithm comes after hundreds of tenants are accepted, while in Oktoups the first rejection occurs during the first dozens of requests. The performance of the allocation algorithms degrades with higher bandwidth heterogeneous index because more bandwidth fragments will be caused. The reason that our homogeneous allocation algorithm and the Oktopus algorithm perform much closer to each other than the result in homogeneous bandwidth demand is that the calibration of the bandwidth requests leads to too high bandwidth demand and thus leaves little room for our homogeneous allocation algorithm to improve over Oktopus. Fig. 9 illustrates the results for the bipartite bandwidth pattern, in which we vary the high bandwidth/low bandwidth ratio,, from 2 to 7. The result is consistent with previous groups of simulations. Our heterogeneous allocation algorithm obviously outperforms Oktopus and the homogeneous allocation algorithm. It is worth noting that the first tenant rejection in Oktopus allocation algorithm occurs extremely early in this case due to false-negative acceptability determination. D. Comparison to Offline Optimization Since our heterogeneous allocation algorithm is heuristic, we compare it to the offline optimal solution by programming Fig. 10. Comparing the three online VM allocation algorithms to the offline optimal solution.,,and Mb/s. solvers. We also compare it to the homogeneous allocation algorithm and the Oktopus algorithm. Since it takes an unaffordable long time (approximately several days to handle a single request) to run the programming solution in large-scale networks (same as that used in the simulation), we use a three-layer 5-port tree with 5 VM slots on each server and link capacity of 500 Mb/s. We generate heterogeneous bandwidth requests with,, and vary from 50 to 200 Mb/s. Fig. 10 shows the result. We can see that our heterogeneous allocation algorithm accepts about 90% VMs of what offline optimization does. When gets higher, the gaps between the programming solution and the two homogeneous allocation algorithms gets larger because more bandwidth is wasted in calibrating the requests.

12 LI et al.: GUARANTEEING HETEROGENEOUS BANDWIDTH DEMAND IN MULTITENANT DATA CENTER NETWORKS 1659 VI. EXPERIMENTS To further demonstrate the performance of our allocation algorithm in a real system, we carry out experiments on a small data center testbed. The testbed is composed of 16 physical servers connected by a nonblocking switch. The network capacity of every link is 1 Gb/s. All the servers are desktops with AMDAthlonIIX GCPUand2GBDRAM.Each server is installed with 4 VMs, so there are 64 VM slots in total. We generate 8 tenant requests, each requesting for 8 VMs. The heterogeneous bandwidths of VMs in each tenant request follow the Gaussian pattern with the bandwidth heterogeneity index varying from 1.5 to 2.5 and mean bandwidth of 300 Mb/s per VM. We run both bandwidth-guaranteed and bandwidth-oblivious VM allocations. For bandwidth-guaranteed VM allocation, we use our heterogeneous algorithm and limit the outbound traffic rates of the VMs (enforced by the traffic control module in Linux kernel). For bandwidth-oblivious VM allocation, we just use localized VM allocation and let the VMs compete for all the link bandwidths. To decouple the network performance from disk I/O constraints, we use iperf to generate the traffic. The experiment result is shown in Fig. 11, with the job completion time in the -axis. We find that even using a small network topology and 8 tenants, bandwidth-guaranteed VM allocation can reduce the task completion time, especially when there is high heterogeneity among the requested bandwidths in a tenant. The fundamental reason is that under bandwidth-guaranteed VM allocation, we make coordinated utilization of the network resource among the tenants and VMs. However, if using bandwidth-oblivious VM allocation, VMs with high bandwidth demands may be located in the same physical server (or subtree), which causes the bandwidth bottleneck while the network resource in other places is wasted. The benefit is more obvious when the bandwidths requested by VMs deviate further from each other. VII. RELATED WORK In current cloud services such as Amazon EC2 [6], the network bandwidths allocated to multiple tenants depend on the TCP-based competition among the tenants. Due to vulnerability to denial-of-service (DoS) attacks and lack of bandwidth guarantee, recently there is growing interest in the community to design bandwidth-sharing mechanisms in data center networks. Generally speaking, the proposed mechanisms can be divided into two categories, i.e., competition-based and allocation-based. The competition-based mechanisms extend the flow-level competitions to VM-level or tenant-level ones by assigning weights to VMs or tenants/services for competition. Competition-based bandwidth sharing can make efficient utilization of the network resource, but the network performance of a tenant depends on the number and weights of other tenants in the network. Typical examples of this type of mechanism include Seawall [2], Netshare [11], FairCloud [12], and EyeQ [15]. Seawall [2] designs (sender) VM-level congestion control to avoid unfair bandwidth sharing that depends on the number of TCP flows on a VM. Netshare [11] proposes a hierarchical weighted max-min fair sharing mechanism that the network manager allocates weights to services, and Fig. 11. Task completion times under bandwidth-guaranteed and bandwidthoblivious VM allocation. then each service allocates the bandwidth equally to TCP flows. Faircloud [12] analyzes the dilemma met in today's weight-based bandwidth-sharing solutions and brings forward a weight assignment method. EyeQ [15] enforces admission control on traffic at the end-hosts and provides a minimum bandwidth guarantee to VMs, while the VMs compete for the residual bandwidth in the links. Moreover, some most recent works like Hadrian [17] expand the network resource sharing from intratenant traffic to intertenant traffic. However, in this paper, we still focus on the scenario that network communication occurs among VMs within the same tenant. In contrast, allocation-based mechanisms abstract a bandwidth request model for a tenant, assign VMs of the tenant into proper locations of the network to meet the bandwidth requirement, and use rate limiters at hypervisors to enforce the bandwidth reservation and isolation. Though allocation-based mechanisms may fail to make full utilization of the network resource, it provides predictable network performance for a tenant without concern of other competing tenants. Representative solutions of this category include SecondNet [1], Oktopus [5], and TIVC [13]. SecondNet [1] proposes using a bandwidth matrix to describe the tenant's bandwidth request and designs heuristic algorithms to allocate VMs based on VM's requirements on both network bandwidth and computation resource. Instead, Oktopus [5] uses a hose model [14] to abstract the tenant's bandwidth request, including both virtual clusters and oversubscribed virtual clusters. They also propose VM allocation algorithms to deal with homogeneous bandwidth request, in which all the VMs request for equal bandwidth. TIVC [13] follows the basic idea of Oktopus, but improves the bandwidth utilization efficiency by capturing the time-interleaving dynamical bandwidth requirement from a tenant. Our work can be regarded as a complementary work to TIVC since we focus on the spatial heterogeneity in bandwidth demand between VMs while TIVC targets at temporal heterogeneity for a certain VM. Bazaar [16] provides more flexibility when assigning the VMs to guarantee bandwidth requirements by adjusting the required number of VMs and bandwidth for each VM,soastomoreefficientlyutilize the cloud resource. VIII. CONCLUSION In this paper, we studied the VM allocation problem in virtualized data center networks concerning the bandwidth

13 1660 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 23, NO. 5, OCTOBER 2015 guarantee. We improve the existing allocation algorithm for homogeneous bandwidth request by considering the in-continuity in the solutions space. More importantly, we propose a heterogeneous bandwidth request model that provides more flexibility for tenants to abstract their needs, and we design VM allocations algorithms to solve this NP-complete problem. Evaluation results show that our algorithm can better utilize the VM resources in the data center than existing algorithms, significantly postpones the first tenant rejection, and performs close to the optimal offline allocation. It shows that simply applying homogeneous algorithm can result in suboptimal results, which can be significantly improved by our heterogeneous algorithms. ACKNOWLEDGMENT A previous version of this paper was published in the Proceedings of the IEEE International Conference on Network Protocols (ICNP), Austin, TX, USA, October 30 November 2, In this version, we make the following extensions. First, we take the waste capacity into consideration when making acceptance determination in heterogenous bandwidth allocation. Second, we simplify the AR aggregation algorithm in heterogenous bandwidth allocation based on the feature of the allocation enforcement algorithm. Third, we add more simulations (particularly the simulation for bandwidth requests with bipartite pattern) to evaluate our allocation algorithm. Fourth, we add an example to show the problem with existing allocation algorithms to better motivate our work. [15] V. Jeyakumar et al., EyeQ: Practical network performance isolation for the multi-tenant cloud, in Proc. 4th USENIX Workshop Hot Topics Cloud Comput., 2012,p.8. [16] V. Jalaparti et al., Bridging the tenant-provider gap in cloud services, in Proc. ACM Symp. Cloud Comput., 2012, Art. no. 10. [17] H. Ballani et al., Chatty tenants and the cloud network sharing problem, in Proc. 10th USENIX Conf. Netw. Syst. Design Implement., 2013, pp Dan Li (M 10) received the Ph.D. degree in computer science from Tsinghua University, Beijing, China, in He is an Associate Professor with the Computer Science Department, Tsinghua University. His research interests include Future Internet architecture and data center networking. Jing Zhu received the B.S. degree from the International Institute of Informatics and Systemics, Tsinghua Univerisity, Beijing, China, in 2011, and is currently pursuing the Ph.D. degree in computer science at Tsinghua University. His research interests include Internet architecture and protocol designing, routing strategy designing, and data center network. REFERENCES [1] C. Guo et al., SecondNet: A data center network virtualization architecture with bandwidth guarantees, in Proc. 6th CoNEXT, 2010,Art. no. 15. [2] A. Shieh et al., Sharing the data center network, in Proc. 8th USENIX Conf. Netw. Syst. Design Implement., 2011, pp [3] J. Crowcroft and P. Oechslin, Differentiated end-to-end Internet services using a weighted proportional fair sharing TCP, Comput. Commun. Rev., vol. 28, no. 3, pp , [4] R. Pan, B. Prabhakar, and A. Laxmikantha, QCN: Quantized congestion notification, 2007 [Online]. Available: 1/files/public/docs2007/au-prabhakar-qcn-description.pdf [5] H. Ballani et al., Towards predictable datacenter networks, in Proc. ACM SIGCOMM, 2011, pp [6] Amazon, Seattle, WA, USA, Amazon Elastic Compute Cloud, [Online]. Available: [7] Google, Mountain View, CA, USA, Google App Engine, [Online]. Available: [8] Microsoft, Redmond, WA, USA, Windows Azure platform, [Online]. Available: [9] E. Keller, J. Szefer, and J. Rexford, NoHype: Virtualized cloud infrastructure without the virtualization, SIGARCH Comput. Archit. News, vol. 38, pp , [10] T. Ristenpart et al., Hey, you, get off of my cloud: Exploring information leakage in third-party compute clouds, in Proc. 16th ACM Conf. Comput. Commun. Security, 2009, pp [11] T. Lam, S. Radhakrishnan, and A. Vahdat, NetShare: Virtualizing data center networks across services, UCSD, La Jolla, CA, USA, Tech. Rep., [12] L. Popa et al., FairCloud: Sharing the network in cloud computing, in Proc. ACM SIGCOMM, 2012, pp [13] D. Xie, N. Ding, Y. Hu, and R. Kompella, The only constant is change: Incorporating time-varying network reservations in data centers, in Proc. ACM SIGCOMM, 2012, pp [14] N. G. Duffield et al., Resource management with hoses: Point-tocloud services for virtual private networks, IEEE/ACM Trans. Netw., vol. 10, no. 5, pp , Oct Jianping Wu (M 04 SM 05 F 12) received the master and doctoral degrees in computer science from Tsinghua University, Beijing, China, in He is now a Full Professor with the Computer Science Department, Tsinghua University. In the research areas of the network architecture, high-performance routing and switching, protocol testing, and formal methods, he has published more than 200 technical papers in academic journals and proceedings of international conferences. Junjie Guan received the B.E. degree from Beijing University of Posts and Telecommunications, Beijing, China, in 2013, and is currently a graduate student in computer science at Dartmouth College, Hanover, NH, USA. He joined the NASP research group, Tsinghua University, Beijing, China, during He is interested in distributed system and data center networking. Ying Zhang received the Ph.D. degree in electrical engineering and computer science from the University of Michigan, Ann Arbor, MI, USA, in She is a Researcher with the IP and Transport Research Group, Ericsson Research Silicon Valley, San Jose, CA, USA. Her research interests are in networking and systems, including software-defined networking, cloud, Internet, and cellular network management, Internet routing and measurement, and network security.