Resource Allocation in the Network Operator's Cloud: A Virtualization Approach

Transcription

1 Resource Allocation in the Network Operator's Cloud: A Virtualization Approach J. Soares,2, J. Carapinha, M. Melo,2 Portugal Telecom Inovação Aveiro, Portugal {joao-m-soares, jorgec, marcio-mmelo}@ptinovacao.pt Abstract The access infrastructure to the Cloud is usually a major drawback that limits the uptake of Cloud services. Attention has turned to rethinking a new architectural deployment of the overall Cloud service delivery. We argue that it is not sufficient to integrate the cloud domain with the operator s network domain based on the current models. In this work we envision a full integration of the Cloud and the network, where cloud resources are no longer confined to a data center, but are spread throughout the network and owned by the network operator. In such an environment, challenges arise at different levels, such as at the resource management, where both cloud and network resources need to be managed in an integrated approach. We particularly address the resource allocation problem through joint virtualization of network and cloud resources, by proposing an algorithm to allocate cloud and network resources in an integrated way. This algorithm is evaluated through both simulation and experimental results in a real virtualization platform. Keywords-Cloud; Network; Resources; Server Stress; Node Stress; Link Stress I. INTRODUCTION Cloud Computing has no universally accepted definition but there is a consensus regarding its fundamental properties: scalability, elasticity, on-demand self-provisioning, pay-peruse, and anywhere, anytime access. It is clear that CC brings with it requirements at two different levels: at the data center (DC) level; and at the access level. Depending on the type of service, different Quality of Service (QoS) guarantees are required on the DC and in the network (Wide Area Network - WAN). Moreover, the scalability and elasticity of the Cloud may suggest variations on the requested network resources as the Cloud scales up or down. However, Cloud and network are two completely different entities, and usually the access to Cloud services is done over best-effort Internet. In some cases, when typical best effort Internet model is not enough, an independent network service that fulfills the Cloud service requirements can be purchased, connecting the user and the Cloud hosting the service. This typically happens in the enterprise sector, namely through operator-managed VPN service. Nevertheless, today these network services are statically provisioned and cannot follow the dynamics of the Cloud. The access component of the Cloud is extremely important [] and is also a major drawback that is limiting the uptake of Cloud services: an increasingly large number of heavy services, that are not able to be supported in a best-effort way, are emerging (e.g. Netflix, OnLive). The key role that the networking layer needs to play in the CC service R. Monteiro 2, a Sargento 2 2 Instituto de Telecomunicações University of Aveiro Aveiro, Portugal {romeumonteiro7, susana}@ua.pt delivery has been exposed, and attention has turned to rethink a new architectural deployment of the overall Cloud service delivery [] [2] [3]. However, we believe that it is not sufficient to integrate the cloud domain with the operator s network domain based on the current models, where Cloud resources are confined to big DCs. Today, CC relies on the power of big DCs, which has proven to reduce costs [4]. However, this is not the best solution for every problem, in particular when: Many individuals or organizations in a certain geographical area need to access the same resource/content/service. In such a case, it seems more appropriate that this object is located, moved or cached near the users, rather than being repeatedly transported across the network (negatively impacts the network performance) []; The access to a shared resource/content/service often requires low latency, therefore, it might be appropriate to locate the data closer to the users. In such cases, instead of relying on a single DC, Cloud Providers (CPs) can use Content Delivery Networks (CDNs), e.g. Akamai, to improve the delivery of their services. This is indeed true, but the level of dispersion of DCs in the network will be always limited to some degree. Considering all the above mentioned factors, the network operator (NO) is on the right path to play a strong and active role in the CC business. In this work we envision the full integration of Cloud and network, where cloud resources are no longer confined to DCs, but are spread throughout the network. Cloud and network resources will then be provisioned in an integrated way to provide a specific service support. Therefore, a joint management of cloud and network resources will be required, along with requirements at other levels, e.g. security. In this work we tackle the resource management topic. To provide a flexible and costeffective infrastructure for this integration, we consider the provisioning of cloud and network resources through virtualization (virtualized computing, storage and network), enabling the support of virtual infrastructures (VIs). In this virtual integrated environment, discovery, allocation, adaptation and re-optimization of both network and cloud resources, are main challenges of joint resource management. In particular, we address the allocation problem by proposing an algorithm to allocate the resources in an integrated way, complementing it with simulation and experimental results in a real virtualization platform. This paper is organized as follows. Section II describes the relevant work in the area and the way it relates to our work. Section III presents an algorithm for mapping cloud and network resources based on node and link stress, and /2/$3. 22 IEEE 8

2 studies different approaches for node and server stress calculation, evaluating the impact of different parameters in the joint embedding process in section IV. Further, in section V we present the results of an experimental testbed. Finally, section VI presents the conclusions and indicates directions for future work. II. RELATED WORK The importance of the WAN role in the cloud is becoming increasingly evident. Standardization bodies and enterprise efforts have highlighted the need for cloud and network resources to be handled together. IBM offers enterprises a cloud data backup supported by Verizon s VPN services, and Cisco has recently presented CloudVerse []. The Open Grid Forum (OGF), with the pyocni [6] is one of the standardization bodies actively involved in the subject. From the research community, with few exceptions, network and cloud resource allocation are usually addressed as two different problems. Works in [7], [8], and [9] address the allocation of virtual machines (VMs). More recently, [] addresses the problem of resource allocation in a large-scale cloud under CPU and memory constrains. The VN mapping optimization problem, which strives to map virtual nodes to physical nodes and virtual links to physical links in a way that maximizes the amount of virtual resources that can be embedded in the physical infrastructure, can be formulated as an unsplittable flow problem, known to be NP-hard [], and therefore it is only tractable for a small amount of nodes and links. Proposals such as [], [3], [4] propose heuristics and specific approaches to reduce the complexity of the mapping approach. In [] it is presented a fair joint multiple resource - computational and network - allocation; however, it makes a clear separation between both. Ongoing EU-funded projects such as SAIL [6] and GEYSERS [7] are additional examples of work in this area. The presented approaches and achievements made so far are of great interest; however, none in fact covers the joint provisioning of cloud and network resources. Our work addresses the subject, having cloud resources scattered in the operator s network. III. THE CLOUD INSIDE THE OPERATOR S NETWORK A VIRTUALIZATION APPROACH This section identifies the challenges, possible implementations, and its associated constrains, for a scenario where cloud and network resources are provisioned in an integrated way by the NO (sub-section A). Through the proposed approach, we present an algorithm to provide the allocation of VIs, i.e. infrastructures that combine cloud and network resources. Note that we consider cloud resources as virtual servers (VS); however, this can be extended to DCs. A. The Challenge and the Approach When coupling network and CC resources, several resource management challenges arise: discovery, allocation, adaptation and re-optimization of both resources in an integrated approach. The management of these resources lays upon concepts of virtual resource mapping in the physical infrastructure with self-organized reconfiguration of resources, devices and associated network, according to the service/user requirements, policies (e.g location) and changes in the infrastructure [8]. There are several challenges, and in this work we mainly target resource allocation. Virtual resources should be provisioned and placed in an optimal location according to the available physical resources and the service requirements, based on a number of possible criteria from both cloud and network, e.g.: type of VMs and possible restriction on location of these VMs; latency, bandwidth topology, geographical places where users will access the service. In order to map these resources, a combined algorithm, able to perform balanced decisions taking into account the abovementioned requirements of both network and cloud resources, is needed. This algorithm must be able to determine a possible solution, i.e., physical hosts able to allocate the cloud resources which, at the same time, can have an associated network service able to fulfill the requirements in the access to the cloud. Moreover, agility is a key feature of CC. The key enabler behind it is virtualization, by allowing the decoupling of operating systems (OSs) and applications from the underlying physical infrastructure. It is thus fundamental that this agility is preserved when bringing the network into the picture. NO managed VPNs (e.g. BGP/MPLS) are among the available network offers today for enterprises. This type of services has been conceived to work in a relatively stable network environment (which is the case with most enterprise VPNs today), but is not appropriate to cope with the typical dynamics of Cloud services. Part of the reason lies in the characteristics of the BGP protocol, in charge of handling intra-vpn routing, which suffers from well-known slow convergence issues [9]. On the other hand, network virtualization has been explored by different research initiatives in multiple contexts and application scenarios, where its flexibility is pointed out as a major feature. With these considerations in mind, our approach builds upon a scenario in which all resources can be virtualized. B. Virtual Infrastructure Allocation Algorithm Resource management functions work according to management objectives, resource capacity and user preferences/constrains. Examples of management objectives for resource allocation are the fair resource allocation, minimizing energy consumption, or ensuring QoS, such as guaranteeing minimum amounts of exclusive bandwidth or maximum packet delays. These objectives are not mutually exclusive and can, therefore, be combined. However, it usually results in an NP-hard problem. In [], the authors propose a heuristic to map virtual resources in the physical infrastructure, which supports the network heterogeneity, in both links and nodes. The algorithm is inspired by the concepts of node and link stress, i.e., links and nodes with less stress are more prone to accepting new virtual resources. Its management objective is to minimize the stress of the resources and to balance the stress among the resources. We will use these concepts to include the joint mapping of network (routers and links) and cloud (VSs) resources /2/$3. 22 IEEE 8

3 There are special characteristics of cloud resources that have to be included in the mapping approach, such as CPU load (Load), memory (MEM) and storage (STG) capacity. We consider that CPU frequency is not a limiting parameter when choosing the server host (an analysis on the subject is presented in [2]). This way, we now have 3 requested parameters, CPU, MEM and STG, that will limit the available physical resources for cloud hosting. We consider that virtual servers (VSs) are treated in a similar way to virtual routers (VRs), but with the following differences: () their candidates are restricted to physical servers (and VRs candidates are restricted to physical routers); (2) the VS parameters and the way VS stress is determined. Server stress is an indicator of how likely a physical server should be to host a VS in comparison with other servers. This indicator is used on the mapping algorithm to calculate the potential of a certain candidate server to host a VS. However, the joint cloud and network approach requires that the potential of a candidate is not calculated only using server stress, but also considering the stress of the physical links which might be used to host virtual links (VLs). Algorithm shows a pseudo-code description of the proposed algorithm that we explain throughout this subsection. The algorithm starts by calculating link and node stress. In [], link stress is the value of the bandwidth in use in the physical link: = () This option is based on the comparative analysis performed via simulation in [2]. We showed that considering either the fraction of bandwidth in use in the physical link or a non-linear approach results in a very similar performance in terms of VIs successfully mapped. The interplay between server and link stress allows for a server placement that considers both the servers' characteristics as well as the network characteristics. In a first approach the (router) node stress was given by: =. (. ) This means that a physical router running 3 VRs, with 7MB of RAM not allocated, a processor working at 2GHz, 4 CPUs and a workload which corresponds to the use of. CPUs, would have a stress given by: = (. ) =8 (3) MEM and Load are parameters that represent amounts of limited resources that are progressively occupied. In equation 2, the stress of the physical nodes is inversely proportional to the amount of free Load and MEM. This option is based on the consideration that a physical network with a balanced amount of free resources in the nodes will be able to maximize the amount of virtual resources it can map and embed. We argue that this is a good approach to balance the amount of free resources in the nodes, but it will have as trade-off longer physical paths for the VLs between the nodes (since more distant nodes might be chosen if they have more free resources). Since virtual nodes always occupy the same amount of physical resources (memory, storage, (2) processor cycles) in any physical node, special attention should be paid to the amount of physical links used to map the VI, since one might save bandwidth by using shorter paths and, as stated in [2], the bandwidth of the physical links represent the main limiting constraint to the amount of embeddable virtual resources. In order to prevent long physical paths from being used when the occupation of the nodes is not at a critical level, we propose the stress node expression 3 for router node stress calculation. = (3) In this approach, represents the average MEM of the virtual nodes, and represents the average load increase for each virtual node embedded; k represents a constant value. This way, when calculating the potential (π) of the candidate nodes (which is a product of the link cost and node stress), link cost will be the most important parameter, until the considered nodes achieve a critical occupation state (that can be adjusted through the constant k). After a thorough analysis through simulation in [2], we reached the best values for k = 3. The proposed server stress expression, in equation 4, has a similar structure to the one in equation 2, proposed in [2], and considers STG in a similar approach as the MEM. This is referred to the proportional stress approach. =. (. ) Equation presents the non-proportional stress approach, which is based on equation 3. As in equation 3, this approach prevents long physical paths from being used when the occupation of the nodes is not at a critical level. Since we are dealing with servers, equation extends (4) /2/$3. 22 IEEE 82

4 equation 3 by considering STG and not taking into account the CPU frequency for the reasons previously stated. = () Note that, in expressions (2) to (), is a small constant to avoid dividing by. After calculating the nodes and links stress ( and ), the proposed algorithm performs a pre-selection of possible physical hosts (candidates), where hosts that do not have a CPU frequency equal or higher than required for the virtual node, or do not have enough free RAM or storage for the virtual node's demands, are excluded from its candidate list. Afterwards, the algorithm verifies if there is at least one physical path available, with the virtual link QoS requirements, between each pair of candidates to the virtual source and destination of the link. These possibilities of connection are registered; the candidates that do not possess at least one possible connection to one candidate of each virtual neighbor node to the virtual node they are applying are removed. Note that each candidate removal along the algorithm is followed by a check of the remaining possible connections for the candidates that had possible connections to the removed candidate, and a removal of those candidates will take place if appropriate, and so on. At this point, the algorithm has removed some of the inadequate candidates; however, as virtual nodes are mapped to the candidates, more inadequate candidates may appear. At this point, the choice of a candidate to host one virtual node may compromise the search for a solution. To prevent this, when choosing a candidate, the algorithm verifies if there is still one candidate for the other virtual nodes; if not, the selected candidate is removed from the candidate list. In the end, the mechanism determines a possible allocation solution for the VI ( ). IV. SIMULATION RESULTS In order to evaluate the algorithm and the different proposed stress expressions, several simulations were performed. For a deeper evaluation study, please refer to [2]. A. Simulator In order to evaluate the proposed algorithm and stress approaches, a Matlab simulator was built. For each run, the program designs a random physical network infrastructure according to a pool of parameters; it also simulates a set of requests of VIs, also according to a pool of parameters, with Markov-modulated inter-arrival and inter-departure times. These pools of parameters are described in Table I. The use of the physical resources over time, as well as the ability to map the VIs, allows us to calculate different indicators of the algorithm performance when using different stress approaches. In the simulations both physical substrate and VIs have 2% of the nodes as servers (rounded to the higher integer) and the remaining 8% as routing nodes. The same substrate and VI requests are used for the study of both stress approaches. When not used as independent variables, the VI request rate is λ = 2 virtual networks per time unit (Poisson arrivals), and the average duration is /µ = 2 time units (exponentially distributed duration), where µ is the average service rate. The VSs' characteristics are based in the Amazon's EC2 instance types [2]. Each scenario runs times, each with time units. The results presented have a 9% confidence interval. Router Nodes Server Nodes Links TABLE I. SIMULATION PARAMETERS Physical Networks Virtual Networks N. CPUs {2; 4; 6; 8} {; 2; 3; 4 } CPU Freq(Hz) { /.2 steps} { /. steps} Memory {2; 4; 6}(GB) {64; 28; 26; 2}(MB) N. CPUs {8; 6; 32; 64} {; 2; 4; 8; 6; 32; 64} {64; 28; {; 2; 4; 8; Storage (GB) 26} 6} Memory (GB) {26; 2; 24} {2; 4; 8; 6; 32; 64} Bandwidth (Mbps) {8; 2} { } B. Analysis of results In this section we present a set of results obtained through simulation. In the following figures, the Stress approach corresponds to equation 4 of section III, where server stress is inversely proportional to the server's free resources; the Stress approach corresponds to equation, where server stress is calculated in a non-linear way. We analyze the following performance indicators: ) Acceptance Ratio - the fraction of VIs that are successfully mapped; 2) Bandwidth Ratio - the ratio between the time-averaged sum of the VLs' bandwidth and the sum of the physical links' bandwidth; 3) STG Ratio - the ratio of the time-averaged virtual STG in use and the total STG of the physical network; 4) VR ratio - the timeaveraged number of VRs per physical router; ) VS Ratio - the time-averaged number of VSs per physical server. Most of the results are presented as a function of the number of VI requests per time unit. However, we also include one where the indicator is plotted as a function of the number of physical nodes. Accepted VI requests / Total VI requests Acceptance ratio Number of Substrate nodes Figure. Acceptance ratio of VI requests. From the presented results, it is visible that the Non- Stress approach performs better than the Stress one in most of the evaluated Accepted VI requests / Total VI requests Acceptance ratio /2/$3. 22 IEEE 83

5 parameters and scenarios. For example, in Figure, when the number of substrate nodes is (44 routing nodes and server nodes), the acceptance ratio for the Non- approach is 93%, while for the approach it is about 87%. It can be seen that the difference grows larger as the frequency of the VI requests increases, and particularly, as the number of substrate nodes increases. Virtual BW / Substrate BW Average number of VRs per substrate routing node Figure 2. Virtual bandwidth and HDD vs VI requests. Figure 3. Average number of virtual routers and virtual servers vs the number of VI requests Analyzing the trends in the graphs of Figures, 2 and 3, it is expected that, for larger numbers of substrate nodes and/or higher VI request frequencies, the performance difference will increase even more, thus making the Non- Stress approach the best approach, particularly in an environment with high number of VI requests and a large physical network. This approach considers the average characteristics of the requests of VSs, and uses a non-linear approach to model the relation between node occupation and node stress. The consideration that stress should increase more as the remaining resources approach a critical level, while not changing much when there is a lot of free resources, seems to be a successful approach for the calculation of server stress. TABLE II. Bandwidth ratio VR ratio CHARACTERISTICS OF THE TESTBED MACHINES Name CPU Freq. (GHz) CPU Cores HDD Memory (GB) RAM Amount (GB) Virtual HDD / Substrate HDD Average number of VSs per substrate server node Storage Ratio VS ratio V. TESTBED &EXPERIMENTAL RESULTS In this section we present experimental results of the proposed approach over a real testbed. A. Experiment Description The testbed is composed by 6 physical nodes and is connected according to Figure 4. Table II presents the characteristics of each physical node. Figure 4. Virtualization testbed In this experiment, we have designed a standard VI to be mapped, which is presented in Figure. Virtual node VR is restricted to be mapped either in or in, by using geographical restrictions, while VR2 is restricted to be mapped on. VS server A can be mapped either in and in. We erased all pre-existing VIs and mapped 39 of these virtual networks in sequence, and repeated this process 3 times. We present the results of node occupation as a function of the number of pre-existing VIs, both for the server and the routing nodes. B. Results and Discussion Figure 6 presents the occupation of the server machines. In each mapped VI there is a VS that can be mapped either in or in. We can see that has significantly more free STG memory than, thus expectantly making it more prone to host the VS of each mapped VI. This is confirmed by the experimental results, where we can see that the VSs tend to be hosted in, while only gets a smaller portion of them. As s free STG gets smaller, the frequency with which it hosts new VSs is also reduced. Meanwhile, and dispute one of the VRs. Figure presents the occupation of the router machines. We observe that both and get a similar number of VRs, with a regular difference in occupation, where is slightly more occupied than. This is to be expected, as has the double number of cores of, thus making more prone to accept more virtual resources. It should be noted that has a higher CPU frequency than, but it is not even near double the CPU frequency of. In this section we analyzed mapping decisions for VIs and showed that machines with more resources or with better connections for other physical nodes are able to host more resources. It was possible to also observe how network and node stress interplayed and influenced mapping /2/$3. 22 IEEE 84

6 decisions, and how physical machines with a critical level of free resources tended not to be used. Figure. Mapped VI VI. CONCLUSION AND FUTURE WORK This paper presented an algorithm for the allocation of cloud and network resources in an integrated way based on node and link stress. The algorithm assumes a NO s infrastructure where cloud resources are spread over the infrastructure, and where all resources can be virtualized. Active VSs Active VRs Active VSs vs Figure 6. Use of resources in the physical server nodes as a function of the number of mapped VIs Available Storage (GB) Active VRs vs Figure 7. Use of resources in the physical routing nodes as a function of the number of mapped Vis Two approaches for the evaluation of server stress were presented and an analysis over them was performed. Through this analysis, we concluded that a non-linear approach to calculate server stress shows better results in terms of number of VIs successfully mapped than the linear approach, which considers server stress to be inversely Available Memory (MB) Available Storage vs Available Memory vs Available Memory (MB) Available Memory vs proportional to the amount of free resources of the physical servers. By focusing on using network resources in an efficient way, we were able to increase the VI mapping. Knowing that the addressed problem is NP-hard, we plan in the future to provide the optimal solution for comparison purposes. We also plan to extend the algorithm s management objectives to include green objectives. REFERENCES [] Akamai, Can Cloud and High Performance Co-Exist?, Whitepaper, May 2. [2] Cisco, IP Next-Generation Network [Online]. Available: cisco.com/en/us/netsol/ns37/networking_solutions_solution_catego ry.html [Mar. 8, 22]. [3] Cisco, The Cisco Powered Network Cloud: An Exciting Managed Services Opportunity, 29. [4] M. Armbrust, et. al., Above the Clouds: A Berkeley View of Cloud Computing, Technical Report No. UCB/EECS-29-28, University of California, Berkeley, Feb., 29. [] Cisco, CloudVerse [Online]. Available: [Mar. 8, 22]. [6] Institut Télécom, pyocni [Online]. Available: [Mar. 8, 22]. [7] K. Bouyoucef, et. al."optimal allocation approach of virtual servers in cloud computing," Next Generation Internet (NGI), 2 6th EURO-NF Conf. on, vol., no., pp.-6, 2-4 June 2 [8] M.J. Csorba, et. al., Ant system for service deployment in private and public Clouds, Proceeding of the 2nd workshop on Bio-inspired algorithms for distributed systems, ACM, 2, p [9] I. Houidi, et. al., Adaptive virtual network provisioning, In Proceedings of the second ACM SIGCOMM workshop on Virtualized infrastructure systems and architectures (VISA '). ACM, New York, NY, USA, [] R. Yanggratoke, et. al.,"gossip-based resource allocation for green computing in large clouds," Network and Service Management (CNSM), 2 7th Int. Conf. on, vol., no., pp.-9, Oct. 2 [] J. Nogueira, et. al., Virtual network mapping into heterogeneous substrate networks, in ISCC 2, June 2. [2] J. Lischka and H. Karl, A virtual network mapping algorithm based on subgraph isomorphism detection, in VISA 9: Proceedings of the st ACM workshop on Virtualized infrastructure systems and architectures. New York, NY, USA: ACM, 29, pp [3] J. Lu and J. Turner, Efficient mapping of virtual networks onto a shared substrate, Washington University in St. Louis, Tech. Rep., 26. [4] I. Houidi, et. al., A distributed virtual network mapping algorithm, in Communications, 28. ICC 8. IEEE Int. Conf. on, , pp [] Y. Osana and S. -i. Kuribayashi, "Enhanced Fair Joint Multiple Resource Allocation Method in All-IP Networks," Advanced Information Networking and Applications Workshops (WAINA), 2 IEEE 24th Int. Conf. on, vol., no., pp.63-68, 2-23 April 2 [6] FP7 Project Scalable and Adaptive Internet Solutions (SAIL) [Online]. Available: [Mar. 8, 22]. [7] FP7 Project Generalised Architecture for Dynamic Infrastructure Services (GEYSERS) [Online]. Available: [Mar. 8, 22]. [8] J. Soares, et. al., Building Virtual Private Clouds with Networkaware Cloud, ADVCOMP 2, Lisbon, November 2. [9] Dan Pei, et. al., BGP convergence in virtual private networks, In Proceedings of the 6th ACM SIGCOMM Conf. on Internet measurement (IMC '6). ACM, New York, USA, [2] R. Monteiro, Creation and reconfiguration of virtual networks in the operator perspective, Master Thesis, Dept. Elect. Telecommunication and Informatics, University of Aveiro, Aveiro, Portugal, 2. [2] Amazon, Elastic Compute Cloud (EC2) [Online]. Available: aws.amazon.com/en/ec2/instance-types/# [Mar. 8, 22] /2/$3. 22 IEEE 8