A Novel Self-Adaptive VM Consolidation Strategy Using Dynamic Multi-Thresholds in IaaS Clouds

Transcription

1 future internet Article A Novel Self-Adaptive VM Consolidation Strategy Using Dynamic Multi-Thresholds in IaaS Clouds Lei Xie 1,2, *, Shengbo Chen 1,2, Wenfeng Shen 1,3 and Huaikou Miao 1,2 1 School Computer Engineering and Science, Shanghai University, Shanghai , China; schen@shu.edu.cn (S.C.); wfshen@shu.edu.cn (W.S.); hkmiao@shu.edu.cn (H.M.) 2 Shanghai Key Laboratory Computer Stware Testing and Evaluating, Shanghai , China 3 Shanghai Institute for Advanced Communication and Data Science, Shanghai University, Shanghai , China * Correspondence: leixie@shu.edu.cn; Tel.: Received: 11 May 2018; Accepted: 11 June 2018; Published: 13 June 2018 Abstract: With rapid development cloud computing, demand for infrastructure resources in cloud data centers has furr increased, which has already led to enormous amounts energy costs. Virtual machine (VM) consolidation as one important techniques in Infrastructure as a Service clouds (IaaS) can help resolve energy consumption by reducing number active physical machines (PMs). However, necessity considering energy-efficiency and obligation providing high quality service (QoS) to customers is a trade-f, as aggressive consolidation may lead to performance degradation. Moreover, most existing works threshold-based VM consolidation strategy are mainly focused on single CPU utilization, although resource request on different VMs are very diverse. This paper proposes a novel self-adaptive VM consolidation strategy based on dynamic multi-thresholds (DMT) for PM selection, which can be dynamically adjusted by considering future utilization on multi-dimensional resources CPU, RAM and Bandwidth. Besides, VM selection and placement algorithm VM consolidation are also improved by utilizing each multi-dimensional parameter in DMT. The experiments show that our proposed strategy has a better performance than or strategies, not only in high QoS but also in less energy consumption. In addition, advantage its reduction on number active hosts is much more obvious, especially when it is under extreme workloads. Keywords: self-adaptive VM consolidation; dynamic multi-thresholds; energy consumption; QoS; IaaS clouds 1. Introduction Infrastructure as a Service (IaaS) has been very popular in cloud computing area over past few years. Popular IaaS cloud providers, such as Rackspace and Amazon EC2, are delivering se virtual resources to customers in different data centers over Internet. However, due to huge demand for cloud computing, thousands large-scale cloud computing centers have been established, which leads to a great deal power consumption [1]. The main reason such high energy consumption is not power consumption by large quantities hardware, but inefficient usage se cloud resources. Therefore, trying to develop adaptive strategies to dynamically adjust each resource in cloud data centers is very important from an energy-efficiency perspective. One solutions for improving energy efficiency is to leverage capabilities virtualization technology [2], which improves efficiency resource utilization by sharing a physical machine(pm) among multiple virtual machines (VMs). The live migration [3] virtual machine to dynamically scheduled resource can transfer VMs from one PM to anor, while keeping services provided by Future Internet 2018, 10, 2; doi: /fi

2 Future Internet 2018, 10, corresponding VM still available. This technique makes it possible to dynamically optimize placement VMs in different purpose energy efficiency [4] or load balance [], according to resource usage at any time. The key mechanism to improve energy efficiency in cloud data centers is called VM consolidation [6], which aims at migrating VMs into a lower number PMs to increase utilization resources in cloud data centers. However, in some scenarios, packing too many VMs into one single PM may lead to poor Quality--service (QoS); since VMs on one PM always share same underlying physical resources. Therefore, VM consolidation strategies, designed to dynamically adjust VMs placement, should comprehensively ensure reliable QoS, which is ten defined via Service Level Agreements (SLAs) [7]. Besides, energy consumption [8] and VM migration costs [1] during each VM consolidation step should also be taken into consideration. Threshold-based strategy [6] uses upper and lower threshold to select proper source PM, which is ten considered as first step VM consolidation. The key point se threshold-based strategies can be concluded as comparing each PM s current status with defined threshold that is eir static or adaptive [1]. Although researches on static threshold [9] show that set 0.6 as upper threshold CPU utilization could achieve a better performance in energy consumption and VM migrations, it still cannot achieve good performance in SLA violation, due to dynamic resource usage in an IaaS environment. Different from static method, adaptive threshold strategy can leverage real-time usage different resource to dynamically adjust its threshold. Most existing studies [1,4,9,10] only consider one resource demand such as CPU utilization while ignoring or infrastructure resource requests like memory or bandwidth, etc. However, resource requirement different workloads, like computing-intensive, input and output (I/O)-intensive or hybrid-intensive, are very diverse [], as computing-intensive workloads usually call for more CPU/RAM resources, while I/O-intensive workloads prefer consuming more network bandwidth. Besides, selecting a VM only based on one resource will cause saturation in terms this resource and can lead towards no furr improvement in utilization with or types resources underutilized [11]. Based on that, we propose a novel self-adaptive VM consolidation strategy based on dynamic multi-thresholds adjustment mechanism, which is conducted by comparing predicted future requests each multi-dimensional infrastructure resources with ir current environment status. The dynamic multi-thresholds are used for PM selection, as first step in VM consolidation. In addition, an improved VM selection algorithm and a modified VM allocation algorithm are proposed, each which belongs to next two steps VM consolidation. Our contributions this work are as follows: (1) We define each parameter our proposed multi-thresholds, as well as a dynamic adjustment mechanism for se parameters. The selection each overload or underload PM (source PM) based on multi-thresholds strategy can perfectly meet different VM consolidation requests. (2) We improve VM selection algorithm MW-MVM (multi-weight VM migration) by utilizing multi-dimensional parameters our proposed multi-dynamic threshold. The selection target VM based on this algorithm can get a better performance in migration costs and energy consumption. (3) We modify VM placement algorithm MW-BF (multi-weight Best Fit) for allocation target migration VMs and new VMs requested by users on physical nodes. The experiments show that our modified algorithm can efficiently reduce SLA violation, as well as number active hosts.

3 Future Internet 2018, 10, Preliminaries 2.1. Virtual Machine Consolidation The procedure VM consolidation can be comprised three core components: (1) Source PM selection, (2) Target VM selection and (3) VM placement. In (1), a set PMs are selected for VMs to migrate Future Internet out. 2018, This10, component x FOR PEER REVIEW takes all PMs and VMs as input and selects one or more PMs as 3 17 source PM. In (2), one or more VM(s) are selected from a source PM for migration. This component takes source PM as input which has been selected by step (1) and is going to select one or more VMs from this PM to a different PM. In (3), a PM is selected to hold VM selected from (2) or new VM requests. As portrayed in Figure 1, VMs are scattered in multiple PMs before VM consolidation is applied, with PM 2 only holding one VM while PM 1 holds VMs at high overload. On one side, PMs at overload status can easily cause high SLA violation. On or side, underloaded PM 2 with only one VM in it it still needs to to remain alive, alive, which which increases unnecessary energy energy costs. costs. Therefore, Therefore, if noif VM no consolidation VM consolidation strategy strategy is interposed is interposed to to inefficient inefficient usage usage computing computing resources, resources, much extra much energy extra will energy be wasted, will be wasted, well as lower well as QoS. lower QoS. VM1 VM2 VM3 VM4 VM VM6 VM7 VM8 Before PM 1 PM 2 PM 3 Initial status Overload Underload Moderate VM consolidation Source PM Target VM Target PM PM 1 VM 3 PM 3 PM 2 VM 6 PM 3 Infrastructure CPU & RAM & Network Bandwith & etc VM1 VM2 VM4 VM VM3 VM 6 VM7 VM 8 After PM 1 PM 2 PM 3 Finial status Moderate Shut down Moderate Figure 1. Virtual machine (VM) consolidation. Based on different threshold strategies VM consolidation, status each PM can be divided Based on different threshold strategies VM consolidation, status each PM can be divided as overload, moderate or underload. If PM is defined as underload, n all VMs within it would as overload, moderate or underload. If a PM is defined as underload, n all VMs within it would be migrated out and it will n be set to sleep mode or shut down to reduce energy costs (PM 2). be migrated out and it will n be set to sleep mode or shut down to reduce energy costs (PM 2). While VMs in moderate PM will not be migrated, this PM would become one target PMs While VMs in moderate PM will not be migrated, this PM would become one target PMs for some VMs to migrate in (PM 3). Besides, in order to prevent a potential high SLA violation, an for some VMs to migrate in (PM 3). Besides, in order to prevent a potential high SLA violation, overload PM would migrate some selected VMs to ors (PM 1). Therefore, after adding VM an overload PM would migrate some selected VMs to ors (PM 1). Therefore, after adding consolidation strategy, re are only two servers alive in system an both m are at moderate VM consolidation strategy, re are only two servers alive in system an both m are at status. In addition, underload server (PM 2) is shut down, which furr decreases energy moderate status. In addition, underload server (PM 2) is shut down, which furr decreases consumption. energy consumption System Architecture 2.2. System Architecture We We consider consider that that cloud cloud data data center center in this in paper this paper contains contains m heterogeneous m heterogeneous PMs, PM PMs, = < PM PM(1), = PM(2), PM(1),, PM(2), PM(m) >.... Each, PM(m). PM consists Each PM a local consists manager, a a local Virtual manager, Machine a Virtual Monitor Machine (VMM), some virtual machines and ir physical resources, such as CPU, RAM and Network bandwidth. The VMM in every local host (Figure 2) are used to supervise ir real-time status, as different VM consolidation strategies can be operated within each VMM by contact global manager and local manager, yet without any manual adjustment.

4 Future Internet 2018, 10, Monitor (VMM), some virtual machines and ir physical resources, such as CPU, RAM and Network bandwidth. The VMM in every local host (Figure 2) are used to supervise ir real-time status, as different VM consolidation strategies can be operated within each VMM by contact global manager and local manager, yet without any manual adjustment. Future Internet 2018, 10, x FOR PEER REVIEW 4 17 request Cloud Services SaaS PaaS IaaS 1 Global manager Host Host Host n 2 3 Local Manager 4 VMM VM 1 VM 2 VM n Local Manager 4 VMM VM 1 VM 2 VM n... Local Manager 4 VMM VM 1 VM 2 VM n Physical resources Physical resources Physical resources Figure Centralized model cloud resource management. 1. User submit request Infrastructure resource on cloud, including CPU, RAM, etc. 1. User submit request Infrastructure resource on cloud, including CPU, RAM, etc. 2. The global manager collects information from local ones to maintain whole system resource 2. The global manager collects information from local ones to maintain whole system utilization. resource utilization. 3. The global manager optimizes placement VM in reference VM consolidation strategy, 3. The global manager optimizes placement VM in reference VM consolidation strategy, as well as whole system status. as well as whole system status. 4. The local managers resize VMs according to ir resource needs and decide when and which 4. The local selected managers VMs resize should VMs be migrated according out. to ir resource needs and decide when and which. VMMs selected in each VMs node should execute be migrated operation out. resizing and migrating to VMs, as well as. adjustment VMMs in each ir node power execute states. operation resizing and migrating to VMs, as well as adjustment ir power states. After repeating se steps, all nodes in this cloud data center can be optimized by different userdefined After consolidation repeating se strategies. steps, Most all nodes in idle this nodes cloud data are n center kept canswitched be optimized f, whereas by different some user-defined temporary nodes consolidation are kept strategies. in sleep mode Most(or shut idle down) nodes to are allow n kept system switched to rapidly f, whereas respond some to temporary load peaks. nodes are kept in sleep mode (or shut down) to allow system to rapidly respond to load peaks Formulations and Assumptions 2.3. Formulations and Assumptions Let us consider that a large cloud data center provides cloud resources in form virtual machine Let us instances. consider Usually, that a large re cloud are three data main center steps provides for VM cloud consolidation: resources in Source form PM selection, virtual machine instances. Usually, re are three main steps for VM consolidation: Source PM selection, Target VM selection and Target PM selection. For first step selecting Source PM, thresholdbased method is used to determine wher a PM is overloaded or underloaded. If a PM is considered Target VM selection and Target PM selection. For first step selecting Source PM, threshold-based method is used to determine wher a PM is overloaded or underloaded. If a PM is considered to be to be overloaded, which means its resource utilization is over corresponding upper threshold, overloaded, n it will be which selected means as its resource Source PM utilization and some is potential over corresponding VMs in it will be upper migrated threshold, out to n maintain it will be selected QoS; if a as PM is Source determined PM and to be some underloaded potential VMs based in on it will comparation be migrated out lower to maintain threshold, QoS; n if a PM is determined to be underloaded based on comparation lower threshold, n all VMs in all VMs in this PM will be migrated out to minimize energy consumption. this PM To formulate will be migrated problem out to (Table minimize 1), a energy physical consumption. server can be uniquely identified in form PM(j) To = formulate < VM List, P j, Status problem >, (Table where: 1), VM a physical List is server list can all be VMs uniquely on PM(j); identified P in j represents form all PM(j) = VM resource utilization List, P j, Status, where: VM ratio on PM(j) and can List is list all VMs on PM(j); P be divided into different kinds j represents all resource resources CPU, RAM and network bandwidth in form < P c j, P r j, P b j >; Status shows current workload this PM, such as overloaded, moderate or underloaded; and j unique identifier a server which arrange from 1 to n. Thus, we can denote m active servers in a cloud center as PM = < PM(1), PM(2),, PM(m) >. Similarly, VM instance is represented in form VM(i) =<

5 Future Internet 2018, 10, 2 18 utilization ratio on PM(j) and can be divided into different kinds resources CPU, RAM and network bandwidth in form Pj c, Pr j, Pb j ; Status shows current workload this PM, such as overloaded, moderate or underloaded; and j unique identifier a server which arrange from 1 to n. Thus, we can denote m active servers in a cloud center as PM = PM(1), PM(2),..., PM(m). Similarly, VM instance is represented in form VM(i) = PM(j), V j, where PM(j) shows PM where this VM belongs to, V j represents each resource utilization on it, and i is unique id a VM which arrange from 1 to m. Notations Table 1. Notations and descriptions. Descriptions VM(i) The i th VM (virtual machine), i [1, m] PM(j) The j th PM (physical machine), j [1, n] PM List list all PMs VM List list all VMs on PM(j) Pj utilization ratio on PM(j) parameter * Vi utilization ratio on VM(i) parameter * t current lower threshold on parameter * T current upper threshold parameter * Upmj allocated resource * on PM(j) C pmj total capacity resource * on PM(j) Up j v i allocated resource * on VM(i) PM(j) R p j v i request resource * on VM(i) PM(j) P j all resource utilization ratio on PM(j) W VM migrate weights all VMs on PM(j) W PM allocate weights all PMs Source PM PMs need to execute migration Select VM list selected VMs to migrate out Target PM list target PMs to migrate in * can represent CPU(c), RAM(r) or Bandwidth(b). In this paper, what we mainly focus on is to propose a novel strategy for VM consolidation, aiming at saving much energy consumption and reaching high QoS. While in process virtual machine integration, a frequent and indispensable step is VM migration, which will affect actual computing resource utilization in each PM to some extent. Thus, we made following assumptions VM migration in process our consolidation strategy: Assumption 1: The VM selected to execute migration does not cause significant resource consumption during its migration process. Assumption 2: The time for dynamic migration VMs is not counted to compare efficiency different strategies. For Assumption 1, in view numerous details in process VM migration, many scholars specialize in this area to research efficient strategy for dynamic VM migration. Yet this article does not consider too much detail it and we mainly focus on VM consolidation strategies. For Assumption 2, length migration time each VM is not used as standard for comparing advantages or disadvantages different strategies, as re is no correlation between migration time and consolidation strategies. Based on that, when we calculate cost VM migration, we mainly consider number VM migrations as an important basis to measure merits different consolidation strategies, and following section will describe more details our proposed VM consolidation strategy.

6 Future Internet 2018, 10, Novel Self-Adaptive VM Consolidation Strategy 3.1. Dynamic Multi-Thresholds (DMT) As mentioned earlier, different types applications may share one physical resources, fixed thresholds are not suitable for environments with dynamic workloads. Thus, system should adaptively adjust its behavior by considering different system workloads. Based on that, we propose a novel adjustment mechanism for threshold-based VM consolidation by comparing current threshold each defined parameter with ir future value, which is predicted by forecasting each VM s future utilization on CPU, RAM and Network bandwidth. To furr efficiently minimize consumption energy during each consolidation, two or parameters SLA and VM migration costs are taken into consideration. Thus, our proposed dynamic multi-thresholds can be represented as follows: T = T l, T u, T SLA, T m (1) The multi-thresholds T is defined by a combination threshold lower utilization T l, upper utilization T u, SLA violation ratio T SLA and number VM migration T m. Below t c, t r, t b represents lower thresholds CPU utilization ratio, RAM utilization ratio and Network bandwidth consumption, respectively, while T c, T r, T b represents upper ones. T l = t c, t r, t b, T u = T c, T r, T b (2) The adjustment dynamic multi-thresholds to timely fit whole cloud resources can be simplified into two main parameters, including T l and T u. These two thresholds can be efficiently calculated by forecasting each future utilization ratio based on Linear Regression (Section 3.1.1). While parameter T SLA can be viewed as a limitation factor dynamic threshold by deciding wher to change current threshold or not (Section 3.1.2). T m will be discussed to furr improve selection VM in Section Predict Future Utilization on Linear Regression Since nonlinear regression has larger time and computational overhead than linear regression, which affects overall system load to a certain extent, value obtained by prediction algorithm is only used as a reference for threshold adjustment. If too much computational overhead is spent on prediction while subsequent adjustment mechanism is ignored, this is not conducive to dynamic adjustment threshold. Based on above concern and experiment results, we select linear regression as our prediction algorithm and next section will give details mechanism for dynamic adjustment specific threshold. As Linear Regression [12] is a popular approach to statistically estimate relationship between different inputs and outputs, it can approximate a regression function by modeling relationship between input and output variables in a straight line, where α 1 and α 2 are regression coefficients. y = α 1 x + α 2 (3) To measure goodness a regression function is to compare predicted output variable (ŷ) with real one (y) in data point i, as ir difference ε i can be considered as magnitude residual at each n data points. ε i = y i ŷ i (4) The proposed LR algorithm approximates a prediction function based on linear regression method. The function shows linear relationship between future and current CPU

7 Future Internet 2018, 10, utilization in all hosts as follows: where α 1 and α 2 are regression coefficient parameters, while p c and p c are expected and current total CPU utilization values all hosts, respectively. p c = α 1 p c + α 2 () Here α 1 and α 2 are calculated by estimating k last CPU utilization in all hosts. In our experiment, value k is set to 1 in our simulation, because interval utilization measurements is eight minutes. The history value each resource utilization over two hours ago is significant to forecast its short-time future utilization. Moreover, value obtained by prediction algorithm is only used as a reference for threshold adjustment, and predicted value is not completely used as a future threshold value. Therefore, following adjustment mechanism can be seen as a secondary selection and adjustment predicted value. That is to say, dynamic adjustment mechanism thresholds shown in Section not only considers value predicted by linear regression obtained in this section, but also compares current overall workload system DMT Adjustment Mechanism To efficiently balance consumption energy cost and SLA violation, rules dynamically adjusting parameter t c and T c within T l and T u are restricted to SLA violation threshold T SLA. We divided problem into following two situations, where Current SLA can be calculated by statistic Interquartile range SLA violation on all hosts. Situation 1: Current SLA < T SLA Since Current SLA is below threshold SLA violation (Table 2), our purpose this situation can mainly focus on energy saving. When predicted utilization p c all hosts is greater than current threshold T c in case parameter CPU, this means that re will be many CPU requests by VMs in future. In such a case, we choose to replace T c by p c, since higher upper threshold, less VM migration will take place and more energy will be saved. At same time, when p c is below T c, keep previous value T c instead replacing. If p c is below current lower threshold T l in CPU parameter t c, this means that re will be fewer requests CPU resources in future. In such a case, we choose to replace t c by p c, since lower T l, more hosts will be switched to sleep mode to eliminate idle power consumption, and all VMs that are on low resource utilization host will be migrated to or suitable hosts. Orwise, keep t c as its previous value. Table 2. DMT adjustment when Current SLA < T SLA. Predicted Utilization (if) p c > T c p c < T c p c < t c p c > t c Threshold Adjustment (n) T c = p c T c t c = p c t c Situation 2: Current SLA > T SLA Unlike situation 1, when Current SLA is higher than T SLA, this means that situation 2 has already caused much SLA violation in current stage (Table 3). Since too much SLA violation will lead to a lower QoS, key to dealing with this problem is to reduce workload high load hosts. Based on goal dropping SLA violation, we choose to keep previous upper value T c when p c is higher than T c and replace T c p c when p c is lower than T c because higher upper threshold may easily lead to high SLA violation. On or hand, when p c is lower than t c, keeping

8 Future Internet 2018, 10, precious lower threshold will help prevent low load hosts from migrating all VMs to or hosts which will help reduce SLA violation. Similarly, when p c is higher than t c, replacing t c p c could drop SLA violation and thus help improve QoS. Table 3. DMT adjustment when Current SLA > T SLA. Predicted Utilization (if) p c > T c p c < T c p c < t c p c > t c Threshold Adjustment (n) T c T c = p c t c t c = p c For a brief description, we only take threshold CPU utilization (t c, T c ) all hosts as an example, showing how it dynamically adjusts its current thresholds by comparing its current value with predicted one. Similarly, procedure adjusting or two parameters RAM (t r, T r ) and Bandwidth (t b, T b ) can also following se steps. Many details for selecting source PM based on this DMT are shown below Source PM Selection Based on DMT To select a proper source PM, first step is to detect ir workload based on comparation each parameter in DMT (Algorithm 1). Algorithm 1 Load Detection (LD) 1 Input: PM(j) 2 Output: load_status 3 foreach PM(j) in PM List 4 if (some parameter * in PM(j) fit P j > T ) load_status = 1 6 else if (all parameter * in PM(j) fit P j < T ) 7 load_status = 2 8 else 9 load_status = 0 10 return load_status Here we take three parameters (CPU, RAM, Bandwidth) whole system to reflect its total resource utilization by each PM with different numbers virtual machines in it (Algorithm 2). For each host, if one its parameters is over upper threshold or all m are below lower threshold, it will be selected as one target physical machines in Source PM which contains migration list all target hosts (Table 1). At same time, PMs with higher SLA violation than threshold T SLA will first be selected to Source PM, and will be set to higher priority.

9 Future Internet 2018, 10, Algorithm 2 Source PM Selection 1 Input: PM List 2 Output: Source PM 3 foreach PM(j) in PM List 4 if (PM(j) SLA > T SLA ) n Source PM PM(j) 6 else 7 if (LD (PM(j)) == 1) n 8 Source PM PM(j) 9 Select VM select VMs on VM selection policy 10 if (LD (PM(j)) == 2) n 11 Source PM PM(j) 12 Select VM all VMs in PM(j) 13 set PM(j) to sleep mode or switch f 14 return selected source PMs To avoid too much energy consumption caused by frequent migration VMs and low QoS caused by short shut down during each migration, we decided to prioritize implementation order VM consolidation. That is, first migrate some VMs from overload hosts to lower load hosts, and n set all VMs on hosts with low load to migrated out and switch se hosts to sleep mode or shut down. Thus, DMT has been set to adaptively fit cloud environment source PM selection until now, while parameter T m in multi-thresholds will later be taken into consideration in Section 3.2 for helping selection target VM during VM consolidation. Once a PM is detected to be overloaded or low load by our defined dynamic multi-thresholds, next step is to select proper VMs to migrate from this host and we will furr discuss adaptive VM consolidation strategies on improvement VM selection and placement based on dynamic multi-thresholds in Sections 3.2 and The Improved MW-MVM Algorithm for VM Selection To efficiently choose proper target VMs on each selected Source PM with fewer VM migrations, [11] first proposed MVM (Minimization VM Migration) algorithm. The main idea this method is to sort CPU utilization each VM in descending order, and n select targeted VM to migrated out in two criterions. One criterion is that VM s CPU utilization should be higher than difference between upper threshold and each PM s present overall CPU utilization. Anor criterion is that, compared to values on all VMs, difference between new utilization and upper threshold on each selected VM is minimum. If re is no suitable VM that satisfied se two criteria, VM in Source PM that has highest resource utilization will be selected. Unless new utilization this source PM is under upper threshold, all above processes will not be repeated any more. The inadequacy this method is that it only takes one parameter (CPU utilization) into consideration, while different overload PMs usually have different requests on or resources. Based on that, we try to improve this method within our multi-thresholds by using different weights to efficiently allocate VMs. Below is an example showing detail our improved method.

10 Future Internet 2018, 10, Assume that current multi-thresholds are T = T l, T u, T SLA, T m, and load thresholds are T l = t c, t r, t b, T u = T c, T r, T b. The utilization ratio resource * on PM(j) can be calculated as P j = U pmj C pmj = m i=1 U p j v i C pmj (6) where * represents CPU in c, RAM in [ r, Bandwidth ] in b, respectively. Thus, resource utilization ratio on PM(j) can be expressed as P j = Pj c, Pr j, Pb j. We define formula for relative difference Q j between upper threshold and actual utilization resource * on PM(j) as { P Q j = j T, i f (Pj > T ) (7) 0, else [ ] Thus, relative different PM(j) is Q j = Q c j, Qr j, Qb j, which can n be used as weights for choosing minimum number VMs since weights showing maximize corresponding overload each resource on PM(j). The weights all VMs on PM(j) can n be calculated as W VM = Up T j v i Q j [ = Up c j v i Up r j v i Up b j v i ] Q c j Q r j = W pj v 1 W pj v 2. Q b j (8) W pj v m Here U p j v i represents allocated resource in each VM, which indicates number various resources that virtual machine has occupied and Q j T calculated by (7) shows difference each threshold, as higher value each resource means more corresponding migration requests. Thus, by multiplying U p j v i and Q j T, weight all VMs in PM(j) can be calculated. For VM(i) in PM(j), its weight is represented as W pj v i, which is used to decide selection target VM, as well as number VMs that need to be migrated out. To minimize VM migration times, all weights are sorted in descending order. As higher weight is, higher corresponding request is, lower migration times will be. The host (PM(j)) first select VM with highest weights and add it to waiting list, n it will be checked again for being overload (Algorithm 3). If it is still considered as being overloaded, VM selection policy is applied again to select anor VM to migrate from host. This will be repeated until host is considered as being not overloaded. The last step is to check migration number se selected VMs in waiting list. If migration number some selected VMs has gone over set threshold T m, it will not be added, orwise, VMs in waiting list will be added to Select VM which contains all VMs that need to be migrated.

11 Future Internet 2018, 10, Algorithm 3 Improved Multi-weights MVM policy (MW-MVM) 1 Input: Source PM, VM List 2 Output: Select VM 3 foreach PM(j) in Source PM 4 Select VM = NULL calculate W VM using formula (8) 6 order = W VM. sortdescendingorder() 7 foreach VM(i) in PM(j) 8 waitinglist VM(i) with highest value in order 9 PM(j) PM(j) VM(i) 10 update P j PM(j) 11 if (LD (PM(j)) = 0) n 12 Break 13 foreach VM(i) in waitinglist 14 if (migration_num(vm(i) < T m )) n 1 Select VM VM(i) 16 if (Select VM = NULL) 17 return Select VM Without loss generality, all overload hosts can follow se steps to select minimum number VMs. While for low load hosts, all VMs within it will be migrated out and re is no need to execute this VM selection policy before VM allocation and placement se selected VMs in waiting list to which PM is discussed in next section VM Placement Using Modified MW-BF Algorithm The VM placement is similar to a bin packing problem on variable bin sizes and prices, where bins represent physical nodes; items are VMs that had to be allocated; bin sizes are available resource capacities (CPU, RAM, Bandwidth, etc.) nodes; and prices correspond to power consumption by nodes. As bin packing problem is NP-hard, to solve it we apply a modification Best Fit (BF) algorithm by considering multi-weight VM that needs to be allocated. In our modification (MW-BF), we sort all PMs in ascending its resource utilization ratio, multiplied by weights allocating VM. This allocates each VM to a host that provides least increase power consumption and chooses most power-efficient one by leveraging heterogeneity each node. Use matrix P to represent each resource utilization ratio all PMs, defined as: P = P 1 P 2. P m = P c 1 P b 1 P r 1 P c 2 P b 2 P r 2... Pm c Pm b Pm r The product multi-weights and resource utilization each PM for allocating VM selected from PM(j) can be calculated as: W PM = P Q T j = P c 1 P b 1 P r 1 P c 2 P b 2 P r 2... Pm c Pm b Pm r Q c j Q r j Q b j (9) (10)

12 Future Internet 2018, 10, The pseudo-code for algorithm is presented in Algorithm 4. The complexity algorithm is n m, where n is number nodes and m is number VMs that have to be allocated. After se steps, waiting list Target PM n contains PMs that need to be migrated in. Algorithm 4 Modified Multi-weights Best fit (MW-BF) 1 Input: PM List, Select VM 2 Output: placement VMs 3 Select VM. sortdescendingutilization() 4 foreach VM in Select VM do calculate each W PM using Formula (10) 6 W PM. sortascendingutilization() 7 if PM has enough resource for VM n 8 Target PM PM 9 minenergy MAX 10 allocatedpm NULL 11 foreach PM in Target PM do 12 energy estimateenergy (PM, VM) in (13) 13 if energy < minenergy n 14 allocatedpm PM 1 minenergyr energy 16 if allocatedpm = NULL n 17 allocate VM to allocatedpm 18 return placement VMs The VM first selects PM with least W PM and adds it to waiting list (Target PM ), n it will be checked for being overload. To maximize utilization remaining resource on se destination PMs, all product W PM are sorted in descending order. In general, least product resource utilization and weights is, more corresponding resource it will leave, which means that VM would have enough remaining resources to use and thus it could efficiently decrease cost frequent migration VMs. 4. Experiments and Results 4.1. Experiment Setup It is essential to evaluate proposed VM consolidation strategy on a large-scale experiment environment with a real infrastructure since target system is an IaaS Cloud. Yet to conduct repeatable experiments on a real infrastructure is extremely difficult due to requirements evaluating and comparing proposed algorithm. In contrast to alternative simulation toolkits (SimGrid or GangSim), CloudSim [13] allows modeling virtualized environments, supporting on-demand resource management and configuration. Therefore, we choose CloudSim toolkit 4.0 to simulate our proposed strategy, as it is a modern simulation framework aimed at Cloud computing environments. The experiments selected 0 heterogeneous physical hosts. Each physical node is allocated with 3 to virtual machines initially. When a node is determined as overload or underload compared to our defined dynamic multi-thresholds, proposed self-adaptive VM consolidation strategy will be triggered. The CPU MIPS (millions instrument per seconds) ratings are equivalent to Amazon EC2 instance types. The users submit requests for provisioning 200 heterogeneous VMs. Each VM is randomly assigned a workload trace from one servers from workload data. Initially, VMs are allocated according to ir parameters assuming 100% utilization. The configuration is shown in Table 4.

13 Future Internet 2018, 10, Our proposed dynamic multi-thresholds strategy (DMT) is compared with four threshold adjustment strategies. The first strategy uses a static threshold (ST) in which threshold never changes during consolidation. The threshold sets to 70% for detecting an over-loaded host and 30% for detecting a low-loaded one. The next two strategies adjust utilization threshold dynamically based on statistic method by median absolute deviation (MAD) and interquartile range (IQR) while fourth method (LiRCUP) considers prediction CPU utilization on each PM, as it selects higher one to be next source PMs instead adjusting threshold [14]. Among this strategy, first three are benchmark algorithms presented in CloudSim (ST, MAD, IQR) [1] and last one (LiRCUP) is an improved single-parameter prediction algorithm. To compare performance proposed dynamic multi-thresholds strategy, algorithm on VM selection and placement above five strategies (ST, MAD, IQR, LiCUP, DMT) are same, with Minimum VM Migration algorithm (MVM) for VM selection and Best Fit algorithm (BF) for VM placement. For sixth strategy DMT (MW) in our experiment, its VM selection algorithm is MW-MWM, a modified algorithm MVM shown in Section 3.2, and its VM placement algorithm is MW-BF shown in Section 3.3. The DMT (MW) strategy is added to furr compare performance with DMT, in terms improved VM selection algorithm (MW-MWM) and placement algorithm (MW-BF). Table 4. Experiment configuration table. Parameter Unit Value Number PMs 0 Number VMs 200 CPU core performance PMs MIPS {2000, 200, 3000, 4000} Memory each PM G {1, 2, 4, 8} Bandwidth each PM Mbps {00, 800, 1000} CPU core performance VMs MIPS {1000, 100, 2000, 3000} Memory each VM G {0., 1, 2, 4} Bandwidth each VM Mbps {100, 200, 400} initial threshold T l < t c, t r, t b > < 0.3, 0.2, 0.2 > initial threshold T u < T c, T r, T b > < 0.7, 0.8, 0.8 > initial threshold T SLA 4% initial threshold T m Evaluation Metrics SLA violation In our experiment, QoS is defined via SLA violation, as ratio unallocated resources demanded by applications and total requested resources. The unallocated resource can be calculated as difference between requested resource (R p j v i ) all VMs and actual allocated resources (Upmj ) all PMs (1). Here * can be represented as CPU, RAM and Bandwidth, respectively and each parameter this metrics considers whole life time on each VM. T SLA = N j=1 t (Upmj (t) M i=1 R p j v i )dt T t Upmj (t)dt (11) N j=1 where M is number VMs in PM(j), N is number PMs. Energy consumption Our method will take CPU utilization in priority when considering energy-efficiency, compared to RAM and Bandwidth. Research on [1] shows that an idle host consumes approximately 70% power a fully utilized one and energy costs each server can be described on both

14 Future Internet 2018, 10, power cost and CPU cost, which can be calculated within a linear relationship. Based on that, we defined our power consumption metrics as follows (2), where P max is set to 20 W and c represents Future Internet 2018, 10, x FOR PEER REVIEW CPU utilization. P(c) = 0.7 P max P max c = P max ( c) (12) P(c) = 0.7 P max P max c = P max ( c) (12) Furrmore, according to each different workload, changing utilization CPU can be represented Furrmore, as a function according time: to each c(t), Therefore, different workload, E in (3) is used to changing define utilization total energy consumption CPU can be represented by a host. as a function time: c(t), Therefore, E in (3) is used to define total energy consumption by a host. T T E = P(c(t))dt (13) E = P(c(t)) dt (13) t t 4.3. Experiment Results and Analysis The simulations have been run on 10 h each workload category to determine algorithm that delivers best best energy energy consumption, consumption, SLA SLA violation violation and number and number VM migrations VM migrations over different over different workload workload types. Thetypes. optimization The optimization algorithms have algorithms been run have on each been iteration run each iteration simulation time. simulation The resultstime. are shown The results in Figure are 3shown as below: in Figure 3 as below: Energy cost (KWh) SLA violation (%) VM migrations low workload medium workload high workload low workload medium workload high workload low workload (a) (b) (c) medium workload high workload Figure Three Three aspects aspects performance performance on each onvm each consolidation VM consolidation strategy over strategy different overworkload different types. workload (a) energy types. (a) consumption. energy consumption. (b) SLA violation. (b) SLA(c) violation. number (c) number VM migrations. VM migrations. From simulation results in Figure 3, we can see that when entire system is under low From simulation results in Figure 3, we can see that when entire system is under low workload or high workload, our proposed algorithm is superior to or four algorithms in workload or high workload, our proposed algorithm is superior to or four algorithms in energy energy consumption, SLA violation rate, and VM migrations. When overall system load is consumption, SLA violation rate, and VM migrations. When overall system load is medium medium workload, consumption energy and VM migrations this algorithm are not much workload, consumption energy and VM migrations this algorithm are not much different different from that or algorithms, but rate SLA violation is obviously lower. from that or algorithms, but rate SLA violation is obviously lower. In terms energy saving, when system is under a low workload, DMT strategy improves In terms energy saving, when system is under a low workload, DMT strategy improves energy saving on about 41.9% (3.1 KWh) over MAD algorithm and improves 2.8% (4. KWh) energy saving on about 41.9% (3.1 KWh) over MAD algorithm and improves 2.8% (4. KWh) over LiRCUP when system is at a high load. For SLA violation rate, when system is at a over LiRCUP when system is at a high load. For SLA violation rate, when system is at high load, DMT algorithm and DMT (MW) algorithm both effectively reduce violation rate a high load, DMT algorithm and DMT (MW) algorithm both effectively reduce violation rate overall system in almost 2%. While for VM migration costs, strategy this article results in fewer overall system in almost 2%. While for VM migration costs, strategy this article results in VM migrations, and DMT (MW) strategy furr reduces number virtual machine fewer VM migrations, and DMT (MW) strategy furr reduces number virtual machine migrations and thus reduces migration costs. migrations and thus reduces migration costs. Figure 4 shows number active hosts at each moment (that is, number PMs that still Figure 4 shows number active hosts at each moment (that is, number PMs that service VM tasks). When VMs are consolidated by using different strategies, original hosts with still service VM tasks). When VMs are consolidated by using different strategies, original hosts low loads will migrate out all VMs y own and will be set to sleep mode or shutdown. Thus, with low loads will migrate out all VMs y own and will be set to sleep mode or shutdown. counting current number active hosts can effectively reflect effectiveness VM consolidation strategy. It can be clearly seen from Figure 4 that when whole system is under low workload (a), our proposed VM consolidation strategy can shut down a larger number hosts and can reach a steady state faster than any or strategies within less than h; when system workload is moderate (b), number shut-down hosts among se six strategies is not much different, with LiRCUP having

15 Active hosts Active hosts Active hosts Future Internet 2018, 10, Thus, counting current number active hosts can effectively reflect effectiveness VM consolidation strategy. It can be clearly seen from Figure 4 that when whole system is under low workload (a), our proposed VM consolidation strategy can shut down a larger number hosts and can reach a steady state faster than any or strategies within less than h; when system workload is moderate (b), Future Internet 2018, 10, x FOR PEER REVIEW number shut-down hosts among se six strategies is not much different, with LiRCUP having lowest lowest number number active active hosts; hosts; when when overall overall system system is is at at high high load load (c), (c), all all se se algorithms algorithms can can only only shut shut down down a few a few hosts. hosts. Although Although our our proposed proposed algorithm algorithm cannot cannot reach reach a stable stable state state quickly quickly in in this this situation, situation, it it still still has has least least number number active active hosts, hosts, which which effectively effectively reduces reduces energy energy consumption consumption system. system. Therefore, Therefore, our our proposed proposed VM VM consolidation consolidation strategy strategy based based on DMT on has DMT more has obvious more advantages obvious advantages in case high in case workload high and workload low workload, and low workload, especially when especially it is when combined it is combined with MW-MVM with MWand MVM MW-BF and algorithm MW-BF algorithm on MW-DMT. on MW-DMT. This result This is result also is consistent also consistent with with conclusion conclusion in in Figure Figure 1, 1, which which furr furr demonstrates demonstrates superiority superiority our our proposed proposed VM VM consolidation consolidation strategy strategy based based on on dynamic dynamic multi-thresholds. multi-thresholds low workload Time (h) ST MAD IQR LiRCUP DMT DMT (MW) medium workload high workload Time (h) ST MAD IQR LiRCUP DMT DMT (MW) (a) (b) (c) Time (h) ST MAD IQR LiRCUP DMT DMT (MW) Figure 4. The number active hosts when adapting each consolidation strategy during 10 h. (a) when Figure 4. The number active hosts when adapting each consolidation strategy during 10 h. (a) when under low workload. (b) when under medium workload. (c) when under high workload under low workload. (b) when under medium workload. (c) when under high workload. Related Work. Related Work Virtual machine consolidation [6] is one techniques in a cloud resource management Virtual machine consolidation [6] is one techniques in a cloud resource management system system to increase energy-efficiency IaaS Cloud, since failure existing PMs or addition to increase energy-efficiency IaaS Cloud, since failure existing PMs or addition new new PMs are continuous events happening in cloud data centers. Nathuji and Schwan [16] first PMs are continuous events happening in cloud data centers. Nathuji and Schwan [16] first proposed proposed an architecture energy management system for virtualized data centers where resource an architecture energy management system for virtualized data centers where resource management management is divided into local and global policies. Srikantaiah et al. [17] n proposed a heuristic is divided into local and global policies. Srikantaiah et al. [17] n proposed a heuristic method to method to handle optimization over multiple resources, as y considered VM consolidation handle optimization over multiple resources, as y considered VM consolidation as a bin as a bin packing problem. Cardosa et al. [18] have leveraged min, max and shares parameters packing problem. Cardosa et al. [18] have leveraged min, max and shares parameters Xen s Xen s VMM to represent minimum, maximum and proportion CPU allocated to VMs sharing VMM to represent minimum, maximum and proportion CPU allocated to VMs sharing same resource. same resource. Threshold-based VM consolidation strategy use upper and lower threshold values to identify a Threshold-based VM consolidation strategy use upper and lower threshold values to identify PM as overloaded or underloaded, respectively. Secron [19] considers an upper threshold to prevent a PM as overloaded or underloaded, respectively. Secron [19] considers an upper threshold to PM from reaching 100% utilization with performance degradation. To select proper PMs for prevent PM from reaching 100% utilization with performance degradation. To select proper consolidation, Feller et al. [10] propose a static CPU threshold to detect under-loaded and over-loaded PMs for consolidation, Feller et al. [10] propose a static CPU threshold to detect under-loaded and PMs. However, although setting static thresholds is simple and appealing, it is not efficient for an over-loaded PMs. However, although setting static thresholds is simple and appealing, it is not efficient environment with dynamic workloads, since different types application may be running on one for an environment with dynamic workloads, since different types application may be running on single PM. Therefore, threshold values should be adjusted to each workload type and level, which will help optimize performance VM consolidation efficiently. To dynamically consolidate virtual machines in procedure selecting source PM, Beloglazov [1] proposed adaptive upper and lower thresholds based on statistical analysis historical data while Farahnakian [14] propose a regression-based prediction model to forecast resource utilization both PMs and VMs. Recent research on adaptive threshold-based dynamic VM

16 Future Internet 2018, 10, one single PM. Therefore, threshold values should be adjusted to each workload type and level, which will help optimize performance VM consolidation efficiently. To dynamically consolidate virtual machines in procedure selecting source PM, Beloglazov [1] proposed adaptive upper and lower thresholds based on statistical analysis historical data while Farahnakian [14] propose a regression-based prediction model to forecast resource utilization both PMs and VMs. Recent research on adaptive threshold-based dynamic VM consolidation algorithms [20,21] is using a similar solution, in which threshold value adapts with change different resource utilization. However, above methods only consider single-resource utilization infrastructure, yet do not consider or corresponding sources. To try to consider or multiple resources, Lu Liu [22] researched load balance each application among different virtual machines, but he did not consider placement each VM. The research Xu Jing [23] is consistent with ours; y use multi-resource to optimize placement virtual machine but in order to realize goals performance on power and temperature, y do not focus on study thresholds as it gives only two sets thresholds each parameter, but each threshold is set in static value. In general, although above researches all consider multi-resources, y do not focus on optimization threshold-based VM consolidation in step for PM selection. While in second step VM consolidation VM selection, different algorithms are proposed. VMs that have highest correlation resource utilization with or VMs are selected, which is called MC algorithm [4]. Anor algorithm called MVM proposed in [24] shows that, it first sorts VMs in descending order with respect to CPU demand and n selects VM that satisfies two author-defined criterions. HPG algorithm was also discussed by same author, that is selecting VM with lowest ratio actual resource usage to its initial claimed resource demand. The VM which requires minimum time to complete migration is selected for migration called MMT in [1], while migration time is estimated as amount RAM utilized by a VM divided by spare network bandwidth available for PM. However, apart from MMT, rest VM selection algorithms only consider CPU demand VMs but ignore memory or network bandwidth requirement VMs. As argued by [11], selecting a VM only based on CPU will cause saturation in terms CPU and can lead towards no furr improvement in utilization while leaving or types resources underutilized. It is highly challenging to determine a single converging point representing equivalent total resource demand a multitude resource types, while different types resources represent different dimensions [9]. 6. Conclusions and Future Work In this paper, we present a novel self-adaptive VM consolidation strategy based on dynamic multi-thresholds (DMT) in IaaS Clouds. First, we define each parameter multi-thresholds, including CPU, RAM, Bandwidth, SLA, VM migrations, as well as a dynamic adjustment mechanism for se parameters threshold. After that, we improve VM selection algorithm (MW-MVM) based on this dynamic multi-threshold, which fully utilizes multi-dimensional parameters our proposed dynamic threshold by predicting ir future values. To furr reduce cost VM migration and improve efficiency VM consolidation, a modified VM placement algorithm is n discussed (MW-BF). Finally, experiments proposed strategy are performed on CloudSim, compared with three benchmark strategies and one prediction strategy. The simulation results show that our proposed dynamic multi-thresholds strategy (DMT and DMT (MW)) for VM migration has a good performance, not only in lower SLA violation, but also in less energy cost and VM migrations. Particularly, se advantages our proposed VM consolidation strategy are much superior than or strategies when whole cloud system is at high or low workload, as well as reduction in number active hosts. Furrmore, since experimental objects in this paper are only 0 heterogeneous servers, number PMs in real cloud system is ten as high as tens thousands. Thus, effect energy-efficiency and QoS for VM consideration using our proposed strategy will be much more

17 Future Internet 2018, 10, obvious, which will help furr decrease carbon dioxide and overall energy consumption in IaaS clouds. In future, we plan to consider or different parameters in multi-thresholds strategy and will furr explore impact weights each parameter. At same time, we will also focus on modification or VM selection and placement algorithms to furr improve migration-efficiency and energy-efficiency VM consolidation in IaaS clouds. Author Contributions: L.X. and S.C. conceived and designed experiments; L.X. performed experiments; W.S. analyzed data and provided technical support; H.M. contributed analysis tools and revised this paper; L.X. wrote paper. Funding: This work is partially supported by National Natural Science Foundation China (NSFC) under grant No , The National Key Research and Development Program China (No. 2017YFB ), Shanghai Innovation Action Plan Project under grant No , Shanghai University Material Genetic Engineering Institute (No. 14DZ ), and Japan Society for Promotion Science under Grants-In-Aid for Scientific Research 1H Conflicts Interest: The authors declare no conflict interest. References 1. Beloglazov, A.; Buyya, R. Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation virtual machines in cloud data centers. Concurr. Comput. Pract. Exp. 2012, 24, [CrossRef] 2. Barham, P.; Dragovic, B.; Fraser, K.; Hand, S.; Harris, T.; Ho, A.; Neugebauer, R.; Pratt, I.; Warfirld, A. Xen and Art Virtualization. Proc. SOSP 2003, 37, Clark, C.; Fraser, K.; Hand, S. Live migration virtual machines. In Proceedings Symposium on Networked Systems Design and Implementation, Berkeley, CA, USA, 2 4 May 200; DBLP: Trier, Germany, 200; pp Beloglazov, A.; Abawajy, J.; Buyya, R. Energy-aware resource allocation heuristics for efficient management data centers for Cloud computing. Futur. Gener. Comput. Syst. 2012, 28, [CrossRef]. Xu, M.; Tian, W.; Buyya, R. A survey on load balancing algorithms for virtual machines placement in cloud computing. Concurr. Comput. Pract. Exp. 2016, 29. [CrossRef] 6. Khan, M.A.; Paplinski, A.; Khan, A.M.; Murshed, M.; Buyya, R. Dynamic Virtual Machine Consolidation Algorithms for Energy-Efficient Cloud Resource Management: A Review. In Sustainable Cloud and Energy Services; Springer: Berlin, Germany, 2018; pp Wu, L.; Garg, S.K.; Buyya, R. Service Level Agreement (SLA) Based SaaS Cloud Management System. In Proceedings International Conference on Parallel and Distributed Systems, Melbourne, VIC, Australia, December 201; IEEE Computer Society: Washington, DC, USA, 201; pp Arroba, P.; Moya, J.M.; Ayala, J.L.; Buyya, R. Dynamic Voltage and Frequency Scaling-aware dynamic consolidation virtual machines for energy efficient cloud data centers. Concurr. Comput. Pract. Exp [CrossRef] 9. Beloglazov, A.; Buyya, R. Adaptive threshold-based approach for energy-efficient consolidation virtual machines in cloud data centers. In Proceedings 8th International Workshop on Middleware for Grids, Clouds and e-science, Bangalore, India, 29 November 3 December 2010; ACM: New York, NY, USA, 2010; pp Feller, E.; Rilling, L.; Morin, C. Snooze: A scalable and autonomic virtual machine management framework for private clouds. In Proceedings th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), Ottawa, ON, Canada, May 2012; pp Ferdaus, M.H.; Murshed, M.; Calheiros, R.N.; Buyya, R. Virtual Machine Consolidation in Cloud Data Centers Using ACO Metaheuristic. In Proceedings Euro-Par 2014 Parallel Processing, Porto, Portugal, August 2014; Springer International Publishing: Berlin, Germany, 2014; pp. 162a 167a. 12. Olive, D.J. Linear Regression Analysis. Technometrics 2003, 4, [CrossRef] 13. Calheiros, R.N.; Ranjan, R.; De Rose, C.A.F.; Ruyya, R. CloudSim: A Novel Framework for Modeling and Simulation Cloud Computing Infrastructures and Services. arxiv 2009, arxiv:

18 Future Internet 2018, 10, Farahnakian, F.; Liljeberg, P.; Plosila, J. LiRCUP: Linear Regression Based CPU Usage Prediction Algorithm for live migration virtual machines in data centers. In Proceedings th EUROMICRO Conference on Stware Engineering and Advanced Applications (SEAA), Santander, Spain, 4 6 September 2013; pp Kusic, D.; Kephart, J.O.; Hanson, J.E.; Kandasamy, N.; Jiang, G. Power and performance management virtualized computing environments via lookahead control. Clust. Comput. 2009, 12, 1 1. [CrossRef] 16. Nathuji, R.; Schwan, K. Virtualpower: Coordinated power management in virtualized enterprise systems. In Proceedings ACM SIGOPS Operating Systems Review, Stevenson, WA, USA, October 2007; Volume 41, pp Srikantaiah, S.; Kansal, A.; Zhao, F. Energy aware consolidation for cloud computing. In Proceedings Conference on Power Aware Computing and Systems, San Diego, CA, USA, December 2010; USENIX Association: Berkeley, CA, USA, 2010; p Cardosa, M.; Korupolu, M.; Singh, A. Shares and utilities based power consolidation in virtualized server environments. In Proceedings 11th IFIP/IEEE Integrated Network Management (IM 2009), Long Island, NY, USA, 1 June Murtazaev, A.; Oh, S. Sercon: Server Consolidation Algorithm using Live Migration Virtual Machines for Green Computing. IETE Tech. Rev. 2011, 28, [CrossRef] 20. Farahnakian, F.; Bahsoon, R.; Liljeberg, P.; Pahikkala, T. (Eds.) Self-Adaptive Resource Management System in IaaS Clouds. In Proceedings 2016 IEEE 9th International Conference on Cloud Computing (CLOUD), San Francisco, CA, USA, 27 June 2 July Deng, D.; He, K.; Chen, Y. (Eds.) Dynamic virtual machine consolidation for improving energy efficiency in cloud data centers. In Proceedings th International Conference on Cloud Computing and Intelligence Systems (CCIS), Beijing, China, August 2016; pp Lu, L.; Zhang, H.; Smirni, E.; Jiang, G.; Yoshijira, K. Predictive VM consolidation on multiple resources: Beyond load balancing. In Proceedings International Symposium on Quality Service, Montreal, QC, Canada, 3 4 June 2013; pp Xu, J. A multi-objective approach to virtual machine management in datacenters. In Proceedings International Conference on Autonomic Computing, Karlsruhe, Germany, June 2011; pp Voorsluys, W.; Broberg, J.; Venugopal, S.; Buyya, R. Cost Virtual Machine Live Migration in Clouds: A Performance Evaluation. Lect. Notes Comput. Sci. 2012, 931, by authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under terms and conditions Creative Commons Attribution (CC BY) license (