VMbuddies: Coordinating Live Migration of Multi-Tier Applications in Cloud Environments

Transcription

1 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS VMbuddies: Coordiatig Live Migratio of Multi-Tier Applicatios i Cloud Eviromets Haiku Liu, ad Bigsheg He Abstract Eabled by virtualizatio techologies, various multi-tier applicatios (such as Web applicatios) are hosted by virtual machies (VMs) i cloud data ceters. Live migratio of multi-tier applicatios across geographically distributed data ceters is importat for load maagemet, power savig, routie server maiteace ad quality-of-service. Differet from a sigle-vm migratio, VMs i a multi-tier applicatio are closely correlated, which results i a correlated VM migratios problem. Curret live migratio algorithms for sigle-vm cause sigificat applicatio performace degradatio because itermediate data exchage betwee differet VMs suffers relatively low badwidth ad high latecy across distributed data ceters. I this paper, we desig ad implemet a coordiatio system called VMbuddies for correlated VM migratios i the cloud. Particularly, we propose a adaptive etwork badwidth allocatio algorithm to miimize the migratio cost i terms of migratio completio time, etwork traffic ad migratio dowtime. Experimets usig a public bechmark show that VMbuddies sigificatly reduces the performace degradatio ad migratio cost of multi-tier applicatios. Idex Terms Cloud, Live migratio, Multi-tier applicatio, Virtual machie INTRODUCTION I teret applicatios have bee prosperous i the era of cloud computig, which are usually hosted i virtual machies i geographically distributed data ceters. Live migratio of Iteret applicatios across data ceters is importat for differet scearios icludig load maagemet, power savig, routie server maiteace ad quality-of-service [8][][38][42]. Additioally, Iteret applicatios ted to have dyamically varyig workloads that cotai log-term variatios such as time-of-day effects i differet regios. It is desirable to move the iteractive/web applicatio to the data ceter that has better etwork performace to users for lower respose time [4]. Also, workloads ca be migrated across differet data ceters to exploit time-varyig electricity pricig [32]. The recet advace of VM live migratio techiques [][3] is able to relocate a sigle VM across data ceters with acceptable migratio cost [34][38]. Typical Iteret applicatios employ a multi-tier architecture, with each tier providig certai fuctioality. Specific to multi-tier applicatios, we eed to migrate several tightly-coupled VMs i multi-tiers, istead of a sigle VM. Previous studies have demostrated the potetial performace pealty of multitier applicatios durig migratio [7][23]. I this paper, we ivestigate whether ad how we ca reduce the migratio cost without sufferig applicatio performace degradatio. A typical multi-tier web applicatio cosists of three tiers: presetatio layer (Web tier), busiess logic layer (App tier) ad data access layer (DB tier) [8]. Differet layers usually ru o differet VMs ad have differet memory H. Liu is with North Chia Electricity Power Uiversity, Baodig, 73, Chia, ad Nayag Techological Uiversity, , Sigapore. haikuliu@gmail.com. B. He is with Nayag Techological Uiversity, , Sigapore, he.bigsheg@gmail.com access patters. VMs are correlated because oly whe all VMs of the multi-tiers are migrated to aother data ceter, they ca completely ad efficietly serve requests i that data ceter. We call this problem correlated VM migratios. Correlated VM migratios ca cause sigificat performace pealty to multi-tier applicatios. Cosider the followig sceario: if the middle tier is first migrated, the the other two tiers must redirect the commuicatio ad data access traffic to aother data ceter ad wait for the processig results to be set back. Moreover, because the multi-tier applicatio ad migratio processes share the same lik for data trasferrig, give the data-itesive ature of multi-tier applicatios ad limited etwork badwidth betwee two data ceters, etwork badwidth cotetio may cause sigificat performace degradatio for both applicatios ad VM migratios. While live migratio of VMs provides the ability to relocate ruig VMs from oe physical host to aother without perceivable service dowtime [][3], the state-ofthe-art VM migratio techiques maily target a sigle VM (either withi a data ceter [][6][24][3] or across differet cloud data ceters [8][38][42]). These techiques caot fudametally solve the correlated VM migratios problem. We eed effective ad efficiet mechaisms to coordiate correlated VM migratios across distributed data ceters. Ideally, the coordiatio system should avoid splittig multiple tiers betwee data ceters so that the performace pealty of VM migratios o the multi-tier applicatio is miimized. Meawhile, we should dimiish the VM migratio cost i terms of migratio completio time, etwork traffic ad migratio dowtime. To achieve these goals, we eed to address the followig techical challeges. First, migratio completio time of differet VMs may vary sigificatly due to VM cofiguratios ad workload characteristics i differet tiers. It is challegig to orches- Mauscript received December 7, 23. xxxx-xxxx/x/$xx. 2x IEEE Published by the IEEE Computer Society

2 2 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS trate their migratio progress ad complete the migratios simultaeously. Secod, etwork badwidth is the critical resource of VM migratio across data ceters. So far there is little work o progress maagemet for correlated VM migratios so that the total migratio cost is miimized. The last challege is how to implemet a coordiatio system i existig virtualizatio platforms such as Xe ad demostrate its practical value. To address the aforemetioed challeges, we propose a coordiatio system called VMbuddies to solve the correlated VM migratios problem for multi-tier applicatios. VMbuddies embraces a sychroizatio protocol to esure that all VMs complete their migratios simultaeously, avoidig itermediate data exchage across distributed data ceters. I order to reduce the migratio cost, we develop etwork badwidth allocatio algorithms accordig to memory access patters of differet workloads. For VMs with stable memory access patter, we propose a aalytical cost model of pre-copyig algorithm that is widely used for live VM migratios i commo virtualizatio platforms such as VMware, Xe, KVM ad Hyper-V. Based o the cost model, we formulate the badwidth allocatio as a distributed costrait optimizatio problem ad give a optimal solutio so that the total migratio cost is miimized. We further develop a adaptive badwidth allocatio algorithm for VMs with dyamic memory access patters. With the sychroizatio protocol ad badwidth allocatio mechaisms, we desig ad implemet a prototype called VMbuddies i Xe. We ivestigate the effectiveess ad efficiecy of VMbuddies by usig a public multi-tier applicatio bechmark RUBBoS []. The experimets are coducted with two complemetary approaches: real private data ceters ad simulated Wide Area Network (WAN). The evaluatio compares a umber of metrics (such as migratio completio time, etwork traffic, migratio dowtime ad applicatio performace degradatio) with several alterative approaches. The results show VMbuddies ca sigificatly reduce the performace degradatio of multitier applicatios durig migratios, while oly experiecig slight overhead of sychroizatio cost i terms of migratio completio time ad etwork traffic. Particularly, i compariso with Xe, VMbuddies achieves lower average respose time by 77% ad higher throughput by 8%, ad reduces total migratio cost by 36% o average. The rest of this paper is orgaized as follows. Sectio 2 gives a brief itroductio to our groudwork ad formulates the correlated VM migratio problem. Sectio 3 presets the sychroizatio protocol ad badwidth allocatio algorithms. Sectio 4 describes the implemetatio details of VMbuddies. We preset the evaluatio methodologies ad experimetal results i Sectio 5. We review the related work i Sectio 6 ad coclude i Sectio 7. 2 CORRELATED VM MIGRATION PROBLEM t t t 2 Memory image roud 2 3 source host A trasmittig dirty pages target host B Memory image roud 2 3 This sectio itroduces the backgroud of sigle-vm migratio techiques ad correlated VM migratios. Sigle-VM migratio. Live migratio of VMs has bee a effective approach to maage workloads i a odisruptive maer. Amog differet implemetatios of VM live migratio, pre-copyig algorithm [][3] is the most popular ad widely used i today s virtualizatio platforms. Thus, this paper focuses o pre-copyig algorithm i Xe, ad the methodology i our paper ca be exteded to other VM migratio techiques. The basic workflow of pre-copyig algorithm i Xe is described as follows. As show i Fig., memory precopyig is coducted i several iterative rouds. The VM s physical memory is first trasferred from host A to host B, while the source VM cotiues ruig i host A. Pages dirtied i each roud must be iteratively re-set i the ext roud to esure memory cosistecy. That is, the data to be trasmitted i each roud are dirty pages geerated i the previous roud. After several rouds of precopyig, a stop-ad-copy phase is performed to trasmit the remaiig dirty pages while the source VM temporarily stops executio. Whe the fial data trasferrig is doe, the VM o host B resumes ad takes over the VM o host A. There are a umber of factors affectig the migratio cost i terms of migratio dowtime, migratio completio time ad total etwork traffic. The major factors iclude the size of VM memory, memory dirtyig rate, etwork trasmissio rate ad cofiguratio of migratio algorithm (e.g., coditios for startig the stop-ad-copy phase). Amog these factors, the size of VM memory ad the memory dirtyig rate are mostly determied by the VM ad workloads. That meas, they usually caot be cotrolled by migratio algorithms. Thus, this paper focuses o the optimizatios o the latter two factors: etwork trasmissio rate ad cofiguratio of migratio algorithms. We ote that CPU resource show moderate performace impact o VM migratios. Previous study demostrated that the migratio process oly takes aroud 3% of oe CPU to attai the maximum etwork throughput over the gigabit lik [3]. Based o this observatio, we thik etwork badwidth is usually more importat tha CPU resource, especially for VM migratio i WAN eviromets. Moreover, this paper focuses o the algorithms of etwork badwidth allocatio for VM migratio, so we assume the CPU resource is always sufficiet for migratio processes. Correlated VM migratios. The objective of this paper is to solve the correlated VM migratios problem raised i multi-tier applicatios. As illustrated i Fig. 2, a multitier web applicatio is migrated from DC to DC 2. Each tier is ruig o multiple VMs, ad thus the VMs across differet tiers are correlated with data depedecy. Network badwidth is a critical resource across distributed data ceters. It is usually much smaller tha the etwork badwidth withi a data ceter. Previous studies (e.g., t stopad-copy resume Fig.. Live migratio algorithm performs pre-copyig i iterative rouds.

3 LIU ET AL.: VMBUDDIES: COORDINATING LIVE MIGRATION OF MULTI-TIER APPLICATIONS IN CLOUD ENVIRONMENTS 3 cliet request respose Performace degradatio? Web tier Applicatio tier Database tier [38]) assumed that the etwork badwidth betwee two data ceter was 465 Mbps. Without loss of geerality, we assume that the peak etwork badwidth betwee DC ad DC 2 reserved for the migratio processes is B. As discussed i Itroductio, the applicatio traffic ad migratio traffic share the same liks betwee two data ceters. The badwidth cotetio betwee them may result i sigificat performace degradatio i both applicatio ad VM migratio. That motivates us to develop VMbuddies to coordiate the VM migratio of multi-tier applicatios, ad let the iter-cloud etwork badwidth to be exclusively used by the migratio traffic. 3 SYSTEM DESIGN I this sectio, we first preset the sychroizatio protocol for coordiatig live migratio of VMs i a multi-tier applicatio. Next, we preset badwidth allocatio algorithms to reduce the migratio cost of all VMs. Particularly, we first cosider the sceario where the memory dirtyig rate is stable i the etire VM migratio. We start with a cost model i estimatig the migratio completio time of all VMs, ad develop a optimal badwidth allocatio algorithm to miimize the migratio completio time. With this optimal algorithm as a buildig block, we idetify the stable phases i memory dirtyig rate, ad develop a adaptive algorithm for badwidth allocatio i VMbuddies. I the adaptive algorithm, the badwidth allocatio for each stable phase ca use the optimal algorithm developed i the sceario for stable memory dirtyig rates. 3. Sychroizatio Protocol DC DC 2 Commuicatio traffic Migratio traffic Commuicatio traffic Fig.2. Performace pealty due to live migratio of a multi-tier web applicatio across distributed data ceters. To avoid iter-data ceter etwork traffic betwee correlated VMs, we develop a sychroizatio protocol to orchestrate all VMs to proceed the stop-ad-copy phase at the same time. As show i Fig. 3, each VM migratio may reach its stop-ad-copy phase at differet poits of time (called pseudo-sychropoit). The pseudo-sychropoit depeds o the termiatio coditios of pre-copyig algorithm. I our sychroizatio protocol, we postpoe the stop-ad-copy phase util all VMs reach the stop-ad-copy phase (called sychropoit). However, all VMs are still ruig durig the sychroizatio, ad the dirtied memory pages still eed to be trasmitted to the destiatios. We call this phase wait-ad-copy. The badwidth cosumed i this phase is determied by the memory dirtyig rates of the Web VM App VM DB VM Arbitrator reach_pseudosychropoit pseudo-sychropoit sychropoit VMs. Algorithm shows the pseudo code of the sychroizatio protocol for correlated VM migratios. The sychroizatio protocol relies o a arbitrator for cotrol. The arbitrator implemets a message-passig mechaism for cotrollig the VM migratios. Whe a VM reaches the pseudo-sychropoit, it should immediately sed a message reach_pseudo-sychropoit to the arbitrator, ad the proceed the wait-ad-copy phase util it receives the start_stop-ad-copy message from the arbitrator. The arbitrator uses a variable p to record the umber of VMs that have reached the pseudo-sychropoit. Oce all VMs have reached the sychropoit, the arbitrator broadcasts a message start_sotp-ad-copy to all VMs. To hadle the potetial migratio failures, we adopt a simple approach for fault tolerace. We view the coordiated migratio processes as a trasactio i a batch. I case of failures, all correlated VM should resume at their origial host, abortig the migratio. More advaced fail-tolerat VM migratio techiques will be studied i our future work. The sychroizatio protocol allows differet resource allocatios ad VM schedulig mechaisms. Without a effective arbitratio of resource allocatio ad schedulig, the sychroizatio may cause sigificat migratio cost 2 4 start_stop -ad-copy 3 wait-ad-copy Fig.3. Sychroizatio of live migratio of multi-tier applicatios. Algorithm : Sychroizatio protocol for correlated VMs live migratio. let m be the umber of VMs to be migrated 2. let p /*the umber of VMs that reach the pseudosychropoit */ 3. Begi Migrate (VM[i]) 4. while VM[i] does ot reach the pseudo-sychropoit do 5. pre-copy the memory of VM[i]; 6. edwhile 7. sed message reach_pseudo-sychropoit to arbitrator; 8. proceed wait-ad-copy phase; 9. if receive message start_stop-ad-copy the. proceed stop-ad-copy phase;. edif 2. Ed Migrate 3. Begi Arbitrator( ) 4. if receive message reach_pseudo-sychropoit form VM[i] the 5. p p+; 6. if p == m the /*All VMs reach the sychropoit */ 7. for i= to m do 8. sed message start_stop-ad-copy to VM[i]; 9. edfor 2. edif 2. edif 22. Ed Arbitrator

4 4 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS T ad performace degradatio. To address this challegig problem, we ivestigate differet badwidth allocatio algorithms to miimize migratio cost of correlated VMs. 3.2 A Naive Badwidth Allocatio The sychroizatio protocol guaratees that all the correlated VM migratios start the stop-ad-copy phase at the same time. A aive badwidth allocatio method is schedulig the VM migratios oe by oe. Each VM migratio gais the majority of the etwork badwidth for the best migratio performace. As show i Fig. 4, a VM starts live migratio oly whe aother VM has reached the pseudo-sychropoit. However, whe the VM reaches the pseudo-sychropoit, it should proceed the waitad-copy phase to postpoe the stop-ad-copy phase. This stage still cosumes a portio of badwidth (determied by its memory dirtyig rate) to re-sed the dirty pages. Clearly, the order of VM migratios affects the badwidth allocatio. To reduce the sychroizatio cost, a ituitive schedulig is to start VM migratios i the ascedig order of their memory dirtyig rates. The VM with the largest memory dirtyig rate should be migrated last. Thus, we call this schedulig algorithm Largest memory Load VM Last (LLL). LLL is simple but caot effectively use the etwork badwidth. The VMs i the wait-ad-copy phase still cosume etwork badwidth, causig excessive etwork traffic betwee data ceters. This overhead is exaggerated whe the umber of correlated VMs icreases, ad for the memory-itesive workloads like multi-tier applicatios. To address the overhead caused i the wait-ad-copy phase, we propose a parallel VM migratio scheduler ad adaptive badwidth allocatio algorithms for all the VM migratios, as described i the ext sub-sectios. 3.3 Optimal Badwidth Allocatio for Static Workloads T 2 Stop-ad-copy Fig.4. LLL schedulig of multiple VM migratios. Ulike LLL, VMbuddies performs VM migratios i multiple tiers simulataeously, ad smartly decide the badwidth amog VMs. A primary goal of VMbuddies is to determie how much etwork badwidth should be allocated to each migratio process to miimize total migratio cost of multi-tier applicatios. We first cosider a sceario that the memory access patter of each VM is stable (i.e., memory dirty rate is stable ad ca be modeled as a costat), ad the exted this algorithm to dyamic workloads Modelig Static Workloads Migratio Our model o static workload migratio uses a sigle VM as the buildig block. I the followig, we preset a aalytical model of pre-copyig algorithm to estimate the migratio cost of a sigle VM. Table summarizes the T 3 TABLE PARAMETERS FOR VM LIVE MIGRATION MODELING Symbol Descriptio parameters ad otatios used throughout this paper. Assume the pre-copyig algorithm proceeds i ( N) rouds. Let v i ( i ) deote the data volume trasmitted at each pre-copyig roud, ad t i ( i ) deote the elapsed time at each roud. v equals the VM memory size M. t represets the time cosumed for trasferrig the VM memory image ad t i ( i ) is the time of trasferrig the dirty pages geerated durig previous rouds. The data trasmitted i roud i is calculated i Eq. (). M, if i ; vi () d ti, otherwise. The elapsed time at roud i is calculated i Eq. (2). dti ti. (2) r We ca derive the expressios of v ad t step by step as follows. ()v = M, t = ; (2) v = d t =, t = = ; (3) v = d t =, t = = M r M V T r d B Vthd N W Size of VM memory image. Total etwork traffic durig migratio. Migratio completio time. Network trasmissio rate durig migratio. Memory dirtyig rate durig migratio. Maximum etwork badwidth reserved for VMs migratio traffic. Threshold of the remaiig dirty memory that should be trasferred durig the stop-ad-copy phase. The pre-defied maximum umber of rouds for iterative pre-copyig i migratio algorithm. The Writable Workig Set (WWS) of applicatios. i ; ad fially we get M d vi d * ti, M d t i i. (3) i r r The the total etwork traffic durig the migratio ca be summed up i Eq. (4). i d V vi M. (4) i i i r Cosequetly, the migratio completio time ca be summed up i Eq. (5). i M d T ti. (5) i i r i r The migratio completio time is the duratio that has egative effect o applicatio performace. It is a key performace metric of migratio cost, especially for multi-tier applicatios migratig i cloud eviromets. To evaluate the covergece rate of VM migratio algorithm, we calculate the total umber of rouds by the iequality v V. It is the coditio to termiate the iterative pre-copyig ad to start the stop-ad-copy phase. Furthermore, it should ot be larger tha the pre-defied parameter N. As a result, the umber of pre-copyig iteratios becomes: Vthd d. (6) mi log log, N i

5 LIU ET AL.: VMBUDDIES: COORDINATING LIVE MIGRATION OF MULTI-TIER APPLICATIONS IN CLOUD ENVIRONMENTS 5 For a give VM, M ad V thd (determied by migratio algorithms) ca be viewed as costats. Cosequetly, the iterative pre-copyig would coverge faster if d/r is smaller. We therefore defie d/r as the covergece coefficiet of VM live migratio. The above aalysis assumes that the memory dirtyig rate is smaller tha the memory trasmissio rate o average. However, i practice, there are still scearios that the memory dirtied at a higher speed tha the etwork trasfer. Oe observatio is that, the memory dirtied i each roud should ever exceed the applicatios writable workig set (WWS) []. I this case, the pre-copyig termiates oce the umber of iteratios exceeds N or the etwork traffic exceeds the VM memory size by a factor of (by default, the factor is 3 i Xe migratio algorithm). Fially, i case where d/r >, the total etwork traffic ad migratio completio time should be estimated by Eq. (7) ad Eq. (8), respectively. V mi{ M N * W, M W}, (7) T mi{ M N * W, M W}/ r, (8) where αw is the amout of dirty memory to be trasferred i each roud of pre-copyig, ad ( < < ) is a coefficiet of proportioality that ca be leart by liear regressio techique [26]. M + N αw deotes the total memory trasferred i maximum umber of rouds, ad βm + W deotes the total memory trasferred if migratio traffic exceeds times of the VM memory size. As the average memory dirtyig rate d ad W are both determied by the applicatios ad ca be measured i a log time widow before VM migratios, we oly eed to determie the etwork trasmissio rate (i.e., parameter r), which ca be referred to Eq. (5) or Eq. (8). With the cost model of a sigle VM migratio, we formulate the problem of correlated VM migratios i cloud eviromets. Assume the multi-tier applicatio is comprised of m VMs. We should orchestrate the VM migratios to miimize the maximum migratio completio time of all the VMs by allocatig appropriate etwork badwidth to each migratio process. As the etwork badwidth betwee geographically distributed data ceters is shared ad limited resource, this issue ca be abstracted as a distributed costrai optimizatio problem (DCOP), as show i Eqs. (9). Note, we istatiate the parameter T, V, M, d, r for VM k to be T k, V k, M k, d k, ad r k, respectively. mi{max{ T, T2,, Tm}} i Mk dk Tk f ( rk ) ti. (9) i i rk i rk m s. t., r k k B s. t., dk rk B I case where d r, T = f(r ) i Eqs. (9) should be replaced by Eq. (8) Problem Solvig DCOP is origially a problem i which a group of agets must distributedly choose values for a set of variables so that the cost of a set of costraits over the variables is either miimized or maximized. It is kow that DCOPs are NP-hard problems [44]. There are o polyomial-time algorithms to solve this NP-hard problem. However, observig that the polyomial i Eqs. (9) is a mootoic fuctio of r, we ca simplify the problem by Theorem. Theorem. Whe a set of VMs are migrated cocurretly, if there exists a optimal badwidth allocatio to miimize the maximum migratio completio time of the VMs, the the badwidth resource should be fully used, i.e., r = B, ad all VMs have equivalet migratio completio time, i.e., i, j, T = T. Proof (sketch). Assume live migratio of each VM is performed by a migratio daemo cocurretly. Give a arbitrary badwidth allocatio for these processes with costrait r = B, the migratio completio time of each VM ca be figured out by T = f(r ), which is a mootoically decreasig fuctio that T always decreases as r icreases i the ope iterval (, B). I case there is a differece of migratio completio time betwee two arbitrary VMs, we deprive the migratio daemo with a shorter completio time some badwidth ad re-allocate it to the migratio daemo with a loger completio time. By iteratively usig the stepby-step approximatio method to adjust badwidth resource for each migratio process, all VMs should have the same migratio completio time fially. Isight: Based o Theorem, the sychroizatio cost betwee differet VMs is avoided as all VMs fiish their migratio at the same time. Our objective mi(max(t, T 2, T m )) ca be trasformed to a problem of solvig multivariate polyomial equatios over a fiite field as Eqs.(). T T2 Tm i Mk dk Tk f ( rk ) ti. () i i rk i rk m.., k k.., k k s t r B s t d r B I case where d r, the T = f(r ) i Eqs. () should be replaced by Eq. (8). We ote that if the above equatios ca be solved, the objective mi ( V ) is also satisfied because the badwidth resource is fully utilized ad the migratio completio time is miimized. However, Eqs. () are still hard to solve because T = f(r ) is a polyomial with high order. Fortuately, as T = f(r ) is a mootoically decreasig fuctio i the field (, B), there exists oly oe solutio to satisfy the above equatios. As r = B is a multivariate liear equatio, if we search the solutio by chagig the multi-variate r, there should be umberless combiatios that eed to verify equatios T = T = = T. I cotrast, observig that T = f(r ) cotais oly oe variate ad their curves are mootoic ad similar, we desig a work backward strategy to solve Eqs. (). That is, we tetatively search the solutio set of equatios T = T = = T to satisfy r = B. We propose a quasi-secat method [2] to approximate the Eq. r = B iteratively, as show i Fig. 5 ad Algorithm 2. The solutio cosists of three steps. First, we figure out the miimum of f(r ) i the field (, B], deoted by S = mi{f(r ), f(r ),, f(r )}. Secod, we use horizotal lies T i = S + i ( i > ) to sweep the curves T = f(r ),

6 6 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS T T =f(r ) T 2 =f(r 2 ) T 3 =f(r 3 ) Workload P P2 P3 P4 P WW2 W3W4 W5 W T * T 3 =S+ T 2 =S+ T =S+ T =S+ r * r 3 32 r 3 r 3 r Fig. 5. A quasi-secat method to solve the multivariate polyomial equatios. VM- VM-2 VM (a) O-Off workload VM- VM-2 VM (b) Irregular workload Fig. 6. Badwidth allocatio accordig to workload patters: (a) model o-off workloads i several stable phases of page dirtyig rate; (b) model irregular workloads i fix-sized time widows through average page dirtyig rate. Algorithm 2: Optimal badwidth allocatio for static workloads () S mi(f(r ), f(r ),, f(r )), i, λ, λ, is a radom umber bigger tha, so λ = ; (2) Let f(r ) S + λ, ( m), solve these equatios ad fid out a solutio set R = (r, r,, r ); (3) Verify whether the solutio satisfy equatio r = B by calculatig the distace d = r + r + + r B, if d > δ, the i i +, λ λ + λ, where λ = d λ (d d ), (i > ), go to (2); otherwise, Ri is the target solutio ad the searchig termiates. (4) Allocate the badwidth r for live migratio of VM-k. where λ is the step size to icrease the horizotal lie T i. Let r deotes the root of f(r ) = S + λ i the iteratio i. We use Newto's method [3] to approximate the root of these equatios oe by oe ad get a solutio set, for example R = (r, r, r ). Third, we check whether the solutio R satisfies r = B by calculatig the error d = r B. The iteratios will be termiated whe the error d δ, where δ is a very small costat. The problem is how to determie the step size λ to speed up the roots searchig. We illustrate the iterative approximatig process by a sketch diagram i Fig. 5. Take T = f(r ) as a example, the lie T ad T sweepig it gets two poits (r, T ) ad ( r, T ), respectively. Suppose the fial solutio of T = f(r ) is r, the lie passig through these two cosecutive poits itersectig the vertical lie r = r determies λ. The the ext root r ca be solved. Without lose of geerality, λ ca be deduced from the proportioal relatioship of three cosecultive roots, as show i Eq. (): di ( i i ) i i, () di di we defie λ = λ λ, ad the we have: dii i. (2) di di By iteratively chagig the step size i Algorithm 2, it should approximate the target solutio fially. As the covergece order of quasi-secat method is the golde ratio (.68), Algorithm 2 coverges fast i practice. Actually, there are C = ( ) equatios i Eqs. (). However, step (2) i Algorithm 2 oly eeds to solve m equatios i each roud of iteratio. The computatio cost is liearly proportioal to the umber of VMs to be migrated. I this way, Algorithm 2 simplifies the problem complexity of root fidig. I our experimets, the proposed approach is able to solve more tha high orders equatios withi oe secod, which is sufficiet for typical multiple tier applicatios i practice. I costrast, a geeral solve method provided by MATLab caot give out the solutio i 2 hours. 3.4 Adaptive Badwidth Allocatio for Dyamic Workloads The model ad algorithm discussed i the precedig Subsectio assume that the memory dirtyig rates are steady for all VMs. However, i practice, multi-tier applicatios ca be service-orieted ad dyamic (e.g., due to the daily or seasoal chages). The memory access patter of differet applicatios may vary i respose to the load chage of the service requests. We desig our badwidth allocatio algorithm adaptig to the temporal chages of page dirtyig rates. Particularly, we rely o the predictio o the page dirtyig rate i a short-term period (amely widow). We idetify the relatively stable periods (amely phases), ad view the badwidth allocatio problem for each phase as a istace of optimal badwidth allocatio problem i the sceario of stable page dirtyig rate. That meas, we ca leverage the model ad the optimal algorithm i the previous subsectio to achieve local optimum i each phase. Prior to startig VM migratios, we first measure the page dirtyig rate of each VM durig a log-term ru. We the model memory access patter of each VM based o the profilig rus. The modellig methodology is orthogoal to this paper. There are a umber of existig learig methods ad models [3][39], which have demostrated high accuracy ad efficiecy. For modelable workload such as o-off memory access patters, as show i Fig.6 (a), we formulate the page dirtyig rate ito phases [39]. We first divide the workload curves ito several small splies evely, ad the combie two successive splies iteratively if their differece is smaller tha a pre-defied threshold. We recogize a sigificat pulse as a ew phase if its successive splie has slight variatio. As show i Fig. 6(a), the migratio daemo orchestrates live migratio of correlated VMs usig the optimal badwidth allocatio algorithm i each stable phase. The migratio daemos track the pages dirtied ad aalyze the phase chage accordig to the historical

7 LIU ET AL.: VMBUDDIES: COORDINATING LIVE MIGRATION OF MULTI-TIER APPLICATIONS IN CLOUD ENVIRONMENTS 7 VM VM... VM VM migratio daemo migratio optimizatios sychroizati o Dom protocol dirty memory tracker data trasmit rate cotroller dom VMM statistics. Oce a sigificat chage of page dirtyig rate is detected, they immediately report to the arbitrator for badwidth re-allocatio i respose to the phase chages. The remaiig data to be trasferred also should be reported to the arbitrator for decisio makig. For simplicity, the badwidth allocatio takes effect at the ew roud of VM migratio. At last, the sychroizatio protocol described i Subsectio 3. is used to achieve the ultimate sychroizatio of correlated VM migratios. We also eed to hadle the scearios where some workloads may ot show distict phase chages, as show i Fig.6 (b). We ca oly measure the average memory dirtyig rate i each time widow, ad thus the adaptive badwidth allocatios is sesitive to the sizes of time widow. A smaller widow meas higher accuracy of memory dirtyig rate predictio, but causes higher overhead due to the computatio of badwidth reallocatio. We eed to carefully determie the size of widows to make tradeoff betwee predicatio accuracy ad computatio overhead. We experimetally study its impact i the evaluatio (Subsectio 5.5). I this sceario, the arbitrator periodically performs badwidth reallocatio i the ed of each time widow, so the migratio daemos oly eed to sum up the pages to be trasferred i ext time widow, icludig the remaiig pages i curret roud of pre-copyig ad ew pages dirtied i curret widow. The arbitrator agai leverages the optimal algorithm described i the previous subsectio to allocate badwidth to each migratio deamo. 4 IMPLEMENTATION I this sectio, we preset the details of desig ad implemetatio o VMbuddies. We desig ad implemet VMbuddies based o Xe 4. platform. We cojecture that the techiques ad optimizatios proposed i this paper are also applicable for other virtualizatio platforms. Our implemetatio carefully cosiders efficiecy ad trasparecy of correlated VM migratios across data ceters. 4. System Architecture Other VMs Fig.7. Block diagram of system architecture. VM migratio schedulig VM workload modelig migratio cost predictio badwidth allocator Arbitrator Fig. 7 shows the system architecture of VMbuddies. The system is composed of two major modules, amely migratio daemo ad arbitrator. The migratio daemo i domai cosists of several compoets such as dirty memory tracker, data trasmissio rate cotroller, sychroizatio protocol, ad some migratio optimizatios. k Rouds of iterative pre-copyig The arbitrator cosists of VM memory iformatio collector, migratio cost predictio model, badwidth allocator, ad VM migratio schedulig compoets. Whe a multitier applicatio eeds to be migrated, the arbitrator eeds to coduct the followig actios for badwidth allocatios: ) activates the dirty memory tracker i each migratio daemo, 2) collects the page dirtyig iformatio of each VM, 3) performs workload predictios ad calculates the memory dirtyig rate, 4) estimates the migratio cost, ad 5) fially allocates the etwork badwidth to each migratio process. These actios are implemeted with messagepassig mechaisms amog the arbitrator ad other compoets i VMbuddies. Next, the VMs are migrated at the speed of the allocated badwidth i multiple phases, accordig to Sectio 3.4. I the followig, we first preset two migratio optimizatios ad the describe some other implemetatio details. 4.2 Optimizatios of VM Live Migratio Although there is a lot of existig migratio optimizatios for cost reductio, we choose the most effective ad simple oes for our sceario: multi-tier applicatios migratio i WAN eviromets. Particularly, we highlight two techiques -- ballooig to evict uused pages from VMs memory image, ad a itelliget pre-copyig termiatio to adjust the umber of iteratios i the pre-copyig algorithm. Ballooig. It is desirable to flush o-essetial data out of memory, ad to trasfer oly a smaller workig set durig migratio. VMbuddies leverages ballooig mechaism [6][36] to reduce memory volume trasferred durig VMs live migratio. We first profile a VM s memory usage to detect its memory footprit, ad the the size of free memory ca be determied. The balloo driver the requests the size of free memory from the guest OS kerel ad returs it to the uderlyig hypervisor. I this way, ballooig mechaism deflates a VM s memory size by evictig free pages from the VM s memory image. This is extremely useful for VMs with large memory size ad small applicatio workig set. Itelliget pre-copyig termiatio. Multi-tier applicatios ca have very differet memory ad computatio characteristics. Istead of usig static coditios o precopyig termiatios as modeled i Sectio 3, VMbuddies termiates the pre-copyig phase itelligetly whe the pre-copyig iteratios caot reduce the migratio dowtime. Curret Xe termiates the iterative precopyig till either () the remaiig dirty memory becomes smaller tha 5 pages; or (2) the umber of iteratios exceeds 3 times; or (3) the etwork traffic exceeds Number of dirty pages 6k 5k 4k 3k 2k App-VM Number of dirty pages 8k 6k 4k 2k k DB-VM 8k Rouds of iterative pre-copyig Fig. 8. Dirty pages geerated i each roud of pre-copyig.

8 8 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS Web tier App tier DB tier VM6 Group Group 2 VM VM3 VM7 VM4 the VM memory size by a factor of 3. However, our experimets o multi-tier applicatios idicate that the first coditio is seldom satisfied, ad the migratio algorithm ofte performs may uecessary iteratios. Fig. 8 shows the umber of dirty pages geerated i the each roud of iterative pre-copyig whe we migrate a 3-tier web bechmark RUBBoS over a lik with 4Mbps badwidth. The migratios of applicatio-tier ad DB-tier termiate the precopyig phase accordig to coditios (2) ad (3), respectively. However, oly the first several rouds of precopyig results i reductio of memory remaiig to be set i the ext roud. This idicates that the migratio algorithm should itelligetly determie the time to stop pre-copyig to avoid uecessary iteratios. VMbuddies leverages a heuristic to termiate iterative pre-copyig. We observe that memory dirties approximately as fast as it ca be trasferred after a umber of iteratios. To detect this poit, we moitor the umber of pages set ad dirtied i each roud util the differece betwee them becomes very small. At this time, the migratio daemo should immediately otify the arbitrator that the VM reaches the pseudo-sychropoit. This simple optimizatio ca sigificatly reduce the migratio completio time ad etwork traffic, with up to 45% reductio of migratio cost o average i our experimets. 4.3 VM Groupig VM8 VM2 VM5 VM9 (a) Duplicated VMs i each tier ad bidirectioal commuicatio betwee two tiers VM6 Group Group 3 VM VM2 Group 2 VM3 VM7 VM4 VM8 VM5 VM9 (b) VMs with topology of directed acyclic graph Fig.9. Groupig VMs of differet tiers to lower the parallelism degree of VM migratios i the cloud. Today s eterprise multi-tier applicatios are usually comprised of multiple VMs i each tier. Fig. 9(a) shows a web applicatio with 2/3/4 cofiguratio (i.e., 2, 3 ad 4 VMs for Web tier, Applicatio tier ad database tier, respectively). If these VMs are migrated cocurretly, the performace of VM migratios would be sigificatly degraded due to etwork badwidth cotetio. To address this scalability problem, we propose VM groupig mechaism to lower the parallelism degree of multiple VM migratios. The other cosideratio is the layout of VMs. The topology of a multi-tier applicatio (such as tree or directed acyclic graph) usually affects the decisio makig of groupig ad orderig correlated VM migratios. Overall, we cosider the followig two scearios. First, we cosider a sceario where multiple VMs i each tier have the same fuctio, ad thus the multiple replicas are used for load balacig ad fault tolerace, as show i Fig. 9(a). Actually, if we keep at least oe VM of each tier i both source data ceter ad destiatio data ceter, the the service is available at both sides after a recofiguratio of coectio maagemet. A feasible groupig scheme is show i Fig. 9(a). If the VMs i Group are the first to be migrated cocurretly, the the VMs i Group 2 must be migrated i the ed. This guaratees that at least oe miimal orgaism ca serve the requests at both sides without iter-dataceter commuicatio. The other VMs excluded from these two groups ca be migrated oe by oe to fully use the badwidth resource. Secod, we cosider a sceario where the applicatios show complicated topologies such as tree or directed acyclic graph (DAG), ad multiple VMs have differet fuctios, as show i Fig. 9(b). I this case, we eed to miimize the iter-dataceter etwork traffic. We model the applicatio s depedecies as a graph, where VMs are vertices ad data flows as edges. The problem ca be formulated as a uiform graph partitioig problem. We eed to fid the miimum cuts of the graph so that the cut sets have the smallest umber of edges (uweighted case) or smallest sum of weights. We choose the most commoly used Fiduccia-Mattheyses algorithm [5] to solve this multilevel graph partitioig problem due to its high efficiecy. For example, a 3-way partitioig ca split the DAG i Fig. 9(b) ito three subgraphs. We ote that such processig ca be doe prior to VM migratios so that o computatio overhead is imposed o the migratio daemos ad applicatios. 4.4 Other Implemetatio Details There are still may other implemetatio details about VMbuddies, icludig memory dirtyig rate measuremet, etwork badwidth cotroller, disk migratio, ad etwork coectios maitace. We provide these details i the Appedix of the supplemetary file. 5 EVALUATIONS I this sectio, we evaluate applicatio performace degradatio ad VM migratios cost i terms of the followig metrics: () Service level degradatio: the respose time ad throughput of multi-tier applicatios durig migratio. (2) Migratio completio time: the elapsed time from startig the first VM migratio to the time that all VMs fiish migratio. (3) Network traffic: the total data volume trasmitted durig the VM migratios. (4) Migratio dowtime: it is the duratio that the services of multi-tier applicatios are etirely uavailable. 5. Experimetal Setup Testbeds: We evaluate VMbuddies i both real WAN ad i-lab testbeds. We deploy our prototype i data ceters at Nayag Techological Uiversity (NTU) ad Natioal Uiversity of Sigapore (NUS). Our measuremet has showed a stable throughput of 4 Mbps ad a roud trip latecy of 2ms betwee the two sites coected via VPN. At both sites, we have HP Z4 workstatios equipped with Itel Xeo W GHz 6 processors, 2GB DDR3 memory, oe 5GB 72rpm SATA hard disk, ad Gbit Etheret iterface. The host machies are with 64-bit Fedora 6 Liux distributio ad the hypervisor is Xe 4.

9 LIU ET AL.: VMBUDDIES: COORDINATING LIVE MIGRATION OF MULTI-TIER APPLICATIONS IN CLOUD ENVIRONMENTS 9 Dirty pages (4KB/page) with Liux 3. kerel. The hosts are allocated with 2 CPU ad 4 GB memory to isure that the resource for migratio daemos is sufficiet. The VMs ru i paravirtualizatio mode, with the OSes havig the same distributio of Liux as the hosts. By default, all VMs are cofigured to use CPU ad GB RAM. This allows us to easily simulate the overloaded sceario i the bechmark. Because the real WAN testbed caot cover all test cases due to the limited badwidth, we also simulate a WAN testbed i our lab by usig the Liux traffic shapig iterface, which ca freely cotrol the badwidth, latecy ad package loss experieced i WAN eviromets. Workload: We use a commo multi-tier web bechmark RUBBoS (Rice Uiversity Bulleti Board System) [] to ivestigate the performace of VMbuddies. RUBBoS simulates a olie ews forum like slashdot.org. It is a typical ope-sourced bechmark that is widely used i academic commuities. We ca freely modify the workload geerator ad report more statistical iformatio as we eed. We deployed the bechmark i a 3-tier architecture usig Apache 2.2, Tomcat 7., ad MySQL 5.5. We deploy the bechmark with VMs i m//o cofiguratio for the three tiers, however, we oly eed to focus o the results of // cofiguratio. This is because the result is represetative for a geeral m//o cofiguratio through VM groupig. I our experimets, each tier rus withi a sigle VM. To geerate workload i the servers, some other servers are used to ru cliet workload geerators at our lab. We simulate three typical workload patters [3]: Static ad Light Workload (S&L): 2 cocurret users access the website i a stable arrivig rate, as show i Fig. (a). Dyamic ad Heavy Workload (D&H): 5 ad cocurret users alterately access the website for 3 secods so as to simulate a cyclical o-off workload patter, as show i Fig.(b). k 8k 6k 4k 2k (a) Static ad light workload Elapsed time ( secod) Dirty pages (4KB/page) 8k 5k 2k 9k 6k 3k Web-VM App-VM DB-VM Dirty pages (4KB/page) Light workload with Symmetric Waves patter 6k 5k 4k 3k 2k k (c) Light workload with symmetric waves Web-VM App-VM DB-VM (b) Dyamic ad heavy workload Web-VM App-VM DB-VM Elapsed time ( secod) Elapsed time( secod) Fig.. The umber of dirty pages observed for three types of RUBBoS workload. miute widows. I each widow, we first icrease the cocurret users from to 35 evely, ad the we decrease the users iversely. We repeat such widows to simulate a wave workload i which the memory dirtyig rate chages slowly over time, as show i Fig. (c). The workload shows almost equivalet average memory dirtyig rate compared to the S&L. This patter simulates the workload fluctuatios i real world (e.g., due to daily chages). Methodology: We evaluate the migratio performace of RUBBoS by comparig the followig approaches: No Sychroizatio (NS): The multiple VMs are migrated sequetially usig default Xe migratio algorithm without sychroizatio. This is the baselie approach. Migratio Optimizatios (MO): The multiple VMs are migrated oe by oe without sychroizatio, usig optimized Xe migratio algorithm (Sectio 4.2). Largest memory Load VM Last schedulig (LLL): the simple schedulig of VM migratios based o sychroizatio protocol (Sectio 3.2). Optimal Badwidth Allocatio (OBA) i Sectio 3.3 (L&SW): This workload patter cosists of four- TABLE 2 MEASUREMENT OF WRITABLE WORKING SET Workloads Writable Workig Set (MB) Web-VM App-VM DB-VM S&L D&H L&SW ad 3.4. LLL+MO, OBA+MO: the multiple VMs are migrated usig MO combiig with LLL ad OBA algorithm, respectively. OBA+MO is the algorithm implemeted i VMbuddies. We choose NS as the baselie because it has o sychroizatio costs ad the migratio performace is the best so that we ca clearly study sychroizatio overheads caused by LLL ad OBA. Whe the VMs are migrated i WAN testbeds, we first replicate the VMs disk images to the destiatio before the memory migratio begis. The results reported below are the average of the three trials (except failed migratios) ad refer to the cost of memory migratio ad disk sychroizatio durig VM live migratios. 5.2 Memory Dirty Rate ad WWS Measuremet The shadow model of Xe allows us to track the umber of dirty pages withi ay time widow. We coducted a experimet to measure the memory dirtyig rate of each tier whe RUBBoS rus D&H, S&L ad L&SW workloads. For each VM, we read the dirty bitmap ad the clea it every oe secod. Fig. plots the umber of dirty pages geerated by three workloads i differet time widows. The x-axis shows elapsed time ad the y-axis measures the umber of pages dirtied withi oe secod. The memory dirtyig rate of each tier retais at relative low level of fluctuatio for S&L workload. I cotrast, the fluctuatio is sigificat i terms of phase chages as the

10 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS Average respose time (secods) MO Resp. time MO Throughput stop-ad-copy 6 Web-VM App-VM 4 pre-copyig pre-copyig before 2 migratio Elapsed time (3 secods) load chages for D&H workload. Moreover, the phase chages well alig with each tier whe workload chages. This result justifies our desig o characterizig the memory access patter of each VM by observig average memory dirtyig rates periodically. Also, VMs i differet tiers demostrate differet memory dirty rates, idicatig differet requiremets of etwork badwidth durig migratio. To measure the VMs Writable Workig Set (WWS), we first clea the dirty bitmap ad the sample it every oe secod periodically. Note that the dirty bitmap was ever cleaed so that the sampled umber of each peek was the accumulatio of dirty pages from the begiig of the test. If the page umbers i two cosecutive peek are equal, this idicates that the writable workig set becomes stable. Table 2 also shows the writable workig set of differet tiers of S&L, D&H ad L&SW workloads i four miutes. Note that WWS deotes active memory pages i a VM so it is usually smaller tha the VM s workig set. The size of writable workig set implies the opportuity of etwork traffic reductio by usig memory ballooig. 5.3 Service Level Degradatio VMbuddies Resp. time VMbuddies Throughput DB-VM pre-copyig We first cosider a hot spot mitigatio sceario where the RUBBoS applicatio with D&H workload must be migrated from a overloaded data ceter at NTU to aother at NUS. We place the applicatio to light-load servers ad quadrupled each VM s CPU ad mai memory allocatio oce the VMs resume at the ew data ceter. after migratio 3 Fig.. The performace of RUBBoS applicatio usig D&H workload whe it is migrated i a real WAN testbed. Migratio completio time (secs) Usychroized time Sychroized time NS MO LLL LLL+MO OBA OBA+MO Fig.2. The sychroized ad usychroized time measured whe the D&H workload is migrated i a real WAN testbed. Average throughput (request/secs) Fig. shows how the performace of RUBBoS web site is affected whe the VMs were migrated. For MO (without sychroizatio), durig the pre-copyig phase of web-tier, the VM is still ruig at the source data ceter, ad a slight performace degradatio is observed. However, oce the Web-VM is moved to the destiatio data ceter ad App-VM migratio begis, the average respose time sigificatly icreases to 2.2 secods compared to the prior.49 secod. This degradatio aggravates durig the migratio of DB-VM. The average respose time raises 4.8X to 2.85 secods, ad the average throughput drops from 55 req/sec to 26 req/sec, early 8% throughput loss. The reaso behid this observatio is that the co-located service is decoupled i two data ceters durig the migratios, ad the migratio process also coteds agaist the applicatio for etwork badwidth. I cotrast, VMbuddies guaratee the correlated VMs always stay i the same data ceter, o matter before migratio or after migratio. The applicatio oly suffers 3% icrease of respose time ad % loss of throughput due to VM migratio overhead. VMbuddies reduces the average respose time by 77% ad improve 8% throughput compared to MO. Whe the VM migratios complete, the average respose time drops to.3 secod ad the average throughput raises to 75 req/sec i both MO ad VMbuddies because the capacity of the VMs icreases by a factor of 4. As VMbuddies is desiged to mitigate applicatio performace degradatio caused by splittig VMs betwee data ceters, it s etetial to measure the sychroized ad usychroized periods durig VM migratios. As show i Fig. 2, although sequetial migratio approaches icludig NS ad MO show relatively shorter migratio completio time, they suffer log-term usychrioized periods whe App-VM ad Web-VM are migrated. The applicatio performace is sigificatly degraded durig this period, as show i Fig.. I cotrast, LLL, OBA ad their optimizatios ca guaratee the applicatio performace through sychroizatio protocol durig VM migratios, at the cost of reasoable icrease of migratio completio times. Compared to NS, VMbuddies (OBA+MO) reduces the migratio completio time from 478 secods to 88 secods, a 4% reductio of time that the applictio suffer from service level degradatio durig VM migratios. 5.4 Migratio Cost As a result of the performace pealty durig the migra- Migratio completio time (secs) NS LLL OBA MO LLL+MO OBA+MO DH SL LSW (a) 4 Mbit/sec Badwidth Migratio completio time (secs) NS LLL OBA MO LLL+MO OBA+MO DH SL LSW (b) 4 Mbit/sec Badwidth Fig.3. The migratio completio time of D&H, S&L ad L&SW workloads over etworks with 4 ad 4 Mbit/sec badwidth.

11 LIU ET AL.: VMBUDDIES: COORDINATING LIVE MIGRATION OF MULTI-TIER APPLICATIONS IN CLOUD ENVIRONMENTS Network traffic (GB) NS LLL OBA MO LLL+MO OBA+MO DH SL L&SW (a) 4 Mbit/sec Badwidth Network traffic (GB) NS LLL OBA MO LLL+MO OBA+MO DH SL LSW (b) 4 Mbit/sec Badwidth Fig.4. Network traffic of migratig D&H, S&L ad L&SW workloads over etworks with 4 ad 4 Mbit/sec badwidth. Migratio dowtime (secs) NS LLL OBA MO LLL+MO OBA+MO DH SL L&SW (a) 4 Mbit/sec Badwidth migratio dowtime (secs) NS LLL OBA MO LLL+MO OBA+MO DH SL LSW (b) 4 Mbit/sec Badwidth Fig.5. Total migratio dowtime of migratig D&H, S&L ad L&SW workloads over etworks with 4 ad 4 Mbit/sec badwidth. tio, it is importat to miimize the migratio completio time i order to reduce this period of lower performace. Fig. 3 shows the migratio completio times of D&H ad S&L workloads usig differet migratio approaches i real WAN ad i-lab testbeds. The maximum available etwork throughput i the i-lab testbed is 4 Mbit/sec, represetig a high-speed etwork. Fig. 4 shows the correspodig etwork traffic geerated durig the migratios ad Fig. 5 shows the total migratio dowtime. We have the followig observatios: () OBA causes less sychroizatio cost tha LLL. Specifically for D&H workload o high-speed etwork, OBA ca reduce 8% total migratio completio time ad 6% etwork traffic compared to LLL. It early avoid applicatio performace degradatio durig migratio while oly experiecig % icrease of migratio completio time compared to NS; (2) By usig MO strategies, all migratio approaches show sigificat reductio of migratio completio time ad etwork traffic. The reductio is more sigificat for memory itesive workloads or low-speed etwork eviromets. For D&H workload o the real WAN testbed, it ca reduce the migratio cost by 47%. I summary, VMbuddies reduces the migratio cost by 36% o average compared to NS. (3) For the same migratio approach, S&L workload shows much less migratio cost tha D&H workload i both high-speed ad low-speed etworks. The reaso is that higher memory dirtyig rate i D&H results i much more data to be set i each roud of pre-copyig; (4) Although the average memory dirtyig rate of L&SW workload is almost equal to that of S&L, L&SW shows a little higher migratio costs ad higher stadard deviatios tha S&L due to errors of workload patter predictios ad overhead of frequet badwidth re-allocatio. (5) LLL algorithm always causes higher migratio cost tha OBA algorithms as it causes higher sychroizatio cost for the Web-VM ad App- VM migratios, as show i Table 3; (6) The lower speed etwork implies more etwork traffic, as show i Fig. 3. As a result, the reductio of migratio completio time is ot liearly proportioal to the icrease of etwork badwidth; (7) All migratio approaches show slight differece of total migratio dowtime betwee each other. Although VMbuddies leads to less tha 4% icreases of total migratio dowtime compared to NS, it avoids the sigificat applicatio performace degradatio durig live migratio of correlated VMs. We further examie the log of live migratio i NS ad VMbuddies. I NS, the live migratio of DB-VM always geerates more tha 3 GB etwork traffic for D&H workloads whether it is migrated over 4 Mbps or 4 Mbps etworks. The reaso is that the migratio process always termiates the pre-copyig phase oly whe the etwork traffic already exceeds 3 times of its memory size. Similarly, the migratio process of App-VM termiates the precopyig always after 3 rouds, but most of iteratios are uecessary, simply causig more etwork traffic ad migratio time. I VMbuddies, all VMs accelerate the progress of live migratio through our itelligetly termiatig the iterative pre-copyig. For S&L workload migrated over the etwork with 4 Mbps badwidth, we fid that NS works well because the memory dirtied i these VMs relatively slowly. I this case, the ballooig mechaism takes effect o the migratio cost reductio because the VM shows smaller workig set whe it experieces light memory load. 5.5 Algorithms Accuracy ad Rutime Overhead We study the workload modelig errors ad algorithm overhead by comparig L&SW with S&L, as show i Fig.6. We use S&L as a baselie ad illustrate the ormalized migratio cost of L&SW, both are measured i the real WAN eviromet usig OBA + MO approach. L&SW show reasoable higher migratio cost tha S&L due to rutime overhead ad model errors. Whe the time widow of memory dirtyig rate predictio icreases, the frequecy of badwidth re-allocatio decreases ad thus the migratio completio time is reduced. The etwork traffic is ot sesitive to the chages of time widow because live migratio of VMs usually last log i WANs ad positive/egative predicatio errors couteract each other. However, migratio dowtime show larger stadard deviatios whe the time widow icreases. This is because the stop-ad-copy phases are performed at differet time widows ad a larger widow implies higher predictio error. We further study the rutime overhead of Algorithm 2 o a host machie cofigured with 2 CPU ad 4 GB memory. We expoetially icrease the umber of VMs from 3 to 96, i.e., umber of equatios to be solved. Fig. 7 shows that the compute cost of Algorithm2 is liearly proportioal to the umber of VMs. This demostrates that VMbuddies cause acceptable rutime overhead. I practice, we ever eed to solve a large umber of equatios because our VM groupig scheme guaratees oly a small set of VMs is migrated cocurretly.

12 2 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS Normalized migratio cost Baselie(S&L) Migratio completio time Network traffic Migratio dowtime Time widows (secods) Fig.6. The ormalized migratio cost of L&SW compared to S&L. I the ed, we summarize the evaluatio results of differet migratio approaches. MO always shows the lowest cost of VM migratios, however, it cost sigificat applicatio performace degradatio due to splittig VMs i differet data ceters. LLL ad VMbuddies (OBA+MO) both ca sigificatly reduce applicatio performace degradatio through our sychroizatio protocol. However, LLL results i much more sychroizatio cost tha that of VMbuddies. VMbuddies guaratee the applicatio performace durig migratio, at the cost of slight icrease of migratio cost. 6 RELATED WORK We preset the related work i the followig three categories: sigle-vm migratios, multiple VM migratios ad multi-tier applicatio migratio i the cloud. Sigle-VM Live migratio over LANs or WANs: There have bee may approaches proposed for VM live migratio, such as pre-copyig [][3], post-copyig [6], full-system log/replay [24]. Additioally, there have bee a umber of optimizatios to further reduce the overhead of pre-copyig algorithm, icludig memory compressio [2], page de-duplicatio [2] [4] [38], partial migratio [5], parallelizig migratio[33], shared storage acceleratig [22] ad others [9]. Those optimizatios are also applicable to VMbuddies. Ulike VM migratios i LANs where the disk is shared ad etwork coectios are sustaied, the disk ad etwork coectios eed special care i WANs. For disks, block-level disk pre-copyig ad write throttlig mechaisms was proposed to migrate the persistet storage [8][42]. Mashtizadeh et al. evaluated sapshottig, dirty block trackig ad IO mirrorig for storage migratio supported by VMware ESX [29]. For etwork coectios, VPN [38], IP tuelig ad dydns [8] were proposed to guaratee the etwork switchig trasparet to VM applicatios. VM Turtable demostrated VM live migratio over WAN usig dedicated optical liks for Grid computig [34]. CloudNet built a private etwork usig MPLS-based VPN for VM live WAN migratio [38]. Xe-Blaket provided a abstract layer o top of Xe to support VM live migratio across multiple cloud providers [37]. These works all focused o migratig a sigle VM i LANs or over WANs. Noe has cosidered the problem of correlated VM migratio i multi-tiered applicatios across distributed data ceters. Time of equatios solvig (secs) Number of VMs Fig.7. The compute overhead of Algorithm 2 Multi-VM Migratio: There have bee limited proposals o reducig the cost of multiple cocurret VM migratios. Deshpade et al. ivestigated live gag migratio of VMs i LAN eviromets [4]. They proposed page ad sub-page level memory de-duplicatio amog co-located VMs ad compressio strategies to optimize memory migratio of multiple VMs. As the etwork badwidth is sufficietly high i LANs, their work did ot cosider the correlated VM migratio problem. Al-Kisway et al. were also motivated by data deduplicatio, ad proposed VMFlocks to optimize crossdataceter trasfer ad istatiatio of disk images of multiple correlated VMs [4]. Their work focused o exploitig the similarity amog the VM images ad did t cosider the memory live migratio. These works are orthogoal to the theme of this paper. They focus o reducig the data trasfer by de-duplicatio or fie-graied data sharig. I cotrast, VMbuddies employ etwork badwidth allocatio strategies to coordiate the correlated VM migratios so that the total sychroizatio cost is miimized. Multi-tier Applicatio Migratio i the Cloud: A umber of studies had ivestigated the live migratio performace of multi-tier applicatios for both itra- ad iter-dataceter scearios. Voorsluys et al. [35] evaluated the performace degradatio of 2-tier Web 2. applicatios ruig withi VMs. The similar evaluatios were coducted i [7][23]. Their experimetal results all showed sigificat performace degradatio i terms of request respose time, eve whe the multi-tier applicatios are migrated i the same data ceter. While those studies had give prelimiary results o the performace problem of correlated VM migratios i a multi-tier applicatio, they had ot formulated or solved the problem. Recetly, Zheg et al. proposed Pacer [43], a progress maagemet system for VM migratio i the cloud. Pacer cotrols VM migratio completio time based o aalytic models of progress predictio ad olie adaptatio. Pacer is perhaps the most similar work with VMbuddies. However, Pacer focuses o progress maagemet of sigle-vm migratio, ad the evaluatio result o correlated VM migratios is very limited. This paper cocetrates o the correlated VM migratio problem. We formulate it as a DCOP problem, ad propose a efficiet algorithm to solve it. The evaluatio studies provide more isight o live migratio of complex applicatios. 7 CONCLUSION As more ad more multi-tier applicatios are hosted i virtualizatio eviromets across geographically distributed data ceters, the efficiecy of live migratio of VMs i a multi-tier applicatio becomes importat. I this paper, we formulate the VMs live migratio of multi-tier applicatios as a correlated VM migratios problem, ad idetify the low efficiecy of the basic approach i curret virtualizatio platforms such as Xe. Therefore, we propose a coordiatio system called VMbuddies for correlated VM migratios. To miimize the migratio cost, VMbuddies coordiates VM migratios with a sychro-

13 LIU ET AL.: VMBUDDIES: COORDINATING LIVE MIGRATION OF MULTI-TIER APPLICATIONS IN CLOUD ENVIRONMENTS 3 izatio protocol ad a optimal etwork badwidth allocatio algorithm. Experimets usig a public bechmark show that VMbuddies achieves lower average respose time by 77% ad higher throughput by 8%, ad reduces average total migratio cost by 36% i compariso with Xe. We also ote that the performace models of VM live migratio would facilitate data ceter admiistrators i may scearios, such as, schedulig log-term VM migratio with deadlie costraits, makig tradeoffs amogst differet migratio performace metrics through etwork resource cotrol, makig decisio for groupig ad orderig correlated VMs to reduce migratio cost. I our future work, we pla to explore VMBuddies for more complicated scearioes, such as workflow [4], MapReduce ad MPI applicatios that have tree or directed acyclic graph (DAG) topologies. ACKNOWLEDGMENT This work is supported by the Sigapore Natioal Research Foudatio uder its Evirometal & Water Techologies Strategic Research Programmes ad admiistered by the Eviromet & Water Idustry Programme Office (EWI), uder project 2-IRIS-9. The work of Haiku Liu was doe whe he visited Nayag Techological Uiversity. REFERENCES [] [2] ml [3] [4] S. Al-Kisway, D. Subhraveti, P. Sarkar, ad M. Ripeau, VMFlock: Virtual Machie Co-migratio for the Cloud, Proc. ACM Symp. High Performace Parallel ad Distributed Computig (HPDC'), pp.59-7, Ju. 2 [5] N. Bila, E. Lara, K. Joshi, H. A. Lagar-Cavilla, M. Hiltue, ad M. Satyaarayaa, Jettiso: Efficiet Idle Desktop Cosolidatio with Partial VM Migratio, Proc. ACM Europea Coferece o Computer Systems (EuroSys 2), pp , Apr. 22 [6] P. Barham, B. Dragovic, K. Fraser, S. Had, T. Harris, A. Ho, R. Neugebauer, I. Pratt, ad A. Warfield, Xe ad the Art of Virtualizatio, Proc. ACM Symp. Operatig Systems Priciples (SOSP 3), pp.64-77, Oct. 23 [7] J. Billaud, ad A. Gulati, hclock: Hierarchical QoS for Packet Schedulig i a Hypervisor, Proc. ACM Europea Coferece o Computer Systems (EuroSys 3), pp , Apr. 23 [8] R. Bradford, E. Kotsovios, A. Feldma, ad H. Schioeberg, Live Wide-Area Migratio of Virtual Machies Icludig Local Persistet State, Proc. Third Cof. Virtual Executio Eviromets (VEE 7), pp.69-79, Ju. 27 [9] H. Ballai, K. Jag, T. Karagiais, C. Kim, D. Guawardea ad G. O'Shea, Chatty Teats ad the Cloud Network Sharig Problem, Proc. th USENIX Symp. Networked Systems Desig ad Implemetatio (NSDI'3), pp.7-84, Apr. 23 [] C. Clark, K. Fraser, S. Had, J. G. Hase, E. Jul, C. Limpach, I. Pratt, ad A. Warfield, Live Migratio of Virtual Machies, Proc. Secod Symp. Networked Systems Desig ad Implemetatio (NSDI 5), pp , May 25 [] C. Che, B. He ad X. Tag, Gree-aware Workload Schedulig i Geographically Distributed Data Ceters, Proc. IEEE 4th Cloud Computig Techology ad Sciece (CloudCom 2), pp.82-89, 22 [2] J. Chiag, H. Li, T. Chiueh, Itrospectio-based Memory Deduplicatio ad Migratio, Proc. ACM Iteratioal Coferece o Virtual Executio Eviromets(VEE'3), pp. 5-6, Mar. 23 [3] G. Daiel, R. Jerry, C. Ludmila ad K. Alfos, Workload Aalysis ad Demad Predictio of Eterprise Data Ceter Applicatios, Proc. IEEE th Symp. o Workload Characterizatio (IISWC'7), pp.7-8, Sep. 27 [4] U. Deshpade, X. Wag, ad K. Gopala, Live Gag Migratio of Virtual Machies, Proc. iteratioal Symp. High Performace Distributed Computig (HPDC ), pp , Ju. 2 [5] C. M. Fiduccia, R. M. Mattheyses, A Liear-time Heuristic for Improvig Network Partitios, Proceedigs of the 9th Desig Automatio Coferece (DAC '82), pp. 75-8, Ju. 982 [6] M. Hies ad K. Gopala, Post-Copy Based Live Virtual Machie Migratio Usig Adaptive Pre-Pagig Ad Dyamic Self- Ballooig, Proc. ACM Cof. Virtual Executio Eviromets (VEE 9), pp.5-6, Mar. 29 [7] H. Hlavacs ad T. Treuter, Predictig Web Service Levels durig VM Live Migratios, Proc. DMTF Workshop o Systems ad Virtualizatio Maagemet (SVM ), pp.-, Oct. 2 [8] D. Huag, B. He ad C. Miao, A Survey of Resource Maagemet i Multi-Tier Web Applicatios, to appear i IEEE Commuicatios Surveys ad Tutorials, 24 [9] K. Z. Ibrahim, S. Hofmeyr, C. Iacu, ad E. Roma, Optimized Pre-copy Live Migratio for Memory Itesive Applicatios, Proc. Iteratioal Cof. High Performace Computig, Networkig, Storage ad Aalysis (SC ), Nov. 2 [2] H. Ji, L. Deg, S. Wu, X. Shi, ad X. Pa, Live virtual machie migratio with adaptive memory compressio, Proc. IEEE Cof. Cluster Computig (Cluster'9), pp. 3-22, Aug. 29 [2] V. Jeyakumar, M. Alizadeh, D. Mazi`eres, B. Prabhakar, C. Kim, ad A. Greeberg, EyeQ: Practical Network Performace Isolatio at the Edge, Proc. th USENIX Symp. Networked Systems Desig ad Implemetatio (NSDI 3), pp.297-3, Apr. 23 [22] C. Jo, E. Gustafsso, J. So, ad B. Egger, Efficiet Live Migratio of Virtual Machies Usig Shared Storage, Proc. ACM Cof. Virtual Executio Eviromets(VEE'3), pp. 4-5, Mar. 23 [23] S. Kikuchi ad Y. Matsumoto, Impact of Live Migratio o Multi-tier Applicatio Performace i Clouds, Proc. IEEE Fifth Cof. Cloud Computig (Cloud 2), pp , Ju. 22 [24] H. Liu, H. Ji, X. Liao, L. Hu ad C. Yu, Live Migratio of Virtual Machie Based o Full System Trace ad Replay, Proc. 8th Iteratioal Symp. High Performace Distributed Computig (HPDC'9), pp.-, Ju. 29 [25] H. Liu, H. Ji, X. Liao, C.Yu, ad C. Xu, Live Virtual Machie Migratio via Asychroous Replicatio ad State Sychroizatio, IEEE Trasactio o Parallel ad Distributed System, vol. 22, o. 2, pp , Dec. 2 [26] H. Liu, C. Xu, H. Ji, J. Gog, X. Liao, Performace ad Eergy Modelig for Live Migratio of Virtual Machie, Proc. ACM Iteratioal Symp. High Performace Parallel ad Distributed Computig (HPDC'), pp.7-82, Ju. 2 [27] K. Z. Meth ad J. Satra, Desig of the iscsi Protocol, Proc. IEEE Symp. Mass Storage Systems (MSS 3), pp.6-22, Apr. 23

14 4 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, TPDS [28] V. Ma, A. Gupta, P. Dutta, A. Vishoi, P. Bhattacharya, R. Poddar, ad A. Iyer, Remedy: Network-aware Steady State VM Maagemet for Data Ceters, Proc. th IFIP Cof. Networkig, pp. 9-24, May 22 [29] A. Mashtizadeh, E. Celebi, T. Garfikel, ad M. Cai, The Desig ad Evolutio of Live Storage Migratio i VMware ESX, Proc. USENIX A. Techical Cof. (USENIX'), pp.-4, Ju. 2 [3] M. Nelso, B. H. Lim, ad G. Hutchis. Fast Trasparet Migratio for Virtual Machies, Proc. USENIX A. Techical Cof. (USENIX 5), pp , Apr. 25 [3] W. H. Press, S. A. Teukolsky, W.T. Vetterlig, ad B. P. Flaery, "Chapter 9. Root Fidig ad Noliear Sets of Equatios Importace Samplig", Numerical Recipes: The Art of Scietific Computig (3rd ed.), New York: Cambridge Uiversity Press, 27 [32] A. Qureshi, R. Weber, H. Balakrisha, et al, Cuttig the Electric Bill for Iteret-scale Systems, Proc. ACM Cof. Data Commuicatio (SIGCOMM 9), pp.23-34, Aug. 29 [33] X. Sog, J. Shi, R. Liu, J. Yag, ad H. Che, Parallelizig Live Migratio of Virtual Machies, Proc. ACM Iteratioal Cof. Virtual Executio Eviromets(VEE'3), pp , Mar. 23 [34] F. Travostio, P. Daspit, L. Gommas, C. Jog, C. de Laat, J. Mambretti, I. Moga, B. va Oudeaarde, S. Raghuath, ad P. Wag, Seamless Live Migratio of Virtual Machies Over the MAN/WAN, Future Geeratios Computer Systems, vol. 22, o. 8, pp. 9-97, Oct. 26 [35] W. Voorsluys, J. Broberg, S. Veugopal, ad R. Buyya, Cost of Virtual Machie Live Migratio i Clouds: A Performace Evaluatio, Proc. Iteratioal Cof. Cloud Computig (Lecture Notes i Computer Sciece), pp , Dec. 29 [36] C. A. Waldspurger, Memory Resource Maagemet i VMware ESX Server, Proc. 5th Symp. Operatig Systems Desig ad Implemetatio (OSDI 2), pp.8-94, Dec. 22 [37] D. Williams, H. Jamjoom, ad H. Weatherspoo, The Xe- Blaket: Virtualize Oce, Ru Everywhere, Proc. 7th ACM Europea Cof. Computer Systems (EuroSys 2), pp. 3-26, Apr. 22 [38] T. Wood, P. Sheoy, K. K. Ramakrisha, ad J. V. Merwe, CloudNet: Dyamic Poolig of Cloud Resources by Live WAN Migratio of Virtual Machies, Proc. 7th Iteratioal Cof. Virtual Executio Eviromets (VEE ), pp.2-32, Mar. 2 [39] D. Xie, N. Dig, Y. C. Hu, ad R. Kompella, The oly Costat is Chage: Icorporatig Time-varyig Network Reservatios i Data Ceters, Proc. ACM Cof. Data Commuicatio (SIGCOMM 2), pp.99-2, Aug. 22 [4] A. Zhou ad B. He, Trasformatio-based Moetary Cost Optimizatios for Workflows i the Cloud, to appear i IEEE Trasactios o Cloud Computig, 24 [4] H. Zheg ad X. Tag, O Server Provisioig for Distributed Iteractive Applicatios, Proc. 33rd Iteratioal Cof. Distributed Computig Systems (ICDCS 3), pp. 5-59, Jul. 23 [42] J. Zheg, T. S. Eugee Ng, K. Sripaidkulchai, Workload-Aware Live Storage Migratio for Clouds, Proc. 7th Iteratioal Cof. Virtual Executio Eviromets (VEE ), pp.33-44, Mar. 2 [43] J. Zheg, T. S. Eugee Ng, K. Sripaidkulchai ad Z. Liu, Pacer: A Progress Maagemet System for Live Virtual Machie Migratio i Cloud Computig, to appear i Trasactios o Network ad Service Maagemet, 24 [44] R. Ziva ad H. Peled, Max/Mi-sum Distributed Costrait Optimizatio through Value Propagatio o a Alteratig DAG, Proc. th Iteratioal Cof. Autoomous Agets ad Multiaget Systems(AAMAS 2), pp , Ju. 22 Dr. Haiku Liu is curretly a research fellow at School of Computer Egieerig, Nayag Techological Uiversity. He got the Ph.D. degree i Huazhog Uiversity of Sciece & Techology (27-22). His curret research iterests iclude virtualizatio techologies, cloud computig, ad distributed systems. Bigsheg He received the bachelor degree i computer sciece from Shaghai Jiao Tog Uiversity (999-23), ad the PhD degree i computer sciece i Hog Kog Uiversity of Sciece ad Techology (23-28). He is a assistat professor i Divisio of Networks ad Distributed Systems, School of Computer Egieerig of Nayag Techological Uiversity, Sigapore. His research iterests are high performace computig, cloud computig, ad database systems. He has bee awarded with the IBM Ph.D. fellowship (27-28) ad with NVIDIA Academic Partership (2-2).