Counting Sort for the Live Migration of Virtual Machines

Transcription

1 Counting Sort for the Live Migration of Virtual Machines QingXin Zou 123 ZhiYu Hao 3 Xu Cui 4 XiaoChun Yun 13 YongZheng Zhang 3 1 Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 119, China 2 Graduate University of Chinese Academy of Sciences, Beijing, 149, China 3 Institute of Information Engineering, Chinese Academy of Sciences, Beijing, 193, China 4 China Academy of Aerospace Aerodynamics, Beijing, 174, China {zouqingxin, haozhiyu, cuixu, yunxiaochun, zhangyongzheng@software.ict.ac.cn Abstract The live migration of virtual machines is an important technique in the area of virtualization, and it has been used for load balancing, fault tolerance, and system maintenance in modern data centers, clusters, and cloud computing. The pre-copy algorithm is the most used method for the live migration of virtual machines. However, the existing problem of repeatedly transferring dirty memory pages leads the increase of the transferring data amount, delays of the total migration time as well as the downtime. By analyzing the iteration process of the pre-copy algorithm, we find that the transferring order of memory pages during every middle round has a huge impact on the generation and transferring of dirty memory pages. Further we put forward the concept of the live migration of virtual machines based on the. During every middle round of the iteration process, we do not transfer the memory pages according to their original order, instead we transfer the memory pages according to their times of being dirty. Experiment results show that with different workloads the method could simultaneously decrease the transferring data amount, the total migration time, and the downtime to improve the performance of the live migration. Keywords- live migration;pre-copy; I. INTRODUCTION Live migration is a key feature of virtual machine technology. When the virtual machine s storage is shared (such as SAN or NFS), only the memory image and virtual CPU context will be transferred during the process of live migration. In this case, almost the whole time is spent on the data transferring of the memory image. The pre-copy algorithm is just designed for the data transferring of the memory image. The downtime and the total migration time are contradictory with each other. However in this article, we improve the pre-copy algorithm and simultaneously decrease the total migration time and downtime under different workloads. The main measures taken can be summarized as: First, we transfer the memory pages according to their times of being dirty from low to high. Doing so, we can decrease the repeatedly transferred dirty memory pages. Second, we adopt the method with the time complexity of O (n). Comparing with other sort methods, we can save the migration and sort time. At last, batch size refers to the number of transferred memory pages at a time. Using the method, the batch size is also an affect factor. We diminish the batch size to decrease the transferred memory pages further. The remainder of this paper is organized as follows. Section II introduces the pre-copy algorithm of the live migration. Section III introduces the algorithm design of the method. In section IV we present the experiment results and analysis. In section V some related work is discussed. In section VI, we will present the conclusions and future work. II. THE PRE-COPY ALGORITHM The iteration process of the pre-copy algorithm is shown in Figure 1. 1-Dirty Iteration -Clear (Threshold Number or Max Round) First Round Middle Rounds Critical Round Final Round Figure 1. Process of the iteration The iteration mechanism is used to periodically check dirty memory pages to resend them during the live migration. During the first round of iteration, all memory pages are marked dirty. During the process of iteration, some memory pages which have been modified are also marked dirty and others which have been unmodified are marked clear. When the dirty memory pages during one round of iteration are less than the predefined threshold value or the max round of iteration is attained, the iteration process would be terminated and the virtual machine will be suspended, and then moved into the final round of iteration. For one memory page, we denote the dirty state with 1 and the clear state with. Then for one memory page during adjacent two rounds of iteration, there are four state combinations, which are 11, 1, and 1. The meaning of the state combination 1 is that one memory page is dirty during the front round and clear during the rear round. We should only send the memory page for the state combination of 1, because this memory page has been modified and in the visible future it could not be modified again. For the /13/$ IEEE /13/$ IEEE

2 other three state combinations, we should not send the memory pages. The pre-copy algorithm will repeat the process until the total dirty memory pages during one round is below one threshold value, or the max round of iteration is attained, this round of iteration becomes the critical round. After transferring the dirty memory pages during the critical round, the virtual machine will be suspended and then step into the final round. Now we see during the iteration process of the pre-copy algorithm that there are many memory pages which are transferred to the destination host repeatedly. If we could reduce the transferred memory pages for the pre-copy algorithm, we would save the migration time and decree the transferring data amount, consequently improving the migration efficiency. III. ALGORITHM DESIGN During one round of the iteration, if we could change the transferring order of the memory pages, we could delay the transferring time of one memory page. And the states combination of one memory page could become 11 from 1. Due to the state change, the memory pages which should be sent during one round of the normal pre-copy algorithm will not be sent, consequently reducing the retransmission of the dirty memory pages. Supposing during the time internal of T1 the dirty probability of one memory page k is A k1 and the dirty probability of one memory page w is A w1. During the time internal of T2, the dirty probability of the memory page k is A k2 and the dirty probability of the memory page w is A w2. Then during the time internal of Tn. the dirty probability of memory page k is A kn and the dirty probability of memory page w is A wn. On average, the dirty probability of memory page k is A kavg =( A k1 + A k2 + +A kn )/n and the dirty probability of memory page w is A wavg =( A w1 + A w2 + +A wn )/n. During the whole time interval of (T1+T2+ +Tn), the memory page k dirty times are C k and the memory page w dirty times are C w. We could not gauge the dirty probability of one memory page during every time interval. But if we could replace the different specific values during every time interval with the same average value; suppose (A kavg >=A wavg ) and (C k <C w ). According to this supposition, if the memory page w became dirty during one round, the memory page k would become dirty too. Because the two memory pages go through same lasting time during one round, and the case of (A kavg >=A wavg ) exists. So during the whole time interval, there will be (C k >=C w ). This is contradictory to the supposing of (C k <C w ). In the case of (A kavg >=A wavg ), only (C k >=C w ) exists. The dirty times reflect the average dirty probability from one flank. Therefore, if possible, we would make transferring decision at the rear stage of each round on one memory page which own more times of being dirty. Then, as possible we could decrease the cases of multiple retransmissions for one same memory page due to being frequently dirty. During every round of the pre-copy algorithm iteration, we break the fixed style of transferring memory pages according to their original order. But at the beginning of each round we sort the memory pages according to their dirty times from low to high. Then we could delay the transferring time of memory pages with high dirty probability and lower the chance of transferring repeatedly, consequently improving the efficiency of the pre-copy algorithm. A. Function of the sort There are two factors of dirty times and delaying time for one memory page on one position. This is similar to the vector amount accumulate. See the equality below., 1 If we regard the delaying time of the respond positions as the Vector A and regard the dirty times of the respond positions as the Vector B. Then the Vector A is relatively fixed. Time must go forward, and in reality, could not go backward or turn. The delaying time of the back position is longer than the front position. The n-dimensional vector A of delaying time arrange from low to high all the time. Only the Vector B is changing. < A,B >= A * B cos B Figure 2. Vector amount accumulates From the geometric perspective in Figure 2, the vector amount accumulated is the product of one vector module and the projector of the other vector. So, the smaller the angle is, the bigger the product is. Also, the smaller the angle is, the more the work is from the mechanic. At the same time the ordered vector A of delaying time is relatively fixed from low to high, so the angle is determined by the ordered vector B of dirty times. To be the smallest angle, the dirty times of the memory pages should be sorted from low to high too. In the live migration of a virtual machine, every memory page has different dirty times. If we transfer them according to their different dirty times, then we will produce different results. To attain the best results, and make the work of becoming dirty for the memory pages during one round biggest, we should transfer the memory pages by their dirty times from low to high. B. Use of the In computer science, is a stable, efficient and simple sort algorithm. Counting sort is an algorithm for sorting a collection of objects according to keys which are small integers. It is an integer sorting algorithm. It operates by counting the numbers, and using arithmetic on those counts, to determine the positions of each key value in the output sequence. Its running time is linear in the number of items and the difference between the maximum and minimum key values. Because uses key values as indexes into an array, it is not a comparison sort. The O(n log n) lower bound for comparison sorting does not apply to it [6]. Because there is a limitation on the maximum round of iteration using the pre-copy algorithm for the live migration, this hints that the dirty times of every memory page are not more than the maximum round of iteration. So we can use θ A

3 the method then transfer the memory pages by their times of dirty from low to high. The algorithm description is shown in Table I. Table I. Algorithm based on the tmax: the max round of iteration p2m_size: the size of the memory pages i, j: temp variants. A[p2m_size]: store dirty times of every memory page. D[p2m_size]:record original position of memory pages C[tmax+1]: auxiliary storage initialize every item of A[] with ; if(not the final round of iteration){ for(i=;i<p2m_size;i++){ if(dirty of page i){ Increase A[i] by 1; if(not the first round of iteration){ for(i=; i <= tmax; i++){ initialize the auxiliary storage C[i] with ; for(j=; j<p2m_size; j++){ Counting process, increase C[A[j]] by 1; Positions start from, so decrease C[] by 1; for(j=1; j <= tmax; j++){ Add C[j] by C[j-1] so that memory pages will occupy different position according to their dirty times; for(j=(p2m_size-1);j>=;j--){ Make D[C[A[j]]] equal to j so that we can record the original position, and decrease C[A[j]] by 1 to move one position; transfer the memory pages from D[] to D[p2m_size-1]; By the handling of the method, we don t transfer the memory pages from position to (p2m_size-1); instead we transfer them from D[] to D[p2m_size-1]. IV. EVALUATION We bring about the algorithm with the virtualization platform of Xen on the operating system of Ubuntu and every experiment is done five times. We take the average value which has been rounded up as the experiment data. The CPU of our destination host is an Intel(R) Core(TM) 3.3GZ and our source host is an Intel(R) Core(TM) 2 Duo The RAM of the two hosts is 2GB. The source host and destination host are connected by a FR 4 router of 1Mbps LAN. And on the source host, we share disk images with the destination host by the NFS service. A. Workloads test In our experiments we take the followed scenes as the workloads. 1)Idle: An idle Ubuntu OS with no special applications running in it. 2)Tpcc[7]: Tpcc is an on-line benchmark of transaction processing, simulates complete environment of a population of terminal operators executing transactions against a database. We use tpcc-mysql, and configure ten connections. 3)Dbench[8]: It is an open source benchmark and tool for generating I/O workloads to a file system. We configure one connection to generate suitable disk workload. 4)Linpack[9][1]: Linpack of HPL performs massive vector and matrix operations, which can produce CPU and memory pressure. Four processes exist in our experiments, and we use three order matrix. 5)Webbench[11]: A very simple tool for benchmarking WWW or proxy servers. We configure five connections for a static web page of 263 bytes. We have allocated 512MB RAM for this virtual machine and configure two virtual CPUs. The max round of iteration is set by 3. A.1. Fix the batch size Xen accumulates the memory pages to the batch size of 124 then transfer them all together. Considering the influence of the batch size for the method, we use different batch sizes to test the idle scene at first in order to find a proper value. Memory pages (num) Batch Size(Num) Figure 3. Transferred memory pages of different batches Migration time (us) Batch Size(Num) Figure 4.Total migration time of different batches When the batch size is between 2 and 128 we could transfer less memory pages and above 16 there will be less total migration time. In subsequent experiments we fix the middle batch size of 64 between 16 and 128. A.2. Influence of the workloads. First it is the comparing results of downtime in Figure 5. Downtime(ms) Figure 5.Downtime of different workloads For the idle scene, using both of the default pre-copy method and the method, the small threshold value has been met during the critical round, and the total

4 rounds of iteration are less too. For the Debench scene, adopting the default pre-copy method, all the five tests attain the max round, but in adopting the method, there are three tests which meet the threshold value and the other two tests attain the max round. For the other three scenes of Linpack, Tpcc, and Webbench, the max round is attained with the two kinds of methods. Time(ms) Pages(num) Figure 6.Total migration time of different workloads Figure 7.Total memory pages of different workloads Figure 6 and Figure 7 show the total migration time and transferred memory pages. On average the total migration time of the five scenes has been decreased by 23% and the total migration memory pages has been decreased by 16%. After the using of the method we can simultaneously decrease the total transferred memory pages and those during the final round. Then we can decrease the total migration time and the downtime. B. RAM Test We adopt the TPCC for the test of different kinds of RAM. The TPCC scene attains the max round of iteration. Downtime(ms) MB 512MB 124MB Figure 8. Downtime of different RAM Figure 8 shows the downtime. The transferred memory pages during the final round all decrease after using the method. Then they all bring about the decreasing of the downtime Time(ms) MB 512MB 124MB Figure 9. Total migration time of different RAM Pages(num) MB 512MB 124MB Figure 1. Total memory pages of different RAM Figure 9 shows the total migration time and Figure 1 shows the total transferred memory pages. From left to right the respective decreasing ratios of the total migration time are 17%, 11% and 6% and the total transferred memory pages are 23%, 15% and 8%. At the same time the curves Figure 11 shows the changing trend of the transferred memory pages. Memory pages(num) (All rounds) (middle rounds) (All rounds) (middle rounds) Size of RAM(MB) Figure 11. Transferred memory pages From Figure 11 we see that bigger RAM has more memory pages. But for both of the two methods, the curves of the total transferred memory pages are nearly parallel. It indicates that for the same workload and different kinds of RAM, the decreased total transferred memory pages are almost equal. We then subtract the transferred memory pages during the first and the final rounds. For the same scene and different kinds of RAM, the curves of the transferred memory pages during the middle rounds are nearly parallel with the X axis. The transferred memory pages during the middle round decreases 55% for the RAM of 256 MB, decreases 42% for the RAM of 512 MB and decreases 44% for the RAM of 124 MB. In order to observe the details of changing, we gauge memory pages for sending and omitting during every round for one experiment. See Figure 12 below. Memory pages(num) (send) (send) (omit) (omit) rounds 2 (num) 25 3 Figure 12. Details of different memory pages We choose the experiment which is done on the RAM of 512 MB and the total transferred memory pages is closest to the average value from the five tests. Using the method, the omitted memory pages are more than the omitted memory pages using the default pre-copy algorithm, and also the sent memory pages using the

5 method are less than the sent memory pages using the default pre-copy algorithm. On average, for the default pre-copy algorithm, the ratio of the omitted memory pages and the sent is But for the method, the ratio of the omitted memory pages and the sent is 2.9. The memory pages omitted during the middle rounds and not accumulated to the final round are increased by the method. V. RELATED WORK Since [1] had first mentioned the live migration of virtual machine, researchers have paid much attention to this issue and many studies were related to the live migration of virtual machines. Live migration is an important mechanism available to virtual machines. The improved time-series based pre-copy approach is proposed in [2] to accelerate the live migration of virtual machines. It identifies high dirty memory pages via historical statistics by n-size TO_SEND_H bitmap. In other words if the modified times of one dirty memory page in the TO_SEND_H bitmap were more than a given number k, it is a high dirty memory page, and then it will be sent during the last round. The paper of [3] presents the matrix bitmap algorithm that collects a great deal of information of dirty memory pages and allocates different weights for them. They then adds up the weights to decide whether to transfer the page or not. All the methods mentioned above have a common feature in that there are memory pages accumulating into the final round and could influence the downtime. But using the method, the transferred memory pages decrease and there are not memory pages accumulating into the final round. Ordered propagation to transfer dirty memory pages according to their rewriting rates is proposed in [4]. But recording rewriting rates of every page is prohibitively impossible. So instead of counting rewriting rate of every memory page, it divides memory pages into stripe groups, and use statistical sampling and memory stripe groups for approximation. It uses a simple method to statistically sample the rewriting rates of memory stripe groups. When a round of iteration is finished, it counts the dirtied pages during the round of iteration for each stripe group. This number is then divided by the lasting time of the round. The divided result is the rewriting rate of the stripe group. This method could decrease the downtime efficiently, but the algorithm consumes more time, and could increase the total migration time. Using the method we did not increase the total migration time. The compression technique for migrated memory was adopted in [5]. The transferred memory pages in each round would be compressed on a source physical host at first; next they will be sent to the destination host. After receiving the whole compressed memory pages on the destination host, they would be decompressed. This method does accelerate the live migration from the data transferring style, and is different from the method that improves the live migration from the algorithm. In reality, we could use both at the same time to improve the performance of live migration. The related works mentioned above do research on the live migration of virtual machines from the data transferring style, algorithms and so on. With some success, the counting sort method still improves the live migration of virtual machines from the algorithm and memory transference. VI. CONCLUSION AND FUTURE WORK Using the method, we decrease the transferred memory pages during every middle round of the iteration, shorten the interval between adjacent two rounds of iteration including the critical round and the final round, and decrease the transferred memory pages during the final round. At the same time the batch size can influence the method. Experiments demonstrate: under different workloads the downtime, the total migration time, and the total transferred memory pages could decrease simultaneously when we fix the batch size of 64. On average, the downtime decreases by 18%, the total migration time decreases 23%, and the total transferred memory pages decrease 16%. For the same workload with different kinds of RAM, the decreased memory pages during middle rounds are basically equal. In terms of Tpcc, it decreases 47% on average. In the future we will probe other new styles or algorithms of transferring memory pages to improve the performance of the live migration. ACKNOWLEDGMENT The authors would like to give thanks to the reviewers and helper for their valuable comments and suggestions on this paper. This work also is supported by The National Natural Science Foundation of China (No , No and No ). REFERENCES [1] C. Clark, K. Fraser, S. Hand, J. G. Hansen, E. July, C. Limpach, I.Pratt, and A. Warfield, Live Migration of Virtual Machines, Proc. The 2nd conference on Symposium on Networked Systems Design &Implementation, 25, pp [2] Bolin Hu, Zhou Lei, Yu Lei, Dong Xu, Jiandun Li. A Time-Series Based Precopy Approach for Live Migration of Virtual Machines. 211 IEEE 17th International Conference on Parallel and Distributed Systems. [3] Wentian Cui, Meina Song. Live Memory Migration with Matrix Bitmap Algorithm 21 IEEE 2nd Symposium on Web Society (SWS). [4] Yuyang Du Hongliang Yu Guangyu Shix Jian Chenx Weimin Zheng. Microwiper: Efficient Memory Propagation in Live Migration of Virtual Machines th International Conference on Parallel Processing. [5] H. lin,l. Deng,S. Wu, et al, Live Virtual Machine Migration with Adaptive Memory Compression," In Proc. IEEE Int'l Conf. CLUSTER,pp.I-IO,29. [6] [7] [8] [9] [1] [11]