Optimizing Virtual Machine Live Migration without Shared Storage in Hybrid Clouds

2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems Optimizing Virtual Machine Live Migration without Shared Storage in Hybrid Clouds Shan He, Chunming Hu, Bin Shi, Tianyu Wo, Bo Li State Key Laboratory of Software Development Environment (NLSDE) School of Computer Science and Engineering Beihang University, Beijing, China, 100191 Email: {heshan,hucm,shibin,woty,libo}@act.buaa.edu.cn Abstract Virtual machine live migration technology allows a running VM migrates from one physical host to another with no impact on users. As the scale of distributed computing gets larger and larger, hybrid cloud, which is integrated cloud service utilizing both private and public clouds to perform distinct functions within the same organization becomes a hotspot for both academia and industry. Thus, it will be vital to migrate virtual machines in hybrid clouds. The biggest issue which VM migration in hybrid clouds faces today is how to transfer the large amount of storage data. Storage migration of virtual machine with existing mechanisms of memory migration has been studied for a long time. Unfortunately, no one can perform perfectly without modification. It is compelling to consider a new migration model and optimization methods. In this paper, we analyze the advantage and disadvantage of classical methods, and put forward a method which combines the strengths of pre-copy and post-copy migration model and we also use disk working set to optimize virtual machines storage migration. We have developed and implemented our approach to the QEMU/KVM hypervisor and run a series of experiments. The presented technique shows a reduction of up 20.5% on average of the total transfer time for most of the selected scenario. Our approach can also reduce the migration downtime in most cases. Index Terms virtual machine; live migration; storage; working set I. INTRODUCTION Nowadays cloud computing is a popular topic, more and more companies have migrated their business to cloud environments. Most of these cloud infrastructure can be divided into two categories. One is private clouds, which is a particular model of clouding computing that cloud infrastructure can be operated by a single organization, the other is public clouds in which service providers make resources available to the general public over the Internet such as EC2 [1] and Azure [3]. Recently, hybrid clouds, a combination of public and private clouds, become increasingly popular. Virtual machine live migration allows a running VM to move from one physical host to another with no impact on the availability of virtual machine, that makes it possible to be leveraged for a variety of tasks such as load balancing, system maintenance, fault tolerance, and so on. Live migration is an importance feature of virtualization technology for administrators of data centers and clusters, because it can significantly increase the mobility of data centers and manageability of clusters. In a cloud environment, live migration only transfers memory and devices state between two machines that share a common local network and storage system such as Storage Area Network (SAN) [4], Network Attached Storage (NAS) [11]. SAN is a high-speed network of storage devices which provides block-level storage that can be accessed by applications running on any networks servers. With the development of the technology of XVLAN [17], OpenFlow [20] and other virtual networks, it is possible for virtual machine migration in wide area networks (WAN) with no disruptions on network connectivity. however, it is less likely to use shared storage system in hybrid clouds for some reasons. First, existing share storage systems do not provide high efficient I/O performance in WAN as they do in LAN. Second, once a shared storage system has been implemented among several clouds, it is necessary to rely on the storage devices outside of cloud environment, which can bring security problems and difficulties of data management. Consequently, the ability to migrate VM across hybrid clouds without shared storage system is increasingly compelling. WAN VM migration is considered as hotspots of research by the academics and business.some solutions have been proposed by moving existing technologies for migrating memory and device state to transfer storage. In this work, we analyze the advantage and disadvantage of classical methods, such as pre-copy [10], post-copy [12] and hybrid copy. We propose a method migrating virtual machines storage combine the strengths of pre-copy and post-copy with disk working set which is the set of blocks that may be read later. This method can have negligible downtime, shorter total time of migration and minimal impact on the application. We present an integrated memory and storage migration system with our disk working-set. Before we introduce our approach, we analyze the disadvantage and advantage of existing mechanisms of live migration. We found that combining the advantage of pre-copy and post-copy can reduce the amount of data being transferred and have a minimal impact on application at the same time. We do a series of experiments to research disk working-set according to different applications. In some scenario, it can work well. Then, we have implemented our approach to the KVM hypervisor and do some experiments. This research is meaningful, although its performance is not always so good for some cases. 978-1-5090-4297-5/16 $31.00 2016 IEEE DOI 10.1109/HPCC-SmartCity-DSS.2016.143 921

The contributions of this paper are as follows: We propose a method of live storage migration with the disk working set which is the set of blocks to be read in the post-copy stage. This method can minimize total migration time, downtime,volume of data transferred, and the impact on the application. We show the feasibility of the proposed technique by providing an implementation of this method in KVM hypervisor; We demonstrate the effectiveness of the proposed technique by running a series of benchmarks. We achieve an average improvement in the total migration time of over 15.45% with up to 29.38% for certain scenarios; Our migration solution can be applied to various networks environment, compared with the migration with IO Mirroring, has better robustness; The rest of this paper is organized as follows: Section II gives an overview of the background and our motivation. In section III, design a series of experiments to find out disk working-set and analyzes the feasibility of live storage migration with disk working set. Section IV and V contains the design and implementation of the proposed live migration framework. Section VI evaluates our techniques. Section VII introduces the related work on the live migration. Section VIII concludes the paper. II. BACKGROUND AND MOTIVATION The technologies of live migration can be divided into three migration models: the pre-copy model, the post-copy model, and the hybrid copy model. Each of the models has its advantages and limitations. This section introduces and analyze these models,after comparing theirs merits and shortcomings, gave the goals our method requires to meet. A. Pre-copy model In the pre-copy model, the target VM starts running after entire VM is copied to the target host.one of the most famous pre-copy models is the iterations pre-copy method. As it is implemented by KVM [14]with Dirty Block Tracking (DBT) [15] [16], which uses a bitmap to mark dirty blocks. The disk file is copied to the target host from beginning to end, and the memory is migrated at the same time. During the migration, the source VM keeps running. And all write operations are intercepted and trigger the operation of set the bitmap. The blocks are transmitted in several iterations. In each iteration, only the dirty blocks marked in the previous round need to be sent again. Another prevalent approach of the pre-copy is IO Mirroring [18], which will interpose on all disk write operations and sent data to the target host immediately. When all blocks have been copied, the migration ends. The IO Mirroring method just one round and synchronize dirty blocks when write operations happen. The advantage of this model is all read and write operations have good performance when the VM runs at the destination. The disadvantage is dirty blocks are transferred repeatedly, which will lead to extra traffic and the increase of the transmission and migration time. The iterations pre-copy method requires transmission in the last round to converge so that it can enter into the stop and copy stage. But for large size of disk, it takes a long time for a round to finish, while the number of blocks marked in this round is large. So it is hard to obtain a converged value. B. Post-copy model In the post-copy model, the target VM will start running as long as the basic data to support running of VM have been forwarded to the target host. After the state of the target VM is running, storage migration begins. This model includes three steps. First,live memory migration is executed until the number of dirty pages converges. Second, stop the source VM and copy rest of dirty pages to the target host. Last, start the target VM and then begin storage migration. When read operations at the destination are on the block that has not been sent to the target host, the target host will send a request to get those blocks from the source host. This operation is remote reads which incur extra delays, the application must wait until it gets the required data. So, in this model, each block is transferred at most once so that the total transmission of data for blocks is minimized. However, the impact caused by remote reads on the applications is the most serious of three migration models. C. Hybrid-copy model We migrate the blocks, not in the Disk working set, by the Post-Copy with on-demand fetching. Before the beginning of the Post-Copy stage, we start a server which listens to HTTP requests. It starts a separate TCP connection for every request. During the Post-Copy stage, the destination VM is running and the source VM is paused. Each I/O read operations will be caught by the I/O Working module, which will call the ondemanding client to get the read data if these blocks have not been in the destination. When issuing the write operations, we record the write blocks to remark the blocks are the newest which means that it wont be updated by the Block Copy module. Size of chunk requested is 1M, which is the same with the base unit. D. Efficient storage migration model Moving a VM to the another cloud requires substantial downtime and migration time when the entire disk is copied to the target host with current available technology in hybrid clouds. An efficiently live migration model should have a negligible downtime, keep the consistency of data and transparent to the user. So four kinds of measure values to evaluate the performance of the migration methods are the downtime, total migration time, the performance of VM, total data transmission. As mentioned previously, in the pre-copy model the target VM starts running after migration is over, so there is no additional performance degradation when the VM runs at the destination, and remote reads cannot happen. As table Ishows, the strength of the pre-copy model is the minimal impact on the application, but it has the largest total data transmission. The migration time increases as the transmission increases. On the contrary, the post-copy model 922

TABLE I: comparison of three migration model Migration model Down time Total migration time Total transmission Performance impact Pre-copy 1s most most least Post-copy least least least most Hybrid copy 1s middle middle middle has the minimum data transmission. While its shortcoming is the heavy impact on the applications because of remote reads. In an ideal situation, we expect the negligible downtime, the minimum transmission, the shortest migration time and the negligible impact on the user. However, it is impossible to achieve but we can get very close to it. The hybrid copy model is a compromise which can have both strength of the pre-copy model and the post-copy model. It is possible to seek a satisfactory balance between the migration time and the performance of virtual machines by using a hybrid copy model. (a) CPU workload s trace (b) memory workload s trace III. DISK WORKING SET AND CHARACTERISTICS This section analyzes the characteristics of disk working sets from different workloads. To understand whether the history I/O operations can be useful for optimizing storage migration, we collect and analyze various IO trajectories of diverse workloads. The idea of disk working set comes from the concept of memory working set, which defines the pages accessed in a given time interval. Analogous to the working set in memory, disk working set is the approximation of the set of blocks that the process will access in the future for a certain time. In the latter part of this section, we put forward a hybrid copy model with disk working set. A. Trace collection In order to analyze the relevant of working set characteristics and storage migration. We choose four kinds of diverse workloads, high CPU load, memory overhead, file I/O intensive and OLTP workloads which are common in web services. All the benchmark workloads we collect are simulated by sysbench benchmark [8] which is one of the most common benchmark tools. For example, the workload of high CPU load is the CPU performance test which is to verify prime numbers by doing standard division of the number by all numbers between two and square root of the number. And we can specify the maximum number to determine the running time. In our experiment, we set the maximum is 40000, which can be running for twenty minutes in VM. The configuration of VM is 1 CPU, 2GB RAM and 10GB disk. Two machines with 16GB of RAM and 8 Intel Core i7 processors are utilized to run VMs. We get the I/O trace through the virtual machine monitor (VMM) which can interpose all virtual disk read and write operations. We intercept all the I/O operations and record the read or write property, the access time, the number of sectors and quantities of sectors requested. Each 512 bytes means a block. We run all workloads in the same environment. B. Temporal locality and Spatial Locality Characteristics Temporal locality and Spatial Locality Characteristics are basic types of the principle of locality, which used to describe (c) File IO workload s trace (e) OLTP workload s trace (d) local graphs of File IO workload s trace (f) local graphs of OLTP workload s trace Fig. 1: Trace collection of four workloads simulated by sysbench. The horizontal axis shows time in milliseconds, while the vertical axis represents the index of blocks. The red means the read operations to disk, while the green dots are write operations to disk a phenomenon in which the same values and related storage locations, are frequently accessed. Temporal locality Characteristics means that the same values are likely to be accessed again in the future. Spatial locality Characteristics refers to that if a particular storage location is accessed. Then it is very likely that adjacent positions will be referenced in the future. As Figure 1shows, we record I/O traces from four kinds of workloads. The horizontal axis shows time in milliseconds, while the vertical axis represents the index of blocks. The red means the read operations to disk, while the green dots are the write operations to disk. Temporal locality and Spatial Locality Characteristics are obvious. Take the workload of high CPU as an example, even though the I/O operations of this workload are not so many, two lines appeared in the figure 923

1a. Suggest the same blocks and neighbors are accessed in the near future. File IO workloads have complicated traces as show in figure 1c.First, It creates a large number of test files, then produces a lot of IO operations according to the set mode. We can make a choice between the test mode from sequential reads and writes or random reads and writes or a combination. Random read and write is used. In the figure 1d, we provide a local image of the figure for showing I/O traces in details. From this local image, it is evident that read operations gather in some chunks while write operations gather in other chunks. From these figures, across all workloads, the principle of locality works well. So it is possible to take advantage of disk working set in the migration. Whats more, the working set of read operations and the working set of write operations have different impacts on the process of storage migration. One of the most important differences, In the post-copy stage, remote reads will reduce the performance of service, while write operations at the destination have no impact on the applications. However, write operations in the pre-copy stage will increase the number of dirty blocks, which it expects to be a very small number. C. A hybrid copy model with disk working set To optimizing storage migration, we propose a hybrid copy migration with disk working set. History, the first round of virtual disk copy and memory migration belongs to the pre-copy stage. During the pre-copy stage, dirty blocks are maintained by a block bitmap which will be sent to target host at the stop and copy stage. Dirty blocks are transferred in the post-copy stage. The hybrid copy model is a compromise method that cant overcome all disadvantages. The long migration time and large transfer volume are the shortcomings of pre-copy model. While the performance of virtual machine affected by remote reads is the weakness of the post-copy model. Retransmission of dirty blocks is the major reason to increase time. In the hybrid copy methods, only dirty blocks have to be copied again and all blocks are copied twice at most. So compared with the pre-copy model, it can reduce the quantity of data transmission. When compared to the post-copy model, the hybrid copy model causes fewer times of remote reads but have to carry more blocks. Most of hybrid copy methods don t make full use of the advantage that coordinates the balance between the migration time and the impact on the applications. Some optimum methods can be taken to get the best results. First of all, If the blocks that will be accessed by remote reads in the post-copy stage can be sent to the target host in the pre-copy stage, the number of remote reads can significantly decrease. It is clear that the set of blocks have to be sent in the pre-copy stage is as small as possible. The experimental result tells that the disk working set exists in the most workloads, and the set of blocks written can be different from the set of blocks to be read, what s more, these two sets may overlap a small part of books in most cases. If we make the set of blocks corresponding to read operations as the set which has to be migrated in the pre-copy stage, it can meet the requirement that the blocks of remote reads is copied in the pre-copy stage. So we transfer working set of blocks accessed by read operations in pre-copy stage and other data in the post-copy stage, we can get the best trade-offs between the total time and the impact on the applications. Sometimes a compromise is the best methods. In addition, our solution can be applied in various networks environment, compared with the migration with IO Mirroring, has better robustness. The convergence of dirty data in the pre-copy model is required to ensure it works normally. IV. DESIGN Our system provides virtual machines live migration without shared storage in the hybrid clouds. Our goals are to optimize the storage migration which is an important issue in a wide area network. We wish to present a method that can have minimal total migration time and downtime time, the least impact on the application. This section will present the architecture of our migration system and the process of the entire migration. A. Architecture overview Our system is based on the hybrid copy model and uses IO Mirroring mechanisms to sync dirty blocks. Living memory migration is implemented using an iterative pre-copy which is the most popular approach. Living storage memory is a hybrid copy method which determines blocks sent in the pre-copy stage by disk working set. First, a block bitmap for marking disk working set is generated depending on the historical data at the beginning of migration. Then, in the pre-copy stage, the migration of disk working set is prior to the memory migration. The migration of disk working set uses IO mirroring to keep consistency. In the stop and copy stage, the dirty pages and the dirty blocks in working set are transferred to the target host. Last, in the post-copy stage, the target VM starts running, then the blocks which are not included in disk working set are copied from the beginning of the disk file to the end. At the same time, if a read operation at the destination accesses to the blocks which have not yet been transferred, the target host will fetch these blocks from the source host on-demand. Figure??depicts the system architecture. The Live Migration module is responsible for the tasks of memory migration implemented by the method of iterative pre-copy. The Transport module handles memory pages and disk blocks transfers between the two hosts. The I/O Working is a multi-functional module that intercepts all I/O operations and deals with these differently according to the stage, such as update the information that will be used to generate a block bitmap of working set in the non-migration stage. During the non-migration stage, the I/O Working intercepts I/O operations for recording the historical information. In the pre-copy stage, the I/O Working interposes on all disk writes and mirrors these to the Block Copy module. The I/O Working on the destination is responsible for remote reads. The I/O Working module only works on the virtual machine whose state is running. For an example, in the postcopy stage, the source VM is stopped so that no I/O operations can be intercepted at the source host. 924

Fig. 2: Architecture Overview: shows the main components involved in migration The I/O Write/Read module is responsible for writing and reading data from and to the virtual disk. Two major functions of Block Copy module in the source is to copy the data into the disk buffer and synchronize the mirror. In the pre-copy stage, Block Copy module copies disk working set through the I/O Write/Read module from the beginning to end. At the same time, Block Copy module receives the disk writes from the I/O Working module. If the write operation is to a block that has been copied, Block Copy module then copied these data to the Disk Buffer. In the postcopy stage, Block Copy module proceeds linearly across the disk, copying the disk region that has not been copied to the Disk buffer. While Block Copy module at the destination is working to copy the received data to the virtual disk file. The on-demand fetching Server module and the on-demand fetching client are responsible for issuing remote read in the post-copy stage. On-demand fetching Server gets data from disk by the I/O Read/Write model, then transfers these data to the on-demand client. The on-demand client module can be called by the I/O Working module at the destination when the operation of read data is on the disk block which has not been transferred. B. Process of migration Our migration methods can make up of four stages: nonmigration stage, pre-copy stage, stop and copy stage, post-copy stage. As figure 3 shows, this section describes the work of each stage. 1) Non-migration stage a) The I/O Working module intercepts all the I/O operations and records the I/O sequence. all disks are divided into many chunks size of 1M, and number these chunks from the beginning to the end. Then we record the number of chunks instead of the number of the sector when we record the I/O sequence. We define the max size of working set 1/10 of disk size. 2) Pre-copy stage a) At the beginning of migration, a bitmap for marking disk working set is generated according to the historical records b) The Block Copy module on the source transfers blocks belong to disk working set sequentially, while receives the disk writes from the I/O Working and mirrors those blocks that have been copied; c) Until all blocks of working set have been copied, living migration module starts memory migration. Block Copy module still issues disk writes during the memory migration; d) It is the time to stop the source VM When memory migration converges and the quantity of data transmission include disk blocks and pages can be transferred with a limited downtime; 3) Stop and Copy stage a) Remaining data is copying to target host; b) Start the on-demand fetching server that issues remote reads; c) After the steps above completed, a sign represents end of this stage will be sent to the target host. The target VM starts running after receiving sign. 4) Post-copy stage a) The Block Copy module on the source host transfers blocks that have not been copied to the Disk buffer from the beginning to end; b) The I/O Working at the destination intercepts I/O operations, if the operation is read blocks that have not been received, the I/O Working module get data of blocks by the On-demand fetching client which requests those blocks from the source; 925

TABLE II: Five kinds of applications Application Idle OLTP File Server Compress file Kernel compile Description No application Sysbench benchmark Postmark benchmark Compressing a large file(3.7g) compiling the Linux kernel Fig. 3: Shows the migration process c) When all blocks have been copied to the target host, the migration is over; V. IMPLEMENTATION We have implemented the proposed technique in the opensource virtual machine monitor KVM, and visualizer QEMU [7], which intercepts disk I/O request. The section discusses the details and issues of our implementation. A. Disk working set Disk working set is the set of blocks that moved to the target host in the pre-copy stage. As we discuss in?? We expect that blocks will be read in the post-copy stage can be in our working set. The less the requests of remote read are, the better performance of VM is. When the target VM issue disk writes, it is safe to update blocks directly. So we only concern the disk working set of reads operations. We record the I/O read requests sequence to generate a bitmap of disk working set. To take use of spatial locality characteristics, we consider 1MB chunk instead of 512B Block as the basic unit of counting accesses. First, migration methods in KVM use 1MB as the basic unit of transferring, and work of [25] shows that 1MB chunk can have a good effect than 512B. Second, in the implement of migration in KVM, each basic unit are copied to target host with the overhead costs of a few bytes, such as the flags to tell target host what the following data represent, device name and its length,the address of block and so on. The smaller size of transmission chunk is, the greater the overhead. We record the most recent N the index of chunks in a FIFO queue. We set the size of working set is M which is one-tenth of the size of the disk. N is larger than M for a lot of duplicated chunk. At the beginning of migration, we get M different chunks from the N sequentially, then generate bitmap. B. Pre-Copy and IO Mirroring We migrate the Disk working set by the Pre-Copy method and synchronize dirty blocks with IO Mirroring. We use a bitmap to mark the disk working set which use a bit to represent a block. IO Mirroring reflects any additional changes to the source disk that take place during storage migration. Our Block Copy module is implemented to linearly across the disk and copying one disk region in working set at a time. The I/O Working on the source interposes on disk writes and sends these to the Block Copy module. We record the max value of the number of blocks that have been copied. All synchronization works can fall into three types: 1) write to a chunk that has been copied; 2) write to a chunk being copied; 3) write to a chunk that has not been copied. In the first case, Block Copy module enqueues data to the disk buffer if the chunk belongs to disk working set. In the second case, Block Copy will enqueue data when the copy of this chunk has finished. The disk buffer is implemented by a queue. In the last case, we do nothing. C. Post-Copy and on-demand fetching We migrate the blocks, not in the Disk working set, by the Post-Copy with on-demand fetching. Before the beginning of the Post-Copy stage, we start a server which listens to HTTP requests. It starts a different TCP connection for every request. During the Post-Copy stage, the destination VM is running and the source VM is paused. Each I/O read operations will be caught by the I/O Working module, which will call the ondemanding client to get the read data if these blocks have not been in the destination. When issuing the write operations, we record the write blocks to remark the blocks are the newest which means that it wont be updated by the Block Copy module. Size of chunk for on-demand fetching is 1M, which is the same with the basic unit. VI. EVALUATION We implemented IO Mirroring and our prototype implemented in the KVM and QEMU. All experiments are performed on two identical hosts with a 16GB memory of RAM and 8 Intel Core i7 processors. The configuration of VM for migration is 1CPU, 2 GB RAM and 40GB disk. For lacking a standard benchmark suite for live migration of virtual machines, we have selected several general application scenarios listed in II. We use five workloads, an idle VM, OLTP simulation using sysbench benchmark, File server simulation using postmark [13], compressing a large file and compiling the Linux kernel. The OLTP workload used sysbench to generate to IO pattern that simulates an OLTP workload with random access. We use postmark benchmark to simulation File Server workload with a 30% write and 70% read. The Postmark is a file-system benchmark to measure performance for the applications such as e-mail, netnews, and web-based commerce. A. Downtime All live migrations are expected to achieve minimal downtime which has minimal impact on the service. In general, 926

Fig. 4: The downtime of migration Fig. 5: The total migration time of five workloads Fig. 6: The amount of data transferred the max downtime is one second which is a common value set for downtime. We migrate the VM with different types of workload. From Figure 4, we can know that our prototype can decrease downtime on poly-type our selected application workloads. Our disk working set is set of blocks corresponding to the I/O read operations so that there would be less dirty blocks that have to be transferred to the target host during the migration. In the stop and copy stage, dirty blocks that have to be copied repeatedly are produced by write operations. The size of the intersection between the set of blocks corresponding to read operations and the set of blocks corresponding to write operations is smaller, the lower the number of dirty blocks synchronized is. However, when the dirty blocks are so much that the downtime is one second that is the maximum value. B. Migration time Migration time is the total time of virtual machine migration. Figure 5 shows the total migration time of virtual machines with five different workloads. Our results show that our migration times are less than the method of IO Mirroring in five workloads. But Two places in our implementation have extra time. One place is when a bitmap to mark disk working set generates at the beginning of migration. It will take some time to read the records and initialize the bitmap. The other is that some seconds will be consumed for starting the ondemand fetching service. As figure 6shows that we can reduce the transmission.only the block which belongs to the working set and is modified during the pre-copy stage has to be transferred for many times. The other blocks are only transferred for one time. C. Performance overhead Performance overhead should be minimized during migration. However, the evaluation of performance overhead during migration is quite complicated, because the overhead is influenced by the comprehensive influence of downtime, total migration time and overhead incurred during different stages of data transfer. On the one hand, our method incurs more overhead than IO Mirroring because we use more ondemand fetching whose response time is long. On the other hand, our method performs better since that it incurs shorter downtime and total migration. In this case, we demonstrate two experiments to evaluate the overhead. First, we run the same workload tasks with different migration method and compare their elapsed time. As is shown in Figure 7a, our method brings 5.4% more overhead than IO Mirroring with file compression workload. Compared with the promoted metrics mentioned above, it is acceptable. Second, we execute the IO intensive workload both when IO intercept on and off. Figure 7b shows that IO intercept only incurs 7.3% overhead. When the workload has less IO operations, the overhead will be even much lower. (a) Compress file during(b) Compress file when live VM migration IO intercept on and off Fig. 7: File compression time D. Limitation and Future work The experimental results show that our migration method reduces the downtime and migration time in some situations. In all experiments, downtime is reduced and the transmission is generally decreased compared with other methods. As for performance overhead, as the increase of hit rate, the overhead reduce. There is still space for improving. In future, we will validate our method in real-world applications. VII. RELATED WORK Virtual machine migration is no longer a new issue for the reason that living migration technologies in LAN are mature enough. And it has been implemented in all major hypervisors including Xen [9], KVM, Microsoft Hyper-V [6] and VMware ESX [5]. The major approach of these systems is a pre-copy that iteratively copies memory pages from the source host to destination host. The live blocks migration also has been 927

implemented in most major hypervisors. But, up to now, no methods have been widely accepted as one of the best methods for migration entire virtual machines. Live storage migration can be divided into three kinds of migration models. The major approaches are dirty block tracking and IO mirroring, both two we have introduced. Especially, IO mirroring is implemented in Microsoft Hyper- V, VMware ESX, and Xen with DRBD [24]. However, live storage migration of virtual machines using IO mirroring does not apply in KVM. In general, live virtual machine migration considers three respects, memory and device state, storage and network. So studies of live migration over WAN focused on either storage or network concerns. With the development of network technologies, the technologies of tunneling and network virtualization are alleviating the networking limitations that previously prevented moving VM across WAN. Such as Cisco OTV [2], VXLAN and SDN [21]. CloudNet [22] [23] put forward a cloud computing platform that seamlessly connects resources at multiple data center, and a centralized virtual private network. There are so many studies about pre-copy based storage migration. For example, Bradford [10] implements a disk block level iterative pre-copy method in Xen. It can support live storage migration in the WAN. zheng [25] had explored some optimal improvements on the per-copy model to migrate storage over WAN. And XvMotion [19] do some improvements in storage migration implemented by IO Mirroring in VMware ESX. The common optimization methods including compression, de-duplication, change the transport order and so on. XvMotion is deployed in VMware ESX using IO Mirroring and integrates support for memory and storage migration over the local and wide area. CloudNet addresses many of shortcomings of migration in Xen with DRBD. CloudNet use synchronous disk replication and XvMotion use asynchronous replication. VIII. CONCLUSION In this paper, we present a novel method to optimize the migration in hybrid clouds. We combine the strengths of pre-copy and post-copy migration model and also use disk working set to optimize virtual machines storage migration. We have designed and implemented our approach to the QEMU/KVM hypervisor and ran a series of experiments. The presented technique shows a reduction of up 20.5% on average of the total transfer time in most of the selected for some certain poly-type scenario. Our approach can also reduce the migration downtime in most cases. IX. ACKNOWLEDEMENT The work in this paper has been supported in part by the China 863 program under Grant No. 2013AA01A213. REFERENCES [3] Microsoft azure: Cloud computing platform and services. http://azure. microsoft.com/. [4] Storage area network. http://www.webopedia.com/term/s/san.html. [5] vsphere hypervisor. https://www.vmware.com/products/ vsphere-hypervisor.html. [6] Windows server 2008 r2 hyper-v live migration. http://www.microsoft. com/en-us/download/details.aspx?id=12601, 2009. [7] Qemu. http://qemu.org/, 2016. [8] K. A. Sysbench: a system performance benchmark. http://sysbench. sourceforge.net. [9] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In ACM SIGOPS Operating Systems Review, volume 37, pages 164 177. ACM, 2003. [10] R. Bradford, E. Kotsovinos, A. Feldmann, and H. Schiöberg. Live widearea migration of virtual machines including local persistent state. In Proceedings of the 3rd international conference on Virtual execution environments, pages 169 179. ACM, 2007. [11] G. A. Gibson and R. Van Meter. Network attached storage architecture. Communications of the ACM, 43(11):37 45, 2000. [12] T. Hirofuchi, H. Ogawa, H. Nakada, S. Itoh, and S. Sekiguchi. A live storage migration mechanism over wan for relocatable virtual machine services on clouds. In Proceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, pages 460 465. IEEE Computer Society, 2009. [13] J. Katcher. Postmark: A new file system benchmark. Technical report, Technical Report TR3022, Network Appliance, 1997. [14] A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori. kvm: the linux virtual machine monitor. In Proceedings of the Linux symposium, volume 1, pages 225 230, 2007. [15] Y. Luo. A three-phase algorithm for whole-system live migration of virtual machines. In Proceedings of the 2007 Asian technology information program s (ATIP s) 3rd workshop on High performance computing in China: solution approaches to impediments for high performance computing, pages 172 173. ACM, 2007. [16] Y. Luo, B. Zhang, X. Wang, Z. Wang, Y. Sun, and H. Chen. Live and incremental whole-system migration of virtual machines using blockbitmap. In 2008 IEEE International Conference on Cluster Computing, pages 99 106. IEEE, 2008. [17] M. Mahalingam, D. Dutt, K. Duda, P. Agarwal, L. Kreeger, T. Sridhar, M. Bursell, and C. Wright. Virtual extensible local area network (vxlan): A framework for overlaying virtualized layer 2 networks over layer 3 networks. Technical report, 2014. [18] A. Mashtizadeh, E. Celebi, T. Garfinkel, M. Cai, et al. The design and evolution of live storage migration in vmware esx. In USENIX ATC, volume 11, pages 1 14, 2011. [19] A. J. Mashtizadeh, M. Cai, G. Tarasuk-Levin, R. Koller, T. Garfinkel, and S. Setty. Xvmotion: Unified virtual machine migration over long distance. In 2014 USENIX Annual Technical Conference (USENIX ATC 14), pages 97 108, 2014. [20] N. McKeown, T. Anderson, H. Balakrishnan, G. Parulkar, L. Peterson, J. Rexford, S. Shenker, and J. Turner. Openflow: enabling innovation in campus networks. ACM SIGCOMM Computer Communication Review, 38(2):69 74, 2008. [21] M.-K. Shin, K.-H. Nam, and H.-J. Kim. Software-defined networking (sdn): A reference architecture and open apis. In 2012 International Conference on ICT Convergence (ICTC), pages 360 361. IEEE, 2012. [22] T. Wood, K. Ramakrishnan, P. Shenoy, and J. Van der Merwe. Cloudnet: dynamic pooling of cloud resources by live wan migration of virtual machines. In ACM Sigplan Notices, volume 46, pages 121 132. ACM, 2011. [23] T. Wood, K. Ramakrishnan, P. Shenoy, J. Van der Merwe, J. Hwang, G. Liu, and L. Chaufournier. Cloudnet: Dynamic pooling of cloud resources by live wan migration of virtual machines. IEEE/ACM Transactions on Networking (TON), 23(5):1568 1583, 2015. [24] H. Xu, L. Xing, and R. Robidoux. Drbd: Dynamic reliability block diagrams for system reliability modelling. International Journal of Computers and Applications, 31(2):132 141, 2009. [25] J. Zheng, T. S. E. Ng, and K. Sripanidkulchai. Workload-aware live storage migration for clouds. In ACM Sigplan Notices, volume 46, pages 133 144. ACM, 2011. [1] Amazon elastic compute cloud. https://aws.amazon.com/ec2/?nc1=h ls. [2] Cisco active network abstraction. http://www.cisco.com. 928