Combined Server Rejuvenation in a Virtualized Data Center

Transcription

1 Combined Server Rejuvenation in a Virtualized Data Center Fumio Machida, Jianwen Xiang, Kumiko Tadano, and Yoshiharu Maeno NEC Knowledge Discovery Research Laboratories Kawasaki, Japan {f-machida@ab, j-xiang@ah, k-tadano@bq, y-maeno@aj}.jp.nec.com Abstract This paper presents a high-availability solution to a virtualized data center which consists of a cluster of physical servers and hosted virtual machines (VMs). VMs are typically generated through software called virtual machine monitor (VMM) running on physical servers. Both VMs and VMMs face the risks of software aging which are caused by agingrelated bugs in the software and may result in failures in long time execution. To prevent such failures proactively, we propose a combined server rejuvenation technique that performs VM rejuvenations simultaneously with VMM rejuvenation. In order to maximize the resource utilization of the data center, the placement of VMs in the data center is rearranged by live VM migration each time when VMM rejuvenation is performed. Through the simulation experiments, we show that the proposed technique can enhance the availability of VMs in a virtualized data center while achieving high resource utilization. Keywords-component; availability; data center; placement; rejuvenation; virtual machine. I. INTRODUCTION Virtualized data center is established on server virtualization technology which virtualizes hardware resources in physical servers to generate multiple execution environments called virtual machines (VMs). Server virtualization is often implemented by software called virtual machine monitor (VMM) such as Xen [1], VMware ESXi [2] and Microsoft Hyper-V [3], and used as an essential infrastructure of large-scale computing systems. Since both VMs and VMMs stand on software technology, software reliability in server virtualization has significant impacts on the availability of a virtualized data center. For long running software such as OS and VMM, the aging-related bug is one of the major causes of software failures. Aging-related bugs accumulate errors during software execution and may eventually cause unacceptable performance degradations or failures due to resource depletion. The phenomena of software aging are observed in a wide variety of software such as Unix OS [4], Linux [5], telecommunication system [6], Apache web server [7], and Apache Axis web application server [8]. In this paper, we also report a software aging problem observed in Xen and address the risk of software aging in both VMs and VMMs. Software rejuvenation has been proposed as an effective countermeasure to software aging which cleans up aging states by restarting the software execution environment [9]. Since it is difficult to remove aging-related bugs completely in development and testing phases, software aging can be observed after software release and deployment. After the deployment, identification and removal of the root cause of software aging might be cumbersome tasks. In some cases, users cannot even access the source code of the software. Software rejuvenation is applicable even in such cases as it simply restarts the software to mitigate the adverse effect of the aging. Although software rejuvenation involves downtime overhead due to restart, rejuvenation scheduling for maximizing the system availability has been studied extensively in the literature (e.g., [10][11][12][13][14]). In case of software rejuvenation to VMM, the downtime cost for hosted VMs also need to be considered because the executions of the hosted VMs depend on the underlying VMM. As presented in [14], Migrate-VM rejuvenation is useful to reduce the downtime of VMs caused by VMM rejuvenation using live VM migration as long as there is free capacity on the other hosts. In this paper, we propose a new rejuvenation scheduling technique which rejuvenates aged VMs simultaneously with VMM rejuvenation so as to minimize the downtime of the VMs within the limited capacity of a data center. The two objectives of our rejuvenation scheduling technique are i) improving the VM availability and ii) achieving high resource utilization in a virtualized data center. For data center owners, resource utilization is one of the major concerns associated with the total cost of ownership. We evaluate the resource utilization by the ratio of the total number of hosted VMs to the total capacity provided by available server resources. In Migrate-VM rejuvenation [14], some extra free capacities need to be prepared for live VM migration, but they are wasteful in terms of the resource utilization. Our proposed technique does not introduce any additional servers for VMM rejuvenation, and instead rearranges the VM placement to perform VM rejuvenations simultaneously with VMM rejuvenation within the limited capacity. The rejuvenation is triggered automatically following to the detection of software aging, and hence the proposed technique can provide an effective self-healing solution to a virtualized data center. We evaluate the proposed rejuvenation technique by simulation experiments. The rest of the paper is organized as follows. Section II presents an empirical observation of software aging in Xen. Section III describes a general model for virtualized data center. Section IV introduces the proposed rejuvenation scheduling technique for VMM with VMs. Section V shows the results of the simulation study which represent the effectiveness of the proposed approach. Section VI discusses the related work and Section VII gives our conclusion.

2 Free Disk Space [GB] Free memory [KB] monitoring scripts Domain 0 monitoring scripts Domain U Xen hypervisor Physical host server start/stop Domain U suspend/resume migration Time [minutes] Category CPU Memory Swap Disk Process File system Figure 1. Experimental system configuration II. TABLE I. METRICS OBSERVED Metrics User, System Free, Buffer, Cached Used, Free, Cached Free Space, Read Blocks, Write Blocks Running, Sleeping, Stopped, Zombie i-node SOFTWARE AGING IN XEN Xen is an open source implementation of VMM and is widely used for from desktop virtualization to virtualized infrastructure for cloud services. An aging-related bug in Xen has been reported in [15], which causes gradual decrease in available heap memory of Xen server by rebooting the hosted VMs repeatedly. The bug might lead to a serious performance degradation or server crash after long time execution. The authors also mention the risk of software aging in the privileged VM that is the special VM providing management capabilities for VMM. If the privileged VM fails due to software aging such as memory leak, it also affects the entire server including all the hosted VMs. This section reports another aging-related bug found in Xen and shows actual aging phenomena observed by experiments. In order to characterize the aging phenomena in a Xen server, an experimental system is prepared as shown in Figure 1. Xen has a privileged VM called Domain 0 and can generate some user VMs called Domain U. The host server installs Xen and a Domain U is created on it. Monitoring scripts are installed in both the Domain 0 and the Domain U to collect resource utilizations of the host server and the VM. The metrics observed by the monitoring script are shown in TABLE I. These metrics values do not change frequently as long as there are no operations to the host server or the Domain U. Then we conduct a set of accelerated tests to test whether aging phenomena are appeared in long lifetime. The acceleration test is carried out by invoking specific operations repeatedly in a short time. VM start/stop, VM suspend/resume and live VM migration are chosen as the accelerated operations. VM start/stop and VM suspend/resume are executed on another Domain U on the same host while live VM migration is performed between two different host servers. The acceleration tests are conducted for the two versions of Xen; Xen 3.0 and Xen 3.1. Through the experiments, we observe two suspected aging phenomena that can be accelerated by repeated Figure 2. Free memory aging by repeated live VM migrations Time [minutes] Figure 3. Disk space aging by repeated VM suspends/resumes operations. The first suspected aging trend is observed in available memory space in Xen 3.0 by the tests with repeated live VM migration. The size of free memory in Domain 0 gradually decreases as shown in Figure 2, where we execute live VM migration a hundred times during this period. The decreasing trend of the available memory is observed only in this test case. Hence, the memory aging is highly likely to be associated with live VM migration. The second aging trend is observed in free disk space in the host of Xen 3.1 by repeated VM suspend/resume operations. As shown in Figure 3, the free disk space in Domain 0 is monotonically decreasing. This is caused by a residual temporal file created each time VM is suspended. The temporary file should be removed after VM resume but the file is not cleared by resume operation as the bug is reported in [16]. In our experimental environment, 185MB of temporal file is generated each time when VM is suspended, and in total 18.5 GB of disk space is consumed during the experiments by a hundred times of VM suspend operations. This aging phenomenon is visible and seems to be easy to detect before causing any serious troubles. However, the temporal file is not appeared on the file system, and hence it is not easy for users to identify the root cause of gradual shrinkage of disk space. An important fact obtained from the above experimental observations is that the different version of Xen might contain different type of aging-related bugs. Software version-up does not guarantee the complete removal of aging-related bugs. Furthermore, as examined in the two above examples, it is not easy to characterize the aging phenomena associated with the root causes without experiments and investigations. These difficulties motivate us to take more lightweight but effective solutions that can be incorporated in the routine work of system administration.

3 Fortunately, both of the two observed aging phenomena can be cleared by restarting Xen. Software rejuvenation for VMM works to prevent possible causes of server failures in advance. When VMM is going to restart for rejuvenation, however, we need to care about the hosted VMs running on the VMM. One of the solutions to the running VMs is that the VMs are moved out from the host server using live VM migration prior to VMM rejuvenation. Live VM migration is available only when there is a sufficient capacity to receive the migrated VM in any other hosts. To precisely track the behavior of VMM rejuvenation using live VM migration in consideration with data center capacity, we present a virtualized data center model in the next section. III. VIRTUALIZED DATA CENTER MODEL Virtualized data center consists of a cluster of physical servers connected to the same network. These servers are virtualized by VMM for executing multiple VMs on top of them. To enable live VM migration in the data center, a shared storage system is equipped to store the VM images which can be accessed from any physical host servers. The storage system is connected to each server by a dedicated network connection such as FC-SAN (Fibre Channel-Storage Area Network). To simplify the following discussions, we focus on a homogeneous data center configuration which consists of physical servers having the same capacity and VMs reserving the same amount of resources (e.g., one vcpu, 1GB of RAM, 100GB of disk space etc.). Under the above assumptions, the VM placement of the datacenter can be represented by the placement matrix defined as where and. Since VM i is assumed to be unique in the data center, the matrix elements satisfy which restricts each VM is placed on at most one host server (i.e., VM is not replicated on multiple servers). On the other hand, the capacity limitation for a host server is expressed by where C represents the capacity of a host server in the number of VMs (i.e., each host server can host at most C VMs). As long as the above capacity condition is satisfied, a live VM migration can be represented by the replacement of the values of the corresponding matrix elements. For example, if VM i is migrated from host j to j, p ij is changed from 1 to 0 and p ij is changed from 0 to 1, respectively. If a server in the data center suffers from software aging in the VMM and is applied software rejuvenation, the state transitions of a host server can be modeled as shown in Figure 4. It starts from the UP state which represents a robust status of the server. After certain time period, the server enters into the Failure Probable (FP) state due to software aging in the VMM. If the sever fails caused by resource depletion, the state is changed to the Failed (F) state and then returns to the UP state after failure recovery. On the other hand, if the aging status is detected before a failure, we can perform software rejuvenation which leads to the Rejuvenation (RJ) state from the FP state. After the completion of VMM restart, it returns to UP state. In the host model, we focus on VMM failure caused by software aging and we do not consider other causes of failures such as hardware failure. The state transition model for software aging and rejuvenation is presented in [9] where all state transition times are assumed to be exponentially distributed. In this paper, the trigger of VMM rejuvenation is controlled by scheduling method in consideration with VM placement matrix. Therefore, the time to trigger VMM rejuvenation does not follow exponential distribution. The control of rejuvenation schedule is discussed in the next section. VMs also face the risks of software aging depending on the software running in the VMs, such as OS, middleware or user applications. Figure 5 shows the state transition model for a VM. In addition to the similar state transitions as of the VMM, the transition to the Down (DW) state is occurred when the underlying host server goes down. When the host server is down due to a failure or software rejuvenation, all the hosted VMs are forced to shut down and they cannot restart until the host server becomes available. In other words, the state transitions to the DW state in the VM model depend UP VMM aging VMM restart RJ Host recovery Host down UP VM aging VM restart RJ VMM recovery FP F VMM failure Trigger VMM rejuvenation DW Host down VM recovery FP F VM failure Trigger VM rejuvenation Figure 4. State transition diagram for host (VMM) Figure 5. State transition diagram for VM

4 on the state of the hosted server as shown in the host model. As the hosted relationships between VMs and host servers are represented by the VM placement matrix, the state transition for VM i can be specified by the state model for VM i and the corresponding state model for host j with p ij =1. In our data center model, live VM migration is not represented as a state of VM. Since live VM migration can be performed with a very short downtime [17] and VM state is not changed on the destination host, we neglect the state for live VM migration. Although we assume a general aging model for VM, the aging trends of VMs are highly-diverse in practice depending on the software running in the VMs. For instance, some applications have aging trends associated with its workload which are better to be captuered by some more explicit aging models [18][19]. Now, let us define VM state vector and define host state vector The state of a virtualized data center can be represented by the set. The state of the data center is changed in response to the VM state changes, the VMM state changes or the changes in the VM placement. IV. COMBINED SERVER REJUVENATION In this section, we propose a new rejuvenation scheduling technique for virtualized data center. Although live VM migration is useful to move the hosted VMs to other hosting server prior to VMM rejuvenation, it requires additional free capacity to migrate VMs. It is not cost effective to prepare such a redundant capacity only for rejuvenation purposes. For data center owners, maximizing the utilization of resources is one of the primary concerns. Thus we propose an efficient rejuvenation scheduling technique using live VM migration without introducing additional redundant servers. A. Approach The basic idea behind the proposed technique is to perform VM rejuvenations simultaneously at VMM rejuvenation. If both of VMs and VMMs suffer from software aging and require rejuvenations, the rejuvenation can be performed together to reduce the downtime caused by rejuvenations. The approach is similar to Cold-VM rejuvenation which forces all the hosted VMs to be shutdown at the time of VMM rejuvenation [14]. In Cold-VM rejuvenation, however, VMs are shutdown even though they are in robust states (i.e., they are not in the FP state). In contrast, our technique forces shutdown to only aged VMs which are to be rejuvenated in the near future. The robust VMs are moved out from the host server by live VM migration. To prepare the free capacity for live VM migrations, the proposed technique searches any aged VMs on other host servers and shut down them. The shutdown VMs are restarted on the rejuvenated host after VMM rejuvenation. The steps of the proposed technique are detailed as below. 1. Detection of VMM aging The procedure is initiated by the detection of a VMM aging. To detect the aging state of VMM, periodic resource monitoring and analysis for all VMMs are required. If a VMM is decided to be rejuvenated, proceed to the following steps. 2. Preparation for VMM rejuvenation First, the list of the hosted VMs on the VMM is collected. The list is easily obtained through the search of the column in VM placement matrix for the corresponding host. For each VM i in the obtained list, perform the following operations according to the state of VM i. a) if v i =FP Shutdown the VM i to clear aging state. b) if v i =F or v i =RJ Keep the VM i in down state until VMM rejuvenation completes. c) if v i =UP Search available free capacity in the data center to migrate the VM i. If there is available capacity on a host server j, reserve the capacity and perform live VM migration to move VM i to the host j. If no available capacity for migration is found, alternatively search any aged VMs on another host server in robust state. If the VM i is found as a search result, shutdown VM i and subsequently reserve the capacity used by VM i for the migration of VM i. When VM i is stopped, perform live VM migration of VM i to the reserved place. If any aged VMs are not found, shutdown VM i. 3. Restart VMM When all VMs on the VMM are shut down or moved out as a result of the previous step, restart the VMM to clean up the aging state. 4. Restart VMs When the VMM rejuvenation completes, restart all the VMs shut down in the step 2. The VMs shut down in the other hosts are also restarted on this host (i.e., the VM placement is changed during the VMM rejuvenation). In the step 2, the robust VMs which are in UP states could be shut down if there is no available capacity or aged VMs in any other host servers. Consequently, the proposed technique behaves the same as Cold-VM rejuvenation in the worst case. However, the proposed technique can reduce the unnecessary robust VM shutdowns caused by VMM rejuvenation and hence it improves the total availability of VMs without introducing any additional capacity for rejuvenation. B. An example senario Figure 6 shows an example scenario for the combined server rejuvenation. Suppose that there are three host servers

5 VM0 VM0 VMM0 host 0 VMM0 host 0 VM1 VM3 VM aging VM2 VMM1 host 1 VM3 VMM aging VM4 (a) Before combined server rejuvenation VM1 VMM1 host 1 VM4 (b) After combined server rejuvenation VMM2 host 2 VMM2 host 2 VM2 Figure 6. An example scenario for combined software rejuvenation (host0, host1 and host2) and each server can host two VMs at most. At a certain time instant, as depicted in Figure 6(a), two VMs are hosted in both of host0 and host1, and one VM is hosted on host2. When software aging in the VMM of host1 is confirmed, the procedure of combined server rejuvenation starts. The original state of the virtualized data center can be represented by In the step 2, VM2 and VM3 are found as hosted VMs on the host2. According to the data center state, both of the VMs are in UP states, and hence they need to migrate to other host servers. Following to the step 2-c), first available capacity for VM migration is searched and, as a result, an available capacity in the host2 is found. Then the capacity on the host2 is reserved for live VM migration of VM2. In the next search, no available capacity is found for VM migration of VM3. Therefore the procedure searches any aging VMs in the other host servers and reaches the VM1 as a candidate. When we shut down VM1, we obtain a new free capacity on host0 for live VM migration for VM3. After live VM migration of VM3 to host0, VMM rejuvenation in host1 can be started as in the step 3. Finally, in the step 4, downed VM1 is restarted on host1. After the combined server rejuvenation, data center configuration is changed as shown in Figure 6(b) and the state of the data center is presented as no robust VM (in UP state) is sacrificed during the above scenario. C. Limitation The proposed rejuvenation technique requires the knowledge of the states of VMs as well as VMMs in order to trigger the rejuvenations and migrations. It is feasible if all the VMs are owned by the same user. However, if the data center provides a hosting service on which VMs are leased to many users, the state of VMs may not be observable by the data center administrator. Even in such cases, the users who suffer from software aging in the VMs and require rejuvenation services can get the benefit of the proposed solution at the cost of the disclosure of their VM states. V. EXPERIMENTS In this section, we evaluate the proposed rejuvenation technique by discrete event simulations on the data center model. We call the proposed technique as combined server rejuvenation () and evaluate the system with in comparison to that without rejuvenation (WR), that with Cold-VM rejuvenation () and that with Migrate- VM rejuvenation (). The simulation program is implemented by Ruby, and it contains 1K lines of code in total. The program adopts event-oriented simulation in which events are processed in a sequence of simulated time. In, we assume that a shared standby server is prepared to guarantee the extra free capacity for live VM migration. The standby server is used to host VMs which are moved from a host server whose VMM is going to be rejuvenated. During the VMM rejuvenation, the standby server keeps the migrated VMs running. When the VMM rejuvenation completes, all the hosted VMs are returned on to the rejuvenated host server by live VM migration again. If two or more host servers need to be rejuvenated at the same time, the standby server is used in series. We assume that the standby server is also affected by aging and can fail like other host servers. In other words, works only when the standby server is in the UP state or the FP state. However, we do not consider VMM rejuvenation for the standby server in the experiments. A. Measures of interests In the experiments, we compute three measures; average VM availability, resource utilization, the number of VM migrations and the number of robust VM shutdowns. For VM availability, let T be the total time simulated in the experiments and u i be the total time VM i is in UP state or FP state. The average VM availability is defined by The resource utilization can be defined by the ratio of the number of VMs to the total capacity provided by the host servers in a virtualized data center. where all aging states are cleared and VM1 is replaced with VM3 from the original state. Unlike Cold-VM rejuvenation,

6 VM availability VM availability In our experiments, the number of hosts m is determined by the given set of VMs and the capacity C for each host server (i.e., n and C are given). For WR, and, the number of hosts are set to the minimum value given by For, the number of hosts is computed by where is the number of standby servers. The number of VM migrations and the number of robust VM shutdowns are counted in each experiment. The number of VM migrations indicates the cost of rejuvenation techniques using live VM migration because live VM migration might involve performance overhead in the data center. On the other hand, the number of robust VM shutdown measures the cost of VMM rejuvenation especially for in which the hosted VMs are forced to shut down by the corresponding VMM rejuvenation even though they are in robust states. B. Parameter values and assumptions TABLE II shows the default parameter values used in the experiments. Since the simulation focuses on the comparison of the rejuvenation techniques, we do not take into account the detailed resource allocation issues at VM placement. As modeled in Section III, we assume that each VM requires the same amount of resources and all the host servers have the same capacity (C=4). The state transition times in the VM/host model are assumed to follow exponential distributions with the rates specified by the values in TABLE II except for the rejuvenation trigger transitions. The trigger of rejuvenation is controlled by each rejuvenation technique and hence it depends on the state of the data center. The state can be observed by a periodic inspection whose interval is given by. If either a VM or a VMM is aged as presented in the FP state, a VM or VMM rejuvenation is triggered, respectively. The inspection interval is deterministic, while each rejuvenation is triggered based on some condition (i.e., condition-based rejuvenation). In the first experiment, we set to one day. The simulated time TABLE II. Default parameter values for simulation Symbol Parameter Value Simulated time length 5 years Variable Capacity of a host server 4 Number of standby hosts 1 Mean time to VM aging 1 month Mean time to VM failure 1 week Mean time for VM recovery 1 hour Mean time for VM restart 15 minutes Mean time to host aging 3 months Mean time to host failure 1 month Mean time for host recovery 2 hours Mean time for host restart 15 minutes Inspection interval 1 day WR Figure 7. Average VM availability by varying the number of VMs Inspection interval [days] Figure 8. Average VM availability by varying the inspection interval length is set to five years and we compute the average of each measure of interest over a hundred times of experiments. C. Average VM availability First, the average VM availability is computed by varying the number VMs from 10 to 100 and the results are shown in Figure 7. For all of the rejuvenation techniques, the average VM availabilities are improved by rejuvenations (compared with WR). Among three rejuvenation techniques, achieves the highest VM availability, marks the second highest with subtle difference from and follows them. The order is not changed by the number of VMs within the range of 10 to 100 (We will study larger cases later). The result indicates that improves the VM availability close to the performance of without introducing additional standby server for live VM migration. The VM availability is also affected by the length of the inspection interval. Figure 8 shows the average VM availabilities simulated in the different inspection intervals by fixing the number of VM to 100. As can be seen, the VM availability decreases according to the length of the inspection interval. Interestingly, the gap between and becomes smaller as the inspection interval becomes longer. When the inspection interval is set to 10 days, achieves the almost the same VM availability achieved by.

7 VM availability Resource utilization Number of live VM migrations Number of robust VM shutdowns (a) 1 (b) 4500 (c) Figure 9. (a) Resource utilization, (b) the number of VM migrations and (c) the number of robust VM shutdowns by varying the number of VMs D. Resource utilization For resource utilization, we only compare with because the utilizations for WR and Cold- VMR are the same to the results of. As shown in Figure 9(a), has an advantage in resource utilization compared to. The difference is significant especially when the number of VMs is small, which could be the case of small data centers. E. Number of VM migrations The number of VM migrations should be minimized in terms of performance overhead in the data center. Figure 9(b) shows the number of VM migrations counted in the experiments. As can be seen, the number of VM migrations by monotonically increases by the number of VMs. can reduces the number of VM migrations because live VM migration is used only when the hosted VM is in the UP state. F. Number of robust VM shutdowns One potential drawback of is the unnecessary shutdown of VMs during VMM rejuvenation. The drawback becomes clear by counting the number of robust VM shutdowns as shown in Figure 9(c). In Cold- VMR, a number of robust VMs are sacrificed by VMM rejuvenation. has an advantage in the reduction of the number of robust VM shutdowns. In terms of the number of robust VM shutdowns, is the best option in which no robust VMs are shutdown by VMM rejuvenation. G. Scalability is potentially a suitable rejuvenation technique for large-scale virtualized data center. In, it is essential to make combinations of aged VMs with aged VMMs as much as possible so as not to shut down robust VMs during VMM rejuvenation. We increase the number of VMs from 100 to 1000 and compute the VM availabilities by simulation. For reducing the evaluation time in this experiment, we conduct each experiment ten times and plot the average values. As shown in Figure 10, the average VM availability by improves as the number of VMs increases. When the number of VMs reaches 1000, achieves higher VM availability than. Since a large-scale virtualized data center might have thousands of VMs with the similar orders of VMMs, the system suffers from the same order of aged VMs and aged VMMs. can take an advantage of such a situation because it is more likely to see successful matching of aged VMs with aged VMM in a larger data center. VI. RELATED WORK Figure 10. Scalability benefit of in VM availability Software aging issue in server virtualization is first presented in [15]. The authors point out the risk of VMM failures caused by aging-related bugs in Xen, and propose a fast VMM rejuvenation technique using in-memory VM suspend mechanism. An analytic model for evaluating the effectiveness of the VMM rejuvenation technique using VM suspend is studied in [14]. Analytic models for the combination of VM rejuvenation and VMM rejuvenation are also presented in [20]. In this paper, we have presented a new rejuvenation technique using live VM migration under the consideration of VM placement and have studied the performance and scalability of the proposed technique. High-availability solutions for virtualized data center are not limited to software rejuvenation. VM high-availability is a commonly adopted quick reactive recovery technique for VMs at a host server failure and is supported in commercial VMM implementations [21]. Remus also supports an

8 effective recovery of VMs at a host server failure by asynchronous VM replication to a backup server [22]. ReHype provides a promising failure recovery technique for VMM which uses microreboot to recover the VMM while preserving the state of running VMs [23]. These techniques can be categorized as reactive recovery techniques for failures in host server or VMM, while our approach complementally provides a proactive solution which can be combined with the reactive recovery techniques. Virtual machine placement is another important design perspective for providing fault-tolerant virtualized data center. The optimum VM placement can be applied in the initial data center configuration [24]. When a data center encounters a component failure, the hosted VMs should be reconfigured to maintain their availabilities by regenerating VMs on an appropriate host server [25]. In consideration with software rejuvenation for VMs in a virtualized data center, mathematical programming based formulation for VM allocation problem is presented in [26]. In our experiments, redundancy or replication of VMs has not been taken into account but the proposed technique is applicable to such data center configuration as well. VII. CONCLUSION In this paper, we have presented a rejuvenation scheduling technique for a virtualized data center in which both VMs and VMMs confronting software aging are maintained proactively by software rejuvenation. Whenever a VMM rejuvenation is required, the proposed technique searches aged VMs to be rejuvenated together with the VMM and replace them with healthy VMs on the VMM by using live VM migration. As a result we can reduce the unnecessary shutdowns of robust VMs and hence improve the VM availability. The experimental results show that the proposed technique improves the VM availability close to the performance of without introducing any additional standby host servers. The proposed technique is potentially a suitable rejuvenation technique for large-scale virtualized data center as it achieves almost the same VM availability to when the data center consists of a thousand of VMs. REFERENCES [1] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, A. Warfield, Xen and the Art of Virtualization, In Proc. of 19th ACM Symp. on Operating Systems Principles (SOSP19), pp , [2] VMware ESXi, [3] Microsoft Hyper-V, [4] S. Garg, A. van Moorsel, K. Vaidyanathan, and K. S. Trivedi, A methodology for detection and estimation of software aging, In Proc. of Int'l Symp. Software Reliability Engineering (ISSRE98), pp , [5] D. Cotroneo, R. Natella, R. Pietrantuono, and S. Russo, Software Aging Analysis of the Linux Operating System, In Proc. of Int'l Symp. on Software Reliability Engineering (ISSRE2010), pp , [6] A. Avritzer and E. J. Weyuker, Monitoring smoothly degrading systems for increased dependability, Empirical Software Engineering, vol. 2, no. 1, pp , [7] M. Grottke, L. Lie, K. Vaidyanathan, and K. S. trivedi, Analysis of software aging in a web server, IEEE Trans. Reliability, vol. 55, no. 3, pp , [8] L. Silva, H. Madeira and J. G. Silva, Software aging and rejuvenation in a soap-based server, In Proc. of the 5th IEEE Int'l Symp. on Network Computing and Applications (NCA 2006), pp , [9] Y. Huang, C. Kintala, N. Kolettis, and N. D. Fulton, Software rejuvenation: Analysis, module and applications, In Proc. Int'l Symp. on Fault Tolerant Computing (FTCS 1995), pp , [10] S. Garg, A. Puliafito, M. Telek, and K. S. Trivedi, Analysis of software rejuvenation using Markov regenerative stochastic Petri nets, In Proc. of Int'l Symp. on Software Reliability Engineering (ISSRE 1995), pp , [11] T. Dohi, K. Goseva-Popstojanova, and K. S. Trivedi, Estimating software rejuvenation schedules in high assurance systems, Computer Journal, vol. 44, no. 6, pp , [12] K. Vaidyanathan and K. S. Trivedi, A Comprehensive Model for Software Rejuvenation. IEEE Trans. on Dependable Sec. Comput. vol. 2, no. 2, pp , [13] D. Wang, W. Xie, and K. S. Trivedi, Performability analysis of clustered systems with rejuvenation under varying workload, Performance Evaluation, vol.64, no. 3, pp , [14] F. Machida, D. Kim, and K. S. Trivedi, Modeling and Analysis of Software Rejuvenation in a Server Virtualized System, In Proc. of the 2nd Int'l Workshop on Software Aging and Rejuvenation, [15] K. Kourai and S. Chiba, Fast software rejuvenation of virtual machine monitor, In IEEE Trans. on Dependable Sec. Comput. vol. 8, no. 6, pp , [16] Bug Suspend/resume of a managed domain leaks checkpoint file descriptor, [17] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield, Live Migration of Virtual Machines, I n Proc. Symp. Networked Systems Design and Implementation (NSDI 2005), pp. 1-11, [18] P. Reinecke, and K. Wolter, A simulation study on the effectiveness of restart and rejuvenation to mitigate the effects of software aging, In Proc. of the 2nd Int'l Workshop on Software Aging and Rejuvenation (WoSAR 2010), [19] A. Bonvenzi, D. Cotroneo, R. Pietrantuono, S. Russo, Workload characterization for software aging analysis, In Proc. of 22nd Int'l Symp. on Software Reliability Engineering (ISSRE 2011), pp , [20] A. Rezaei and M. Sharifi, Rejuvenation high available virtualized systems, In Proc. of 5th Int'l Conf. on Availability, Reliability and Security (ARES 2010), pp , [21] VMware, VMware Infrastructure Automating High Availability (HA) Services with VMware HA, VMWARE Technical Note, [22] B. Cully, G. Lefebvre, D. Meyer, M. Feeley, N. Hutchinson, and A. Warfield, Remus: High Availability via Asynchronous Virtual Machine Replication, In Proc. of 5th Symp. on Networked Systems Design and Implementations (NSDI2008), pp , [23] M. Le, Y. Tamir, ReHype: Enabling VM Survival Across Hypervisor Failures, In Proc. of 7th ACM Int'l Conf. on Virtual Execution Environments (VEE11), pp , [24] F. Machida, M. Kawato, Y, Maeno, Redundant virtual machine placement for fault-tolerant consolidated server clusters, In Proc. of 12th IEEE/IFIP Network Operations and Management Symposium (NOMS2010), pp [25] G. Jung, K. R. Joshi, M. A. Hiltunen, R. D. Schlichting, C. Pu, Performance and Availability Aware Regeneration For Cloud Based Multitier Applications, In Proc. of Int'l Conf. on Dependable Systems and Networks (DSN2010), pp , [26] J. Alonso, I. Goiri, J. Guitart, R. Gavalda and J. Torres. Optimal resource allocation in a virtualized software aging platform with software rejuvenation, In Proc. of the 22nd Int'l Symp. on Software Reliability Engineering (ISSRE 2011), pp , 2011.