Modeling VMware ESX Server Performance A Technical White Paper. William L. Shelden, Jr., Ph.D Sr. Systems Analyst

Transcription

1 Modeling VMware ESX Server Performance A Technical White Paper William L. Shelden, Jr., Ph.D Sr. Systems Analyst

2

3 Modeling VMware ESX Server Performance William L. Shelden, Jr., Ph.D. The Information Systems Manager, Inc. The performance implications of consolidating Windows systems under VMware ESX Server are analyzed using a simulation model. First VMware ESX Server overhead is measured based on data from Windows systems running as virtual machines under VMware ESX Server. Then a simulation model is used to estimate the contention for the physical CPUs by the virtual CPUs being dispatched from the Windows virtual machines. The simulation results are compared to some benchmark runs to validate the assumptions of the simulation model. Introduction There is currently a great deal of interest in the notion of virtualization of computing system hardware. For example, in the June 20, 2005 edition of Business Week, an article entitled A Virtual Revolution quotes a vice-president of a large Midwest bank holding company as saying that he expects to save $10 million over the next five years as the result of a major consolidation of hundreds of web servers and other applications based on virtualization software. One software product available to virtualize X86 hardware is ESX Server from VMware, an EMC company. ESX Server transforms physical systems into a pool of logical computing resources. Operating systems and applications are isolated in multiple virtual machines that reside on a single physical server. System resources are dynamically allocated to virtual machines based on need and administrator set guarantees, providing mainframe-class capacity utilization and control of server resources. ( eatures.html) ESX Server thus creates an environment of multiple virtual machines, each running a copy of an operating system and its applications, sharing the physical resources of a single physical server. The virtualization of the physical resources is intended to be completely transparent to the operating system and its applications running inside a virtual machine. Two performance issues come to mind when considering this environment: (1) the CPU overhead required to manage the virtualization of the physical resources and (2) the potential for contention for the physical CPUs residing in the physical server among the virtual CPUs residing in the virtual machines. VMware ESX Server CPU Overhead The CPU overhead required to manage the virtualization of the physical resources was measured in a number of VMware ESX Server environments at a number of different utilization levels. The CPU overhead referred to here is the difference between the total utilization of the processor and the sum of the utilization of the virtual machines. As such, it includes the CPU utilization of the Service Console virtual machine. The results are shown in Figure 1 where the overhead CPU utilization for VMware ESX Server is graphed against the total CPU utilization of the physical server. A straight line fitted through the data has equation: y=0.05x We conclude from this analysis that the overhead CPU utilization for VMware ESX Server is generally between 1% and 6% and is a linear function of the total CPU utilization of the physical server.

4 VMware ESX Server Overhead v. Total CPU Utilization Measured Fitted Overhead % Utilization Total CPU % Utilization Figure 1 ESX Server Overhead Contention for the Physical CPUs in the Physical Server When an operating system like Windows or Linux that is running in a virtual machine under VMware ESX Server dispatches work on a CPU, it is really dispatching work on a virtual CPU. Before that work can actually execute instructions on a physical CPU, VMware ESX Server must dispatch the virtual CPU onto a physical CPU. If there are more virtual CPUs in the environment than physical CPUs (as will almost always be the case), there is the potential that the virtual CPU may be delayed before it is dispatched onto a physical CPU. It is also clear that the frequency and duration of such delays will be a function of the number of physical CPUs in the physical server, the number of virtual machines and the number of their virtual CPUs running under VMware ESX Server and the demand for CPU resource presented by the virtual machines. It is also clear that any such delays experienced by an operating system having a virtual CPU dispatched on a physical CPU will be transparent to that operating system. For example, a unit of work dispatched on a virtual CPU by a Windows operating system that takes 0.10 seconds to execute but that experiences 0.05 seconds of delay in being dispatched on a physical CPU, will appear to the Windows system that dispatched it to have taken = 0.15 seconds. As a consequence, in this situation the CPU utilization seen by the Windows operating system will be higher than the actual CPU utilization. To see this assume the rate of CPU completions described above is 10 per second on a server with 2 physical CPUs. Then the real CPU utilization = 10 x 0.10 / 2 x 100% = 50%, but the CPU utilization seen by the Windows operating system = 10 x 0.15 / 2 x 100% = 75%. This is just a consequence of the fact that the delay in dispatching the virtual CPU onto a physical CPU is not seen by the Windows operating system. A similar situation exists in a logically partitioned mainframe system where the logical processors of multiple logical partitions compete for the physical processors of the mainframe. A model of this environment was suggested by Buzen and Shum in [1]. They applied this model to estimating the degradation experienced by individual logical partitions as a result of CPU contention generated by other partitions. This conceptual model can be applied to model the contention for the physical CPUs in a VMware ESX Server environment.

5 The Simulation Model A simulation model was built to model the contention for the physical CPUs in a physical server among the virtual CPUs of the virtual machines running under ESX Server. See Chapter 7 of [2] for a discussion of simulation modeling. In the simulation model each virtual machine is represented by a class of customers. The population of each class of customers is the number of virtual CPUs in the virtual machine represented by the class. If there are N classes of customers in the model (i.e. N virtual machines), there are N+1 service centers in the model. The first service center is the CPU service center and the other N service centers are wait centers in the model, one for each class in the model. The inputs to the model consist of target CPU utilizations for each virtual machine. The mean of the exponential service time distribution at the CPU resource is set to 0.1 second for each virtual CPU and the model is solved iteratively varying the time spent at the wait centers until the target utilizations are achieved. The number of virtual CPUs in a virtual machine is currently limited to two by VMware ESX Server. The terminology used when a virtual machine has two or more virtual CPUs is virtual SMP. A future release of ESX Server will support more than two virtual CPUs per virtual machine. Benchmark Design Before using the simulation model to estimate the performance impact of consolidating Windows systems running under VMware ESX Server, nine simple benchmarks were run. The plan was to run the nine benchmarks and then use the VMware ESX Server measured virtual machine CPU utilizations as input to the model comparing the model s estimate of Windows CPU utilization in each virtual machine to that of the benchmark runs. In each benchmark, two virtual machines (VM-1 and VM-2) were configured to run under VMware ESX Server. Each virtual machine ran a copy of Windows Server 2003 Standard Edition. In each Windows system, two threads were run that were capable of varying the amount of CPU time they used from one benchmark run to another. These threads measured the amount of CPU time they used as well as counted the number of times a specific section of code was executed. This latter metric is used as a measure of throughput. The nine benchmarks were all run on a Dell PowerEdge 2550 (Pentium III 100 MHz) with two physical CPUs. The nine benchmarks are described as: 1. VM-1 executed two threads each trying to execute 100% of the time. VM-2 was powered OFF. VM- 1 and VM-2 each had two virtual CPUs. This benchmark tried to drive each of VM-1 s two virtual CPUs to 100% busy with VM-2 powered OFF. 2. VM-1 executed two threads each trying to execute 100% of the time. VM-2 was powered ON, but did not execute either thread. VM-1 and VM-2 each had two virtual CPUs. This benchmark tried to drive each of VM-1 s virtual CPUs to 100% busy with VM-2 powered ON, but idle. 3. VM-1 executed two threads each trying to execute 100% of the time and VM-2 executed two threads each trying to execute 100% of the time. VM-1 and VM-2 each had two virtual CPUs. This benchmark tried to drive each of VM-1 s two virtual CPUs and each of VM-2 s two virtual CPUs to 100% busy. 4. VM-1 executed two threads each trying to execute 50% of the time and VM-2 executed two threads each trying to execute 50% of the time. VM-1 and VM-2 each had two virtual CPUs. This benchmark tried to drive each of VM-1 s two virtual CPUs and each of VM-2 s two virtual CPUs to 50% busy. 5. VM-1 executed two threads each trying to execute 50% of the time. VM-2 was powered ON, but did not execute either thread. This benchmark tried to drive each of VM-1 s two virtual CPUs to 50% busy with VM-2 powered ON, but idle. 6. VM-1 executed two threads each trying to execute about 55% of the time. VM-2 was powered ON, but did not execute either thread. VM-1 and VM-2 each had two virtual CPUs. This benchmark tried to drive each of VM-1 s two virtual CPUs to about 55% busy with VM-2 powered ON, but idle. 7. VM-1 executed two threads each trying to execute about 55% of the time and VM-2 executed two threads each trying to execute about 55% of the time. This benchmark tried to drive each of VM- 1 s two virtual CPUs and each of VM-2 s two virtual CPUs to about 55% busy. 8. VM-1 executed one thread trying to execute 100% of the time and VM-2 was powered OFF. VM-1 and VM-2 each had one virtual CPU. This benchmark tried to drive VM-1 s one virtual CPU to 100% busy with VM-2 powered OFF.

6 9. VM-1 executed one thread trying to execute 100% of the time and VM-2 executed one thread trying to execute 100% of the time. This benchmark tried to drive VM-1 s one virtual CPU and VM-2 s one virtual CPU to 100% busy. Table 1 summarizes the characteristics of each of the benchmark runs. The last column in the table, titled Chart Legends contains character strings that are used in the legends of charts describing the results of the benchmark runs. Physical Virtual Virtual CPUs Total Virtual VM-1 Target VM-2 Target VM-2 Idle No. CPUs Machines per VM CPUs % Utilization % Utilization Status Chart Legends OFF 2x100% OFF ON 2x100% ON x100% 2x100% x50% 2x50% ON 2x50% ON ON 2x55% ON x55% 2x55% OFF 1x100% OFF x100% 1x100% Table 1 Benchmark Descriptions Terminology In order to accurately discuss the results of the benchmark and model runs to follow, we define the terminology used to describe the measurements both from the benchmark runs and the model runs. From the benchmark runs: Time = Seconds of benchmark run time For a particular virtual machine: Throughput = Number of executions / Time where an execution is the execution of a particular section of code in the benchmark driver program. Throughputs are shown as relative throughputs by dividing them by the throughput in the first benchmark. VMware CPU Utilization = (Number of used CPU secs) / (Time x No. Phys. CPUs) x 100% Windows CPU Utilization = 100 Idle% where Idle% is the average of the Idle% on each CPU the Windows system sees From the model runs: Time = Seconds of simulated time For a particular virtual machine: Throughput = Completions of service at the CPU / Time VMware CPU Utilization = CPU Busy time / (Time x No. Phys. CPUs) x 100% Windows CPU Utilization = (CPU Busy + Queue Time) / (Time x No. Phys. CPUs) x 100%

7 Benchmark Results VMware CPU Utilizations and Throughputs The results of the nine benchmark runs are summarized in the charts in Figures 2 and 3 below. The chart in Figure 2 shows the VMware CPU utilizations of the two virtual machines, VM-1 and VM- 2, as well as the overhead VMware CPU utilization. As indicated in the definitions above, the VMware CPU utilizations shown are of the 2 CPU physical server. The chart in Figure 2 also shows the relative throughput for each benchmark graphed against the right-hand y-axis. VMware CPU Utilizations and Throughputs VM-1 VM-2 Overhead Relative Throughput % Utilization Relative Throughput x100% OFF 2x100% ON 2x100% 2x100% 2x50% 2x50% 2x50% ON 2x55% ON 2x55% 2x55% 1x100% OFF 1x100% 1x100% Benchmarks Figure 2 Benchmark VMware ESX Server CPU Utilizations and Throughputs

8 Windows CPU Utilizations The chart in Figure 3 shows the Windows CPU utilizations of the two virtual machines, VM-1 and VM- 2. The Windows CPU utilizations shown on this chart are utilizations of 2 CPUs even in the last two benchmarks where VM-1 and VM-2 each had only one virtual CPU. Note that on this chart in the third benchmark called 2x100% 2x100% that each of the Windows system running in VM-1 and VM-2 respectively perceive that they are using 100% of 2 CPUs. This, of course, is impossible. The explanation for this is that for about half of the time that each of the virtual machines VM-1 and VM-2 think they are using 100% of 2 CPUs, they have been preempted by VMware ESX Server who has given the CPU resource to the other virtual machine or is using it for overhead processing. Benchmark Windows CPU Utilizations VM-1 VM % Utilizations x100% OFF 2x100% ON 2x100% 2x100% 2x50% 2x50% 2x50% ON 2x55% ON 2x55% 2x55% 1x100% OFF 1x100% 1x100% Benchmarks Figure 3 Benchmark Windows CPU Utilizations

9 Simulation Model Runs Next the series of nine VMware ESX Server environments benchmarked above were modeled with the simulation model. In each model, the physical server had 2 physical CPUs. There were two virtual machines defined in each model with a third virtual machine representing the VMware ESX Server overhead. The VMware CPU utilizations for each of the three virtual machines measured in the associated benchmark run were used as target CPU utilizations in the model run. The chart in Figure 4 shows the VMware CPU utilizations measured in the benchmark runs and modeled in the model runs. Since the measured benchmark values were inputs to the model runs, it is not surprising that the modeled values match the measured values very closely. The values for the virtual machine VM-2 are not shown but were similarly close. VMware CPU Utilizations for VM-1 Measured Modeled % Utilization x100% OFF 2x100% ON 2x100% 2x100% 2x50% 2x50% 2x50% ON 2x55% ON 2x55% 2x55% 1x100% OFF 1x100% 1x100% Benchmarks/Model Runs Figure 4 VMware CPU Utilizations for VM-1

10 Dispatch When Ready The chart in Figure 5 shows the Windows CPU utilizations measured in the benchmark runs and modeled in the model runs. Note here that the modeled values do not match the measured values very well except for the first two and last two runs. It is suggested that the reason that the modeled values of Windows CPU utilization do not match the measured values is that the model is not dispatching virtual CPUs the same way that VMware ESX Server dispatches them. In particular, the model dispatches a virtual CPU when one is ready and there is a physical CPU available. We refer to this in the chart subtitle as Dispatch When Ready. VMware ESX Server, on the other hand, dispatches virtual CPUs differently when a virtual machine has more than one virtual CPU (i.e. the virtual machine is operating in virtual SMP mode). In particular, in this case VMware ESX Server dispatches virtual CPUs two at a time. In other words, for a virtual CPU from a particular virtual machine to be dispatched on a physical CPU, either (1) another virtual CPU from the same virtual machine must already be dispatched or (2) there must be two physical CPUs available on which to dispatch. An option was implemented in the simulation model to dispatch in this way and is referred to as Dispatch in Pairs. Windows CPU Utilizations for VM-1 Measured v. Modeled (Dispatch When Ready) Measured Modeled % Utilization x100% OFF 2x100% ON 2x100% 2x100% 2x50% 2x50% 2x50% ON 2x55% ON 2x55% 2x55% 1x100% OFF 1x100% 1x100% Benchmarks/Model Runs Figure 5 Windows CPU Utilizations for VM-1 (Dispatch When Ready)

11 Dispatch in Pairs The chart in Figure 6 shows the Windows CPU utilizations measured in the benchmark runs and modeled in the model runs when the simulation model was using its option to Dispatch in Pairs. Note here that the modeled values do match the measured values reasonably well. Keeping in mind that the Windows CPU utilizations reflect some contention time for the physical CPUs as well time actually spent executing on the physical CPUs, it is clear that the higher Windows CPU utilizations obtained in the Dispatch in Pairs model runs reflect more contention for the physical CPUs. It is suggested that this is because in a Dispatch in Pairs model run, there will be times when one physical CPU is idle and there is ready work to do. This will occur when a virtual CPU from one virtual machine is dispatched, the other virtual CPU from the same virtual machine is not ready and there is at least one virtual CPU from the other virtual machine that is ready. Windows CPU Utilizations for VM-1 Measured v. Modeled (Dispatch in Pairs) Measured Modeled % Utilization x100% OFF 2x100% ON 2x100% 2x100% 2x50% 2x50% 2x50% ON 2x55% ON 2x55% 2x55% 1x100% OFF 1x100% 1x100% Benchmarks/Model Runs Figure 6 Windows CPU Utilizations for VM-1 (Dispatch in Pairs) Observations Quantifying CPU contention experienced by a virtual machine We can use the measures of CPU utilization from the VMware and Windows perspectives respectively to quantify the amount of CPU contention experienced by a particular virtual machine. For a particular virtual machine, let, ST V = Average CPU service time from VMware point-of-view QT V = Average CPU queue time from VMware point-of-view U V = CPU utilization from VMware pointof-view L = CPU throughput (same from both VMware and Windows perspectives) ST W = Average CPU service time from Windows point-of-view U W = CPU utilization from Windows pointof-view Now,

12 Also, So, ST W = ST V + QT V U V = L x ST V U W = L x ST W ST W / ST V = (U W /L) / (U V /L) = U W /U V ST W / ST V = (ST V + QT V ) / ST V (ST V + QT V ) / ST V = U W /U V 1 + (QT V / ST V ) = U W /U V QT V = ST V x ((U W /U V ) 1) We can interpret this last formula as follows: to obtain 1 second of real CPU service time from VMware (ST V =1.0), the virtual machine experiences QT V = (1 x (U W /U V ) 1) seconds of CPU queue time. For example, in our third benchmark run ( 2x100% 2x100% ) for virtual machine VM-1 we have U V =47.21% and U W =100%. Therefore, in this case, QT V = (100/47.21) 1 = 1.12 seconds. This says that in this run, virtual machine VM-1 experiences 1.12 seconds of CPU queue time on average in order to get 1.0 second of real CPU time. Conclusions 1. There is a minimal amount of overhead CPU time required by VMware ESX Server required to manage its environment. Our somewhat limited analysis suggests that overhead CPU utilization for VMware ESX Server is generally between 1% and 6% and is a linear function of the total CPU utilization of the physical server. 2. There is the potential in a VMware ESX Server environment for contention among virtual machines for the physical CPUs. This contention is experienced by the virtual CPUs being dispatched from operating systems running in virtual machines under VMware ESX Server. Similar potential contention for the physical CPUs also exists in a system running Windows natively. For Windows systems running in virtual machines under VMware ESX Server, this contention can be quantified by comparing the Windows CPU Utilization to the VMware CPU utilization of the virtual machine. 3. The approach of the analysis and modeling undertaken in this paper is based on the assumption that the CPU utilization reported from a guest Windows system can be interpreted as measuring both the CPU time that the Windows system actually used plus the time that the Windows system was preempted by VMware ESX Server or other virtual machines running under VMware ESX Server. Due to certain timekeeping issues within a virtual machine in a VMware ESX Server environment, this is not always the case. See the VMware whitepaper in [3], especially the section on page 17 titled Time Measurements Within a Virtual Machine for further discussion and clarification of this issue. References 1. Buzen, Jeffrey P., Shum, Annie W., CPU Contention in Logically Partitioned Systems, CMG Proceedings, Sauer, Charles H., Chandy, K. Mani, Computer Systems Performance Modeling, Prentice-Hall, VMware Whitepaper, Timekeeping in VMware Virtual Machines,

13