Evaluation and improvements of I/O Scalability for Xen. Jun Kamada, Fujitsu Limited Simon Horman, VA Linux Systems Japan

Transcription

1 Evaluation and improvements of I/O Scalability for Xen Jun Kamada, Fujitsu Limited Simon Horman, VA Linux Systems Japan

2 Agenda Part : I/O Scalability Improvements (Bandwidth Control) Part2 : I/O Scalability Evaluation

3 Part I/O Scalability Improvements (Bandwidth Control) 2

4 Background Bandwidth control is one of the means to provide high I/O scalability. It can also provide safety for almost all guests against malicious guest who try to occupy all bandwidth of I/O. We have Implemented bandwidth (b/w) control feature using Intel NIC s b/w control mechanism and SR-IOV on Xen. Evaluated the effect of the feature briefly. 3

5 Bandwidth Control using SR-IOV Assign each Virtual Functions (VF) virtualized by SR-IOV to different guests using VT-d. Set b/w parameters (e.g. throughput) for each VF through PF. NIC s b/w mechanism controls throughput for each VF according to the settings. Dom0 Guest # Guest #2 Guest #3 PF Driver VF Driver VF Driver VF Driver Hypervisor (Xen) B/w Setting Physical Function (PF) SR-IOV NIC Virtual Function (VF) VF B/w Control Mechanism VF Direct Assign by VT-d Physical Port 4

6 Implementation (/2) Bandwidth Control of Intel GbE VM Bandwidth Allocation Config - VMBAC (0x3608; RW) Bandwidth for each VF can be set through this register. Intel Gigabit Ethernet Controller Datasheet 5

7 Implementation (2/2) Added support to the igb driver to access the hardware bandwidth allocation support of the This has not been merged but is publicly available 6

8 How to use. Enable SR-IOV Add pci=assign-busses on kernel boot option 2. Create Virtual Functions (VFs) # modprobe igb max_vfs=3 3. Assign VFs to pciback driver # echo -n 0000:02:0.0 > /sys/bus/pci/drivers/pciback/new_slot # echo -n 0000:02:0.0 > /sys/bus/pci/drivers/pciback/bind 4. Set Bandwidth Parameter to NIC # echo " > /sys/class/net/eth/device/ bandwidth_allocation ; 600, 300 and 00 are throughput for st Virtual Function ; (VF#), VF#2 and VF#3 respectively by Mbps. 5. Allocate VFs to Guests using VT-d Passthrough 7

9 Evaluation Environment Create three guests which VF is passthroughed. Allocate 600, 300 and 00Mbps of bandwidth for each guests respectively. Execute netperf with various message size for sending packets on each guests. Measure throughput of each guests. Guest # Guest #2 Guest #3 netperf netperf netperf Xen Direct I/O using VT-d Intel GbE (82576) Physical Function (PF) Virtual Function(VF) # (weight = 600 Mbps) VF #2 (weight = 300 Mbps) VF #3 (weight = 00 Mbps) 8

10 Evaluation Results (b/w Control Disabled) Throughput (Mbps) Guest #3 Guest #2 Guest # Message Size (Bytes) Three guests share the bandwidth fairly. 9

11 Evaluation Results (b/w Control Enabled) Throughput (Mbps) Guest #3 Guest #2 Guest # Message Size (Bytes) Three guests share the bandwidth according to the settings (6:3:). 0

12 Conclusion Throughput for each guests are controlled very well by NIC s bandwidth mechanism. It can provide safety on virtualized environment in order that some guest s behavior does not affect another guest. Very usable for physical server consolidation, such as cloud computing.

13 Part 2 I/O Scalability Evaluation 2

14 Background Motivation: I/O scalability is key point for physical server consolidation. What should be clarified: How many server can be consolidated without I/O performance degradation? Assumed Situation: Consolidation of DB server aimed for mission critical environment 3

15 Evaluation Environment DB Server DB Server # #2 DB Server #n Consolidate Dom0 (RHEL5.3 UP TO xen) Guest # Guest #2 DB Server DB Server Guest #n DB Server VBD BE VBD FE VBD FE VBD FE RHEL5.3 (PV and HVM) 4 Cores for Dom0 Xen3.4.0 Tigerton(Xeon7350), 2.93GHz, 4Sockets, 6Cores (FSB:066MHz, L2=8MB, TDP=30W, Mem=64GB, DDR2-667ECC) 2 Cores for guests FC Switch 384 Disks used SAN RAID5(sys) RAID0(data) HUB Traffic Server # for OLTP Traffic Server #2 for OLTP Issue OLTP Req. 4

16 Evaluation Models Test A: (WH=80 USER=6) each guest (DB) is adjusted to use 33% of cpu. vcpus=, mem=024, not pinned, no cpu_cap, # of guests are, 2, 4, 8, 6, 24, 32 Dom0=4core(pinned), DomU (all)=2core Test B: (WH=80 USER=) each guest (DB) is adjusted to use 0% of cpu. vcpus=, mem=024, not pinned, cpu_cap=33% # of guests are, 2, 4, 8, 6, 24, 36, 42 Dom0=4core(pinned), DomU (all)=2core 5

17 Results of test A (/2) Expectation: The total throughput will be increased linearly for about 36 guests. (each guest use 33% of CPU, so 2cores x (00% / 33%) = 36 guests.) Results: a. Total tpm was saturated at 6 guests and Dom0 s CPU load was so high. b. CPU time per transaction increased by about 2 times at 6 guests. Guests (a.) tpm (total) PV Guests dom0 CPU% (Xentop Total ) Max is 400% domu CPU TIME PER Transaction (μsec) dom0 (b.) Max is 200% domu Guests (a.) tpm (total) dom CPU% (Xentop Total ) Max is 400% HVM Guests domu CPU TIME PER Transaction (μsec) dom0 (b.) Max is 200% domu

18 Results of test A (2/2) PV was about 25% better than HVM on Test A. (Dom0 + DomU) DomU Dom0 7

19 Results of test B (/2) Expectation: The total throughput will be increased linearly for over 42 guests. (each guest use 0% of CPU, so 2core x (00% / 0%) > 42 guests) Results: a. Total tpm was approaching saturated at 36 guests. b. CPU time of DomU/Dom0 increased by about 2 times since 24 guests. Guests (a.) tpm (total) CPU% (Xentop Total ) dom Max is 400% PV Guests domu CPU TIME PER Transaction (μsec) dom0 (b.) Max is 200% domu Guests (a.) tpm (total) dom Max is 400% HVM Guests CPU% (Xentop Total ) domu CPU TIME PER Transaction (μsec) (b.) Max is 200% dom domu

20 Results of test B (2/2) PV was about 30% better than HVM on TestB. (Dom0 + DomU) DomU Dom0 9

21 Performance Analysis (/3) There are no I/O bottleneck between Dom0 and SAN storage, however we encountered an IO bottleneck on DomU since 6 guests. The result data seems: a. DomU is waiting for I/O completion. b. Even though CPU usage was not saturated, there were many waiting processes on Dom0. iostat of DomU (Test A) # of guests rkb/s wkb/s avgqu-sz await IO bottleneck occurred svctm %util sar of Dom0 (Test A) runq-sz (a.) (b.) 0 CPU WAIT occurred

22 Performance Analysis (3/3) It seems the main cause of the overhead is spinlock. Oprofile results TOP0 % app name symbol name 6Guest (WH=80,USER=6) samples 4Guest (WH=80,USER=6) samples The ratio of 4Guests was 7.20 xen-syms _spin_lock lpfc /lpfc xen-syms _spin_trylock xen-syms flush_area_mask vmlinux _spin_lock_irqsave vmlinux schedule xen-syms do_grant_table_op vmlinux.text.lock.spinlock xen-syms csched_schedule xen-syms pre_flush Test A and Guests = HVM 2

23 Evaluation for Nehalem-EP 22

24 Evaluation Environment 2 (Nehalem-EP) DB Server DB Server # #2 DB Server #n Consolidate Dom0 (CentOS5.3 UP TO xen) Guest # Guest #2 Guest #n DB Server DB Server DB Server VBD BE VBD FE VBD FE VBD FE RHEL5.3 (PV) 2 Cores for Dom0 Xen3.4.0 Nehalem-EP(Xeon E5520), 2.27GHz, 2Sockets, 8Cores (FSB:066MHz, L3=8MB, TDP=80W, Mem=72GB,DDR3-066) 6 Cores for guests FC Switch 384 disks used SAN RAID5(sys) RAID0(data) HUB Traffic Server # for OLTP Traffic Server #2 for OLTP Issue OLTP Req. 23

25 Evaluation Models for Nehalem-EP Test A: (WH=80 USER=6) Give the same CPU load as Test A Each guest (DB) is adjusted to use 30% of CPU. vcpus=, mem=024, not pinned, no cpu_cap, # of guests are, 2, 4, 8, 6, 24, 32 Dom0=2core(pinned), DomU (all)=6core Supplement Hyper Threading = off Enhanced Speedstep = off 24

26 Results and Comparison (/2) Expectation: The total throughput will be increased linearly for about 20 guests. (each guest use 30% of CPU, so 6core x (00% / 30%) = 20 guests.) Results & Comparison a. The total tpm was approaching saturated at 6 guests but Dom0 s CPU load were much lower than Tigerton and no CPU wait occurred. b. The overhead on DomU was Greatly improved on Nehalem-EP and almost no overhead on Dom0. Nehalem-EP (Xeon E GHz 2socket8core) Guests (PV) TPM (total) Dom0 (a.) runq-sz (sar) Dom0 Max is 200% CPU% (xentop total) DomU Max is 600% cpu time per trasaction (usec) Dom D o m U (b.) 25 Tigerton (Xeon GHz 4socket6core) TPM (total) (a.) runq-sz (sar) Dom CPU % (xentop total) Dom Max is 400% DomU Max is 200% cpu time per trasaction (usec) Dom D o m U (b.)

27 Results and Comparison (2/2) If there are enough CPU resources for DomU, tpm may be increased linearly since 6 guests. (in case of Nehalem-EP) DomU 26 Dom0 Almost no overhead on Dom0.

28 Consideration According to measurement results, the 8 cores Nehalem provide good scalability than the 6 cores Tigerton on xen Tigerton saturated at 6 guests, however Nehalem seemed to scale over 6 guests. (if there are enough CPU for guests.) We are very interested in how the results will be on Nehalem-EX, which have more CPU cores. Nehalem-EP is more green than Tigerton. Total TDP of Tigerton is 520W, however that of Nehalem-EP is 60W. Dom0 performance bottleneck will affect the whole scalability performance of DomUs. Why the Dom0 bottleneck occurred is under investigation. 27

29 This work was partly funded Ministry of Economy, Trade and Industry (METI) of Japan as the Secure Platform Project of Association of Super-Advanced Electronics Technologies (ASET). 28