Kemari: Virtual Machine Synchronization for Fault Tolerance using DomT

Size: px

Start display at page:

Download "Kemari: Virtual Machine Synchronization for Fault Tolerance using DomT"

Dennis Jennings
5 years ago
Views:

1 Kemari: Virtual Machine Synchronization for Fault Tolerance using DomT Yoshi Tamura NTT Cyber Space Labs. 2008/6/24

2 Outline Our goal Design Architecture overview Implementation Evaluation Conclusion Brush up on Xen Summit 2007 Copyright Nippon Telegraph and Telephone Corporation 2

3 What is Kemari? (Kemari) Kemari is a football game that players keep a ball in the air Don t drop the ball! Copyright Nippon Telegraph and Telephone Corporation 3

4 Our goal Don t drop the ball! Don t drop the VMs! Hardware failure Keep running transparently Kemari: Virtual Machine Synchronization Copyright Nippon Telegraph and Telephone Corporation 4

5 What needs to be done? Virtual Machine Synchronization Primary VM and Secondary VM must be identical Detection of failure Failover mechanism Extension of existing techniques Primary node Failover Secondary node Hardware Failure Apps Guest OS VM Synchronization Apps Guest OS VMM Hardware Network VMM Hardware SAN Copyright Nippon Telegraph and Telephone Corporation 5

6 How to synchronize VMs? Primary VM 1. Pause Primary, and sync with Secondary Secondary VM Need to make the overhead of sync smaller Make sync time shorter t interval t sync 2. Resume Primary after sync Only transfer updated data Sync VMs less often Secondary must be able to continue transparently Sync VMs before sending or receiving Events Events: Storage, network, console Copyright Nippon Telegraph and Telephone Corporation 6

7 What happens if synced on specific intervals? Primary VM Primary VMM Storage Secondary VMM Secondary VM V i V i+1 4. Read request 5. Reply 3. Sync completed 6. Write request 1. Sync VM S j 2. Update Secondary V i-1 V i V i+2 7. Reply S j+1 8. Failover to Secondary VM after detecting HW failure S j+1 9. Read request V i V i : VM s state S j : Storage s state The state between VM and storage isn t consistent Secondary VM won t be able to continue transparently Copyright Nippon Telegraph and Telephone Corporation 7

8 Sync on events from VM to storage Primary VM V i Primary VMM 1. Read / Write request Storage S j Secondary VMM Secondary VM V i-1 2. Sync VM and event 4. Sync completed 3. Update Secondary V i V i+1 6. Reply 5. Resume Read / Write S j+1 Resume point: Just before operating storage V i : VM s state S j : Storage s state Secondary will redo the same operation as Primary Secondary will receive the same reply as Primary Copyright Nippon Telegraph and Telephone Corporation 8

Demo Hardware failure VNC Server VNC Client Guest VM is running on Kemari Guest is running VNC server, and the client accesses via VNC client xclock is launched from the

9 Demo Hardware failure VNC Server VNC Client Guest VM is running on Kemari Guest is running VNC server, and the client accesses via VNC client xclock is launched from the client See what happens to the clock when the primary physical server is shut downed from HP ilo2 management console Copyright Nippon Telegraph and Telephone Corporation 9

synchronization mechanism resides in hypervisor to synchronize virtual machines efficiently

10 Architecture overview Kemari Kemari Dom0 DomT Sync DomT DomT Dom0 Back-end Front-end Front-end Back-end Kemari Xen Xen Hardware Network Hardware SAN The core of the synchronization mechanism resides in hypervisor to synchronize virtual machines efficiently LOC 3000 (hypervisor: 1000, Dom0+Tools: 2000) Copyright Nippon Telegraph and Telephone Corporation 10

11 What is DomT? Domain U pfn PT Domain T Shadow mfn pfn mfn PT PT PT PT Para-virtualized domain which uses shadow page table (auto-translated-mode) Don't have to translate the page tables on transferring DomT patch set for xen was written by Michael A Fetterman from University of Cambridge Copyright Nippon Telegraph and Telephone Corporation 11

12 Implementation of Kemari Event Channel tapping Transferring DomT Restoring para-virtualized devices Copyright Nippon Telegraph and Telephone Corporation 12

13 Event Channel tapping Simple but the key component of Kemari Monitors IN/OUT or Both Registered function is called on specific events Dynamically attachable May be useful for measurements DomT 2 5 Dom0 Front-end ECS_TAP Back-end Kemari Copyright Nippon Telegraph and Telephone Corporation 13

Transferring DomT 1 (VMM) 2 (VMM) 3 (Tools) 4 (Tools) DomT Grant table Shared

region 1. Pauses DomT and locks the grant tables. No need to suspend!

Extracts dirtied pfn from the bitmap, copies pfns and the vcpu to the shared

Maps dirtied pages, transfers pages and vcpu to the secondary 4.

14 Transferring DomT 1 (VMM) 2 (VMM) 3 (Tools) 4 (Tools) DomT Grant table Shared buffer (VMM/Tools) tmp buffer Log dirty bitmap VCPU Primary Secondary DomT region 1. Pauses DomT and locks the grant tables. No need to suspend! Grant tables are mapped at the last 4 pages of DomT region 2. Extracts dirtied pfn from the bitmap, copies pfns and the vcpu to the shared buffer, and notifies Tools via event channel 3. Maps dirtied pages, transfers pages and vcpu to the secondary 4. Secondary prepares temp buffers to rollback when failure is detected during transfer Copyright Nippon Telegraph and Telephone Corporation 14

Attach the Back-end to the Device Channel using BACK_RING_ATTACH

Adjust producer and consumer indexes of the Back-end appropriately

15 Restoring para-virtualized devices 1. Device Channel is stored in DomT region DomT Front-end 2. Attach the Back-end to the Device Channel using BACK_RING_ATTACH macro 3. Adjust producer and consumer indexes of the Back-end appropriately Response consumer Request producer Device Channel Response producer Request consumer Dom0 Back-end Copyright Nippon Telegraph and Telephone Corporation 15

16 Evaluation Evaluation items Performance of the Primary VM (Network and File I/O) using netperf and iozone Test machines Hardware spec CPU: Intel Xeon 3GHz X 2 Memory: 4GB Network: Gigabit Ethernet, InfiniBand SAN: FC Disk Array VM spec VMM: Xen with DomT support Guest OS: Debian Etch Memory: 512MB Copyright Nippon Telegraph and Telephone Corporation 16

Performance of Primary VM DomT Kemari (Ethernet) Kemari (InfiniBand) 100 8 100 90 7 90 Throughput [Mb/sec] 80 70 60 50 40 30 20 Throughput [MB/sec] 6 5 4 3 2 Throughput [MB/sec] 80 70 60 50 40 30 20

17 Performance of Primary VM DomT Kemari (Ethernet) Kemari (InfiniBand) Throughput [Mb/sec] Throughput [MB/sec] Throughput [MB/sec] Network 0 O_SYNC InfiniBand boosted the performance of Network and Buffered + fsync, both of which dirties many pages All benchmarks continued transparently when the primary server was shut downed from the HP ilo 2 management console Copyright Nippon Telegraph and Telephone Corporation 17 0 Buffered + fsync

18 Conclusion Kemari is a Virtual Machine synchronization mechanism to achieve Fault Tolerance Don't drop the ball! Don't drop the VMs! Implemented Kemari using Xen and DomT Thanks to Michael from University of Cambridge Demonstrated Kemari achieved acceptable performance Copyright Nippon Telegraph and Telephone Corporation 18

19 Future work Demonstrate the range of applications Kemari can manage to run transparently Improve the performance of I/O intensive applications that send numbers of events Hosting HVM domains with PV drivers Hosting multiple domains simultaneously Functions to implement for practical use such as detection of HW failure and failover mechanism Copyright Nippon Telegraph and Telephone Corporation 19

Modernization of Kemari using HVM with PV Drivers

Modernization of Kemari using HVM with PV Drivers Yoshi Tamura NTT Cyber Space Labs. 2008/11/20 What is Kemari? Don t drop the ball! Don t drop the VMs! Hardware failure Keep running transparently Kemari: