VEE CompSC: Live Migration with Pass-through Devices ZHENHAO PAN &, YAOZU DONG *, YU CHEN &, LEI ZHANG &, ZHIJIAO ZHANG &,

Transcription

1 CmpSC: Live Migratin with Pass-thrugh Devices ZHENHAO PAN &, YAOZU DONG *, YU CHEN &, LEI ZHANG &, ZHIJIAO ZHANG &, & Tsinghua University, *Intel Asia-Pacific Research and Develpment Ltd. VEE 2012

2 Outline Intrductin CmpSC slutin Experiments Cnclusin

3 Intrductin Backgrund Live migratin Pass-thrugh device SR-IOV spec Experimental result Liv migratin with SR-IOV NIC % mre thrughput 42.9% less dwntime

4 SR-IOV Specificatin Intrductin Start with a single functin device HW under the cntrl f privileged SW IOVM VM 0 VM 1 Includes an SR-IOV Extended Capability Physical Functin (PF) DD DD DD Replicate the resurces needed by a VM MMIO fr direct cmmunicatin RID t tag DMA traffic dedicated PCIM Minimal cnfiguratin space Virtual Functin (VF) Intrduces PCI Manager (PCIM) Cnceptual SW entity Cmpletes the cnfiguratin mdel Translates VF int a full functin Cnfigures SR-IOV resurces Physical Functin Virtual Functin PF VF Shared Resurce VF DMA, PIO, and Interrupts Initializatin and Cnfiguratin

5 Related wrk Bnding driver [Linux Ethernet Bnding Driver HOWTO] Failver/Lad balance NPIA (Netwrk Plug-in Architecture) Hst Linux Kernel Virtual Bridge KVM Native Driver Native Driver Tap/Tun Driver Bnding driver NPIA

6 Related wrk VMDq (Virtual Machine Device Queue) Multiple queue pairs fr partitining Virti-net (Service) pp Xmit vqueue (VQ 0) Add_buf push Get_buf Hst Linux Kernel Virtual Bridge file_peratins pp push Receive vqueue (VQ 1) Add_buf Get_buf KVM Native Driver Tap/Tun Driver One cpy fr transmissin & receive Why nt stre/restre device states directly?

7 Outline Intrductin CmpSC slutin Experiments Cnclusin

8 CmpSC Appraches Requirement challenges The state (such as registers) f the device needs t be efficiently read and written t supprt device state replicatin; The dirty memry written by the device Direct Memry Access (DMA) needs t be efficient and tracked fr lazy memry state transmissin.

9 CmpSC Appraches CPU Memry Bus interface fhardware OS Registers DMA Interrupts Interrupt Cntrller IO W/R DMA address address+data Interrupt Request Bus Interface read write cntrl status Registers Device Cntrller Hardware Cntrller Addressable Memry and/r Queues Memry Mapped Regin

10 Requirement f I/O Register migratin : Mst parts: Read/write, N side effect Sme special: RO/WO, RC/WC, etc., with side effect Register type read-write read-nly write-nly read-write-clear write-clear read-clear CmpSC Appraches Descriptin If written since reset, the value read reflects the value written. Writes t this reg have n effect. Reading this reg returns n meaningful value. A register can be read and written. Hwever, a write f a 1b clears the crrespnding bit. Writing 1b t register clears an event pssibly reprted in anther register. A register bit with this attribute is cleared after read. Writes have n effect n the bit value. read-write-set reserved Register that is set t 1b by sftware, and cleared t 0b by hardware. Reserved field can return any value n read access and must be set t its initial value n write access.

11 State replay fr side effect Methd Recrd every hardware access (Recrding stage) Replay them n the target device (Replaying stage) Optimizatin 1 Recrd last reg writing when this writing brings n side effect Optimizatin 2 CmpSC Appraches Define peratin sets (p set), the p sets is Critical Sectin

12 CmpSC Appraches State replay with p set Op sets in Intel 82576/82599 NIC All initializing peratins All Sending peratins All Receiving peratins ther remaining p states include nly {uninitialized, up, dwn} In this kind f set up, nly the latest peratins n each setting register and whether r nt the interface is up need t be tracked.

13 Self-emulatin fr Read-nly, etc. Regsiters Design fr statistic registers (read-nly/read-clear) Require mathematical attributes (mntnicity) Example: drpped packets cunter CmpSC Appraches = n befre migratin initialized t 0 when migratin = m nw (after migratin) crrect value= n + m

14 CmpSC Appraches Dummy writing fr DMA dirty page DMA dirty page tracking. T replicate the I/O state, memry pages mdified by the device DMA peratins must be efficiently tracked fr efficient live migratin. Unfrtunately, DMA dirty page tracking is nt supprted in the existing I/O MMU. Dummy write the DMAed page after DMA prcess finished.

15 CmpSC Architecture Design & Implemenatatin live migratin 1. Pre-Migratin stage 2. Reservatin stage 3. Iterative Pre-cpy stage 4. Stp-and-cpy stage 5. Cmmitment stage 6. Activatin stage

16 CmpSC Implementatin Xen and SR-IOV NIC drivers Implementatin cmplexity ~2000 LC

17 CmpSC Implementatin Xen and SR-IOV NIC drivers Intel Gbps NIC & Gbps NIC PF/VF drivers Driver changes n IGBVF/IXGBEVF Rlck every hardware peratin Pack igbvf_up/igbvf_dwn and ixgbevf_up/ixgbevf_dwn int peratin sets Restratin after migratin

18 CmpSC Implementatin Xen and SR-IOV NIC drivers Shared memry fr sync rw-lck and versin cunter List f registers fr I/O register migratin List f registers fr self-emulatin Synchrnizatin fr Live Migratin Acquire w-lck befre suspending Increase versin cunter Release w-lck after migratin Invke driver restratin at first r-lck

19 CmpSC Implementatin Xen and SR-IOV NIC drivers Pages dirtied by DMA In x86/x64, memry access by DMA cannt be tracked n page tables by MMU, IOMMU In CmpSC, driver perfrms dummy writes t descriptr/buffer when receive an interrupt May cause packet miss/packet duplicatin during migratin

20 Xen and SR-IOV NIC drivers Descriptr ring CmpSC Implementatin Descriptr ring head index is in read-nly register Altering head index is hard (hard fr state replay) CmpSC intrduces an ffset between the ring in hardware's view and sftware's view During migratin, increase the ffset t make sure ring head index n target hardware is 0

21 Outline Intrductin CmpSC slutin Experiments Cnclusin

22 Physical Envirnment Experiments Intel Cre i5 670 (with VT-x, VT-d, VT-c features) 4GB memry, 1TB hard disk Intel & Intel NICs Virtual Envirnment 4 vcpu 3GB memry PF/VF f Intel r Intel NIC

23 Experiments Evaluatin - Thrughput Intel Gbps int: sel f-emulatin

24 Experiments Evaluatin - SPECweb2009 Intel 82599

25 Experiments Netperf Evaluatin - Live migratin(bnding v.s. CmpSC), NIC Bnding CmpSC

26 Experiments Netperf Evaluatin - Live migratin(pv v.s. CmpSC), NIC PV CmpSC Dm0 and the guest were sharing the physical CPU

27 Cnslutin Prpsed a directly slutns fr live migratin f pass-thrugh device : CmpSC Supprt Live Migratin with SR-IOV NIC Future Evaluate NPIA methd Supprt Checkpint (such as Remus in XEN) Other SR-IOV devices...

28 Thank yu fr yur attentin! Questins?