Virtio 1 - why do it? And - are we there yet? Michael S. Tsirkin Red Hat

Transcription

1 Virtio 1 - why do it? And - are we there yet? 2015 Michael S. Tsirkin Red Hat 1 Uses material from Gcompris, tuxpaint Distributed under the Creative commons license.

2 Lots of work... main-title 300 commits Aug 6 to Aug 6 in each year year

3 Virtio 1: update 3 Documented assumptions More Robust More Extendable

4 Conformance statements Virtio 0.9 Virtio DRIVER_OK status bit is set. - The device can now be used. The driver MUST NOT notify the device before setting DRIVER_OK. drv probe(dev); netif_carrier_on(dev) drv probe(dev); add_status(dev, DRIVER_OK); netif_carrier_on(dev) add_status(dev, DRIVER_OK); 4

5 Virtio 0.9: inflate FF FF FF DRIVER 5 00 FF

6 Virtio 1.0: inflate FF FF FF FF DRIVER 6

7 Generation counter FFFFFFFF FFFFFFFF DRIVER

8 Memory map 0.9 COMMON FEATURES QUEUE 1.0 DEVICE SPECIFIC STATUS ISR COMMON CAPABILITY LIST IO BAR FEATURES QUEUE STATUS MEMORY BAR VIRTIO CAPABILITY #1 8 VIRTIO CAPABILITY #2 DEVICE SPECIFIC ISR...

9 Virtio 0.9: Port IO vs Memory Port IO x86 decode: address x86 decode: data Fast on x86 32/64 bit Page tables Required by PCI 9 Express MM IO

10 Fast MMIO avoid need to decode data ADDRESS NOTIFY DATA VQ NUMBER ADDRESS 10 VQ NUMBER NOTIFY DATA IGNORED

11 Virtio 1: Access times on KVM x86: Cycles per access (lower is better) MMIO Fast MMIO Port IO CPU cycles 11

12 Virtio 1: Port IO vs Memory Port IO x86 decode: address Fast on x86 32/64 bit Page tables Required by PCI Express 12 MM IO

13 Memory Region Aliases CAPABILITY LIST IO BAR MEMORY BAR VIRTIO CONFIG CAPABILITY VIRTIO CAPABILITY 13 Queue Notify VirtQueue Queue Notify Queue Notify

14 soft mac Ethernet MAC DRIVER VirtQueue DRIVER 14

15 Virtio feature negotiation DEVICE FEATURES DRIVER DRIVER FEATURES Defaults must be maintained forever! 15

16 Virtio 1: Error handling DRIVER: set features DRIVER: set FEATURES_OK bit DEVICE: check features DEVICE: clear FEATURES_OK on error DRIVER: check FEATURES_OK bit DRIVER: fail gracefully if not set 16!

17 Error handling: Virtio 0.9 Can't recover from device errors Not very useful? Just stop guest. 17

18 Vhost-user GUEST virtio-net VM RAM DMA SETUP VHOST USER CLIENT Client crash or restart need not cause guest crash! 18

19 DEVICE_NEEDS_RESET Read STATUS; Detect: NEEDS_RESET set DRIVER 19 Write STATUS=0 Will reset device Reconfigure device. Write STATUS=DRIVER_OK Restart operation.

20 Compatibility Transitional Device & Driver Legacy Driver Legacy Device Legacy Modern Legacy Modern Legacy Legacy Modern Legacy Legacy Modern DRIVER 20 DRIVER DRIVER

21 Are we there yet? GUEST BIOS DMA VHOST USER VHOST 21

22 What to expect? Current: Virtio-v1.0-cs03 Next bugfix: Virtio-v1.0-cs04 22 Virtio-blk: writeback / writethrough control More update guidance Next feature: Virtio-v1.1-cs01 Virtio-input Virtio-gpu Virtio-vsock

23 TX: Interrupt avoidance uplink 23

24 TX: Interrupt coalescing uplink 24

25 Pass-through for nested virt Virtio Net (on host) Memory mapped: use page tables IOMMU: translate and protect guest memory 25

26 Virtio as PCI Express device Uses memory mapped IO support Multi-root for NUMA Native hotplug Advanced Error Reporting 26

27 Summary Why do it? 27 Improved robustness for virtual devices Are we there yet? Yes! And there's more to come.

28 Thank you! 28

29 Virtio 0.9: Port IO versus memory on KVM x86: cycles per access (lower is better) MMIO Port IO CPU cycles 29

30 OASIS Virtio TC PCI Virtio 1.0 MMIO (ARM) CCW (PPC) 30

31 Virtio 1.0 Virtio PCI: Replace Port IO with Memory mapped IO PCI Express (hotplug, AER, multi-root, SRIOV) Infinite features Reduced memory requirements Fixed endianness Compatibility 31

32 Port IO: outl EF OUT notify (%DX) VQ# %EAX VM Exit REASON 32 QUALIFICATION STATE

33 Memory mapped IO: writel 3E 89 MOV (%EDI) PTE %RSI VALID? VM Exit REASON 33 GUEST ADDRESS RIP

34 Fast MMIO notify MOV VQ# (%EDI) PTE VALID? VM Exit REASON 34 GUEST ADDRESS %RSI

35 Multiple interfaces CAPABILITY LIST IO BAR MEMORY BAR VIRTIO CAPABILITY #1 VIRTIO CAPABILITY #2 35

36 Memory requirements 0.9 VQ desc avail 1.0 VQ desc 36 used avail used

37 features DEVICE FEATURES - - DRIVER 1 v SEL DRIVER FEATURES DRIVER STATUS = FEATURES_OK....

38 Endianness intel Virtio 0.9 PPC Virtio BE Virtio LE Device LE Device BE Virtio 1.0 Virtio LE Device 38 Device Device

39 compatibility Driver Driver Compatibility Device 39 Device

40 Packet layout Virtio 0.9 INDIRECT next Virtio 1.0 header 40 header

41 Packet layout: transactions per sec (higher is better) virtio 0.9 virtio transactions/sec 41

42 More: virtio 1.0 versus Virtio 9p Virtio blk: WCE Virtio-net Multiqueue Virtio-net dynamic offloads Already upstream (based on spec draft) 42

43 vhost updates Vhost scsi Vhost-net zero copy transmit No need for driver changes 43

44 Kvm networking Openvswitch if time allows Ethernet bridge 44

45 Bridge FDB London Heathrow Paris CDG uplink London Heathrow 45 London Paris CDG Paris

46 Flood: DOS potential London Heathrow Paris CDG uplink Bangkok Heathrow 46 London CDG Paris

47 Disable flood London Heathrow Paris CDG uplink London Heathrow 47 London Bangkok Paris CDG Paris

48 softmac Ifconfig eth0 hw ether 00:12:23:45:67:89 00:12:23:45:67:89 virtio-net MAC 48 NEW

49 Using softmac/non promiscuous London Heathrow Paris CDG uplink Paris Heathrow CDG NEW 49 Paris

50 Work in progress ELVIS (vhost blk/vhost net) Virgl Vhost-net performance 50

51 RX latency NIC HOST VHOST VM 51

52 Fast rx NIC HOST Current? VHOST VM 52 RAM?

53 Fast rx: transactions per sec (higher is better) thread irq transactions/sec 53 Hit Miss 79

54 Vhost-net threading tap RX VHOST VM NIC VHOST VM 54 TX

55 Vhost-net thread pool tap VM WQ VHOST NIC VM 55 VHOST

56 threading: UDP RR transactions/sec (higher is better) thread wq

57 threading: TCP STREAM transactions/sec (higher is better) thread wq

58 summary Performance Manageability Security 58

59 Questions? 59

60 OVS: flow match PACKET FLOW kernel userspace OVS-VSWITCHD 60 VM

61 OVS: wildcard match PACKET FLOW * kernel userspace OVS-VSWITCHD 61 VM

62 Wilcard: netperf CRR (higher is better) match wildcard bi-connections/sec 62

63 63