BlueVisor: A Scalable Real-time Hardware Hypervisor for Many-core Embedded System

Transcription

1 BlueVisor: A Scalable eal-time Hardware Hypervisor for Many-core Embedded System Zhe Jiang, Neil C Audsley, Pan Dong eal-time Systems Group Department of Computer Science University of York, United Kingdom School of Computer National University of Defense Technology China TAS 218 eal-time Systems Group 1

2 Outline Virtualization Technology Networks-on-chip BlueVisor Experimental Evaluation Conclusion eal-time Systems Group 2

3 Motivation Virtualization technology was invented by IBM in the 196 s on server platform esource sharing Currently, virtualization technology has became popular everywhere server, desktop and embedded system esource utilization System volume Cost of hardware Load Balance IBM CP-4 mainframe the first system invented for server virtualization eal-time Systems Group 3

4 In eal-time Systems Virtualization technology brings: Consistent execution environment Isolation Time Analysis VM VM VM VMM Host OS Software Layer Hardware Layer SMP (e.g. x- 86, Intanium or ISC) Memory UAT VGA Ethernet eal-time Systems Group 4

5 However Virtualization technology significantly conflicts to requirements of real-time systems, because of Indirection and interposition of privileged instructions (e.g. I/O request) VM VM VM VMM Host OS Software Layer Hardware Layer SMP (e.g. x- 86, Intanium or ISC) Memory UAT VGA Ethernet eal-time Systems Group 5

6 VM1 VM2 VMn APP APP APP APP APP APP APP APP APP Guest OS I/O Drivers Guest OS Guest OS Virtual-To-Physical Translation Interpose/ Transform VMM I/O Scheduler Host OS I/O Drivers Software Layer Hardware Layer SMP (e.g. x-86, Intanium or ISC) Memory UAT VGA Ethernet eal-time Systems Group 6

7 VM1 VM2 VMn APP APP APP APP APP APP APP APP APP Guest OS I/O Drivers Guest OS Guest OS Virtual-To-Physical Translation Interpose/ Transform VMM I/O Scheduler Host OS I/O Drivers Software Layer Hardware Layer SMP (e.g. x-86, Intanium or ISC) Memory UAT VGA Ethernet Significant software overhead Longer response time of handling privileged instructions More uncertainty Worse predictability eal-time Systems Group 7

8 Outline Virtualization Technology Networks-on-chip BlueVisor Experimental Evaluation Conclusion eal-time Systems Group 8

9 Move to Many-core Systems Breakdown of Dennard scaling In order to meet the year-onyear performance increment: Before: Increase processor s clock speed / frequency After: Increase the number of cores on a chip System: Single core Multi-core (point-to-point, bus) Many-core (NoC) eal-time Systems Group 9

10 Network-on-Chip (NoC) Parallella 16 cores Intel s Knight Landing 72 cores MPPA 256 cores eal-time Systems Group 1

11 Network-on-Chip (NoC) priority ID highest priority with remaining credit data_in credit_out routing & transmission control data_out credit_in I/O I/O I/O eal-time Systems Group 11

12 Vritualization VM1 VM2 VMn APP APP APP APP APP APP APP APP APP Guest OS I/O Drivers Guest OS Guest OS Virtual-To-Physical Translation Interpose/ Transform VMM I/O Scheduler Host OS I/O Drivers Software Layer eal-time Systems Group 12

13 NoC + Virtualization Software Layer Hardware Layer I/O I/O I/O eal-time Systems Group 13

14 NoC + Virtualization With NoC, indirection and interposition of privileged instructions has became worsen Longer response time Decreased system performance Software Layer Hardware Layer More uncertainty Worsen predictability Even some works are focusing on the predictability of NoC, e.g. Leandro et al. Buffer-aware bounds DATE 218. I/O I/O I/O eal-time Systems Group 14

15 NoC + Virtualization Moreover, the architecture also suffers from complicated resource management Scheduling between the applications Software Layer Hardware Layer Scheduling between the VMs Scheduling between the processors Contentions on I/Os Significant overhead, both software + hardware Decreased predictability I/O I/O I/O eal-time Systems Group 15

16 Therefore, We are looking for a hypervisor, which enables: Predictability Improved system performance Decreased software overhead Good Scalability Isolation eal-time Systems Group 16

17 A lot of intelligent are also working on this Sisu Xi, Justin Wilson, Chengyang lu et al. T-Xen: towards real-time hypervisor scheduling in xen Sandro Pinto, Jorge Pereira, Tiago Gomes et al. LTZVisor: TrustZone is the Key Ye Li, M Danish, West et al. Quest-V: A Virtualized Multikernel for High-Confidence Systems eal-time Systems Group 17

18 Outline Virtualization Technology Networks-on-chip BlueVisor Experimental Evaluation Conclusion eal-time Systems Group 18

19 VM1 VM2 VMn APP APP APP APP APP APP APP APP APP Guest OS I/O Drivers Guest OS I/O Drivers Guest OS I/O Drivers VMM Host OS Software Layer Hardware Layer BlueVisor SMP (e.g. x-86, Intanium or ISC) Memory UAT eal-time I/O controller: VGA GPIOCP Ethernet eal-time Systems Group 19

20 I/O I/O I/O BlueVisor eal-time Systems Group 2

21 Standard FreeTOS API() FreeTOS_Open() FreeTOS_Write() FreeTOS_ead() FreeTOS_IOctl() Zhe Jiang CPU Virtualization Each processor (no matter the architecture) is set as an individual guest VM. Bare-metal Virtualization Para-Virtualization Application User Mode Kernel Mode OS Kernel FreeTOS I/O Manager High Layer I/O Driver FreeTOS Low Layer I/O Driver Hardware CPU I/O FreeTOS Kernel eal-time Systems Group 21

22 Standard FreeTOS API() FreeTOS_Open() FreeTOS_Write() FreeTOS_ead() FreeTOS_IOctl() Zhe Jiang CPU Virtualization Each processor (no matter the architecture) is set as an individual guest VM. Bare-metal Virtualization Para-Virtualization Application User Mode Kernel Mode OS Kernel High Layer I/O Driver FreeTOS Hardware CPU FreeTOS Kernel I/O eal-time Systems Group 22

23 CPU Virtualization Currently, BlueVisor system supports the following OS kernels: FreeTOS ucosii XilKernel Customized interface to support a new OS is also provided. eal-time Systems Group 23

24 I/O Virtualization VCDC Virtualized Complicated Device Controller (VCDC) integrates most functionalities of I/O virtualization and I/O drivers educe software overhead significantly Increase I/O performance significantly, i.e. I/O throughput & response time Scalability Jiang, Zhe, and Neil C. Audsley. VCDC: ECTS 217 eal-time Systems Group 24

25 eal-time I/O Controller GPIOCP General-purposed resource efficient coprocessor Using Uart to send at time.. Pre-programable, E.G. Using Uart to send Hello World at a specific time. emove the transmission Enables predictable I/O operations Enables Timng-Accurate I/O operations Jiang, Zhe, and Neil C. Audsley. GPIOCP: DATE 217 eal-time Systems Group 25

26 Interrupt Management Two types of interrupt management based on GPIOCP are provided: Type1: Fast Interrupt Handler Type2: Normal Interrupt Handler eal-time Systems Group 26

27 Inter-VM Communication Two types of Inter-VM communication Packet-based communication (Simple and brief communication) Memory-based communication (Complicated and long communication) eal-time Systems Group 27

28 Outline Virtualization Technology Networks-on-chip BlueVisor Experimental Evaluation Conclusion eal-time Systems Group 28

29 Experimental Platform Platform Xilinx VC79 Many-Core System 4x5 size 2D mesh Type NoC Number of CPUs System Frequency Kernel Version 16 1 Mhz FreeTO S (V9..) To enable comparison, a similar hardware architecture was built, but without the BlueVisor and virtualization technology. BlueVisor I/O I/O I/O eal-time Systems Group 29

30 Experimental Metrics Memory Footprint I/O performance I/O esponse Time I/O Throughput Timing-accuracy Interrupt Handling eal-time Systems Group 3

31 Memory Footprint nfreetos: native FreeTOS; FreeTOS + I/O: FreeTOS + I/O drivers (e.g. UAT, VGA, etc.); vfreetos: simply implemented software virtualized FreeTOS; BV_vFreeTOS: virtualized FreeTOS in BlueViosr system..text.data.bss Total BlueVisor nfreetos 121,39 1,728 35,74 158,741 nfreetos+ I/O 179,652 1,852 36,25 217,754 vfreetos + I/O 189,556 1,882 36,45 227,888 BV_vFreeTOS +I/O 131,969 1,732 35, ,424 eal-time Systems Group 31

32 I/O Performance I/O esponse Time I/O Throughput Timing-Accuracy eal-time Systems Group 32

33 I/O Performance esponse Time Aims to evaluate the performance of the I/O system while CPU and I/O are fully loaded in a BlueVisor and non- BlueVisor system. Number of Active CPUs Evaluated I/O I/O equests Scheduling Policy in non- BlueVisor Scheduling Policy in BlueVisor 16 SPI nor-flash ead 4, 8, 16, 32 Bytes ound-obin (Global) FIFO (Local) ound-obin eal-time Systems Group 33

34 I/O Performance esponse Time Worst Case I/O esponse Time (unit: clock cycle) Written Bytes Non-BlueVisor (FIFO) Non-BlueVisor (ound-obin) BlueVisor (ound-obin) 1 9,357 65, , ,813 1, ,166 4,555,159 23, ,72,565 17,345,151 89,78 Written Bytes Variation of I/O esponse Time (unit: clock cycle) Non-BlueVisor (FIFO) Non-BlueVisor (ound-obin) BlueVisor (ound-obin) 1 1,541 59, As shown in the results: The worst case of I/O response time (max) is smaller; The I/O response time is more stable (max min) - more predictable; 4 7,61 286, ,26 3,823,14 3, ,142 15,475,355 13,711 eal-time Systems Group 34

35 I/O Performance Throughput Aims to evaluate the performance of the I/O system while CPU and I/O are fully loaded in a BlueVisor and non- BlueVisor system. Number of Active CPUs Evaluated I/O I/O equests Scheduling Policy in non- BlueVisor Scheduling Policy in BlueVisor 4 SPI nor-flash Write ound-obin (Global) FIFO (Local) ound-obin FIFO eal-time Systems Group 35

36 I/O Performance Throughput From the results, we can see that, in BlueIO system: The I/O throughput is higher; The variance of I/O throughput is smaller. eal-time Systems Group 36

37 I/O Performance Timing-accuracy Aims to evaluate the timing-accuracy of the I/O operations in a BlueVisor and non- BlueVisor system. Number of Active CPUs Evaluated I/O I/O equests Scheduling Policy in non- BlueVisor Scheduling Policy in BlueVisor 9 GPIO Write I/O pins ound-obin (Global) ound-obin eal-time Systems Group 37

38 I/O Performance Timing-accuracy We define the error of I/O operations timing accuracy as: E = Tx Ty Tx: time at which I/O operation is required; Ty: the actual time that the I/O operation actually occurs. Tx E Ty If E =, this I/O operation occurs exactly at the expected time we say it is totally timing-accurate. t eal-time Systems Group 38

39 I/O Performance Timing-accuracy CPU Index Non-BlueVisor BlueVisor E (unit: ns) E (unit: clock cycle) E (unit: ns) E (unit: clock cycle) Min Med Mean Max Min Med Mean Max Min Med Mean Max Min Med Mean Max (,) (,1) (,2) (1,) (1,1) (1,2) (2,) (2,1) (2,2) eal-time Systems Group 39

40 Interrupt Handling Evaluate the response time of interrupt handling in BlueVisor and non-bluevisor architectures Number of Active CPUs Evaluated I/O I/O equests Scheduling Policy in non- BlueVisor Scheduling Policy in BlueVisor 1 GPIO esponse Interrupt None None eal-time Systems Group 4

41 Interrupt Handling Fast IQ can be always completed at 1 cycles Native FreeTOS BS_vFreeTOS (Fast IQ) Interrupt handling (unit: clock cycles) Best Case Worst Case Mean Normal IQ takes more time than the IQ in native FreeTOS BS_vFreeTOS (Normal IQ) eal-time Systems Group 41

42 Outline Virtualization Technology Networks-on-chip BlueVisor Experimental Evaluation Conclusion eal-time Systems Group 42

43 Conclusion We are looking for a hypervisor, which enables: Predictability Improved system performance Decreased software overhead Good scalability Isolation eal-time Systems Group 43

44 Limitation Extra hardware overhead Communication overhead in the entrance of BlueVisor (Traffic congestion) Hard to be upgraded. eal-time Systems Group 44

45 In the future Memory Virtualization cache coherence Isolation Support SMP OS Support More I/O Devices eal-time Systems Group 45

46 BlueVisor: A Scalable eal-time Hardware Hypervisor for Many-core Embedded System Zhe Jiang, Neil C Audsley, Pan Dong eal-time Systems Group Department of Computer Science University of York, United Kingdom School of Computer National University of Defense Technology China TAS 218 eal-time Systems Group 46