Prof. Daniel Rossier, PhD

Transcription

1 Dealing with Hardware Heterogeneity Using a Virtualization Framework Tailored to ARM Based Embedded Systems Prof. Daniel Rossier, PhD HEIG-VD Institut REDS, Reconfigurable & Embedded Digital Systems rte Cheseaux 1, 1400 Yverdon-les-Bains

2 Outline Background Overview of EmbeddedXEN Protection & Memory Isolation Domain Interactions Device Heterogeneity Conclusions & Future Work 2

3 Background HEIG-VD University of Applied Sciences in Yverdon-les-Bains, Switzerland (CH) Reconfigurable Embedded Digital System Institute Hardware Design, FPGA Embedded Execution Environment, Drivers, BSP, OS/RTOS, etc. Applied Research & Development ARM based Microcontrolers (v4, v5, v6, v7) Low-level interactions between computing cores (CPU, DSP, FPGA, etc.) Towards Cortex-A15 HVM 3

4 Background Embedded Virtualization on ARM XEN: free & easy access to a stable & evolving hypervisor source code Early port of XEN on ARM in 2007 in the context of a Diploma Project Necessity to get a simple, thin, fast, robust, easy-to-deployed virtualization framework Re-use of Linux file organization & build system (Makefiles, scripts, ) Focus on realtime aspects (Linux/Xenomai as RTOS) Different sources of inspiration "Fast Secure Virtualization for the ARM Platform", Daniel Ferstay, Master Thesis, The University of British Columbia, 2006 XEN ARM port, George G. Davis, MontaVista, 2007 Secure Xen on ARM Project, Sang-bum SUH, Samsung,

5 Background Focus on heterogeneity of embedded devices Idea to re-use OS & applications from old devices to recent devices Time-to-market migration to new hardware generation Dealing with various cross-compiled binaries (ARM v5-v7) Dealing with different peripherals Less emphasis on security aspects Publicly available 5

6 Background Hardware constraints Low latency, reactivity (response time) ARM cores do not support virtualization mechanisms not easy to deal with various levels of execution modes Para-virtualization remains attractive About 30 files to be (slightly) adapted Low execution overhead Efficient processing of downcalls/upcalls with support of domain interactions Physical interrupts are quickly processed in dom0. 6

7 Overview of EmbeddedXEN Primary (host) OS known as Dom0 Secondary (guest) OS known as DomU Xen-guest (pv) - Full privilege - Dom0 original drivers - Backend drivers - Native drivers Limited use of hypercalls - Full privilege - DomU original drivers - Frontend drivers Xen-guest (pv) ARM execution modes (USR/SVC) pseudo-user mode with double stack handling / downcalls & upcalls are simple jumps to specific (callback) addresses No subtle use of domain access control (ARM DACR) to protect memory Memory sharing facilitated between primary and secondary OS VCPU sched_flip Scheduler Domain Creation and Setup EmbeddedXEN Hypervisor Hypercalls Handling Prioritized Upcalls Handling Migration Manager Hardware 7

8 Overview of EmbeddedXEN Linux-like source tree and build system (Makefiles) Config & build system tools environnement.conf Common part embeddedxen linux domu xenbus core include console include arch linux dom0 hypervisor include arm arch/arm mach-msm mach-mx35 xen include arch Secondary OS mach-mx35 mx35_fab4.c Hypervisor arch/arm arm kernel mm Primary OS mach-msm 8

9 Overview of EmbeddedXEN Single binary (multi-kernel) image EmbeddedXEN Automatic parsing & image relocation during hypervisor bootstrap Boot Head EOD 0 (Dom-0) EOD 1 (Dom-U) Hypervisor DOM-0 DOM-U vmlinux vmlinux.dom0 vmlinux.domu uimage Dom-0 Filesystem /home/root/squeezeos.rootfs.img: Stored separately in sdcard, for example Dom-U Filesystem 9

10 Protection & Memory Isolation Memory isolation between domains relies on different address space isolation. No further advanced mechanisms to protect hypervisor and guest memory The guest OS kernel runs at the same privilege as the hypervisor. Each domain receives its own (contiguous) physical memory region during domain set up. No pagination of the kernel linear address space is performed. Paravirt of memory management is kept minimal. Guest OS kernel has access to the whole memory. No protection mechanism for strong isolation of VM (but is it really necessary?) 10

11 Protection & Memory Isolation Virtual & Physical Memory Layouts Initial Virtual Address Space Physical RAM dom0 Virtual Address Space 0xffff0000 Interrupt Vectors frame table Interrupt Vectors frame table XEN heap (2 MiB) XEN heap (2 MiB) 0xFF boot allocator Hypervisor code 1st-level page table Linear mapping domu boot allocator Hypervisor code 1st-level page table VMALLOC_END I/O dom0 I/O PAGE_OFFSET 0xC domu dom0 Linear mapping Hypervisor dom0 3GiB 3GiB 0x

12 Domain Interactions Virtualization of peripherals in EmbeddedXEN is quite similar to the existing mechanisms in XEN. Driver split with frontend & backend drivers Communication with xenbus Use of grant tables for sharing/copying pages between domains However, a revisited (simplified) implementation of these mechanisms have been achieved in EmbeddedXEN. XEN store is dynamically allocated at boot time of guest OSes. No user space tools are required to manage XEN store entries or peripherals configs. Hotplugs & dynamic configs of peripherals are less relevant to embedded systems. 12

13 Domain Interactions domu is passed information during bootstrap under control of dom0. xenbus thread xenstore thread subsys OS Subsys subsys OS Subsys event_channel A/B Page Dom0 prod cons Page DomU prod cons xenstore xenbus thread backend drivers frontend drivers event_channel C/D via unpause_domu(store_mfn, store_evtchn) domu Hypervisor start_info->store_mfn start_info->store_evtchn 13

14 Domain Interactions Grant tables are used in a different way Shared pages are possible only in the vmalloc'd area. Kernel linear addresses are not shareable; contents needs to be copied using temporary mappings. Dom0 RAM DomU 0xFF Hypervisor Hypervisor 0xFF VMALLOC_END gnttab_foreign[1] gnttab VMALLOC_START gnttab(domu) gnttab(dom0) has references to has references to gnttab(domu) domu shared pages gnttab(dom0) has references to has references to gnttab(domu) gnttab(dom0) VMALLOC_END gnttab gnttab_foreign[0] VMALLOC_START mem_map_foreign mem_map_foreign(domu) dom0 mem_map_foreign(dom0) mem_map_foreign 14 0xC x GiB shared pages Hypervisor 3GiB 0xC x

15 Device Heterogeneity Different levels of heterogeneity At CPU level: various instructions sets (locks, cache, etc.), various PTEs (MMU) flags, various co-processors, etc. Compatibility ensured via hypercalls At peripherals level: not the same hardware Compatibility ensured via backend driver processing hypervisor include arch/arm built-in.o cache-v6.s cache-v7.s Kconfig Makefile mm.h proc-macros.s tlb-v6.s tlb-v7.s mach-msm mm mach-mx35 xen arch/arm kernel mm 15

16 Device Heterogeneity Example of ARMv6 running on ARMv7 CPU (imx35 -> HTC Desire HD) Original version linux domu/arch/arm/mm/cache-v6.s: ENTRY(v6_flush_kern_cache_all) mov r0, #0 #ifdef HARVARD_CACHE mcr p15, 0, r0, c7, c14, D cache clean+invalidate #ifdef CONFIG_SMP mcr p15, 0, r0, c7, c5, I+BTB cache invalidate #else b v6_icache_inval_all #endif #else mcr p15, 0, r0, c7, c15, Cache clean+invalidate #endif Paravirt (PV) version linux domu/arch/arm/mm/cache-v6.s:.extern xen_flush_kern_cache_all ENTRY(v6_flush_kern_cache_all) b xen_flush_kern_cache_all #if 0 /* paravirt */ mov r0, #0 mov pc, lr #endif /* 0 */ mov pc, lr linux domu//hypervisor.c: void xen_flush_kern_cache_all(void) { struct mmuext_op op; op.cmd = MMUEXT_FLUSH_CACHE; HYPERVISOR_mmuext_op(&op, 1, NULL, DOMID_SELF); } hypervisor-4.0.2/arch/arm/mm/cache_v7.s: ENTRY(xen_flush_kern_cache_all) stmfd sp!, {r4-r5, r7, r9-r11, lr} bl much more complex!! mov r0, #0 mcr p15, 0, r0, c7, c5, I+BTB cache invalidate ldmfd sp!, {r4-r5, r7, r9-r11, lr} mov pc, lr ENDPROC(xen_flush_kern_cache_all) 16

17 Device Heterogeneity Example of framebuffer device heterogeneity Configuration of framebuffer is retrieved from Dom0 via xenbus xenstore xenbus thread User space applications (/dev/fb0) framebuffer core Dom0 xenbus thread User space applications (/dev/fb0) DomU device-specific framebuffer msmfb_pan_update() framebuffer core dom0 fb memory (allocated by dom0) xenfb backend fb_event_dom_register fb_event_dom_switch Query framebuffer params FB config is retrieved from dom0 xenfb frontend domu fb memory (allocated by domu) 17

18 Device Heterogeneity Example of audio device heterogeneity DomU audio buffers are accessed from Dom0 via shared pages. Dom0 DomU xenstore xenbus thread User space applications /dev/msm_pcm_out - or - /dev/pcm/snd/pcm0c0d0p xenstore xenbus thread User space applications /dev/pcm/snd/pcm0c0d0p pcm subsystem pcm subsystem arch-specific audio driver xen-audio backend xenvaud_pcm_start xenvaud_pcm_stop xen-audio frontend Audio buffers allocated in domu Sound device (headset, speakers, etc.) 18

19 Conclusions & Future Work EmbeddedXEN is an embedded virtualization framework which puts emphasis on efficient and heterogeneous hardware. Application environments can be re-used "as such" on modern platforms (Android-based for example) taking advantage of last generation hardware. EmbeddedXEN relies on the main principles of XEN, with a revisited lightweight, but less secure architecture. A single multikernel binary image, easy to deploy on the target platform without additional tools, makes EmbeddedXEN well tailored to embedded systems. 19

20 Conclusions & Future Work Further investigation projects: Elaboration of a domu using a graphical desktop for user applications Support of multicore ARM CPUs (cortex-a9, cortex-a15) Live migration of domu using remote NFS-filesystem (migration within a cloud) Support of hard realtime OS (RTEMS-paravirt) 20

21 Thanks for your attention! Further information: 21