Xenbedded: Xen-based client virtualization for phones and tablets

Transcription

1 Xenbedded: Xen-based client virtualization for phones and tablets Kyle Temkin Chris Patterson Assured Information Security, Inc. August 25 th, 2016 // XenSummit 2016

2 Who are we? Kyle Temkin Hardware Hacker and Security Researcher Freenode: ktemkin Chris Patterson Senior Computer Engineer Freenode: cjp256

3 Why run Xen on Phones/Tablets? First, and foremost: it s cool! } This is the frontier of virtualiztion tech there s lots of new stuff to build! Mobile virtualization also offers more pragmatic benefits: } Separation of critical and non-critical tasks. Thnk Qubes or OpenXT for mobile: different VMs for different types of activity. Potential tools to deal with the whole BYOD mess. Pictured: Google Pixel C Photo: Google Store } Potential to extend benefits beyond phones/tablets; e.g. to devices with integrated touchscreens such as car media centers.

4 Isn t ARM support enough? Q: Xen already supports many ARM processors: shouldn t it be able to run on most phones and tablets? A: Almost. There are still a few things we need: Hardware support for ARM s virtualization extensions. Bootloader support for launching software in hypervisor mode (HYP/EL2). It s easy to find yourself locked out of these modes! First-Stage Bootloader (usually comes with SoC; often in ROM) Second-Stage Bootloader (e.g. u-boot, android bootloader) Final Payload (Linux, Xen) All too often, the FSBL drops down to Supervisor Mode (PL1/EL1)... leaving no way back into PL2/EL2!

5 Landing in PL2/EL2 - CPU mode All forms of interrupts must be masked in PSTATE.DAIF (Debug, SError, IRQ and FIQ). The CPU must be in either EL2 (RECOMMENDED in order to have access to the virtualisation extensions) or non-secure EL1. Q: What if the boot chain can t land me in PL/EL2? A: If you have to use that device, things will get interesting. Best option: convince your vendor to support EL2 on unlocked devices. Last resort: dust off your hardware hacking skills! Samsung CID hacks (developer bootloader) Qualcomm TrustZone exploits (chain to get at HVBAR)

6 Finding the Right Device Finding a good phone or tablet to work with can be tricky. You need a device that: } Supports the ARM Virtualization Extensions, including a full-featured GIC. } Provides an unlockable bootloader entered in NS-HYP or EL2. You want a device that: } Is supported by an open-source Linux variant (for dom0) } Is closely approximated by an available evaluation board. } Has an accessible UART serial port

7 Identify Devices with HYP/EL2 Payloads If your device runs Linux 3.7+ (arm32) or Linux 4.3+ (arm64), checking for HYP/EL2 is easy: # dmesg grep HYP A 1 #arm32 CPU: All CPU(s) started in HYP mode. CPU: Virtualization extensions available. # dmesg grep EL2 #arm64 CPU: All CPU(s) started at EL2 # #Note: these only work bare-metal! Otherwise, you can compile a simple kernel module to check:

8 Fragmentation of Linux Baselines Depending on the device, several Linux variants may be available, including mainline and per-device variants. Mainline Android Variants ChromiumOS variants linux4tegra linux-sunxi linux-exynos linux-xilinx linux-96boards oh, my! Not all Linux forks are created equal: } If possible, select a device that has reasonable mainline support. (e.g. MSI Primo73) } Failing that, you may want to look for a fork that deviates less from mainline: A recent kernel version is usually a good hint. Another good hint: look at the device tree for your target device. Are many of its compatible strings supported in mainline?

9 Approximating HW with Dev Boards Pixel C NVIDIA Jetson TX1 image source: Google Store image source: NVIDIA Dev Blogs Pairing your device with an evaluation board can dramatically speed up development. It helps a lot to have a known good environment to test your hardware in. Is the issue in your code, or with the weird tablet bootloader? Compare and find out! If you can get one, a JTAG debugger make all the difference!

10 Finding UART Consoles Most but not all phones and tablets have a UART surfaced somewhere. Friendlier devices have externally accessible UARTs; some devices require soldering. Internally Accessible Externally Accessible Credit: ssvb Original image. The Xen-compatible Primo 73 s UART is accessible both directly on its PCB or, with some configruation, routed through its internal microsd slot. Embedded Linux and Android communities are a good first resource for UART pinouts: you ll be surprised how often the community has documented how to get UART access!

11 Example: Pixel C UART While the Pixel C has no documented external UART, digging through some of its firmware reveals an internal USB-to-serial-adapter. } When the USB-C connector s CC1 and CC2 pins are pulled down with 5.1Ω resistors, the device enters debug mode, and the convert is connected as below: } Naturally, we went a little overboard, and our intern (@c4757p) developed a full fledged open-source-hardware debug adapter for these ports:

12 Getting Xen onto Your Device Once you ve selected a device, the next step is to get Xen to start. A basic path forward might be: } Boot a simple, custom hello-world kernel. } Boot up the Xen kernel alone get it to crash! } Provide Xen with its required images, and see how things behave.

13 Booting a Hello World Kernel Many devices require their kernels to be encapsulated in payload-specific image formats. } ChromiumOS: FIT image with specific contents, prefixed with signed header } Many Android Devices: Android-specific boot.img format } We ve seen lots of variants on these formats! Booting a Hello World image helps to verify that you have the particulars of your image format correct. } Also a great time for a final check for HYP/EL2! } Good starter example:

14 (Some People Just Want to) Watch Xen Crash Once you can boot an image on your particular device, try swapping out your hello-world kernel for Xen. } Be sure to build Xen with EARLYPRINTK set up for your board! Launching Xen... - UART enabled - CPU booting - Current EL Xen starting at EL2 - Zero BSS - Setting up control registers - Turning on paging - Ready (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) No valid device tree (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... = Victory!

15 Getting Xen its Images Once you successfully have Xen crashing, you re ready to try a real boot, which means passing in several necessary images. This can be tricky. We expect our bootloader to load: } The Xen kernel image, typically a zimage/image; } A flattened device tree (dtb) describing the target device; } The dom0 kernel, typically also a Linux zimage/image; and } An optional ramdisk (initrd/initramfs). but most image formats won t load all of these! } We ll need a next-stage bootloader to make our images available. u-boot is a popular choice.

16 Discharge Boot Adapter We created Discharge, a simple image adapter for phones and tablets. } Combines the images relevant to a Xen boot into the single-image formats expected by most consumer device loaders (Android/CrOS). } Faster and significantly lower-footprint than u-boot; but also significantly less powerful. If u-boot better fits your use case, use it! Full MIT-licensed source code:

17 Xen on Your SoC If things seem broken at first, don t despair! You may have to deal with quirks for your particular SoC or with Xen behaviors (bugs?) that your new configuration has unmasked. } This is where things can get complicated. We won t provide a general method for moving forward (this talk is too short!), but we will provide examples. } In particular, see Julien Grall s slides from Porting Xen on ARM to a new SoC.

18 Example: Tegra Legacy Interrupt Controller SPI[n:32] SPI Tegra LIC Legacy GIC GIC CPU Interface IRQ Interrupt Distributor Controllers GIC CPU Interface IRQ CPU 0 CPU 1 The Tegra LIC, by default, gates all Shared interrupts. On Tegra systems, a set of Legacy Interrupt Controllers must be programmed in addition to the core Generic Interrupt Controller (GIC). } Idea is straightforward: implement Xen support for the Tegra LIC. Figuring out the best way to do this: less straightforward.. } One working implementation is available online: (Very) Simplified View of Tegra Interrupt Routing We re currently working to get a proper solution mainlined!

19 Client Virtualization: New Peripheral Support for Xen Tablets and phones present a huge variety of new peripherals we may want to virtualize for guest domains: Display/GPUs Light Sensors Flash LEDs Touch Sensors Cameras Wireless/Cellular Inertial Sensors Fingerprint Sensors and more every day. Virtualizing these devices require three components: } A device driver running in domain zero; ideally provided by the BSP distribution. } A backend driver to (e.g.) multiplex access to the physical hardware; and } A frontend driver to deliver events to the guest kernel.

20 Display and Input Multiplexing To provide our guests with a high-performance display capability, our group has worked to develop a new display technology for Xen. } Provides a near-native experience for both mobile and desktop devices, including backend and frontend drivers for multi-touch and multi-monitor support. } Includes a full input stack, including PV drivers. Baseline: X11 and VNC } We won t go into detail here. This is covered in full by the next talk in this room ( Display Handler ). Display Handler: Native Look

21 Multi-Touch with openxt-kbdfront Multi-touch input is handled by an extended version of xen-kbdfront. } Entirely open; source is GPLv2 and available on our GitHub. ( } Backwards-compatible with xen-kbd, and thus works with existing backends. } We may submit our changes as patches to xen-kbdfront, if well received.

22 Goal: Android Support Support for Android guests is not far off basic functionality is working on x86 but work towards this support is still underway. Challenges still to be solved: } Blocker: Modern Android versions require* hardware-accelerated video; so we ll need to implement a solution for multiplexing access to the host GPU. } Soft req: Android isn t as useful on a tablet without mobile-centric power mgmt., including intelligent use of host sleep states, paralleling Android s wakelocks. } Soft req: Many android applications require services from TrustZone Applets (Trustlets), but we now need a guest-aware TrustZone, or equivalent. *At least, according to Google. Software rendering may be an alternative, but doesn t work as-is, and likely isn t worth pursuing for performance reasons.

23 Android: Accelerated Graphics Android s SurfaceFlinger compositor and many applications require OpenGL support. Potential paths forward: } On x86, HVM guests can run Android under the VirGL virtualized GPU, which is heavily tied to its QEMU implementation. See Rob Herring s Android-on-QEMU effort. To support ARM PVH guests, we d need to develop a PV transport for the VirGL. } Raw API Sharing solutions such as the emugl library used by the Android emulator also offer potential solutions.

24 Android: TrustZone Implications Single TZ-Enabled Enclave Unused? dom0 dom1 dom2 TEE Service VM dom0 Single TZ-Enabled Enclave Core System Services TEE Service VM dom1 TEE Service VM dom2 dom0 dom1 dom2 In the current implementation, only one trusted enclave is available. } This is definitely not an optimal architecture for per-guest services, such as secure password stores, DRM, and cryptographic services. As a result, we disable guest SMCs: and lose these capabilities altogether! ( No Netflix? ) } Consider the implications of existing TrustZone escalations (e.g. the QSEE bug) do you really want your guests interacting with TrustZone services? } Potential solution: leverage hypervisor separation to create a service-vm trusted execution environment (TEE) for per-guest services.

25 Questions and Demo