Lars Kurth Community Manager, Xen Project Chairman, Xen Project Advisory Board Director, Open Source, Citrix

Size: px

Start display at page:

Download "Lars Kurth Community Manager, Xen Project Chairman, Xen Project Advisory Board Director, Open Source, Citrix"

Candice Hall
6 years ago
Views:

1 Lars Kurth Community Manager, Xen Project Chairman, Xen Project Advisory Board Director, Open Source, Citrix lars_kurth Presentations on

2 Cloud 1. Virtual Machine Introspection (with demo) 2. Vulnerability Management In Xen Project 3. Live Patching (with demo) Embedded & Automotive 5. Why Virtualize Embedded Systems 8. Additional Security Properties of Xen 9. Xen Project In Security Applications 10. Xen Project in Embedded and Automotive (with demo) 12. Conclusion Bonus Material: Embedded & Automotive (Slide 49) 6. Hypervisor Architectures on ARM 7. PV Drivers and Protocols for Embedded Use-Cases 11. Schedulers and interrupt latency Bonus Material: Security (Slide 67) 4. Assessing a FOSS Project s Security Record

7 A new way to protect against malware Developed by Zentific, Citrix, Bitdefender, Intel and others

8 Installed in-guest agents, e.g. anti-virus software, VM disk & memory scanner, network monitor, etc. Can be disabled by rootkits Dom0 VM 2 VM 3 VM n App App App Dom0 Kernel Guest OS Guest OS Guest OS Drivers Agent(s) Agent(s) Agent(s)

9 Uses HW extensions to monitor memory (e.g. Intel EPT) Low Intrusion Register rules with Xen to trap on and inspect suspicious activities (e.g. execution of memory on the dynamic heap) Several Protected area Dom0 Security Appliance VM 1 VM VM 2 2 App VM VM 3 3 App VM VM n n App Dom0 Kernel Drivers Introspection Engine Guest OS Guest OS Guest OS authentication mechanism to protect the IF

10 All malware need an attack technique to gain a foothold Attack techniques exploit specific software bugs/vulnerability Most exploits use one of a small set of attack techniques Buffer Overflows, Heap Sprays, Code Injection, API Hooking, Because VMI protects against attack techniques It can protect against entirely new malware Verified to block these advanced attacks in real-time APT28, Energetic Bear, DarkHotel, Epic Turla, Regin, ZeuS, Dyreza, EternalBlue solely by relying on VMI WannaCry/EternalBlue blocked in real installations 1 1 businessinsights.bitdefender.com/hypervisor-introspection-defeated-enternalblue-a-priori

11 Rootkits Exploit 0-days in Operating Systems/System Software Can disable agent based security solutions (mask their own existence) VMI solutions operate from outside the VM Thus, it cannot be disabled using traditional attack vectors BUT: VMI is not a replacement, for traditional security solutions It is an extra tool that can be used to increase protection

12 Documentation wiki.xenproject.org/wiki/virtual_machine_introspection Products Bitdefender HVI XenServer Protection & Remedial Monitoring & Admin Citrix Ready AIS Introvirt XenServer Zentific Zazen Xen & XenServer & Protection & Remedial Monitoring & Admin Forensics & Data gathering Malware analysis Pratap Flickr

13 Pratap Flickr

14 Result of several community consultations

16 Fixing Security Bugs: Dedicated security team = security experts from within the Xen Project Community R P Security Team: Triage Creation of fix/patches Validation of fix/patches Assignment of CVE Issue description and risk analysis R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org

17 R Fix their systems/software: Eligible Xen Project Users are informed under embargo of the vulnerability P A Eligible Users = Pre-disclosure list members: Product Companies, Open Source & Commercial Distros (e.g. Huawei, Debian) Service/Cloud Providers (e.g. Alibaba) Large Private Downstream (e.g. Google) Allowed to share information via xen-security-issues- discuss@lists.xenproject.org R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa

18 General Publication: Information about vulnerability is made public R P A X Everyone else: Patches their systems either through security updates from distros/products or builds them from source. Users of service/cloud providers will not be impacted R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa

19 Product Vendors: Create and test live patches Product customers: Apply live patches here R P A X Security Team: Can a Livepatch can be created? No? If possible, re-write fix/patches Service Providers: Create, test and deploy live patches Users of service/cloud providers will not be impacted R: Vulnerability reported to security@xenproject.org P: Vulnerability pre-disclosed on xen-security-issues@lists.xenproject.org A: Vulnerability announced on xen-announce@lists.xenproject.org & xenbits.xen.org/xsa

20 A tale of close collaboration within the Xen Project Community

21 Replacing compiled functions with new code, encoded in an ELF file called payload, while the hypervisor is running without impacting running guests. const char *xen_extra_version(void) { return XEN_EXTRAVERSION; } push %rbp mov %rsp,%rbp lea 0x16698b(%rip),%rax leaveq retq const char *xen_extra_version(void) { return Hello World ; } push %rbp mov %rsp,%rbp lea 0x29333b(%rip),%rax leaveq Retq Design: xenbits.xenproject.org/docs/unstable/misc/livepatch.html

22 The exact source tree used to build the running Xen instance. The.config from the original build of Xen. A build-id onto which the livepatch will be applied. A source patch. livepatchbuildtools The exact same compilation toolchain used to build the running Xen. Livepatch payload

23 Supports stacking of different payloads; payloads depend on build-id Functionality: list: lists loaded and applied live patches upload: load & verify a live patch unload: unload a live patch apply: apply a live patch Depends on build-id of XSA 214 Depends on build-id of XSA 213 Depends on build-id of revert: un-apply a live patch Xen XSA 215 XSA 214 XSA 213

24 Target Technology Function + Data Data Structures Inline f() patching XenServer LivePatch Kernel Live Patching Dom0 & Guest Linux Kernel kgraft (SUSE) kpatch (RedHat) via hooks For Dom0 (CentOS) ksplice (Oracle) Hypervisor Xen LivePatch Xen 4.7 Xen 4.8 via hooks Future For Xen Integrates different solutions into a single user experience

Source patches + other build artifacts (livepatch-build or kpatch-build) package Live Live Patch Patch build for each patch level RPM RPM build for most recent patch level Hot Fixes

Verification and Validation: The process of patching a live hypervisor or kernel is not an easy task. What happens is a little bit like open heart surgery.

25 Source patches + other build artifacts (livepatch-build or kpatch-build) package Live Live Patch Patch build for each patch level RPM RPM build for most recent patch level Hot Fixes contain Per valid patch level: a Xen or Dom0 Live Patch Matching RPMs for most recent patch level In case of a reboot or for Xen/Dom0 not capable of Live Patching Extensive Verification and Validation: The process of patching a live hypervisor or kernel is not an easy task. What happens is a little bit like open heart surgery. The patient is the hypervisor and/or Dom0 itself, and precision and care are needed to get things right. One wrong move and it is game over. Hot Fix Verification Publication Hot Fix RPMs LPs Validation Signing RPMs LPs (iso) Q&A

updates (such that after reboot the patches are

updates Dom0 Dom0 Kernel (CentOS) Disk XAPI Toolstack

Running System instance that supports live patching

26 updates (such that after reboot the patches are applied) RPM Hot Fix LPs works out correct LP & updates Dom0 Dom0 Kernel (CentOS) Disk XAPI Toolstack (using native live patching tools or APIs) Hypervisor Running System instance that supports live patching RPM Hot Fix LPs download XenCenter or xe Initiates host update SysAdmin

27 xenbits.xenproject.org/people/larsk/lcc17 - Build LivePatch.mp4 xenbits.xenproject.org/people/larsk/lcc17 - Apply LivePatch.mov Pratap Flickr

29 Xen Project LivePatch Specification & Status xenbits.xenproject.org/docs/unstable/misc/livepatch.html wiki.xenproject.org/wiki/livepatch Xen Project LivePatch Presentations & Videos xenbits.xenproject.org/people/larsk/fosdem17-livepatch.pdf (Short) people/larsk/xpds16-livepatch.pdf (Long) Xen Project LivePatch Videos fosdem.org/2017/schedule/event/iaas_livepatxen/ XenServer xenserver.org Pratap Flickr

31 Consolidation Reduce cost, size, weight, power consumption and heat emission (already an issue today in cars) CAN bus scalability / Architecture complexity Reduce development costs: platform independence Security and Safety Separate safety critical apps from general apps Safety Certification of the Hypervisor Embedded Requirements (Bonus) Minimal IRQ latency Low or 0 scheduling overhead Drivers for special I/O devices Flexible architecture

32 System Partitioning Sandboxing drivers & system components Fine-grain control of VM capabilities Enables multi-layered security approach Other Security Features Trusted Execution Environment (TEE) Virtual Machine Introspection, alt2pm Live Patching

33 Application Storage Domain Network Domain Dom0 Kernel Guest Kernel Guest Kernel* Guest Kernel* NetFront Driver NetBack Driver BlockFront Driver BlockBack Driver Network Driver Disk Driver Xen Project Hypervisor Disk Controller Network Controller Driver Domain Guest OS*: Linux, BSD, MiniOS, unikernel,

VM Fine-grained policy, controlling which hypervisor functionality is accessible to this (class of) VM Effect: limit what an exploit in this VM could do Attack Surface Reduction Similar to Linux

34 VM Fine-grained policy, controlling which hypervisor functionality is accessible to this (class of) VM Effect: limit what an exploit in this VM could do Attack Surface Reduction Similar to Linux Security Modules/SELinux Same policy syntax as SELinux Different types, roles, users and attributes Same tools for policy compilation / verification (checkpolicy) security config passthrough inter-vm communication hypervisor domain(self) domain(other) memory (grant, mmu, shadow)

36 Documentation wiki.xenproject.org/wiki/dom0_disaggregation wiki.xenproject.org/wiki/xen_security_modules_:_xsm-flask Products & Projects Qubes OS OpenXT Crucible:Defense starlab.io Secure OS FOSS Platform for security research, security applications and embedded appliance integration building on Xen & OpenEmbedded Xen Project based virtualization platform for technology protection, cyber-hardening, and system integrity for aerospace & defense systems Pratap Flickr

38 USB Service Domain Banking Domain Personal Domain Firewall VM enforces network policies Network Domain Dom0 Secure UI and sysadmin domain User defined App VMs for individual apps or groups of apps

40 Dornerworks dornerworks.com/xen Consulting Xen Embedded Distros Xen for Xilinx Zynq Xen for NXP i.mx 8 ARLX Hypervisor DO-178 (EAL6+), IEC 62304, ISO MILS EAL FACE, VICTORY, ARINC 653 Starlab starlab.io Crucible and Crucible:Defense Xen embedded hypervisor In progress: DO-178, MILS EAL Uses a minimal Dom0 using MiniOS, disaggregation and XSM/FLASK AIS ainfosec.com BAE Systems baesystems.com Galois galois.com Maintain FreeRTOS Xen Port Developed and maintain HalVM Precedents of military grade certification for Xen based systems & xenbits.xenproject.org/people/larsk/xpds14 - Xen and the Art of Certification.pdf Pratap Flickr

41 GlobalLogic Product: Nautilus bit.do/gl-nautilus First product in production expected in Q Supports: HW: Renesas R-Car Gen2 & Gen3, TI Jacinto6, Intel Apollo Lake, Qualcomm 410C, Sinlinx A33 Guests: Linux up to 4.9 Android M, N, N-Car QNX, ThreadX, FreeRTOS PV Drivers for: GPU, Audio, HW accelerated Video codecs, DRM, Contributions: 27 smaller features from 2013 to 2016 EPAM Demo Next slide Interesting Features: Container based telematics applications running in a Xen VM that can be downloaded from a cloud service Ongoing Contributions: ABIs for PV Sound, PV Display & PV DRM Leading development of co-processor sharing framework LG Electronics Demo bit.do/lg-xen-demo-2016 Bosch Car GmbH Contributions 10 smaller features in 2016 Perseus (stealth) Founded by Xen maintainer bit.do/perseus-2017 Demo at IAA 2017 in Frankfurt, Germany ADIT Joint venture Bosch & DENSO Little known at this stage Pratap Flickr

42 xenbits.xenproject.org/people/larsk/ LCC17 - The Internet of Transportation[1080P].MP4 Pratap Flickr

43 AWS Telematics Simulation Agent ver 1.0 Telematics Simulation Agent ver 2.0 Monitoring Dashboard Driver Behavior Based Insurance Backend Dom0 - Control DomD HW Drivers & Cluster DomU Fusion DomU Linux IVI TrustZone Dom0 Services Cluster Simulation App Wayland/Weston Telematics simulation Agent (Acceleration, Braking, Corning, GPS) IVI Simulation App MW Frameworks Trusted Apps Minimal rootfs Wayland BE (Events/Display) OpenGL ES ALSA w PV_ALSAS_BE Container mgmt tool Containers Minimal rootfs with systems library PV DISPLAY PV EVENTS PV SOUND Linux Kernel w/o HW Drivers Linux Kernel with GPU and other HW Drivers Linux Kernel w/o HW Drivers Linux Kernel with GPU and w/o other HW Drivers Hypervisor OP-TEE OS R-Car H3 Platform TZ monitor

44 Picture by Lars Kurth

45 Xen Project & Security in Cloud: Only Hypervisor with VMI Protection from new classes of malware Several security companies working with XenServer Live Patching Disruption free application of vulnerabilities Used by several cloud providers Used best in commercial products, e.g. XenServer Industry Leading Vulnerability Process Includes QEMU and Kernel XSAs Picture by Lars Kurth

Xen Project in Embedded & Automotive: Extremely Flexible and Versatile Proven in many different markets Easy to port to new environments Easy to develop new PV drivers (see bonus 2) Highly

46 Xen Project in Embedded & Automotive: Extremely Flexible and Versatile Proven in many different markets Easy to port to new environments Easy to develop new PV drivers (see bonus 2) Highly customizable Security and Resilience Isolation, Partitioning, Security Features Safety Examples of Military Grade Certification BUT: looking at ways to make this easier and cheaper Challenges still being addressed Standardization of more I/O devices via PV protocols Standardization of GPU and co-processor sharing RTOS or other minimal OS as Dom0 Testing of embedded Hardware by the project Picture by Lars Kurth

47 Picture by Lars Kurth

48 Picture by Lars Kurth

50 EL0/PL0 least privileged mode used for applications (user mode) EL1/PL1 privileged mode used for running kernels such as the Linux kernel EL2/PL2 This has a higher level of privilege and can be used to run a hypervisor which takes control of the system and can host multiple "guest" operating systems

51 EL0 Guest Userspace Guest Userspace Guest Userspace Host Userspace Guest Userspace Guest Userspace EL1 Guest Kernel Guest Kernel Guest Kernel Guest Kernel Guest Kernel EL2 Hypervisor Native DDs Host Kernel + Hypervisor Native DDs Traditional Embedded Type 1 Hypervisor Type 2 with VHE/ARMv8.1 (e.g. KVM)

52 EL0 Host Userspace Guest Userspace Guest Userspace Host Userspace Guest Userspace Guest Userspace EL1 Guest Kernel Guest Kernel Guest Kernel Guest Kernel Guest Kernel EL2 Hypervisor Native DDs Host Kernel + Hypervisor Native DDs Traditional Embedded Type 1 Hypervisor Type 2 with VHE/ARMv8.1 (e.g. KVM)

53 EL0 Dom0 Userspace Toolstack Guest Userspace Guest Userspace Strong Isolation Device Drivers run in EL1, not EL2 EL1 Dom0 Kernel Guest Kernel Guest Kernel Protected Address Spaces: Grant tables Native DDs EL2 Xen Project Hypervisor Trusted Computing Base (TCB)

54 EL0 Dom0 Userspace Guest Userspace Guest Userspace Toolstack EL1 Dom0 Kernel Native DDs Guest Kernel Guest Kernel Control Plane Server: sysadmin EL2 Xen Project Hypervisor Embedded: config/setup, system health monitoring (watchdog), maintenance, SW updates,

56 Application Dom0 Kernel Guest Kernel *Back Driver Native Driver *Front Driver Xen Project Hypervisor I/O HW

57 Existing net, block, console keyboard, mouse, USB framebuffer, GPU sharing* New in Xen 4.9 9pfs (share a filesystem between VMs) Pvcalls (forward POSIX calls across VMs) multitouch, sound, display, DRM Developing New Ones Easy to write (GPL and BSD samples) Kernel and User Space *) A number of different approaches by different vendors in different market segments are being deployed, which are PV-like, but not strictly a PV protocol

59 Xen supports several different schedulers with different properties.

60 Xen supports several different schedulers with different properties. Hard real-time (ARINC653) Soft real-time (RTDS) Regular VM scheduler (Credit) Dedicated to 1 VCPU via pinning and Null scheduler no scheduler overheads

61 Scheduler Use-cases Today Future plans Credit General Purpose Supported Default Supported Optional Credit 2 General Purpose Supported Default Optimized for lower latency, higher VM density RTDS Soft & Firm Real-time Multicore Embedded, Automotive, Graphics & Gaming in the Cloud, Low Latency Workloads Experimental Better XL support <1μs granularity Supported Hardening Optimization ARINC 653 Hard Real-time Single core Supported Compile time Avionics, Drones, Medical Null Hard Real-time Experimental Supported

62 vcpu 0 vcpu 1 virq 109 pcpu 0 pcpu 1 IRQ injection Always on the CPU running the vcpu irq 109

63 vcpu 0 vcpu 1 virq 109 IF THEN virq target changes or vcpu is moved virq is moved immediately virq 109 pcpu 0 pcpu 1 irq 109

64 vcpu 0 vcpu 1 pcpu 0 virq 109 pcpu 1 Xilinx ZynqMP board (four Cortex A53 cores, GICv2) WARM_MAX (excluding the first 3 interrupts): <2000ns Without Null scheduler See blog.xenproject.org/2017/03/20/xen-on-arminterrupt-latency/ irq 109 IRQs always shadow the virq minimizes latency

Developer Portal: bit.do/xen-devs Xen on ARM whitepaper: bit.do/xenarm-white Xen on ARM wiki: bit.do/xenarm-wiki Port Xen to a new SOC: bit.do/xenarm-porting Add Xen support Xen to your OS: bit.

65 Developer Portal: bit.do/xen-devs Xen on ARM whitepaper: bit.do/xenarm-white Xen on ARM wiki: bit.do/xenarm-wiki Port Xen to a new SOC: bit.do/xenarm-porting Add Xen support Xen to your OS: bit.do/xenarm-os Device Passthrough presentation: bit.do/xenarm-pt OE meta-virtualization Xen recipe: bit.do/xenmeta OpenXT (Xen + OpenEmbedded): openxt.org Xenbedded presentation: bit.do/xenbedded Monthly ARM Community Call: bit.do/xenarm-call

66 Picture by Lars Kurth

67 Security Process Number of Vulnerabilities Media Coverage Other Considerations

68 Responsible Disclosure: fix critical systems/software before publication R F P A X Full Disclosure, post-fix: public disclosure with a fix R F A X Full Disclosure, immediate (no-fix): public disclosure without a fix R A F X R: Vulnerability reported to security@... P: Vulnerability pre-disclosed to eligible users A: Vulnerability announced publicly F: Fix available

69 Responsible only FOSS Project Bug Severity 1 Process Type Days 2 Who? 3 Linux Kernel via OSS-security distros 4 Medium Critical Responsible Disclosure OSS-security 5 Low Full Disclosure, no-fix QEMU (KVM) via QEMU Security Process 6 OpenStack OSSA OpenStack OSSN Xen Hypervisor Includes Linux & QEMU vulnerabilities in supported Xen configurations Medium Critical Low Medium Critical Low Responsible Disclosure Full Disclosure, no-fix Responsible Disclosure Full Disclosure, post-fix D D D, S, P Low Critical Responsible Disclosure 14 D, S, P 1) Is the CVE severity used to handle vulnerabilities differently? 2) Days embargoed (information is classified) 3) D = Distros/Products, S = Public Service, P = Private Downstream 4) 4) New members: 5) or devel list 6) Only handles x86 KVM bugs (no ARM or other bugs)

70 Impacts how long a user (aka you) is at risk Is my distro/vendor on a pre-disclosure list? A surprisingly large number of distros are not: including a few on Linux.com s Best Linux Distros of 2016 list Impacts cloud / service providers As a user, are security issues fixed before public disclosure? Low Severity vulnerabilities can still be High Risk Temporal and Environmental CVSS scores are not covered by CVE databases (neither cvedetails.com or nvd.nist.gov) Vulnerabilities can be chained together, making the combo High Severity (e.g. Hot Potato used 3 old unpatched vulnerabilities to gain root access)

71 cvedetails.com (bit.do/guide-cvedetails) Easy to use interface for vulnerability data Data from several sources Browsable by vendor, product, version, type, date Vulnerability statistics, trends, reports BUT: rigid getting data outside pre-defined vendor/product categories is near-impossible vulners.com (good guides on slideshare.net search for vulners) In many ways more accurate and flexible than cvedetails type:cve AND (description:kvm OR description:qemu) AND published: [2012 TO *] 307 type:cve AND (description:xen) AND published: [2012 TO *] 245 Works best when used through its API (in particular if you want to visualize the data)

350 300 250 222 197 169 517 51 120 7 129 34 143 5 63 Linux Kernel KVM w.

72 Linux Kernel KVM w. QEMU Xen Project Critical High Low 100 Medium 50 Legend: CVSS Score Distribution * *) Data up to Sept 4th, 2017 Vulnerability data from vulners.com

73 % 35% 33% of Xen stories cover Vulnerabilities Number of Vulnerabilities % % 20% 15% 16% Clips covering vulnerabilities (US only) Xen KVM QEMU Linux Xen KVM QEMU Linux 10% 5% 0% Xen 6.5% 2.5% KVM QEMU Linux % of Vulnerability Stories on The Register Data covering September 2016 September 2017: from vulners.com, mention.com and theregister.com

74 Does the Project Look for Vulnerabilities? Approach to Quality and Testing: e.g. Fuzzing, Audits of components Does the project award Bug Bounties (e.g. NetBSD)? Do vendors supporting the project offer Bug Bounties? Infrastructure related to Vulnerabilities Transparency: How well are processes documented Vulnerability Testing: XTF (in Xen) Vulnerability Tooling: XSATool, XSAMatch (in Xen)

Xen Project Automotive and Embedded Overview

Xen Project Automotive and Embedded Overview April, 2018 Lars Kurth Chairman, Xen Project Advisory Booard This work is licensed under a Creative Commons Attribution-Share Alike 4.0 (CC BY-SA 4.0) GENIVI