FUT92715 Solve the Paradox SUSE Linux Enterprise Live Patching Roadmap

FUT92715 Solve the Paradox SUSE Linux Enterprise Live Patching Roadmap Tuesday, Nov 8, 11:30 AM - 12:30 PM Friday, Nov 11, 9:00 AM - 10:00 AM Hannes Kühnemund SUSE Product Management Vojtěch Pavlík Director SUSE Labs

Sound familiar? [ Streaming Service hosted by 3rd party] LIVE SERVER DOWNTIME - 2.15 IS HERE! UPDATE Update: We continue to work on server maintenance in advance of upgrading the game version. Update 2: The issue with our server maintenance continues to take longer than expected. Update 4: So it turns out that it's a bit harder than we all wished. [ Hoster of an online game ] [ Hosting Service for Developers ] 2

Downtime Quiz planned Regular cadence - monthly - quarterly - yearly On the weekend In alignment with all stakeholders Combination of Taks - software updates / configuration - hardware exchange of defect parts - datacenter maintenance / AC Optimizable with - SUSE Manager 3

Downtime Quiz planned unplanned Regular cadence - monthly - quarterly - yearly On the weekend In alignment with all stakeholders Combination of Taks - software updates / configuration - hardware exchange of defect parts - datacenter maintenance / AC Optimizable with - SUSE Manager 4

Downtime Quiz planned unplanned Regular cadence No cadence - monthly - quarterly - yearly On the weekend Usually on Christmas Day In alignment with all stakeholders No alignment with stakeholders Combination of Taks Only one particular problem fixed - software updates / configuration - hardware exchange of defect parts - datacenter maintenance / AC Optimizable with Optimizable with - SUSE Manager - Various technologies available 9

Minimize Unplanned Downtime UPS RAS System Rollback RAID High Availability and GEO Virtualization Load Balancer? 10

Strike the balance? 11

Strike the balance? No Downtime Security 12

Since 2005, more than 75 data breaches in which 1,000,000 or more records were compromised have been publicly disclosed. But what about the non-disclosed ones? 13

Vulnerabilities Year # vulnerabilities 2010 4258 2011 3532 2012 4347 2013 4794 2014 7038 2015 8822 10000 8000 6000 4000 2000 2010 2011 2012 2013 2014 2015 28% Vulnerability type 2015 38% Operating System Browsers Mobile Devices Rank Operating System # vulnerabilities 2015 1 Apple OS X 384 2 Microsoft Windows Server 2012 155 3 Canonical Ubuntu Linux 152 4 Microsoft Windows 8.1 151 18% 16% Applications... 11 The 77 Source: [http://www.cvedetails.com] & [https://nvd.nist.gov/] & [http://www.gfi.com/blog/2015s-mvps-the-most-vulnerable-players/] 14

In a data center, not so long ago 15

In a data center, not so long ago Nov-11, 2015 December 2015 January February March April May June July August September 16

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 December 2015 January February March April May June July August September 17

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE: NVD: Common Vulnerabilities and Exposures It is a standard naming scheme used by the NVD National Vulnerability Database (https://nvd.nist.gov/) December 2015 January February March April May June July August September 18

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 December 2015 January February March April May June July August September 19

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 Reboot Dec-11, 2015 December 2015 January February March April May June July August September 20

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 Dec-11, 2015 CVE--0728 December 2015 January February March April May June July August September 21

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 Dec-11, 2015 CVE--0728 Jan-15, Reboot December 2015 January February March April May June July August September 22

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 Dec-11, 2015 CVE--0728 Jan-15, December 2015 January February March April May June July August September 23

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 Dec-11, 2015 CVE--0728 Jan-15, Reboot Feb-10, December 2015 January February March April May June July August September 24

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0774 CVE--2384 Dec-11, 2015 CVE--0728 CVE--0774 CVE--2384 Jan-15, CVE--0774 CVE--2384 Feb-10, CVE--0774 CVE--2384 December 2015 January February March April May June July August September 25

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0774 CVE--2384 Dec-11, 2015 CVE--0728 CVE--0774 CVE--2384 Jan-15, CVE--0774 CVE--2384 Feb-10, CVE--0774 CVE--2384 Reboot Mar-22, December 2015 January February March April May June July August September 26

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 Dec-11, 2015 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 Jan-15, CVE--0774 CVE--1583 CVE--2384 CVE--3134 Feb-10, CVE--0774 CVE--1583 CVE--2384 CVE--3134 Mar-22, CVE--1583 CVE--3134 December 2015 January February March April May June July August September 27

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 Dec-11, 2015 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 Jan-15, CVE--0774 CVE--1583 CVE--2384 CVE--3134 Feb-10, CVE--0774 CVE--1583 CVE--2384 CVE--3134 Mar-22, CVE--1583 CVE--3134 Jun-09, Reboot December 2015 January February March April May June July August September 28

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Dec-11, 2015 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Jan-15, CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Feb-10, CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Mar-22, CVE--1583 CVE--3134 CVE--4997 Jun-09, CVE--4997 December 2015 January February March April May June July August September 29

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Dec-11, 2015 CVE--0728 CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Jan-15, CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Feb-10, CVE--0774 CVE--1583 CVE--2384 CVE--3134 CVE--4997 Mar-22, CVE--1583 CVE--3134 CVE--4997 Jun-09, CVE--4997 Aug-16, Reboot December 2015 January February March April May June July August September 30

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Dec-11, 2015 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Jan-15, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Feb-10, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Mar-22, CVE--0758 CVE--1583 CVE--2053 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Jun-09, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Aug-16, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 December 2015 January February March April May June July August September 31

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Dec-11, 2015 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Jan-15, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Feb-10, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Mar-22, CVE--0758 CVE--1583 CVE--2053 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Jun-09, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--4997 CVE--5829 Aug-16, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Reboot Sep-12, December 2015 January February March April May June July August September 32

In a data center, not so long ago Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Dec-11, 2015 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Jan-15, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Feb-10, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Mar-22, CVE--0758 CVE--1583 CVE--2053 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Jun-09, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Aug-16, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 CVE--6480 Sample data taken on Sept-15, Sep-12, CVE--6480 December 2015 January February March April May June July August September 33

That reminds me of... 34

CVEs...? So what...? CVE--0728 gain privileges or cause a denial of service local users can bypass intended access restrictions gain privileges or cause a denial of service CVE-2015-7990 allows local users to cause a denial of service CVE-2015-7872 local users can cause a denial of service (OOPS) CVE-2015-6937 local users can cause a denial of service (NULL pointer dereference and system crash) local users to bypass intended AF_UNIX socket permissions or cause a denial of service (panic)... 35

Can t we patch software while it runs? Mankind already flew to the moon 36

Minimize Unplanned Downtime UPS RAS System Rollback RAID High Availability and GEO Virtualization Load Balancer Live Patching 37

Dynamic Software Updates Trinity Test 1945 (Manhattan Project) IBM punch card automatic calculators were used to crunch the numbers A month before the Trinity nuclear device test, the question was: What will the yield be, how much energy will be released? The calculation would normally take three months to complete recalculating any batches with errors Multiple colored punch cards introduced to fix errors in calculations while the calculator was running 38

Modern history of kgraft and other DSU technologies DSU: Dynamic Software Updates the goal is to be able to fix bugs and add features either by - changing some functions or - replacing the whole program kgraft developed as Open Source project by SUSE Labs Upstream project klp Takes best of both kgraft (SUSE) and kpatch (Red Hat) Still in catch up w.r.t. to features required by enterprises PoDUS Gupta Erlang Ginseng UpStare kpatch Ksplice Kitsune kgraft klp 1990 1995 2000 2005 2010 2015 39

ftrace: return address modification mechanism 40

Common Pitfalls Function Inlining DWARF to the rescue Static Symbols kernel keeps list: kallsyms IPA-SRA (optimization like -O2) gcc optimization log Multiple functions / dependencies consistency model Eternal sleepers (getty console 10) send fake signal SIGKGRAFT / ignore State transformation (req. for complex fixes) not in kgraft right now 3rd party kernel modules depends on what the modules does... 41

Consistency Requirement: ensure system consistency when deploying live patches Freezing the system (kpatch, ksplice) Lazy migration (kgraft) 42

Consistency Requirement: ensure system consistency when deploying live patches Freezing the system (kpatch, ksplice) Lazy migration (kgraft) stop_kernel(); check all stacks, whether any thread is stopped within a patched function If yes, resume kernel and try again later If not, flip the switch on all functions and resume the kernel 43

Consistency Requirement: ensure system consistency when deploying live patches Freezing the system (kpatch, ksplice) Lazy migration (kgraft) stop_kernel(); For each thread separately: check all stacks, whether any thread is stopped within a patched function Present the old version of functions to the thread until it leaves the kernel then give it the updated version If yes, resume kernel and try again later Wake sleeping threads up by a special signal. Prevent the signal from reaching userspace If not, flip the switch on all functions and resume the kernel Once all threads have exited the kernel at least once we're DONE 44

Consistency model for KLP? The chosen model is a merge of kpatch and kgraft Combines stack checking and per-thread changes Non-intrusive, fast finishing Works well already but requires both: Reliable stack unwinder (needed by kpatch) Worked on by Josh Poimboeuf @ Red Hat Currently needs FRAME POINTER up 10% slowdown of kernel execution Could use DWARF complex, being developed by SUSE speed is a concern initial implementation removed from upstream Kernel thread model cleanup (needed by kgraft) Worked on by Petr Mladek @ SUSE Touches both kthreads and workqueues These parts are the critical core Needs a lot of good planning and review Takes time Takes time 46

Live Patching on ppc64le? [ http://mpe.github.io/posts//05/23/kernel-live-patching-for-ppc64le/ ] 47

In a SUSE data center, today ;-) 48

In a SUSE data center, today ;-) Nov-11, 2015 December 2015 January February March April May June July August September 49

In a SUSE data center, today ;-) Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 December 2015 January February March April May June July August September 50

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 December 2015 January February March April May June July August September 51

In a SUSE data center, today ;-) Nov-11, 2015 CVE--0728 Dec-11, 2015 CVE--0728 December 2015 January February March April May June July August September 52

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, December 2015 January February March April May June July August September 53

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, December 2015 January February March April May June July August September 54

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, Feb-10, December 2015 January February March April May June July August September 55

In a SUSE data center, today ;-) Nov-11, 2015 CVE--0774 CVE--2384 Dec-11, 2015 CVE--0774 CVE--2384 Jan-15, CVE--0774 CVE--2384 Feb-10, CVE--0774 CVE--2384 December 2015 January February March April May June July August September 56

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, Feb-10, Mar-22, December 2015 January February March April May June July August September 57

In a SUSE data center, today ;-) Nov-11, 2015 CVE--1583 CVE--3134 Dec-11, 2015 CVE--1583 CVE--3134 Jan-15, CVE--1583 CVE--3134 Feb-10, CVE--1583 CVE--3134 Mar-22, CVE--1583 CVE--3134 December 2015 January February March April May June July August September 58

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, Feb-10, Mar-22, Jun-09, December 2015 January February March April May June July August September 59

In a SUSE data center, today ;-) Nov-11, 2015 CVE--4997 Dec-11, 2015 CVE--4997 Jan-15, CVE--4997 Feb-10, CVE--4997 Mar-22, CVE--4997 Jun-09, CVE--4997 December 2015 January February March April May June July August September 60

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, Feb-10, Mar-22, Jun-09, Aug-16, December 2015 January February March April May June July August September 61

In a SUSE data center, today ;-) Nov-11, 2015 CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Dec-11, 2015 CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Jan-15, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Feb-10, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Mar-22, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Jun-09, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 Aug-16, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 December 2015 January February March April May June July August September 62

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Jan-15, Feb-10, Mar-22, Jun-09, Aug-16, Sep-12, December 2015 January February March April May June July August September 63

In a SUSE data center, today ;-) Nov-11, 2015 CVE--6480 Dec-11, 2015 Sample data taken on Sept-15, CVE--6480 Jan-15, CVE--6480 Feb-10, CVE--6480 Mar-22, CVE--6480 Jun-09, CVE--6480 Aug-16, CVE--6480 Sep-12, CVE--6480 December 2015 January February March April May June July August September 64

In a SUSE data center, today ;-) Nov-11, 2015 Dec-11, 2015 Sample data taken on Sept-15, Jan-15, Feb-10, Mar-22, Jun-09, Aug-16, Sep-12, December 2015 January February March April May June July August September 65

In a SUSE data center, today ;-) Nov-11, 2015 CVE-2015-6937 CVE-2015-7872 CVE-2015-7990 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Dec-11, 2015 CVE--0728 CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Jan-15, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Feb-10, CVE--0758 CVE--0774 CVE--1583 CVE--2053 CVE--2384 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Mar-22, CVE--0758 CVE--1583 CVE--2053 CVE--3134 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Jun-09, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--4997 CVE--5829 CVE--6480 Aug-16, CVE--0758 CVE--2053 CVE--4470 CVE--4565 CVE--5829 CVE--6480 Sample data taken on Sept-15, Sep-12, CVE--6480 December 2015 January February March April May June July August September 66

Key Solution Highlights Available for SLES 12 onwards (x86-64) Provides fixes for Kernel bugs which affect Security Stability Data Integrity No runtime performance impact No interruption of applications while patching Allows full review of patch source code Build-in PTF support Patches available for most recent maintenance kernels (last 12 months) Currently based on kgraft OpenSource project 67

Where does SLE Live Patching make most sense?... and where not? What s your guess? 68

Where does SLE Live Patching make most sense?... and where not? What s your guess? (c) creativecommons.org/licenses/by/3.0 69

Where does SLE Live Patching make most sense?... and where not? What s your guess? (c) creativecommons.org/licenses/by/3.0 http://cdn.slashgear.com/wpcontent/uploads/2012/10/google-datacenter-tech-21.jpg 70

Outlook SLE Live Patching for ppc64le SLE Live Patching for IBM z Systems SLE Live Patching for Aarch64 User Space Live Patching Virtualization Live Patching 73

Your chance to win a 3ft (!) SUSE plush chameleon Receive Live Patching 60 day code for FREE here at SUSECON Register & logon to http://scc.suse.com Activate code (60 day counter starts) Add Live Patching to your SLES 12 SPx server by December 31st using eval code This lets you join a drawing for the SUSE chameleon Drawing happens on February 28 Winner will be contacted by email Chameleon will go on a journey After receiving the chameleon, winner sends selfie with chameleon back to SUSE ;-) 74

References One hour of downtime costs $100k for 95% of all enterprises http://itic-corp.com/blog/2013/07/one-hour-of-downtime-costs-100k-for-95-of-enterprises/ Kernel Live Patching for ppc64le http://mpe.github.io/posts//05/23/kernel-live-patching-for-ppc64le/ Forrester Linux vs. Unix Hot Patching have we reached the tipping point? http://blogs.forrester.com/richard_fichera/16-05-20-linux_vs_unix_hot_patching_have_we_reached_the_tipping_point Using Live Patching to patch a running SAP HANA system with zero interruption https://www.youtube.com/watch?v=e9kwtfwevlg 75

Thank you Hannes Kühnemund SUSE Product Management hkuehnemund@suse.com @hakuehnemund www.linkedin.com/in/hanneskuehnemund Vojtěch Pavlík Director SUSE Labs vojtech@suse.com 76