March 10, Linux Live Patching. Adrien schischi Schildknecht. Why? Who? How? When? (consistency model) Conclusion

Transcription

1 March 10, 2015

2 Section 1

3 Why Goal: apply a binary patch to kernel on-line. is done without shutdown quick response to a small but critical issue the goal is not to avoid downtime Limitations: simple changes changes in data structure layout

4 Section 2

5 Similar projects kexec CRIU

6 Live patching project (Suse) kpatch (Red Hat) ksplice (Oracle)

7 Section 3

8 Ftrace Insert a jmp at the beginning of all the functions gcc -pg -mfentry (add 5 bytes, that call mcount) mcount = 13% overhead Store each trampoline location in a section " mcount_loc" scripts/recordmcount use the linker to merge them all into vmlinux At boot: foreach entries in " mcount_loc", replace jmp with 5 NOP Set the call to mcount only if needed insert INT3 and its handler at NOP[0] insert addr at NOP[1-4] insert jmp at NOP[0] send NMI IPI to flush the instruction decoders ftrace_calller invokes the hooks

9 Ftrace Insert a jmp at the beginning of all the functions gcc -pg -mfentry (add 5 bytes, that call mcount) mcount = 13% overhead Store each trampoline location in a section " mcount_loc" scripts/recordmcount use the linker to merge them all into vmlinux At boot: foreach entries in " mcount_loc", replace jmp with 5 NOP Set the call to mcount only if needed insert INT3 and its handler at NOP[0] insert addr at NOP[1-4] insert jmp at NOP[0] send NMI IPI to flush the instruction decoders ftrace_calller invokes the hooks

10 Ftrace Insert a jmp at the beginning of all the functions gcc -pg -mfentry (add 5 bytes, that call mcount) mcount = 13% overhead Store each trampoline location in a section " mcount_loc" scripts/recordmcount use the linker to merge them all into vmlinux At boot: foreach entries in " mcount_loc", replace jmp with 5 NOP Set the call to mcount only if needed insert INT3 and its handler at NOP[0] insert addr at NOP[1-4] insert jmp at NOP[0] send NMI IPI to flush the instruction decoders ftrace_calller invokes the hooks

11 Ftrace Insert a jmp at the beginning of all the functions gcc -pg -mfentry (add 5 bytes, that call mcount) mcount = 13% overhead Store each trampoline location in a section " mcount_loc" scripts/recordmcount use the linker to merge them all into vmlinux At boot: foreach entries in " mcount_loc", replace jmp with 5 NOP Set the call to mcount only if needed insert INT3 and its handler at NOP[0] insert addr at NOP[1-4] insert jmp at NOP[0] send NMI IPI to flush the instruction decoders ftrace_calller invokes the hooks

12 Section 4

13 Subsection 1

14 new functions must be applied at once old function must not be executed after switching to the new one no threads runs on old functions no threads sleeps on them

15 Subsection 2

16 ksplice and kpatch_v1 add ftrace entry stop_machine() stop running processes disable interrupts Safeness check walk through the threads and check the stack enable the hook

17 ksplice and kpatch_v1 + safe + simple - stop_machine() stops all processes a while ( 40ms) - fail to upgrade non-quiescient kernel function (schedule)

18 Subsection 3

19 Functions can be called while patching. atomic reference counter inc at function entry dec_if_pos at function exit Active safeness check at context switch check stack entries safely sleeping tasks can be checked safely

20 + get rid of stop_machine() - kretprobe has no error notification - not incremental (big patch has many functions)

21 Why relying on the stack may be hazardous How Linux retrieve the stack entries (return addresses): get an address on the stack (local variable) while (valid_stack_ptr(addr)) ++addr check if pointer is in.text section use frame pointer if available ret address lies just above a frame pointer the frame pointer chain broke in some rarely used assembler code

22 Subsection 4

23 Reality check trampoline: a per-thread flag is set on each kernel entry/exit set a trampoline which monitor kernel entry/exit and redirect to the old/new function remove the trampoline and set the new function when all the processes have changed their universe at least once

24 - all processes must wake up or execute a syscall - sometimes requires a signal to be sent - kernel thread never leave the kernel + does not rely on the stack entries

25 Section 5

26 Dazed and confused Questions?