Linux for Safety Critical Applications: Hunting down bug patterns

Transcription

1 Linux for Safety Critical Applications: Hunting down bug patterns

2 GNU/Linux for Safety Open-Source procedures and decisions Availability of technology and know-how Built on standards and well defined open interfaces Long-term availability HW support Security and secure interoperability (already certified to EAL4+) The use of GNU/Linux is not really driven by cost issues, there are technical reasons to consider it even more than any potential cost issues. 2

3 SIL2LinuxMP Claim: Risk of the GNU/Linux Kernel/OS inducing a hazard is acceptable. Assumption: Applications are developed according to IEC under consideration of Safety Application Conditions (SAC). Goal: Certify one version of the Linux Kernel for the use in safety critical applications Build a framework of methods and procedures that allows future certification of other (future) versions of Linux. 3

4 A Typical Safety Work flow 4

5 Covering System Safety SIL2LinuxMP is generic how to do this? Solution Turn it around Define what can be done (select a functional subset e.g. PSE51) Define what SIL can be achieved for the set of functions Constrain the usage SACs Non-interference , Annex F Working on generic solutions sometimes mandates introduction of constraints - if these need to be lifted for a specific project partial re-run of safety life cycle needed! 5

6 Misconceptions about Safety Not a checklist approach. It s a set of objectives - how you reach them is more or less your problem. Provides a systematic safety lifecycle (IEC ); Exceptions from this lifecycle if justified (IEC , 4.2 and ) 6

7 Generic Software Layer Kernel latest stable / well defined configuration(s) Selected minimum subset of drivers (based on use-cases of full partners) Core libraries guided by standards (e.g. POSIX supporting threaded RT-applications) Minimum runtime environment (e.g. Busybox based rootfs) (Application on top - not in scope) Well defined toolchain (build and validate) The goal is that the generic components of a GNU/Linux based safety related system can be qualified with low efforts. 7

8 Generic Software Layer 8

9 Procedure for SIL2LinuxMP Use-Cases per participating partner company based on the specific system: 1) Abstract use-case to function set 2) Perform system level hazard assessment Hazard List 3) Estimate risks (severity / probability) 4) Determine mitigation demands of components 5) Allocation of specific mitigation to system SW components and application layer (SACs) 6) Assess residual risk exhibited by SW/OS level failures 9

10 Non-compliant Development Coding Standards Specified tool-chain Defined submission procedures Well defined transition criteria (staging next mainline) Assigned responsibilities (Maintainers) Continuous peer-review from concept to code The DLC of the Linux kernel is maybe one of the most Stringent development cycles currently in use for such a large project. 10

11 Linux DLC Overview 11

12 Documentation/SubmitChecklist 5: Check your patch for general style as detailed in Documentation/CodingStyle. Check for trivial violations with the patch style checker prior to submission (scripts/checkpatch.pl). You should be able to justify all violations that remain in your patch. 9: Check cleanly with sparse. 15: All codepaths have been exercised with all lockdep features enabled. 22: Newly-added code has been compiled with `gcc -W' (use "make EXTRA_CFLAGS=-W"). This will generate lots of noise, but is good for finding bugs like "warning: comparison between signed and unsigned".??: Check cleanly with make coccicheck 12

13 sparse Catch the common problems that happen during development Build it into the default kernel (code annotations) Mandate its use (e.g. by linux-next building with sparse) Continuously update it when new pitfalls emerge 13

14 sparse Constraints Your functions MUST have types - no KR style default int! It does NOT cover all possible extensions to C - it does cover everything the kernel needs though (GNU and C99) If unsure - get on the mailinglist: linux-sparse, vger.kernel.org Integrate sparse into your build-env so that you use it continuously - if you wait until you are done - then it will take you out. 14

15 sparse What sparse can detect. Address_space mismatch (-Waddress-space) Type mismatches (-Wbitwise) Bad casting (-Wcast-truncate) Lock context (-Wcontext) Read man sparse for (a lot) more. 15

16 sparse Example: lock context checks context (expression, in_value, out_value) include/linux/compiler.h: acquires(x) attribute ((context(x,0,1))) releases(x) attribute ((context(x,1,0))) must_hold(x) attribute ((context(x,1,1))) 16

17 lockdep Lock Classes Lockdependency validator operates on lock classes Lock classes (i.e. inode class struct inode) Lock class same locking rules lockdep traces the state of a lock class Rolling profile of state and correctness of the dependencies 17

18 lockdep Lockclass States ever held in hardirq context ( == hardirq-safe ) ever held in softirq context ( == softirq-safe ) ever held with hardirqs enabled ( == hardirq-unsafe ) ever held with softirqs and hardirqs enabled ( == softirq-unsafe ) ever used ( ==!unused ) 18

19 lockdep state bits. +? acquired while irqs disabled acquired in irq context acquired with irqs enabled read acquired in irq context with irqs enabled. Unused locks cannot be the cause of an error 19

20 lockdep Single Lock Rules 1. The softirq-unsafe lock-class is automatically hardirq-unsafe as well. 2. The following states are mutually exclusive: <hardirq-safe> and <hardirq-unsafe> <softirq-safe> and <softirq-unsafe> 20

21 lockdep Multi-Lock Rules 1. The same lock-class may not be acquired twice if you must use the recursive lock versions and provide the nesting information. 2. Locking order matters that means if the locking order <L1> <L2> is used, then <L2> <L1> may never happen. Note that lockdep will fuss at any such ordering even if there is no temporal relation - lockdep can produce false positive (though not very often). 21

22 lockdep Example Code static int my_kthreadb(void *arg) { int i; static int my_kthreada(void *arg) { int i; for (i = 0; i < NUM_CYCLES; i++) { for (i = 0; i < NUM_CYCLES; i++) { mutex_lock(&counter_lockb); mutex_lock(&counter_locka); counter++; mutex_unlock(&counter_lockb); mutex_unlock(&counter_locka); mutex_lock(&counter_locka); mutex_lock(&counter_lockb); counter++; mutex_unlock(&counter_lockb); mutex_unlock(&counter_locka); } } return 0; return 0; } } 22

23 ] ] ] ] ] hello] ] ] ] hello] ] ] ] ] ] ] ] ] ] ] ] ] Institute ] of INFO: possible circular locking dependency detected ] lockdep #3 Tainted: G O my_kthread/4808 is trying to acquire lock: (&counter_locka){+.+...}, at: <ffffffffa091102c>] my_kthreadb+0x2c/0x6f but task is already holding lock: (&counter_lockb){+.+...}, at: <ffffffffa091101e>] my_kthreadb+0x1e/0x6f which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #1 (&counter_lockb){+.+...}: <ffffffff8107b976>] lock_acquire+0xda/0x123 <ffffffff >] mutex_lock_nested+0x5f/0x3b2 <ffffffffa091109b>] my_kthreada+0x2c/0x6f hello] <ffffffff8105cb8d>] kthread+0xba/0xc2 <ffffffff8140abec>] ret_from_fork+0x7c/0xb0 23 lockdep Example Output

24 lockdep Example Output ] -> #0 (&counter_locka){+.+...}: ] <ffffffff8107b11e>] lock_acquire+0xaf8/0xe ] <ffffffff8107b976>] lock_acquire+0xda/0x ] <ffffffff >] mutex_lock_nested+0x5f/0x3b ] <ffffffffa091102c>] my_kthreadb+0x2c/0x6f hello] ] <ffffffff8105cb8d>] kthread+0xba/0xc ] <ffffffff8140abec>] ret_from_fork+0x7c/0xb ] ] other info that might help us debug this: ] ] Possible unsafe locking scenario: ] ] CPU0 CPU ] ] lock(&counter_lockb); ] lock(&counter_locka); ] lock(&counter_lockb); ] lock(&counter_locka); ] ] *** DEADLOCK *** 24

25 ] 1 lock held by my_kthread/4808: ] #0: (&counter_lockb){+.+...}, at: <ffffffffa091101e>] my_kthreadb+0x1e/0x6f hello] ] ] stack backtrace: ] CPU: 1 PID: 4808 Comm: my_kthread Tainted: G O lockdep # ] Hardware name: Apple Inc. MacBookPro7,1/Mac-F222BEC8, BIOS MBP71.88Z.0039.B0B /01/ ] ffffffff81f1bad0 ffff880136c0bbf8 ffffffff813ffcd ] ffffffff81f1bad0 ffff880136c0bc48 ffffffff813fd2b ] ffff8800a05fa1d ffff8800a05fa ] Call Trace: ] <ffffffff813ffcd0>] dump_stack+0x4f/0x7c ] <ffffffff813fd2b1>] print_circular_bug+0x1f8/0x ] <ffffffff8107b11e>] lock_acquire+0xaf8/0xe ] <ffffffff8107b976>] lock_acquire+0xda/0x ] <ffffffffa091102c>]? my_kthreadb+0x2c/0x6f hello] ] <ffffffff8107bc8b>]? mark_held_locks+0x54/0x ] <ffffffff >] mutex_lock_nested+0x5f/0x3b2 25 lockdep Example Output

26 coccinelle SmPL abstracts away irrelevant parts One small semantic patch can affect hundreds of lines of code patches in the linux kernel are already produced by semantic patches (see kernel janitors mailing list for LOTS of examples) make coccicheck checks the kernel source against all semantic patches in scripts/coccinelle/ 26

27 coccinelle Applications Find and Fix Bugs finding code patterns systematically fix bugs Collateral Evolution - Evolution in a library API leads to changes in the client find interactions with the library systematically transform the interactions with the library More esoteric Applications: Finding Security Vulnerabilities Code Obfuscation 27

28 coccinelle Program matching and transformation for unpreprocessed C code. Fits with the existing habits of Systems (Linux) programmers. Semantic Patch Language (SmPL): Based on the syntax of patches, Declarative approach to transformation High level search that abstracts away from irrelevant details A single small semantic patch can modify hundreds of files, at thousands of code sites 28

29 coccinelle Basics coccinelle abstracts irrelevant parts away. Different spacing and indentation Variablenames Irrelevant Code (" ") Isomorphism e.g. if (!y) if (y == NULL) if (NULL == y) 29

30 coccinelle Basics Rules and Metavariable expression rulename is optional, but might be needed later metavariable expression E; could be any expression, e.g.: 1+2 sizeof(struct foobar) strlen(mystring) 30

31 coccinelle Basics Transformation Specifications - in the leftmost column to remove this line + in the leftmost column to add this line * in the leftmost column to mark something of interest (cannot be used with + and -) 31

32 coccinelle Collateral Evolutions In the library: int foo(int x, int y); int foo(int x); Hence, in the clients calls could look like this: foo(3, 5); z = foo(x, y); if (foo(3, my_y)) { foo(bar(42), y); foo(3); z = foo(x); if (foo(3)) { foo(bar(42)); Finding all those occurrences automatically, and fixing them is a lot of work and includes lots of potential for bugs. 32

33 coccinelle Collateral Evolutions expression Ex, Ey; foo(ex -,Ey ) 33

34 coccinelle Simple Example A simple Example: The!& expression -!E1 & E2 +!(E1 & E2) 34

35 #include depends on expression ( - (((n) + (d)) - 1) / (d)) + DIV_ROUND_UP(n,d) - (((n) + ((d) - 1)) / (d)) + DIV_ROUND_UP(n,d) ) Define multiple matching and transformation rules Express that applicability of a rule depends on the success or failure of another rule. 35 coccinelle Depends On

36 coccinelle Missing unlock // // // // A mutex_lock is not matched by a mutex_unlock before an error return. Confidence: Moderate Copyright: (C) Gilles Muller, Julia Lawall, EMN, INRIA, DIKU. GPLv2. URL: expression mutex_lock(l);... when!= mutex_unlock(l) when any when strict ( if (...) {... when!= mutex_unlock(l) + mutex_unlock(l); return...; } mutex_unlock(l); ) 36

37 Thanks! 37