Hunting Down Data Races in the Linux Kernel

Transcription

1 Hunting Down Data Races in the Linux Kernel Eugene A. Shatokhin

2 Data Race "Simultaneous access to the same memory location by multiple threads, where at least one of the accesses modifies the memory location." Hans-J. Boehm "How to miscompile programs with benign data races" LinuxCon Europe, Oct 21-23, /28

3 Data Race Thread 1, my_read(): printk( KERN_INFO "balance: %d.\n", balance); Thread 2, my_write(): balance++; int balance; Thread 3, my_write(): balance++; Thread 4, my_read(): printk( KERN_INFO "balance: %d.\n", balance); LinuxCon Europe, Oct 21-23, /28

4 Example 1 static struct my_dev my_devices[2]; static int my_open(struct inode *inode, struct file *filp) { struct my_dev *dev = &my_devices[iminor(inode)]; filp->private_data = dev; } /* if opened for the 1st time, allocate the buffer */ if (dev->data == NULL) { dev->data = kzalloc(dev->buffer_size, GFP_KERNEL); if (dev->data == NULL) return -ENOMEM; } return 0; LinuxCon Europe, Oct 21-23, /28

5 Example 2 static int my_open(struct inode *inode, struct file *filp) { printk(kern_info "%s", buf); return nonseekable_open(inode, filp); } static int init my_init_module(void) { dentry = debugfs_create_file( "data", S_IRUGO, dir, NULL, &my_fops); buf = kstrdup("hello World!"); return 0; } LinuxCon Europe, Oct 21-23, /28

6 Common Races Racy updates Init VS usage With user space LinuxCon Europe, Oct 21-23, /28

7 Approaches Static analysis Dynamic analysis Hybrid (dynamic + symbolic, ) LinuxCon Europe, Oct 21-23, /28

8 Static Analysis Benefits Safer Many code paths are checked No specific hardware, tests, Drawbacks False alarms "Code path explosion" Requirements for resources LinuxCon Europe, Oct 21-23, /28

9 Dynamic Analysis Benefits Longer code paths are checked Less false positives Less resources needed Drawbacks Not all code paths are checked Specific hardware or emulation Less safe LinuxCon Europe, Oct 21-23, /28

10 Static Analysis: Tools Coverity Scan (?) Coccinelle (?) Smatch RacerX, RELAY, LinuxCon Europe, Oct 21-23, /28

11 Dynamic Analysis: Tools Redflag Stony Brook University Jinx Corensic KernelStrider + ThreadSanitizer KernelStrider - ROSA, ISPRAS ThreadSanitizer - Google RaceHound ISPRAS LinuxCon Europe, Oct 21-23, /28

12 Redflag Compile-time instrumentation GCC plugins Collect data in runtime, analyze "offline" 1 Gb+ memory buffer, 7 m events max Lockset & block-based algorithms Availability -? "Redflag: a Framework for Analysis of Kernel-Level Concurrency" LinuxCon Europe, Oct 21-23, /28

13 Example 3 static int my_open(struct inode *inode, struct file *filp) { printk(kern_info "%s", buf); //? return nonseekable_open(inode, filp); } static int init my_init_module(void) { buf = NULL; //? dentry = debugfs_create_file( "data", S_IRUGO, dir, NULL, &my_fops); buf = kstrdup("hello World!"); return 0; } LinuxCon Europe, Oct 21-23, /28

14 Jinx Commercial Thin hypervisor Stochastic testing; replay Crash early, crash often Dead? Details: LinuxCon Europe, Oct 21-23, /28

15 KernelStrider Instrumentation on load Recording: memory accesses function calls ThreadSanitizer "offline" analysis x86, kernel (3.2+ recommended) GPL v2 Details and code KernelStrider: ThreadSanitizer: LinuxCon Europe, Oct 21-23, /28

16 Main Components Target module Kernel space User space KS Core Instrumented module Function handlers Output system Trace ThreadSanitizer Report LinuxCon Europe, Oct 21-23, /28

17 Example Report WARNING: Possible data race during read of size 4 at 0xffffc : {{{ T2 (L{L3, L4}): #0 readl (io.h:55) #0 e1000_set_rx_mode (e1000_main.c:2237) #1 e1000_configure (e1000_main.c:390) #2 e1000_open (e1000_main.c:1402) Concurrent write(s) happened at (OR AFTER) these points: T1 (L{}): #0 e1000_vlan_filter_on_off (e1000_main.c:4823) #1 e1000_probe (e1000_main.c:1222) Locks involved in this report (reporting last lock sites): {L3, L4} <> Race verifier data: e1000:.text+0x76af,e1000:.text+0x9a5 }}} List of threads: T1 modprobe T2 NetworkManager LinuxCon Europe, Oct 21-23, /28

18 KernelStrider Demo LinuxCon Europe, Oct 21-23, /28

19 KernelStrider Kernel memory (5-6) * sizeof(code) + (20-300) Mb for buffers Performance atl1c, r8169, e1000 No bandwidth loss (100 Mbit LAN, iperf3) CPU load: 5-10% => 35-40% LinuxCon Europe, Oct 21-23, /28

20 KernelStrider False alarms: macros and inlines custom synchronization unhandled relations Possible solutions: annotations (?) trace events (?) compile-time instrumentation LinuxCon Europe, Oct 21-23, /28

21 RaceHound Software & hardware BPs No false alarms Complements KernelStrider Can confirm potential races, esp. repetitive x86, kernel 3.2+ GPL v2 Maintainer: Nikita Komarov (ISPRAS) Code: See also: DataCollider for MS Windows LinuxCon Europe, Oct 21-23, /28

22 RaceHound 1. SW BP triggers 2. Get address, set HW BP 3. Delay 4. HW BP may report race 5. Execute orig. instruction 6. Go on LinuxCon Europe, Oct 21-23, /28

23 RaceHound Kernel memory several Mbytes or less Performance Low freq. - negligible High freq. - may be significant atl1c, IRQ handler, iperf3 Bandwidth loss: 10-12x LinuxCon Europe, Oct 21-23, /28

24 Results Confirmed by RaceHound 6 races in e1000 (Intel) NAPI poll() VS ndo_start_xmit() Under analysis atl1c, r8169, e LinuxCon Europe, Oct 21-23, /28

25 Results: r8169 (Realtek) Races, analysis needed (?) Benign False positives Type Count ndo_open VS probe 1 NAPI VS ndo_start_xmit 2 counters 4 intentional 4 driver-specific sync (IRQ) 1 device registers (IO mem) 2 skb memory reuse 4 Total 18 LinuxCon Europe, Oct 21-23, /28

26 Future Directions WiFi Output compression Merge into KEDR framework More relations to handle (?) Compile-time instrumentation LinuxCon Europe, Oct 21-23, /28

27 Contributions are Welcome Bug reports Suggestions, comments Code esp. new subsystems: USB, FS, video, Mailing list for KernelStrider and KEDR: LinuxCon Europe, Oct 21-23, /28

28 Thank you! LinuxCon Europe, Oct 21-23, /28