Tecniche di debugging nel kernel Linux. Andrea Righi -

Transcription

1 Tecniche di debugging nel kernel Linux

2 Agenda Overview (kernel programming) Kernel crash classification Debugging techniques Example(s) Q/A

3 What's a kernel? The kernel provides an abstraction layer for the applications to use the physical hardware resources Kernel basic facilities Process management Memory management Device management System call interface

4 User space Good for debugging (gdb) Lots of user-space libraries available Unpredictable latency (context switch, scheduler, syscall,...) Overhead Impossibility to fully interact with interrupt routines Impossibility to access certain memory address More difficult to share certain features with other drivers Reliability: user processes can be terminated upon critical system events (OOM, filesystem errors, etc.)

5 Kernel space Written in C and assembly No debugging tool (kgdb, UML,...) Bugs can hang the entire system User memory is swappable, kernel memory can't be swapped out Kernel stack size is small (8K / 4K - THREAD_SIZE_ORDER) Floating point is forbidden Userspace libraries are not available Linux kernel must be portable (this is important if you consider to contribute mainstream) Closed source kernel modules taint the kernel

6 Example kernel module #include <linux/init.h> #include <linux/module.h> /* Module constructor */ static int init hello_init(void) printk(kern_info "Hello, world!\n"); return 0; /* Module destructor */ static void exit hello_exit(void) printk(kern_info "Goodbye\n"); module_init(hello_init); module_exit(hello_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Andrea Righi MODULE_DESCRIPTION("BetterEmbedded hello world example");

7 Kernel problems Kernel panic (fatal error for the system) Kernel oops (non-fatal error) Wrong result (fatal from user's perspective)

8 Kernel panic No recovery is possible Example: exception in an atomic context (i.e., interrupt) Typically result in a system reboot (panic=n), or blinking LED or just hang

9 [ ] general protection fault: 0000 [#1] PREEMPT SMP [ ] Modules linked in: crashtest(o) [last unloaded: crashtest] [ ] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O rc7+ #535 [ ] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ ] task: ffff88003d90a2c0 ti: ffff88003d92e000 task.ti: ffff88003d92e000 [ ] RIP: 0010:[<ffffffff811ab0e5>] [<ffffffff811ab0e5>] kmalloc_track_caller+0xd5/0x2b0 [ ] RSP: 0018:ffff88003e EFLAGS: [ ] RAX: RBX: ffff88003e1d6a20 RCX: be841 [ ] RDX: be801 RSI: RDI: [ ] RBP: ffff88003e0039c8 R08: d6a20 R09: [ ] R10: R11: R12: [ ] R13: R14: R15: ffff88003d [ ] FS: (0000) GS:ffff88003e000000(0000) knlgs: [ ] CS: 0010 DS: 0000 ES: 0000 CR0: b [ ] CR2: ab008 CR3: dc8000 CR4: e0 [ ] DR0: DR1: DR2: [ ] DR3: DR6: ffff0ff0 DR7: [ ] Stack: [ ] be801 ffff88003d92ffd8 ffffffff d ffff880034e3f300 [ ] ffff88003e003a [ ] ffff88003e003a00 ffffffff c ffff880034e3f

10 ] Call Trace: ] <IRQ> ] [<ffffffff d>]? alloc_skb+0x7d/0x ] [<ffffffff c>] kmalloc_reserve.isra.52+0x3c/0xa ] [<ffffffff d>] alloc_skb+0x7d/0x ] [<ffffffff81677e5b>] tcp_send_ack+0x3b/0xf ] [<ffffffff8166ab1e>] tcp_ack_snd_check+0x5e/0xa ] [<ffffffff81671c64>] tcp_rcv_established+0x204/0x6f ] [<ffffffff810e678e>]? put_lock_stats.isra.26+0xe/0x ] [<ffffffff8167c681>] tcp_v4_do_rcv+0x161/0x ] [<ffffffff816fea39>]? _raw_spin_lock_nested+0x79/0x ] [<ffffffff8167dc91>] tcp_v4_rcv+0x731/0x ] [<ffffffff810e706f>]? lock_is_held+0x5f/0x ] [<ffffffff816563d8>] ip_local_deliver_finish+0xc8/0x2f ] [<ffffffff a>]? ip_local_deliver_finish+0x4a/0x2f ] [<ffffffff81656e77>] ip_local_deliver+0x47/0x ] [<ffffffff >] ip_rcv_finish+0x140/0x5e ] [<ffffffff816570e3>] ip_rcv+0x233/0x ] [<ffffffff >] netif_receive_skb_core+0x6a2/0x ] [<ffffffff81625a10>]? netif_receive_skb_core+0x50/0x ] [<ffffffff >] netif_receive_skb+0x21/0x ] [<ffffffff >] netif_receive_skb+0x23/0x1f ] [<ffffffff >] napi_gro_receive+0x98/0xd ] [<ffffffff81565c5a>] e1000_clean_rx_irq+0x18a/0x ] [<ffffffff >] e1000_clean+0x251/0x ] [<ffffffff810e678e>]? put_lock_stats.isra.26+0xe/0x ] [<ffffffff810e6df4>]? lock_release_holdtime.part.27+0xd4/0x ] [<ffffffff >] net_rx_action+0xd5/0x2e ] [<ffffffff81088d17>] do_softirq+0xf7/0x ] [<ffffffff810891d5>] irq_exit+0xb5/0xc ] [<ffffffff >] do_irq+0x63/0xd ] Code: c8 48 8b 55 c0 48 8b e0 ff ff a8 08 0f 85 5f c 8b 23 4d 85 e4 0f d 4a 40 4d 8b 07 <49> 8b 1c 04 4c 89 e f c7 08 0f 94 c0 84 c ] RIP [<ffffffff811ab0e5>] kmalloc_track_caller+0xd5/0x2b ] RSP <ffff88003e003988> ] ---[ end trace baac76a23c6da73c ] ] Kernel panic - not syncing: Fatal exception in interrupt

11 Kernel oops A message is displayed in the log when a recoverable error has occurred in kernel space Example: access a bad address (i.e., NULL pointer dereference) An oops does not mean the system has crashed Current process is killed Oops message is displayed along with a registers dump and a stack trace

12 [ ] BUG: unable to handle kernel NULL pointer dereference at (null) [ ] IP: [<ffffffffa00003c6>] procfs_write+0x2d6/0x320 [crashtest] [ ] PGD 3a78d067 PUD 362be067 PMD 0 [ ] Oops: 0002 [#1] PREEMPT SMP [ ] Modules linked in: crashtest(o) [ ] CPU: 0 PID: 1587 Comm: bash Tainted: G O rc7+ #535 [ ] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ ] task: ffff88003a7ec580 ti: ffff f6000 task.ti: ffff f6000 [ ] RIP: 0010:[<ffffffffa00003c6>] [<ffffffffa00003c6>] procfs_write+0x2d6/0x320 [crashtest] [ ] RSP: 0018:ffff f7e78 EFLAGS: [ ] RAX: RBX: RCX: e [ ] RDX: RSI: ffffffffa RDI: ffff f7eaa [ ] RBP: ffff f7ee0 R08: R09: [ ] R10: ffff88003a7ec580 R11: R12: [ ] R13: a R14: ffff f7f50 R15: [ ] FS: (0000) GS:ffff88003de00000(0063) knlgs: f75f76c0 [ ] CS: 0010 DS: 002b ES: 002b CR0: [ ] CR2: CR3: CR4: f0 [ ] DR0: DR1: DR2: [ ] DR3: DR6: ffff0ff0 DR7: [ ] Stack: [ ] ffffffff811b66cb ffff88003a7ec580 [ ] ffff f7ec8 4f e [ ] b9fa0 ffff fd a [ ] Call Trace: [ ] [<ffffffff811b66cb>]? vfs_write+0x1bb/0x1f0 [ ] [<ffffffff8121a86d>] proc_reg_write+0x3d/0x80 [ ] [<ffffffff811b65d8>] vfs_write+0xc8/0x1f0 [ ] [<ffffffff811b6ad5>] SyS_write+0x55/0xa0 [ ] [<ffffffff81708ce5>] sysenter_dispatch+0x7/0x1f [ ] [<ffffffff813c50ae>]? trace_hardirqs_on_thunk+0x3a/0x3f [ ] Code: e1 f3 6f e1 48 c7 c a0 e8 d5 f3 6f e1 e9 e2 fd ff ff c7 45 d e9 d6 fd ff ff e8 bf fc ff ff e9 cc fd ff ff <c7> e9 bc fd ff ff eb fe 66 c7 07 [ ] RIP [<ffffffffa00003c6>] procfs_write+0x2d6/0x320 [crashtest] [ ] RSP <ffff f7e78> [ ] CR2: [ ] ---[ end trace 33bbddb c ]---

13 Kernel fault classification panic( have a nice day... ;-) ) BUG() / BUG_ON(condition) exception (i.e., invalid opcode, division by zero,...) memory corruption stack overflow/underflow NOTE: in kernel space stack size is limited to 2 pages (8K in almost all architectures) write after free write to a bad address concurrent access without protections (locks, etc.) soft lockup lock a CPU without giving other tasks a chance to run hard lockup lock a CPU without giving other tasks or interrupts a chance to run hung task: task doesn't get a chance to run for more than N seconds scheduling while atomic deadlock use FPU registers in kernel space

14 Useful debugging kernel options Kernel Hacking section -> CONFIG_KALLSYMS_ALL: print function names instead of addresses in kernel messages CONFIG_FRAME_POINTER: get useful stack info in case of kernel bugs CONFIG_DEBUG_ATOMIC_SLEEP: enable sleep inside atomic section checks (i.e., sleep from interrupt handler, sleep when a lock is held, etc...) CONFIG_LOCKUP_DETECTOR: detect hard and soft lockups CONFIG_LOCKDEP: lock dependency enging (deadlock detection) CONFIG_DYNAMIC_FTRACE: enable individual function tracing dynamically (via debugfs /sys/kernel/debug/tracing)

15 Debugging techniques blinking LED printk() procfs SysReq key (Documentation/sysrq.txt) function instrumentation (kprobes) dynamic ftrace (CONFIG_DYNAMIC_FTRACE) debugger (kgdb)

16 printk() Advantages easy to use no need any other system support Disadvantages have to modify and rebuild kernel/modules no interactive debugging

17 printk(): levels printk levels KERN_EMERG: system is unusable KERN_ALERT: action must be taken immediately KERN_CRIT: critical condition KERN_ERR: error condition KERN_WARNING: warning condition KERN_NOTICE: normal condition KERN_INFO: informational KERN_DEBUG: debug message Show kernel messages: # dmesg Redirect all kernel messages to the console # echo 8 > /proc/sys/kernel/printk

18 procfs static int procfs_read(struct seq_file *m, void *v)... static ssize_t procfs_write(struct file *file, const char user *ubuf, size_t count, loff_t *pos)... static int procfs_open(struct inode *inode, struct file *file) return single_open(file, procfs_read, NULL); static int procfs_release(struct inode *inode, struct file *file) return 0; static const struct file_operations procfs_fops =.open = procfs_open,.read = seq_read,.write = procfs_write,.llseek = seq_lseek,.release = procfs_release, ; static int init myproc_init(void) if (!proc_create( myproc, 0666, NULL, &procfs_fops)) return -ENOMEM; return 0; static void exit myproc_exit(void) remove_proc_entry( myproc, NULL);

19 Kprobes (Kernel probes) Kprobes allow to dynamically break into any kernel routine and collect debugging and performance information (CONFIG_KPROBES=y) Trap almost every kernel code address, specifying a handler routine to be invoked when the breakpoint is hit How does it work? Make a copy of the probed instruction and replace the original instruction with a breakpoint instruction (int3 on x86) When the breakpoint is hit, a trap occurs, CPU's registers are saved and the control passes to the Kprobes pre-handler The saved instruction is executed in single-step mode The Kprobes post-handler is executed The rest of the original function is executed

20 Kprobes (example) static int my_handler(struct kprobe *p, struct pt_regs *regs) /* Do something here... */ static struct kprobe my_kp =.pre_handler = my_wrapper,.symbol_name = schedule_timeout, ; static int init my_kprobe_init(void) int ret; ret = register_kprobe(&my_kp); if (ret < 0) printk(kern_info "%s: error %d\n", func, ret); return ret; return 0; static void exit my_kprobe_exit(void) unregister_kprobe(&my_kp);

21 Dump a stack trace static const char function_name[] = "schedule_timeout"; static int my_handler(struct kprobe *p, struct pt_regs *regs) dump_stack(); printk(kern_info "%s called %s(%d)\n", current->comm, function_name, (int)regs->di); static struct kprobe my_kp =.pre_handler = my_wrapper,.symbol_name = function_name, ; static int init my_kprobe_init(void) int ret; ret = register_kprobe(&my_kp); if (ret < 0) printk(kern_info "%s: error %d\n", func, ret); return ret; return 0; static void exit my_kprobe_exit(void) unregister_kprobe(&my_kp);

22 Dynamic ftrace # mount -t debufs none /sys/kernel/debug # cd /sys/kernel/debug # echo sys_nanosleep hrtimer_interrupt > set_ftrace_filter # echo function > current_tracer # echo 1 > tracing_on # usleep 1 # echo 0 > tracing_on # cat trace # tracer: function # # entries-in-buffer/entries-written: 5/5 #P:4 # # _-----=> irqs-off # / _----=> need-resched # / _---=> hardirq/softirq # / _--=> preempt-depth # / delay # TASK-PID CPU# TIMESTAMP FUNCTION # usleep-2665 [001] : sys_nanosleep <-system_call_fastpath <idle>-0 [001] d.h : hrtimer_interrupt <-smp_apic_timer_interrupt usleep-2665 [001] d.h : hrtimer_interrupt <-smp_apic_timer_interrupt <idle>-0 [003] d.h : hrtimer_interrupt <-smp_apic_timer_interrupt <idle>-0 [002] d.h : hrtimer_interrupt <-smp_apic_timer_interrupt

23 KGDB + QEMU Setting up kgdb using kvm/qemu $ kvm -m smp 4 -drive file=debian-6-i386.img -vnc :1 -redir tcp:5190: :22 -kernel /src/linux/arch/x86/boot/bzimage -append "root=/dev/sda1 kgdbwait kgdboc=ttys0" -serial pty char device redirected to /dev/pts/3 (label serial0) $ gdb vmlinux (gdb) target remote /dev/pts/3

24 Debugging workqueues workqueue: asynchronous process execution context kworkers are going crazy (using too much cpu)? Something being scheduled in rapid succession A single work item consumes alots of cpu cycles How to debug? kernel tracepoints: echo workqueue:workqueue_queue_work > /sys/kernel/debug/tracing/set_event kworker stack trace: cat /proc/the_offending_kworker/stack root ? S 12:07 0:00 [kworker/0:1] root ? S 12:07 0:00 [kworker/1:2] root ? S 12:12 0:00 [kworker/0:0] root ? S 12:13 0:00 [kworker/1:0]

25 References J. Corbet, A. Rubini, G. Kroah-Hartman: Linux Device Drivers 3rd Edition Linux documentation Linux weekly news:

26 Q/A You're very welcome! #bem2013