NAME Built-in Kernel Debugger for Linux - v4.4

Transcription

1 NAME Built-in Kernel Debugger for Linux - v4.4 Overview This document describes the built-in kernel debugger available for linux. This debugger allows the programmer to interactively examine kernel memory, disassemble kernel functions, set breakpoints in the kernel code and display and modify register contents. A symbol table is included in the kernel image and in modules which enables all non-stack symbols (including static symbols) to be used as arguments to the kernel debugger commands. Getting Started To include the kernel debugger in a linux kernel, use a configuration mechanism (e.g. xconfig, menuconfig, et. al.) to enable the CONFIG_KDB option. Additionally, for accurate stack tracebacks, it is recommended that the CONFIG_FRAME_POINTER option be enabled (if present). CONFIG_FRAME_POINTER changes the compiler flags so that the frame pointer register will be used as a frame pointer rather than a general purpose register. After linux has been configured to include the kernel debugger, make a new kernel with the new configuration file (a make clean is recommended before making the kernel), and install the kernel as normal. You can compile a kernel with kdb support but have kdb off by default, select CONFIG_KDB_OFF. Then the user has to explicitly activate kdb by booting with the kdb=on flag or, after /proc is mounted, by echo "1" > /proc/sys/kernel/kdb Youcan also do the reverse, compile a kernel with kdb on and deactivate kdb with the boot flag kdb=off or, after /proc is mounted, by echo "0" > /proc/sys/kernel/kdb When booting the new kernel, the kdb=early flag may be added after the image name on the boot line to force the kernel to stop in the kernel debugger early in the kernel initialization process. kdb=early implies kdb=on. If the kdb=early flag isn t provided, then kdb will automatically be invoked upon system panic or when the PAUSE key isused from the keyboard, assuming that kdb is on. Older versions of kdb used just a boot flag of kdb to activate kdb early, this is no longer supported. KDB can also be used via the serial port. Set up the system to have a serial console (see Documentation/serial-console.txt), you must also have a user space program such as agetty set up to read from the serial console. The control sequence <esc>kdb on the serial port will cause the kernel debugger to be entered, assuming that kdb is on, that some program is reading from the serial console, at least one cpu is accepting interrupts and the serial console driver isstill usable. Note: When the serial console sequence consists of multiple characters such as <esc>kdb then all but the last character are passed through to the application that is reading from the serial console. After exiting from kdb, you should use backspace to delete the rest of the control sequence. If the console is sitting on the login prompt when you enter kdb, then the login command may switch into upper case mode. This is not a kdb bug, it is a "feature" of login - if the userid is all upper case then login assumes that you using a TeleType (circa 1960) which does not have lower case characters. Wait 60 seconds for login to timeout and it will switch back to lower case mode. Note: Your distributor may have chosen a different kdb activation sequence for the serial console. Consult your distribution documentation. If you have both a keyboard+video and a serial console, you can use either for kdb. Define both video and serial consoles with boot parameters console=tty0 console=ttys0,38400 Any kdb data entered on the keyboard or the serial console will be echoed to both. If you are using a USB keyboard then kdb commands cannot be entered until the kernel has initialised the USB subsystem and recognised the keyboard. Using kdb=early with a USB keyboard will not work, the USB subsystem is initialised too late. September 17,

2 While kdb is active, the keyboard (not serial console) indicators may strobe. The caps lock and scroll lock lights will turn on and off, num lock is not used because it can confuse laptop keyboards where the numeric keypad is mapped over the normal keys. On exit from kdb the keyboard indicators will probably be wrong, they will not match the kernel state. Pressing caps lock twice should get the indicators back in sync with the kernel. Basic Commands There are several categories of commands available to the kernel debugger user including commands providing memory display and modification, register display and modification, instruction disassemble, breakpoints and stack tracebacks. Any command can be prefixed with - which will cause kdb to ignore any errors on that command, this is useful when packaging commands using defcmd. A line whose first non-space character is # is printed and ignored. The following table shows the currently implemented standard commands, these are always available. Other commands can be added by extra debugging modules, type? at the kdb prompt to get a list of all available commands. September 17,

3 Command Description bc Clear Breakpoint bd Disable Breakpoint be Enable Breakpoint bl Display breakpoints bp Set or Display breakpoint bph Set or Display hardware breakpoint bpa Set or Display breakpoint globally bpha Set or Display hardware breakpoint globally bt Stack backtrace for current process btp Stack backtrace for specific process bta Stack backtrace for all processes btc Cycle over all live cpus and backtrace each one cpu Display or switch cpus dmesg Display system messages defcmd Define acommand as a set of other commands ef Print exception frame env Show environment go Restart execution help Display help message id Disassemble Instructions kill Send asignal to a process ll Follow Linked Lists lsmod List loaded modules md Display memory contents mdwcn Display memory contents with width W and count N. mdp Display memory based on a physical address mdr Display raw memory contents mds Display memory contents symbolically mm Modify memory contents, words mmw Modify memory contents, bytes per_cpu Display per_cpu variables pid Change the default process context ps Display process status reboot Reboot the machine rd Display register contents rm Modify register contents set Add/change environment variable sr Invoke SysReq commands ss Single step a cpu ssb Single step a cpu until a branch instruction stackdepth Print the stack depth for selected processes summary Summarize the system Some commands can be abbreviated, such commands are indicated by a non-zero minlen parameter to kdb_register; the value of minlen being the minimum length to which the command can be abbreviated (for example, the go command can be abbreviated legally to g). If an input string does not match a command in the command table, it is treated as an address expression and the corresponding address value and nearest symbol are shown. Some of the commands are described here. Information on the more complicated commands can be found in the appropriate manual pages. cpu With no parameters, it lists the available cpus. * after a cpu number indicates a cpu that did not respond to the kdb stop signal. + after a cpu number indicates a cpu for which kdb has some data, but that cpu is no longer responding to kdb, so you cannot switch to it. This could be a cpu September 17,

4 that has failed after entering kdb, or the cpu may have saved its state for debugging then entered the prom, this is normal for an IA64 MCA event. I after a cpu number means that the cpu was idle before it entered kdb, it is unlikely to contain any useful data. F after a cpu number means that the cpu is offline. There is currenly no way to distinguish between cpus that used to be online but are now offline and cpus that were never online, the kernel does not maintain the information required to separate those two cases. cpu followed by a number will switch to that cpu, you cannot switch to a cpu marked *, + or F. This command is only available if the kernel was configured for SMP. dmesg [lines] [adjust] Displays the system messages from the kernel buffer. If kdb logging is on, it is disabled by dmesg and is left as disabled. With no parameters or a zero value for lines, dmesg dumps the entire kernel buffer. If lines is specified and is positive, dmesg dumps the last lines from the buffer. If lines is specified and is negative, dmesg dumps the first lines from the buffer. If adjust is specified, adjust the starting point for the lines that are printed. When lines is positive, move the starting point back by adjust lines, when lines is negative, move the starting point forward by adjust lines. dmesg -100 will dump 100 lines, from the start of the buffer. dmesg 100 will dump 100 lines, starting 100 lines from the end of the buffer, dmesg will dump 100 lines, starting 200 lines from the end of the buffer. dmesg will dump 100 lines, starting 100 lines from the start of the buffer. defcmd Defines a new command as a set of other commands, all input until endefcmd is saved and executed as a package. defcmd takes three parameters, the command name to be defined and used to invoke the package, a quoted string containing the usage text and a quoted string containing the help text for the command. When using defcmd, it is a good idea to prefix commands that might fail with -, this ignores errors so the following commands are still executed. For example, defcmd diag "" "Standard diagnostics" set LINES 2000 set BTAPROMPT 0 -id %eip-0x40 -cpu -ps -dmesg 80 -bt -bta endefcmd When used with no parameters, defcmd prints all the defined commands. go id kill lsmod Continue normal execution. Active breakpoints are reestablished and the processor(s) allowed to run normally. To continue at a specific address, use rm to change the instruction pointer then go. Disassemble instructions starting at an address. Environment variable IDCOUNT controls how many lines of disassembly output the command produces. Internal command to send a signal (like kill(1)) to a process. kill -signal pid. Internal command to list modules. This does not use any kernel nor user space services so can be used at any time. per_cpu <variable_name> [<length>] [<cpu>] Display the values of a per_cpu variable, the variable_name is specified without the per_cpu prefix. Length is the length of the variable, 1-8, if omitted or 0 it defaults to the size of the machine s register. To display the variable on a specific cpu, the third parameter is the cpu number. When the third parameter is omitted, the variable s value is printed from all cpus, except that zero values are suppressed. For each cpu, per_cpu prints the cpu number, the address of the variable and its value. September 17,

5 pid <number> Change the current process context, with no parameters it displays the current process. The current process is used to display registers, both kernel and user space. It is also used when dumping user pages. reboot Reboot the system, with no attempt to do a clean close down. stackdepth <percentage> Print the stack usage for processes using more than the specified percentage of their stack. If percentage is not supplied, it defaults to 60. This command is only implemented on i386 and ia64 architectures, patches for other architectures will be gratefully accepted. summary Print a summary of the system, including the time (no timezone is applied), uname information and various critical system counters. INITIAL KDB COMMANDS kdb/kdb_cmds is a plain text file where you can define kdb commands which are to be issued during kdb_init(). One command per line, blank lines are ignored, lines starting with # are ignored. kdb_cmds is intended for per user customization of kdb, you can use it to set environment variables to suit your hardware or to set standard breakpoints for the problem you are debugging. This file is converted to a small C object, compiled and linked into the kernel. You must rebuild and reinstall the kernel after changing kdb_cmds. This file will never be shipped with any useful data so you can always override it with your local copy. Sample kdb_cmds: #Initial commands for kdb, alter to suit your needs. #These commands are executed in kdb_init() context, no SMP, no #processes. Commands that require process data (including stack or #registers) are not reliable this early. set and bp commands should #besafe. Global breakpoint commands affect each cpu as it is booted. set LINES=50 set MDCOUNT=25 set RECURSE=1 bp sys_init_module INTERRUPTS AND KDB When a kdb event occurs, one cpu (the initial cpu) enters kdb state. It uses a cross system interrupt to interrupt the other cpus and bring them all into kdb state. All cpus run with interrupts disabled while they are inside kdb, this prevents most external events from disturbing the kernel while kdb is running. Note: Disabled interrupts means that any I/O that relies on interrupts cannot proceed while kdb is in control, devices can time out. The clock tick is also disabled, machines will lose track of time while they are inside kdb. Even with interrupts disabled, some non-maskable interrupt events will still occur, these can disturb the kernel while you are debugging it. The initial cpu will still accept NMI events, assuming that kdb was not entered for an NMI event. Any cpu where you use the SS or SSB commands will accept NMI events, even after the instruction has finished and the cpu is back in kdb. This is an unavoidable side effect of the fact that doing SS[B] requires the cpu to drop all the way out of kdb, including exiting from the event that brought the cpu into kdb. Under normal circumstances the only NMI event is for the NMI oopser and that is kdb aware so it does not disturb the kernel while kdb is running. Sometimes doing SS or SSB on ix86 will allow one interrupt to proceed, even though the cpu is disabled for interrupts. Ihave not been able to track this one down but I suspect that the interrupt was pending when kdb was entered and it runs when kdb exits through IRET even though the popped flags are marked as cli(). If any ix86 hardware expert can shed some light on this problem, please notify the kdb maintainer. RECOVERING FROM KDB ERRORS If a kdb command breaks and kdb has enough of a recovery environment then kdb will abort the command and drop back into mainline kdb code. This means that user written kdb commands can follow bad pointers September 17,

6 without killing kdb. Ideally all code should verify that data areas are valid (using kdb_getarea) before accessing it but lots of calls to kdb_getarea can be clumsy. The sparc64 port does not currently provide this error recovery. If someone would volunteer to write the necessary longjmp/setjmp code, their efforts would be greatly appreciated. In the meantime, it is possible for kdb to trigger a panic by accessing a bad address. DEBUGGING THE DEBUGGER kdb has limited support for debugging problems within kdb. If you suspect that kdb is failing, you can set environment variable KDBDEBUG to a bit pattern which will activate kdb_printf statements within kdb. See include/linux/kdb.h, KDB_DEBUG_FLAG_xxx defines. For example set KDBDEBUG=0x60 activates the event callbacks into kdb plus state tracing in sections of kdb. set KDBDEBUG=0x18 gives lots of tracing as kdb tries to decode the process stack. You can also perform one level of recursion in kdb. If environment variable RECURSE is not set or is 0 then kdb will either recover from an error (if the recovery environment is satisfactory) or kdb will allow the error to percolate, usually resulting in a dead system. When RECURSE is 1 then kdb will recover from an error or, if there is no satisfactory recovery environment, it will drop into kdb state to let you diagnose the problem. When RECURSE is 2 then all errors drop into kdb state, kdb does not attempt recovery first. Errors while in recursive state all drop through, kdb does not even attempt to recover from recursive errors. KEYBOARD EDITING kdb supports a command history, which can be accessed via keyboard sequences. It supports the special keys on PC keyboards, control characters and vt100 sequences on a serial console or a PC keyboard. PC Special keys Control VT100 key Codes Action Backspace ctrl-h Backspace 0x7f Delete character to the left of the cursor Delete ctrl-d Delete \e[3 Delete character to the right of the cursor Home ctrl-a Home \e[1 Go to start of line End ctrl-e End \e[4 Go to end of line Up arrow ctrl-p Up arrow \e[a Up one command in history Down arrow ctrl-n Down arrow \e[b Down one command in history Left arrow ctrl-b Left arrow \e[d Left one character in current command Right arrow ctrl-f Right arrow \e[c Right one character in current command There is no toggle for insert/replace mode, kdb editing is always in insert mode. Use delete and backspace to delete characters. kdb also supports tab completion for kernel symbols Type the start of a kernel symbol and press tab (ctrl-i) to complete the name If there is more than one possible match, kdb will append any common characters and wait for more input, pressing tab a second time will display the possible matches The number of matches is limited by environment variable DTABCOUNT, with a default of 30 if that variable is not set. AUTHORS Scott Lurndal, Richard Bass, Scott Foehner, Srinivasa Thirumalachar, Masahiro Adegawa, Marc Esipovich, TedKline, Steve Lord, Andi Kleen, Sonic Zhang. Keith Owens <kaos@sgi.com> - kdb maintainer. SEE ALSO linux/documentation/kdb/kdb_{bp,bt,env,ll,md,ps,rd,sr,ss}.man September 17,