Helping Developers Help You Tom Caputi tcaputi@datto.com
The ZFS Code Base 2
The ZFS on Linux* Code Base: Statistics Source code 641 files 383950 lines of code ZFS Test suite (not including ztest) 1327 files 109193 lines of script Contributors 3435 git commits 239 People 15 Sponsor Companies of the 2017 Developer Summit * Not counting Illumos, FreeBSD, OSX code bases! 3
4
5
Mistakes Happen! We re sorry. We are trying our best No one person knows the entire code base Please help us make ZFS better for everyone 6
The ZoL Git Repository 7
What is Github? git is a source code version control software Helps to keep track of code contributions Developers can work simultaneously on different issues Github is a website built around git Acts as the central truth of what ZFS on Linux is Issue tracking and comment threads Release management Hooks for running automated tests Easy(ish) user interface to manage the project 8
9
ZFS repo: https://github.com/zfsonlinux/zfs SPL repo: https://github.com/zfsonlinux/spl 10
Latest commit: contribution 11
Source code (including the last commit that touched that file / directory) 12
Commit log 13
14
Title and description 15
Authors and contributors 16
Diff: list of code changes made in this commit 17
Commit hash: unique identifier for this commit 18
19
Issues / Feature requests submitted by users 20
21
Reporting Issues and Debugging Basics 2 2
Search for existing issues Step 1: Check to make sure your issue hasn t already been filed Read through the ticket to make sure it matches your symptoms If the ticket is closed, read the comments for workarounds / solutions If the ticket is open, skip Step 2 23
Create a new issue ticket Step 2: If you don t find an issue like yours open a new ticket 24
Create a new issue ticket Step 3: Provide any debugging info you may have. If you think that the problem involves a certain feature, you can @ tag the developer of that feature to get their attention 25
Providing Debugging Information: The Basics System information ZFS Version: cat /sys/module/zfs/version Kernel version: uname -r Operating System Distribution CPU architecture: (almost always x86_64) ZFS Configuration Pool layout: zpool status <pool> List of relevant datasets: zfs list Properties of affected datasets: zfs get all <dataset> 26
Identifying Problems: Crashes and Deadlocks Crashes Task never completes Commands can t be killed System may become completely unresponsive Stack trace appears in dmesg Deadlocks Very similar to a crash (commands never complete, can t be killed) System still responsive on other consoles Stack traces may appear in dmesg after 5 minutes or so 27
28
Stack trace: shows where in the code the problem occurred * include a few lines above the trace 29
Identifying Problems: Other Issues Performance Issues Task eventually completes, but takes too much time / system resources Commands can be killed Warnings may be printed in dmesg (probably not) Be sure to provide statistics about what isn t performing well Other Issues Errors: ZFS returns errors when commands are used properly Non-intuitive behavior: ZFS does not function reasonably UI problems: commands / docs are inaccurate or unclear 30
Providing Debugging Information: Extended Steps to Reproduce THE MOST HELPFUL INFO YOU CAN PROVIDE Drastically reduces time to fix Allows developer to work on a fix without waiting for you to reply Most other debug info isn t important if you have this Sequence of commands that trigger the issue If the issue is intermittent, provide approximate reproduction rate Information on ZFS workload If you can t reproduce the info provide info about ZFS usage Kinds of files stored, zfs commands being run, etc 31
Providing Debugging Information: Extended Recompiling with --enable-debug and --enable-debuginfo ZFS has a compilation option that includes added sanity checks Turns on ASSERTs Turns on extra debugging by default Forces debug symbols to be compiled into the binaries Try this if: You are proficient at compiling your own source code Your issue is causing your system to panic You want to ensure that you are providing as much info as possible 32
Providing Debugging Information: Situational ZFS debug message log Does not require debug build echo 1 > /sys/module/zfs/parameters/zfs_dbgmsg_enable echo 0xffffffde > /sys/module/zfs/parameters/zfs_flags cat /proc/spl/kstat/zfs/dbgmsg Line Number from stack trace Requires debug build Allows you to convert a stack trace to a line of code (See next slide) 33
34
Some stack traces will include a RIP: gdb -batch -ex \ 'list *(multilist_sublist_remove+0x10)' \ $(modinfo zfs grep filename awk '{ print $2 }') 35
Issue Examples 3 6
Issue Examples Hard Issues https://github.com/zfsonlinux/zfs/issues/7151 Easy Issues https://github.com/zfsonlinux/zfs/issues/7335 https://github.com/zfsonlinux/zfs/issues/7365 37