Trinity A Linux kernel fuzz tester. Presented by Dave Jones Red Hat. Slides license: CC-BY-SA
Syscall fuzzing.
A short history lesson.. Completely random. Circa 1991: Tsys. SVR4 Circa 2001: kg_crashme.
A short history lesson.. More intelligent. 2005: Ilja van Sprundel: 'sysfuzz' 2006: Clement LECIGNE: netusse 2006: 'scrashme' begins. Malformed, but good enough arguments.
A short history lesson.. Current state of the art. 2010: scrashme becomes 'Trinity'. 2010: Tavis Ormandy: iknowthis. A mix of malformed and plausible arguments.
How it works..
Trinity architecture. Give syscalls what they expect. Annotations. 'Address', 'length', 'file descriptor', 'filename'... Generic fuzz routines based on arg type. Generate 'things' on startup. Multi-process fuzzing using 'things'. Shared mmap between children.
Process model. Startup Main loop. Watchdog Child1 Child2 Child3 Child4
Watchdog. Keeps track of child progress. SIGKILL if 'stuck' Sanity check shm.
Types ARG_RANDOM_INT ARG_FD ARG_LEN ARG_ADDRESS ARG_NON_NULL_ADDRESS ARG_PID ARG_RANGE ARG_OP ARG_LIST ARG_RANDPAGE ARG_CPU ARG_PATHNAME ARG_IOVEC ARG_IOVECLEN ARG_SOCKADDR ARG_SOCKADDR_LEN
Syscall annotation example. struct syscall syscall_mmap = {.name = "mmap",.num_args = 6,.arg1name = "addr",.arg1type = ARG_ADDRESS,.arg2name = "len",.arg2type = ARG_LEN,.arg3name = "prot",.arg3type = ARG_LIST,.arg3list = {.num = 4,.values = { PROT_READ, PROT_WRITE, PROT_EXEC, PROT_SEM }, },.arg4name = "flags",.arg4type = ARG_OP,.arg4list = {.num = 2,.values = { MAP_SHARED, MAP_PRIVATE }, },.arg5name = "fd",.arg5type = ARG_FD,.arg6name = "off",.arg6type = ARG_LEN, };.sanitise = sanitise_mmap,
.sanitise routines. void sanitise_mmap(int childno) { /* no fd if anonymous mapping. */ if (shm->a4[childno] & MAP_ANONYMOUS) shm->a5[childno] = -1; } /* page align non-anonymous mappings. */ if (shm->a4[childno] & MAP_ANONYMOUS) shm->a6[childno] &= PAGE_MASK; else shm->a6[childno] = 0;
Interesting numbers. Instead of completely random numbers.. 0x00000000 0x00000001 Rand() % 256; 0x00000fff // 4095 0x00001000 // 4096 0x00001001 // 4097 0x00008000 0x0000ffff 0x00010000 0x40000000 0x7fffffff 0x80000000 0x80000001 0x8fffffff 0xc0000000 0xf0000000 0xff000000 0xffff0000 0xffffe000 0xffffff00 (rand() % 256); 0xffffffff;
More interesting numbers.. Addresses. 0x0000000100000000; 0x7fffffff00000000; 0x8000000000000000; 0xffffffff00000000; 0x0000000100000000 low; 0x00007fffffffffff; // x86-64 canonical addr end. 0x0000800000000000; // First x86-64 non-canonical addr 0xffff800000000000 (low << 4); // x86-64 canonical range 2 begin 0x7fffffff00000000 low; 0x8000000000000000 low; 0xffff880000000000 (low << 4); // x86-64 PAGE_OFFSET 0xffffffff80000000 (low & 0xffffff); // x86-64 kernel text 0xffffffffa0000000 (low & 0xffffff); // x86-64 module space 0xffffffffff600000 (low & 0x0fffff); // x86-64 vdso
Struct fabrication. Struct size parameter checks or EINVAL Sockaddr Needs knowledge of every protocol. static void gen_ipv4(unsigned long *addr, unsigned long *addrlen) { struct sockaddr_in *ipv4; ipv4 = malloc(sizeof(struct sockaddr_in)); if (ipv4 == NULL) return; } ipv4->sin_family = PF_INET; ipv4->sin_addr.s_addr = random_ipv4_address(); ipv4->sin_port = rand() % 65535; *addr = (unsigned long) ipv4; *addrlen = sizeof(struct sockaddr_in);
Self Protection. Check addresses when passing them to syscalls Blocking syscalls. Trapping signals.
Mo' problems..
Did I break the fuzzer? Proving absence of kernel bug after fix.
Over-sanitising void sanitise_tee(int childno) { if ((rand() % 10) > 0) { shm->a1[childno] = shm->pipe_fds[rand() % MAX_PIPE_FDS]; shm->a2[childno] = shm->pipe_fds[rand() % MAX_PIPE_FDS]; } }
Multiplexed syscalls. Socketcall calls other syscalls. Fcntl, others...
Avoiding OOM. Leaks! Tracking allocations.
Reproducability. Periodic reseeding. System state on startup. Corrupting state.
The results..
Results: Over 150 bugs found last year. +more by others. Bugs in new code found very quickly. Once discovered, bugs usually are repeatable Bugs tend to mask other bugs.
Types of bugs found. Not just syscall code! Lots of networking bugs. VM bugs. Drivers.
Types of bugs found. Very old bugs. (Oldest: Nov 1996. 2.1.8 setsockopt) only libfoo calls this mempolicy: (bug lifetime: 4+ years) VM under stress corner cases. page_alloc failures. OOM killer bugs.
Types of bugs found. CVE fix that needed a CVE. Error path memory leaks. Poor coverage tested code (Weirdo network protocols) Clearly untested code. Broken locks (ATM/BKL)
Types of bugs found. Hardware bugs. Marginal hardware shows up faults quickly under load. SMI handlers.
Types of bugs found. Trinity can break userspace too! trinity -c execve -V /bin
What next?
Next? Extending existing code New syscall support. More.sanitise routines. (currently just 10%) More struct fabrication. Lots to do here. More network protocol support. Most protos already done, but need improving. More flags for ARG_OP/ARG_LIST
Next? Syscall chains. Destructors. Root mode.
Ioctl Worst interface known to man. int ioctl(int fd, int request,...); The second argument is a device-dependent request code. The third argument is an untyped pointer to memory. request has encoded in it whether the argument is an in parameter or out parameter, and the size of the argument argp in bytes Need to annotate every ioctl. Pass down correct structures to right fd. Need to be very careful.
Demo time!
Questions? http://www.codemonkey.org.uk/ Contact: davej@redhat.com Slides license: CC-BY-SA