Seminar report: DieHarder: Securing the heap

Transcription

1 Seminar report: DieHarder: Securing the heap Otto Waltari Helsinki Seminar report UNIVERSITY OF HELSINKI Department of Computer science

2 HELSINGIN YLIOPISTO HELSINGFORS UNIVERSITET UNIVERSITY OF HELSINKI Tiedekunta Fakultet Faculty Laitos Institution Department University of Helsinki Department of Computer science Tekijä Författare Author Otto Waltari Työn nimi Arbetets titel Title Seminar report: DieHarder: Securing the heap Oppiaine Läroämne Subject Computer Science Työn laji Arbetets art Level Aika Datum Month and year Sivumäärä Sidoantal Number of pages Seminar report sivua Tiivistelmä Referat Abstract This paper is a seminar report inspired by the article DieHarder: Securing the Heap [NB10]. Avainsanat Nyckelord Keywords Säilytyspaikka Förvaringsställe Where deposited Muita tietoja övriga uppgifter Additional information

3 Contents ii 1 Introduction 1 2 Heap attacks Preconditions Memory management errors Exploitable allocator Common attack model Allocator implementations DLmalloc Windows PHKmalloc OpenBSD DieHard DieHarder 8 5 Conclusion 9 References 12

4 1 Introduction Heap based attacks are an ongoing threat. As with many security vulnerabilities the exploitable glitch is rst discovered by the attacking party. Pointing out the glitch, creating a patch to x it and getting it to production on all systems using the software may require a vast amount of time, although it should happen as fast as possible. These patches often only x the specic application vulnerability, which leaves the operating system once again exploitable as soon as a similar application level vulnerability is discovered. Bad memory management from the programmer may lead to various errors that could interfere with other processes and enable heap based attacks. Instead of relying on the programmers memory managing skills, an approach to prevent these heap based attacks is to make the memory allocator safe. This basically means that the allocator tries to tolerate the inevitable memory errors and let them not interfere with anything but the processes that caused them. Several widely used memory allocators have no focus on security issues. operating systems have their own implementation of a memory allocator. leaves us with many dierent approaches that eventually do the same thing. Many This The rest of this paper is structured as follows. Section 2 describes briey what heap attacks are and what allows them to happen. Section 3 gives an overview of various allocators used today. Section 4 explains a more secure allocator implementation. And nally, Section 5 concludes the main points of interest in this paper. 2 Heap attacks In a heap attack the attacker targets the memory heap of a process. The heap contains all the memory the current process has allocated. If the attack succeeds the attacker could compromise the system, or simply just cause a crash. In order for the attacker to carry on with a heap attack there are a couple of aws that have to be present in the system that is to be attacked.

5 2.1 Preconditions 2 There are a couple of conditions that are inevitable prior to a heap attack. First of all there has to be a memory management error. Secondly the memory allocator has to be exploitable Memory management errors A memory management error is required. If the target application can be proven to be completely safe from memory errors there is no chance of the attacker getting his hands on the heap. Unfortunately such proofs are hard to make, and therefore we have to assume that memory management errors are possible. Memory management errors usually originate from the programmer. Memory unsafe programming languages such as C and C++ require the programmer to allocate all memory he uses and to free the memory he no longer needs. This allows many kinds of humane errors in the code that even the compiler can not necessarily warn about. There are ve common memory management errors: Heap overow and underow. Overow happens when the allocated memory is too small to store the object. Underow in contrast means reading or writing data to a location that is below the allocated area. Underows do not occur unintentionally very often, but overows are a common mistake. Dangling pointers. A pointer that points to a object that has already been freed. These pointers are also referred to as use-after-free pointers. Use of a dangling pointer might or might not lead to unexpected behaviour. This depends on wether or not the deallocated space has been overwritten yet. Double free. This error happens if the code tries to free the same object twice. Behaviour depends on the runtime implementation. Invalid free. This error happens if the code tries to free an object it never allocated. Once again the following behaviour depends on the runtime implementation.

6 3 Unitialized reads. C and C++ allocation functions malloc() and new() do not initalize fresh allocated space with any default or empty values. Accidental use of the existing arbitrary memory content could lead to unexpected behaviour. These memory management errors should not achievable with memory safe programming languages, such as Java, JavaScript or Flash. This however doesn't remove the threat of heap exploits. For example many browsers store web content objects in the same heap as its own internal data. While for example Java is implemented in C, it by default relies on the same C runtime allocator that the system uses. Attackers have been able to create documents, such as PDF's, images, et cetera, that trigger errors like the ones listed above. Thus the possibility for vulnerabilities exists Exploitable allocator Second requirement for a successful attack is that the memory allocator has to be exploitable. This generally means that the allocator is predictable so that we know how to get it to allocate memory on a certain area. Let us assume that the attacker has found out the memory management error and knows the execution path to the error. The next point of interest is the memory allocator. Allocator exploitation is achieved through driving the allocator to a predictable state. If the attacker manages to predict how to get data executed from a specic location where he is able to manipulate data, for example through overowing, the ingredients are prepared for an actual attack. Memory allocator security is achieved through steps that make the allocator unpredictable. An eective way of doing this is to allocate memory from a random location. If the free space is allocated from a random location it has a great impact on the heap structure and predictability. Additional ways to decrease predictability is to obfuscate and encrypt the heap metadata. We will take a closer look at allocator security later in this paper.

7 2.2 Common attack model 4 Probably the most common attack type is done via writing data beyond allocated space boundaries. In an attack like this the attacker benets from an overow, or underow error. By writing arbitrary data to the adjacent page the attacker could simply crash some process or interfere its behaviour. A more benecial way from the patient attackers point of view would be to try injecting shellcode to the page and hope that the shellcode overwrote a function pointer. If the overwritten pointer is now called, instead of it the attackers shellcode is executed. A second, quite similar way is called heap spraying. If the memory allocator does not erase unallocated data, the attacker could gradually ll, or as the name says spray, the memory with shellcode. If then a process would by itself or with a specic execution path call an uninitialized function pointer, the shellcode would be executed. In a situtation like this it would be extremely hard to know where the pointer would be called. To work around this problem attacks often use so called NOP-slides. NOP-slides are dummy algorithms that drift the instruction pointer to the beginning of the actual harmful shellcode. It is of course important to keep in mind that the attackers have to exploit the vulnerabilities through trial and error. A succesful attack scenario is usually so complicated, that it almost denitely did not succeed on the rst attempt. If the target system or application crashes after a single try, the attacker gets only one attempt per system. Live systems, such as online servers, might or might not restart the crashed service. In either case this is generally considered simply as denial of service. If the attacker is able to trigger many successive attacks at the target system he is more likely to succeed. This is probably the reason why most known heap attacks are done against generic consumer level systems with default applications. This is digestible, because the attacker is able to replicate the target system and investigate and debug the internals on the target system.

8 3 Allocator implementations 5 Roughly speaking there are two types of dynamic memory allocators. Main task for both of them is eventually the same. The dierence between them is in how they allocate and manage the free space that is to be used by applications. Allocators are usually designed to be rapid and to maintain a low fragmentation level. Unfortunately in many cases they tend to lack focus on security issues. One type is a so called freelist allocator. A freelist based allocator maintains a linked list of all the free space within the memory. A linked list is easy to maintain and thus it provides a simple way to allocate, free and merge chunks. The second type keeps track of free and allocated space with the help of a bitmap or a so called memory pool. In this case all the free memory is divided into pages of a xed size, usually 4kB. This means that the smallest allocable size is 4kB while all bigger allocations have to be multiples of 4kB. 3.1 DLmalloc DLmalloc is a freelist based allocator. It is named after its author Doug Lea. Its rst version was written in 1987, and since then it has been probably the most widely used allocator. It works well on systems ranging from embedded devices to large scale computers. It is also widely used for educational purposes because of its open source, straightforward design and exible nature. Doug Lea describes the allocator as follows: "This is not the fastest, most space-conserving, most portable, or most tunable malloc ever written. However it is among the fastest while also being among the most space-conserving, portable and tunable. Consistent balance across these factors results in a good general-purpose allocator for malloc-intensive programs." Original DLmalloc has very little focus on security issues. It stores heap metadata inside the free and allocated chunks. This metadata consists of chunk sizes, usage ags, or if the chunk is not allocated it contains pointers to other free chunks.

9 6 Another drawback is that there is necessarily no space between two adjacent chunks. This basically means that overowing might mess up the adjacent object or even harm the freelist in case a freelist link was overwritten by accident. Due to these various security issues, there is a huge amount of DLmalloc derivatives with improvements. One of them is PTmalloc, which has been used as the allocator in GNU C library since version Windows Due to a closed source many of the details regarding Windows memory allocator are unknown. It is however known that the Windows allocator is freelist based. There have been multiple successful exploit attempts since XP SP2 through Vista [MV09] that have revealed some of the details. For instance, the exploits reveal that a one byte cookie has been used as a key in the header of allocated chunks. 3.3 PHKmalloc PHKmalloc is the former allocator of FreeBSD and NetBSD. It was designed in 1996 to work eciently in virtual memory. The eciency led into various checks which half accidentally exposed dierent types of memory management errors. PHKmalloc is for example able to determine if a pointer passed to free() or realloc() is valid without dereferencing it. Also it is able to detect if a pointer was not returned by malloc() or realloc(). Knowing all this, PHKmalloc is able to detect all double frees and invalid frees. The consequences of these detections depend on the nature of the application. While an unpriviledged process could get away with only a warning, more critical processes might want to abort. Because of PHKmalloc's poor support for multithreading both FreeBSD (since 7.0) and NetBSD (since 5.0) changed their allocators to JEmalloc.

10 3.4 OpenBSD 7 OpenBSD's fundamentals are designed with heavy focus on security issues. Originally OpenBSD implemented PHKmalloc which has in the long run been improved in many ways. Still, many of the PHKmalloc properties listed above apply. Probably the most signicant improvement is the use of address space layout randomization (ASLR) [SPP + 04]. The system call mmap() on OpenBSD benets from a ASLR implementation and thus returns a range of pages from a random location. This is a nice feature and because of it the OpenBSD's allocators randomization is based on mmap(). Even the space for heap metadata is allocated through mmap(), which means that the heap metadata becomes completely segregated from the actual heap objects. OpenBSD's implementation of mmap() has other security focused characteristics. One of them is a sparse page layout. mmap() intentionally leaves empty pages between allocated pages. This makes sure that overowing from a chunk does not interfere with the adjacent chunk. The same thing happens in case of an underow. OpenBSD's allocator even randomizes the placement of objects within a page. This makes the allocator even harder to exploit. Not only is the attacker unable to locate the recently allocated space, but he is also unable to say for sure where inside the page the interesting data lies. Another unpdredictability enhancement is randomized reuse of pages. Once a page is freed it is not likely to be reallocated to another process anytime soon. Freelist based allocators tend to store free pages sorted by size. In the worst case scenario this could end up in the allocator circulating the same page over and over when triggering the same execution path to allocate a page. This kind of predictability is not possible if the freed pages end up in a pool of unused pages and on allocation a random page is pulled from the pool. What OpenBSD's allocator also does to improve exploitability is data overwriting before freeing the page. This so called destroy-on-free feature is discussed more in chapter 4.

11 3.5 DieHard 8 DieHard is a bitmap based allocator and was originally designed to be resistant against memory errors. This resistance was achieved through randomizing both memory allocation and the reuse of freed objects. These properties greatly improved reliability in two dierent ways. One of them being the fact that with randomized allocation it is unlikely that two adjacent objects are next to each other and that the former one would overow onto the latter. Secondly, randomizing the reuse of freed objects makes the allocator unpredictable, and as earlier stated, unpredictability makes the allocator harder to exploit. There is an adaptive version of DieHard that manages memory through so called miniheaps. Each of these miniheaps holds objects of exactly one size. When a specic size page is requested by a process the corresponding miniheap provides the page. Each miniheap size is adaptively adjusted so that a predened ratio between used and allocated space is sustained. It has been suggested that the heap size could be twice the size the process requires [BZ07]. DieHarder, which is presented in chapter 4, is a further improved version of the adaptive variant of DieHard. 4 DieHarder As earlier stated, DieHarder is a DieHard derivative. DieHard is designed to tolerate memory errors, while DieHarder focuses on security. Security vulnerabilities emerge from memory errors, so it is not surprising that both of them benet from the same design. OpenBSD's memory allocator has many eective and secure properties that have been implemented to DieHarder. One of the drawbacks DieHard suers from is that it allocates large continuous areas. This is a bad thing from the security aspect, since there are no guardian pages between heap objects. Although allocating large continuous areas is faster and requires less overhead, a sparse page layout is a great improvement on security, thus making it worth the overhead. DieHarder adapts a sparse page layout mechanism very similar to OpenBSD's randomized mmap().

12 9 Another feature DieHarder adapts from OpenBSD is erasing of freed data. Overwriting every freed page with random data causes of course some overhead. But once again from the security aspect it is well worth it. Take for example a situation where an attacker exploits the allocator and is able to ll an allocated page with arbitrary data. With the help of randomized reuse of used pages, the attacker could gradually ll the memory with for example shellcode. Erasing the unallocated data prevents this scenario. Filling the unallocated chunks with random data leaves the attacker with only one chunk to play with at a time. In addition overwriting unallocated chunks also reveals eectively dangling pointers and other use-after-free errors. On systems with limited or no support for ASLR, DieHarder handles randomizing in the following manner. DieHarder maps a generously large space of virtual memory within which the randomizing is done. While on modern systems the virtual address space is larger than anyone could require, the generous mapping doesn't aect performance. On modern 64 bit systems (x86-64 operating in long-mode) the virtual address space can be up to 16 exabytes (2 64 bytes, or about 16 million petabytes). With this magnitude of address space available mapping excessive address space has no impact on performance. The drawback however is that the process page tables size grow due to uneccessarily mapped pages. Even though physical memory wouldn't be a problem, keeping track of all the mapped pages could have a huge impact on the page table size and cache eciency. All the features presented above do not come for free. Figure 1 shows a chart of runtime overhead while processing some memory intensive tasks. DieHarder generates a noticeable amount of overhead, but it is not however incompetent with other major allocators. On for instance critical systems performance may take a hit, while security is not negotiable. 5 Conclusion A heap attack is an attack against the memory heap of a specic process. There are two preconditions that have to be true in order for a heap attack to be possible. First

13 10 Figure 1: DieHarder runtime overhead of all a memory management error has to be present. These errors usually originate from the programmer, but recently attackers have managed to trigger similar errors through various document types. Probably the most common memory errors are heap under- and overows and dangling pointers, or so called use-after-free errors. Secondly the memory allocator has to be exploitable. Allocator exploitability basically means that the allocator can be driven to a predictable state. If the attacker knows a way to write arbitrary data to a specic location with the help of a memory error, and is afterwards able to exploit the allocator to work on that same location, he is close to a successful attack. There are roughly speaking two types of memory allocators. One type is a so called freelist allocator. These allocators keep track of unallocated space with the help of a linked list. When a process requests memory, the list is searched for a page of proper size, which then is extracted from the list and given to the requesting process. Second type keeps track of allocated and free space through a bitmap. These bitmap bits represent memory pages of the same size, usually 4kB. Bitmap based allocators are considered to be more reliable because the bitmap is segregated from the allocable memory, while the freelist is a data structure located all over the memory, making it fragile. OpenBSD implements an allocator that has a heavy focus on security issues. DieHard

14 11 on the other hand is a allocator that provides high level of reliability and tolerance against memory errors. The authors of DieHard have combined the properties from both of these to create an even more secure allocator called DieHarder. DieHarder features for instance completely segregated metadata, erasing of data when space is freed, randomized page allocation and randomized object positioning inside pages. Secure allocators cause overhead which is an important thing to take into consideration when choosing between either rapid or secure allocators.

15 References 12 BZ07 MV09 Berger, E. D. and Zorn, B. G., Diehard: Ecient probabilistic memory safety, McDonald, J. and Valasek, C., Practical Windows XP/2003 heap exploitation. Black hat, USA, NB10 Novark, G. and Berger, E. D., Dieharder: securing the heap. ACM Conference on Computer and Communications Security, 2010, pages SPP + 04 Shacham, H., Page, M., Pfa, B., Goh, E.-J., Modadugu, N. and Boneh, D., On the eectiveness of address-space randomization. Proceedings of the 11th ACM conference on Computer and communications security, CCS '04, New York, NY, USA, 2004, ACM, pages , URL http: //doi.acm.org/ /