Project 4: Implementing Malloc Introduction & Problem statement

Transcription

1 Project 4 (75 points) Assigned: February 14, 2014 Due: March 4, 2014, 11:59 PM CS-3013, Operating Systems C-Term 2014 Project 4: Implementing Malloc Introduction & Problem statement As C programmers, we rely heavily on malloc and free in our projects. However, we largely take them for granted, not caring about their requirements or optimality. In this project, we will explore how malloc and free work by implementing our own malloc library. We will implement this project in phases, starting out with meh_malloc1, a not-so-great version of malloc, with a free operation, meh_free1, that does not actually free memory. We then use this version as the base for calloc and realloc. Finally, we add real free functionality, making a fully functional malloc and free library. A couple notes: Since we are making a malloc library routine, the only executable will be that of your test program(s). However, those programs will help us see that you implemented malloc and free correctly. Also, note that you will need to use global or static variables to complete this assignment. Try to be careful with them and limit the number used. While the project is described in multiple phases, only the final phase (either Phase 4 or Phase 5) will be submitted and graded. You may work on this project individually or in two-person teams. Please register your team with the cs3013-staff@cs.wpi.edu as soon as possible. Background There are many memory allocation algorithms suitable for implementing malloc() and free(). Two useful references are: The C Programming Language, 2 nd edition, 1988, by Brian Kernighan and Dennis Ritchie, Section 8.7. This is the original malloc from the original UNIX. A web search will reveal a lot of criticism of it, but it has been a workhorse for decades. The assignment below uses this as a starting point. Computer Systems: A Programmer s Perspective, 2 nd edition 2011, by Randall Bryant and David O Hallaron. 1 Section 9.9 is a comprehensive treatment of dynamic memory allocators with an emphasis on high performance and efficient utilization of memory. 1 This textbook is used at WPI in some versions of CS-2011, Machine Organization and Assembly Language. There should be plenty of copies available on campus. Project 4 1 March 4, 2014

2 Sbrk When dynamically allocating memory with malloc, the heap portion of the memory region of the process must be able to grow. While the malloc library provides convenient interfaces for applications, it runs as part of user-space. To request memory from the kernel, the sbrk(int incr) system call must be used. 2 Sbrk asks the kernel to allocate incr bytes of memory to the process, and it then returns a pointer to the memory location the start of the new area. The new end of the heap 3 is thus the return value of sbrk plus the increment. (Providing a negative incr value will cause sbrk to shrink the amount of memory on the heap, but that is not needed for this project.) Trapping to the kernel is computationally expensive and it would be inefficient to simply call sbrk every time the application needs to allocate memory. Instead, malloc requests more memory than it needs and only returns the amount requested, saving the excess. On subsequent calls, malloc doles out memory from this excess if it can, invoking sbrk() only if it needs its pool of available memory. This reserve is particularly useful when applications request multiple allocations in sequence. Free list While it is easy to allocate memory from the heap portion returned by sbrk(), recycling memory returned by free() requires more work. In fact, managing the free list is the central part of any heap memory allocator. Free() needs to know (a) how much memory is being freed and (b) whether it can coalesce that freed memory with adjacent blocks of already free memory. A simple allocator in the style of Kernighan & Ritchie (K & R) allocates eight bytes more than the amount of memory requested by the caller and stores bookkeeping information including the size of the allocation in those excess eight bytes. These eight bytes are stored at the beginning of the allocated block (bytes 0-7) and malloc returns a pointer starting just after this point (i.e., at byte 8). This enables free() to look immediately before the pointer being freed to find the bookkeeping information again. 4 Note that if free() is used on a memory address other than a pointer returned by malloc, bad things are likely to happen. Free would look back eight bytes and then try to free as many bytes as described in the bookkeeping information that it finds there. This could cause strange behavior and lead to a crash. However, if free could tell that the provided address was inaccurate, it could simply return a warning. This is where the second four bytes of the eight-byte bookkeeping buffer come in. They can be set to a checksum or other simple value to help detect when the address provided to free() is wrong. A K&R-style allocator manages free memory blocks as a linked list. A global variable, malloc_head, points to the head of this free list. Initially, malloc_head is NULL. When malloc() is first called, it calls sbrk() to put some memory on the free list. The first few bytes from sbrk hold a small data structure of twelve bytes. The first four bytes of this structure specify the size of the chunk, the second four bytes point to the next chunk in the 2 See the Linux man page for a full description of sbrk() and its companion brk(). 3 By end of the heap, sbrk() means the pointer to the first byte after the heap. 4 It may seem obvious, but if the bookkeeping information were at the end of the allocated, how could free() find it to know how much to free? Project 4 2 March 4, 2014

3 free list, and the last four bytes point to the previous chunk in the list. The following figures show an example of an allocated block on the left and a free block on the right. 5 size checksum size next free previous free payload unused Project phases Start by creating a file called malloc.c. This should define four functions: malloc, free, calloc, and realloc. They should have the same function headers as the real versions of these same functions specified in the official Linux stdlib.h. However, your functions should all be stubs for now. Phase 1: meh_malloc1() Write a function called meh_malloc1(size_t size) to mimic the real malloc(). It should return a chunk of memory to the caller of at least size bytes, where size is greater than zero. meh_malloc1() must maintain a buffer or pool of free memory (obtained from sbrk() as described above) and allocate memory from this buffer. When the buffer is too small, it must allocate a new buffer using sbrk(). The equivalent of free, meh_free1(), should do nothing for now. That is, it remains an empty stub. The allocated memory must be 8-byte aligned, so that floats and doubles will have the appropriate alignment in arrays and structs. Therefore, it makes sense to round up all allocation requests to multiples of eight bytes. meh_malloc1() should add eight bytes to the amount requested by the caller (rounded up to a multiple of eight). As described above, the first four of these bytes should be an integer containing the size of the amount of memory that meh_malloc1() carves off from a free block. The next four bytes should be a checksum to help you debug your allocator. You may design this checksum any way you wish, and you will probably change it as you progress through this project. Finally, meh_malloc1() returns a pointer to the first byte of the payload area i.e., the byte after the checksum. If meh_malloc1() runs out of memory, it must try to get more from sbrk. If sbrk() cannot provide any additional memory, meh_malloc() must return the NULL pointer. Write a small test program to randomly allocate memory via meh_malloc1(), just for debugging purposes. Phase 2: port meh_malloc1() to malloc() When you are confident that meh_malloc1() works, move it to malloc.c in place of the stub of the malloc() function. 5 In this case, the free list is doubly-linked. In the K & R allocator, the free list was singly linked. Project 4 3 March 4, 2014

4 Also, implement calloc() in terms of malloc(), and implement realloc() in terms of malloc() and free(). If you consult the man pages on calloc() and realloc(), you will see they can be implemented with your malloc(). Specifically, calloc() calls malloc() and then bzero() (read the man page). To keep things simple, implement realloc() to always reallocate. That is, let it call malloc(), bcopy(), and then free(), but do not try to optimize. For this phase, you must support each of these operations. At the moment, free() is still a stub that does nothing. This is okay, but programs that use your malloc package will run out of memory faster than those using the Linux version of malloc(). Test your code by linking it with a program that would ordinarily use the real, Linux version of malloc(). For example, if you test it with your implementation of Project 1, link malloc.o and Project1.o together as part of the linking step in your makefile. This will create an executable in which all calls to malloc(), free(), calloc(), and realloc() resolve to your functions rather than the Linux library functions. Debug and test your program. Be aware that your debugging tools may use system routines such as printf() that could, depending upon how you link, resolve their malloc() calls to your malloc package. If this happens, you should use a debugger to analyze the execution of your code. Phase 3: meh_malloc2() and meh_free2() Rewrite meh_malloc() and meh_free() to support the proper freeing of memory. Call these improved versions meh_malloc2() and meh_free2(), respectively. Although you may use any representation that you wish, you are encouraged to use the linked list approach described above. meh_malloc2() should search the free list for a block of memory of sufficient size off and carve a piece off that block for the caller. Use the extra eight bytes bookkeeping space to record the size of the piece allocated and to update the checksum. This will aid in implementing meh_free2(). The remainder, if any, should go back onto the free list. Similarly, meh_free2() should place the freed chunk of memory back onto the free list. It is best to keep the free list sorted in memory order. Upgrade your test program from Phase 1 to test meh_malloc2() and meh_free2() and also to act as heap checker. The purpose of the heap checker is to scan the entire heap and validate its integrity. It should navigate the free list in both directions, making sure that it can cover the entire heap. It should also traverse the allocated blocks by incrementing a pointer to a block by the size to arrive at the next block. It should validate the checksum of each allocated block. You will probably need to figure out how to distinguish allocated from freed blocks. Phase 4: Porting meh_malloc2() to be malloc Now, do as you did in Phase 2. Copy the functions meh_malloc2 and meh_free2() to malloc() and free() of malloc.c. As in Phase 2, link it with a program that aggressively tests all four functions of your malloc package. You are responsible for demonstrating by a combination of test programs and heapchecking that your malloc package actually works correctly. Unless you choose the optional Phase 5 for extra credit, this completes your assignment. Project 4 4 March 4, 2014

5 Phase 5: Bragging rights (optional) Upgrade your implementation of free() to coalesce adjacent free blocks to make them bigger free blocks. Upgrade your heap-checker to test whether adjacent free blocks really have been coalesced. In addition, you must modify realloc() so that it reuses the same block if the new size is smaller than the old size. Deliverables and submission Submit your project via the web-based Turnin system at Zip your files together under the name <name>-malloc.zip, where <name> is your WPI user ID (if submitting as an individual) or your team name as entered in Turnin. Use only standard zip files for compression; other formats such as.rar,.7z, gz, bzip, etc., will not be accepted. Your submission should include: Your malloc.c file, The source code of your test program and your heap checker (which may be one or more programs). A Makefile that compiles (1) your malloc.c into an object file malloc.o, and (2) your test program linked with malloc.o into an executable named test_malloc. The test files or input that you use to convince yourself (and others) that your programs actually work. Output from your tests. A document called README in.doc or pdf format explaining your project, the data representation of the free list and of the allocated blocks, and anything that you feel the instructor and/or graders should know when grading the project. Be sure to include instructions on running your heap-checker. The graders will attempt to grade your malloc.o on their own test programs. If these do not work, you may earn partial credit by providing the interim versions from Phases 1 and 3. Grading rubric (75 points) Points will be assigned to your completed project (at the Phase 4 level) as follows: Compile malloc.c, your test program(s), and your heap-checker without warnings with your Makefile 5 points Ability of graders to link your malloc.o with their test cases 5 points Correct implementation of calloc and realloc in terms of malloc and free 5 points Project 4 5 March 4, 2014

6 Test cases sufficient to show that your malloc and free work correctly, including all boundary conditions 10 points Heap-checker to verify the integrity of your heap at any point during execution 10 points Correct execution with your own test cases 10 points Correct execution with graders test cases 10 points Write-up describing your allocation and free algorithms, heap-checker, data structures, and programming invariants satisfied by your data structures, including test case and heap-checker output 20 points Extra credit (20 points) Coalescing free operation 10 points Test case output showing correct operation of coalescing blocks 5 points Improved implementation of realloc to reuse block if smaller size is requested, including test case output of all boundary conditions 5 points Project 4 6 March 4, 2014