CSE320 Spring Homework 3

Transcription

1 CSE320 Spring Homework 3 Due Friday 11:59pm Introduction In this homework assig nment you are to create a custom dynamic memory allocator instead of using the built-in one provided by glibc. You will incorporate custom build f lags to tweak the dif f erent kind of strategies used f or your dynamic allocator as discussed in class. You will need to use the f unction sf_sbrk which requests memory f rom the heap. Your library will need to manage the heap space itself. The f unctions malloc, realloc, calloc, free, memalign, valloc, etc are NOT allowed in your implementation. We will check your assignment f or these f unctions. If we f ind that you used these f unctions you will receive a ZERO f or the assignment, no exceptions will be made. A real allocator may use the brk / sbrk calls f or small memory allocations and mmap / munmap f or large allocations. To allow your program to continue to use the other f unctions provided f rom glibc, we provide a saf e wrapper around sbrk. This makes it so that the heap breakpoint does not g et altered by the real allocator and destroy the memory manag ed by your allocator. For this assignment, your implementation MUST ONLY use sf_sbrk to move the heap breakpoint. You must read Chapter 9.9 Dynamic Memory Allocation. This chapter will contain all the inf ormation needed to complete this assignment. Since the textbook has suf f icient inf ormation about the dif f erent strateg ies and implementation details, this document will not repeat the contents of those chapters and will instead ref er you to the sections and pages in the book. If you need additional resources, the f ollowing may be helpf ul: 1. What Every Programmer Should Know About Memory 2. Anatomy of a Program in Memory 3. How the Kernel Manag es Your Memory 4. Page Cache, the Af f air Between Memory and Files

2 Takeaways Af ter completing this homework you should: Understand what f ragmentation is and the dif f erent types of f ragmentation that should occur. Memory padding and alig nment. The inner working s of how a dynamic memory allocator operates. Working with structs and linked lists in C. Working with ERRNO numbers in C. Linking with other object f iles. Getting started Add the f iles located at to your repository. DO NOT CLONE T HIS REPO! Please f ollow the directions f rom HW0 which instruct you on how to correctly add the remote repository contents to your repository. If you do not f ollow these directions you will most likely end up with a g it submodule in your directory or a hw3 directory inside of a hw3 directory (you should have neither of these). Directory structure In the hw3 directory are three f olders: lib, src, and include. This directory structure is a typical organization f or C projects. The lib f older contains any non-standard libraries which are used by the project, and the object f iles created f rom the source f iles in the src directory. The src f older contains supporting C source f iles. The include f older contains all header f iles. In this assig nment you are creating the dynamic memory allocator to replace the standard malloc, free, calloc and realloc f unctions. You are responsible f or implementing sf_malloc,

3 sf_free, sf_calloc and sf_realloc in the src/sfmm.c f ile. We have provided a small sample main f ile which tests your sf mm f unctions, sfmmtest.c which perf orms very basic tests. Af ter your allocator passes the very basic tests in this f ile, you must create your own either by extending the sfmmtest.c or creating your own C f ile entirely. You should examine and understand the Makef ile as it is an example of compiling with non-standard libraries. The provided Makefile creates object f iles f rom all the f iles in the src directory and creates libsf util.a with all the object f iles and the given f ile sfutil.o. The f ile libsf util.a is an example of a static library created using the archiver. Your main program WILL NOT be graded and your Makef ile WILL NOT be used. You must not modif y the f ile sf mm.h, as we will replace this f ile when we perf orm our tests f or grading. If you wish to add things to a header f ile in your program, please make an additional header f ile (in the include directory) which does not conf lict with sfmm.h. Initialization and helper functions In the lib directory we have provided object f iles sfutil.o which you will need to link with your project. This f ile exposes the f ollowing f unctions: * This routine will initialize your memory allocator. It should be called * in your implementation ONCE, before using any of the other sfmm * functions. max_heap_size Unsigned value determining the maximum size of * your heap. void sf_mem_init(size_t max_heap_size); * Extends the heap by increment bytes and returns the start address of * the new area. You cannot shrink the heap using this function. * Calling sf_sbrk() with an increment of 0 can be used to find the current * location of the program break. increment The amount of bytes to increase the size of the heap by. On success, sf_sbrk() returns the previous program break. (If the * break was increased, then this value is a pointer to the start of the

4 * newly allocated memory). On error, (void *) -1 is returned, and errno is * set to ENOMEM. void* sf_sbrk(size_t increment); * Function which outputs the state of the free-list to stdout. * Performs checks on the placement of the header and footer, * and if the memory payload is correctly aligned. * See sf_snapshot section for details on the output format. verbose If true, snapshot will additionally print out * each memory block using the sf_blockprint function. void sf_snapshot(bool verbose); * Function which prints human readable block format * readable format. block Address of the block header in memory. void sf_blockprint(void* block); * Prints human readable block format * from the address of the payload. * IE. subtracts header size from the data pointer to obtain the address * of the block header. Calls sf_blockprint internally to print. data Pointer to payload data in memory (value returned by * sf_malloc). void sf_varprint(void *data); The f unctions sf_mem_init and sf_sbrk must be used to initialize and request memory f or your allocator when it is needed. The f unctions sf_snapshot, sf_blockprint, and sf_varprint are helper f unctions which we have provided to help you visualize your f reelist and allocator blocks. We have also provided additional tests which will check certain properties of each f ree block when a snapshot is being perf ormed if you set the snapshot verbose value to true. It would be in your best interests to make sure your allocated blocks pass all these verbose snapshot tests. sf_mem_init

5 This f unction should be used one time in a program which uses your allocations f unctions. The argument it takes describes how large the maximum heap space given to sf _ sbrk will be. Typically you will want this to be one of the f irst f ew lines of your program. sf_sbrk This is the f unction sf _ malloc, sf _ calloc, and sf _ realloc should be using to request heap space f or your allocator. Its operation is similar to the real system call sbrk. Since giving back heap space is tough using sbrk, it does not accept negative values as an argument (size_ t is unsigned, so any negative numbers will be treated as very large positive numbers). At some point in your program, you may want to f ind where the heap breakpoint currently is. To f ind the current position of the heap breakpoint you should call sf _ sbrk with a size of zero and it will return the address of the current heap breakpoint (without moving it). If sf _ sbrk is unable to return the requested size of memory you asked f or, it will return (void *) -1, and errno will be set to ENOMEM. sf_blockprint and sf_varprint These two f unctions are used to print what a memory block looks like. The f unction sf _ blockprint takes the address of a memory block header. You would typically use this while debug g ing your malloc and f ree implementations which have direct access to the address of the block itself. The f unction sf _ varprint should be used with addresses that are returned by sf _ malloc, sf _ realloc, and sf _ calloc. Internally it adjusts the address f rom the value returned by these f unctions to obtain the address of the block s header, and then calls sf _ blockprint with that address. Below we display the f ormat printed by sf _ blockprint and sf _ varprint. Allocated Block: x bytes x bytes 0001 <- Header % p Payload x bytes x bytes x bytes 0001 <- Footer % p Free Block:

6 x bytes x bytes 0000 <- Header % p Next: %p Prev: %p... x bytes x bytes 0000 <- Footer % p sf_snapshot You will use this f unction to help debug/visualize the f ree-list. It starts by printing out inf ormation about the snapshot: Which kind of f reelist is being used (hard coded to Explicit), the size of the memory row (bytes), and the total space (bytes) taken f rom the heap so f ar. It then prints out the address of the f ree block and how many total bytes allocated f or the entire f reeblock. It will print this f or each node in the f ree list. Snapshot: Explicit # :42 0x7ffe06e2a1b If you call sf _ snapshot with the verbose f lag set to true, it will perf orm the f ollowing additional checks per f reeblock. # Examples of the types of checks that may happen WARN: Dumping header value: 0x # This always prints with verbose. WARN: Dumping footer value: 0x # This always prints with verbose. WARN: The allocated bit on the free node footer is set. WARN: Dumping footer value: 0x WARN: The header and footer have different block sizes. WARN: Header block size: 24 WARN: Footer block size: 0

7 WARN: The payload of the block is not aligned on an address divisible by 16. WARN: The total size of the freeblock is < 32 bytes. WARN: This does not meet the minimum requirements for storing WARN: the block header and footer, and the next and free pointers. If you accidentally have an allocated block in your f reelist, snapshot will tell you the address of which node in the f reelist which is pointing to it, and dump the contents using the sf _ blockprint f unction. ERROR: Snapshot encountered an allocated block in the freelist... ERROR: The freeblock 0x7fff5f91dc20 points to this block bytes 0 bytes 0001 <- Header 0x7fff5f91dc Payload: bytes unused 0 bytes 0000 <- Footer 0x7fff5f91dc ERROR: Aborting snapshot function. The above block being displayed probably happened because the memory was not zeroed out and was never set by anything. The large number being printed in the payload leads us to this conclusion. If the node was the head of the f reelist, the message above will be slightly altered to alert you. ERROR: This node is the head of the freelist. If you call snapshot and the f reelist_ head pointer is NULL, it will tell you that the f reelist is empty. WARN: The freelist has no nodes... We discuss more about the f reelist_ head pointer in the implementation section. Allocation functions

8 You will implement the f ollowing f our f unctions in the f ile sfmm.c. The f ile sfmm.h contains the prototypes and documentation f ound here. * This is your implementation of malloc. It creates dynamic memory which * is aligned and padded properly for the underlying system. This memory * is uninitialized. size The number of bytes requested to be allocated. If successful, the pointer to a valid region of memory * to use is returned, else the value NULL is returned and the * ERRNO is set accordingly. If size is set to zero, then the * value NULL is returned. void* sf_malloc(size_t size); * Marks a dynamically allocated region as no longer in use. * Adds the newly freed block to the free list. ptr Address of memory returned by the function sf_malloc, * sf_realloc, or sf_calloc. void sf_free(void *ptr); * Resizes the memory pointed to by ptr to be size bytes. ptr Address of the memory region to resize. size The minimum size to resize the memory to. If successful, the pointer to a valid region * of memory to use is returned, else the value NULL is * returned and the ERRNO is set accordingly. * * A realloc call with a size of zero should return NULL * and set the ERRNO accordingly. void* sf_realloc(void *ptr, size_t size); * Allocate an array of nmemb elements each of size bytes. * The memory returned is additionally zeroed out. nmemb Number of elements in the array. size The size of bytes of each element. If successful, returns the pointer to a valid * region of memory to use, else the value NULL is returned * and the ERRNO is set accordingly. If nmemb or

9 * size is set to zero, then the value NULL is returned. void* sf_calloc(size_t nmemb, size_t size); Make sure these f unctions have these exact names and arguments. They must also appear in the correct f ile. If you do not name the f unctions correctly with the correct arg uments, your prog ram will not compile when we test it. YOU WILL GET A ZERO. Look throug h /usr/include/asm-generic/errno-base.h and /usr/include/asmgeneric/errno.h f or the def initions of the ERRNO codes. Implementation details During compilation it is possible to pass values which will cause dif f erent f eatures of the allocator to be exercised. We will use dif f erent f lag s/pre-processor directives during the testing process to compile your allocator with dif f erent f unctionality. Your implementation MUST only use sf_sbrk calls of size 4 KB (4 096 bytes). Memory Row Size In this assignment we will assume that each Memory row is 64 -bits in size (1 quad word in x86_64 ). In the book f igures, Figure 9.36 f or example each square represents a memory row. C declaration Intel data type GAS suf f ix x86-64 Size (Bytes) char Byte b 1 short Word w 2 int Double word l 4 unsig ned Double word l 4

10 C declaration Intel data type GAS suf f ix x86-64 Size (Bytes) long int Quad word q 8 unsigned long Quad word q 8 pointer Quad word q 8 f loat Single precision s 4 double Double precision d 8 long double Extended precision t 16 Sizes of types on x86-64 Ubuntu Desktop Edition You can f ind more inf ormation about GAS on the wikipedia and tldp. Note the larg est type is long double. Make sure to take this into consideration when determining the alignment and padding f or your payloads. Freelist head In the f ile sf mm.c there is a pointer declared f reelist_ head, which is made available globally via the extern statement in sf mm.h. You MUST store the head node of your explicit f reelist in this pointer. * You should store the head of your free list in this variable. * Doing so will make it accessible via the extern statement in sfmm.h * which will allow sf_snapshot to access the value from a different * file. sf_free_header* freelist_head = NULL; Block Headers & Footer

11 In Chapter Page 84 7 Figure 9.35, the header is def ined as a single 32-bit double word with the block size and allocated bit. In this assig nment, your header additionally records the requested allocation size (in bytes). The block header is def ined as a singe 64 -bit quad word. The struct f or representing the header is f ound in sfmm.h. #define ALLOC_SIZE_BITS 4 #define BLOCK_SIZE_BITS 28 #define REQST_SIZE_BITS 32 struct attribute (( packed )) sf_header { uint64_t alloc : ALLOC_SIZE_BITS; uint64_t block_size : BLOCK_SIZE_BITS; uint64_t requested_size : REQST_SIZE_BITS; }; typedef struct sf_header sf_header; requested_size block_size a The requested_size f ield is 32-bits. It is the number of bytes requested by the call to sf_malloc. The block_size f ield is 28 bits. It is the number of bytes f or the entire block (header/f ooter, payload, padding). The least-signif icant bit is f or the allocated bit (as explained in the textbook). The block size f ield is only 28 bits, but technically it uses the 000a bits as part of the value. It just assumes these bits are always 0 because the address must be divisible by 16 (alignment f or largest type in x86-64 ). This gives us an actual representation of 32 bits f or the block size. For example, if the requested size is 13 bytes and the block size is 16 bytes the header will be set to: Notice the header will always be at an address which is a multiple of 8 (but not 16) due to payload alignment. This allows us to use the LSB (least signif icant bit) as a f lag f or the allocated state of the block. To assist in accessing the next and f ree pointers of a f ree block, an additional structure sf_free_header is def ined.

12 struct attribute (( packed )) sf_free_header { sf_header header; struct sf_free_header *next; struct sf_free_header *prev; }; typedef struct sf_free_header sf_free_header; Lastly, sfmm.h provides a struct f or representing the f ooter. #define ALLOC_SIZE_BITS 4 #define BLOCK_SIZE_BITS 28 struct attribute (( packed )) sf_footer { uint64_t alloc : ALLOC_SIZE_BITS; uint64_t block_size : BLOCK_SIZE_BITS; /* Other 32-bits are unused }; typedef struct sf_footer sf_footer; Use these structs to access the inf ormation in your memory allocator blocks. Explicit Freelist Management Policy You MUST use an explicit f reelist as described in Chapter Pag e 862 to manag e your memory allocations. If your allocator is compiled with no f lags or if the f lag -DLIFO is provided then your f reelist must manag e the inf ormation in a last-in-f irst-out manner. If your allocator is complied with the f lag -DADDRESS then you will instead manage your f reelist by maintaining the allocations in increasing address order (low address in memory to hig h address in memory). Block Placement Policy Your allocator will support either f irst placement or next placement strateg ies as described in

13 Chapter Pag e If your allocator is compiled with no f lags or -DFIRST then it will use the f irst placement strategy f or locating a f ree block. If your allocator is compiled with the f lag -DNEXT then it will use the next placement strateg y f or locating a f ree block. Coalescing Your implementation should perf orm immediate coalescing as described in Chapter Pag e 850. You should use boundary tags as described in Chapter Pag e 851 to assist in coalescing all f ree memory blocks in an ef f icient manner. In Chapter Page 852 Figure 9.39, the boundary tag (f ooter) is def ined as a single 32-bit double word with the block size and allocated bit. Your boundary tag should be a single memory row of 64 - bits, but contains the same f ormat. Unused block_size a Splitting Your allocator must split blocks which are too large to help reduce the amount of internal f ragmentation. You can f ind the details f or this mechanic in Chapter Page Do not make splinters! Splinters are a small (< 16 bytes, in this case) group of f ree bytes lef t af ter inserting a relatively larg e payload into a f ree space. To avoid this you over allocate the amount of memory requested so these small, potentially useless blocks are not inserted into the f ree list. Summary of Memory Block Format

14 Hints HUGE sized allocations: any requests larger than 4GB should return an error. If there s not enough memory to give, f ail and set the ERRNO correctly. Make sure that memory returned is alig ned and padded correctly f or the system in use. Make sure that if no preprocessor directives are g iven that your prog ram still works. Test your program with dif f erent combinations of f lags to see if it still works correctly. What if someone f rees NULL or in the middle of an allocation block or a non-sf _allocated address? Your program should not crash or segf ault if this happens, so you should check and make sure a call to f ree is on a valid address bef ore you f ree it.

15 When encountering edge cases you can make small test programs and observe how malloc and free will behave and emulate accordingly. Hand-in instructions You are expected to hand in at minimum the f ollowing f iles in your git submission. 1. The f iles sfmm.c and sfmm.h. All additional f iles that you may of created *.c or *.h should also be included. 2. Makefile 3. README.md Your assignment is expected to work on the f ollowing platf orms: 1. Ubuntu Desktop x86_ Submitting your assignment REMEMBER Do not submit at the last minute. We will be using the time stamp of the commit associated with the tag to determine if your homework assignment is late or not. On top of that we will NOT GRADE late assignments 1. Make sure all f iles f or this assignment are in a hw3 directory. 2. To tag your submission, make sure f irst that you have pulled f rom the remote and all code changes are merged. Once everything is merged, push it back to the remote server. Be sure you are done making any and all changes bef ore proceeding. T ags cannot be deleted. 3. Log on to your gitlab account and navigate to the tags page f rom your repository s main page. 4. Here you can click the green New T ag button. 5. Enter hw3 f or the tag name, the name of your current branch (typically master ) and an optional completion message. Do not put important inf ormation in the message as we will not necessarily see it. Any relevant inf ormation to your submission should be in your hw3/readme.md 6. To check if you submitted properly, you should now see a hw3 tag in the list of tags f or your repository.

16 When writing your program try to comment as much as possible. Try to stay consistent with your f ormatting. It is much easier f or your TA and the prof essor to help you if we can f igure out what your code does quickly.