OPERATING SYSTEMS ASSIGNMENT 4 XV6 file system

Transcription

1 OPERATING SYSTEMS ASSIGNMENT 4 XV6 file system Introduction In most systems the main weakness of the file system stems from the data access time, which is much longer than accessing the memory. For certain systems, on the other hand, the main bottleneck is not the disk access time, but the required space. One example of such systems is a scenario where many instances of the same data are kept at several places as many different users store identical data on the same storage system. One way of dealing with this problem is that of "data de-duplication", in which duplicates of the same data are recognized and only one instance is actually stored on disk. De-duplication may occur at different places depending on the application. In this assignment, we will implement de-duplication inside the file system by letting different files share blocks. Interested students are encouraged to look up more information online for more realistic implementation methods. Task 0: Downloading XV6 Begin by downloading our revision of xv6, from the os132 svn repository: Open a shell, and navigate to the desired working directory. Execute the following command (in a single line): svn checkout This will checkout the code of XV6 to the directory. Task 1: Block-level deduplication The Linux file system (and xv6 in particular) does not store file contiguously but splits them into blocks of equal size. Using such a scheme, different files may happen to have several equal blocks. Identifying such blocks facilitates the ability to remove duplicates and decreases the space used in the disk. In this task, you are required to change the file system of xv6 such that equal blocks will only be stored once. Since files are represented using the inode data structure, that stores pointers to relevant blocks, 2 pointers of different inodes can be configured to point to the same data block, allowing 2 files to share a block. An 1

2 example of 2 inodes sharing a block is shown in the figure below, where the left part shows the current state of the system and the right depicts a shared block. Inode 1 Inode 2 Inode 1 Inode In order to support this scheme of deduplication, create a program called "dedup" (and place it in the root directory). This program should go over all the used blocks and update the file system such that no equal blocks will remain (and of course, update inodes accordingly and free unused blocks, such as the one marked with "-" in the figure above). Other processes (not the "dedup" process) should not maintain the deduplication, i.e. when a block is written to one should not look for equals, but allow duplicate data to exist (only "dedup" searches for equal blocks and removes duplicates). Given a block, finding blocks equal to it in an efficient matter is not easy, therefore, in this task you are required to implement the naïve solution and simply go over all the existing blocks, comparing the data inside to find equal matches. Of course, changes have to be made in other places too, for example, when a block is not used by any file it must be removed and marked as a free block. For that purpose add a reference counter to every disk block, and be sure to update it as needed. In order to simplify, you may assume that no processes will be run while "dedup" is running. Be that as it may, when using inodes and blocks "dedup" must still make sure they are not read or written to by anyone else. For that purpose lock inodes using the inode cache (icache) and blocks using the buffer cache (bcache). The blocks reference counters may be stored in several places (in the hard drive). Choose whichever one you find most reasonable. You are not required to prevent duplicates among these counters. You are allowed to enlarge the size of the hard drive. You are not allowed to change the size of each block. 2

3 Every block's reference counter should be an unsigned integer of 1 byte. Two blocks are defined as "equal blocks" if the data inside them is equal, i.e., every byte i in the first block is equal to byte i in the second block. While looking for blocks equal to a given block, go over all the blocks in the system only once (that is, do not compare the given block to the same block more than once). Some clarifications: The size of the hard drive may not be reduced (not allowed to reduce to default number of inodes or blocks). Students who choose to implement the bonus (written below) don't need to also implement the naïve search defined above. Please notice that the bonus is given to a working solution, so if you only implemented the bonus and nothing works you may not get the bonus points. For this reason the safest thing to do is write a complete solution with the naïve search and later expand. It is up to you. There is no need to be able to recover from a crash, should it happen while "dedup" is running. Bonus (15 points. Maximal assignment grade is 100): Design and implement a more efficient method of finding block duplicates (think hashing). Compare the performance of the brute force search method described above and of your improved method. Task 2: Sanity test In order to verify that you code works, write these tools to help you understand the current state of the system: 1. Printout of file's block indices Add a program called "fileblocks" that accepts a single parameter, which is the name of a file. This program should read that file's inode and print out all the indices (in the hard drive) of the blocks point from the inode (order them according to their order in the inode). The data inside the block is not needed. 2. Printout of the number of free blocks Add a program called "freeblocks" that prints out the number of free blocks in the hard drive. 3. Shared block rate Add a program called "sharedblocksrate" that prints out the rate of shared blocks. For that see how many blocks were saved due to the deduplication process and divide that by the number of blocks that a system without deduplication would have needed to hold. For example, a 3

4 system with 10 used blocks would need just 1 block we deduplication and 10 blocks without deduplication. The shared blocks rate of that system is 9/10=0.9 Now create files containing the same data (fully or partially) and use these tools to make sure they do share blocks with equal data. Submission guidelines: Make sure that your Makefile is properly updated and that your code compiles with no warnings whatsoever. We strongly recommend documenting your code changes with comments these are often handy when discussing your code with the graders. Due to our constrained resources, assignments are only allowed in pairs. Please note this important point and try to match up with a partner as soon as possible. Submissions are only allowed through the submission system. To avoid submitting a large number of xv6 builds you are required to submit a patch (i.e. a file which patches the original xv6 and applies all your changes). You may use the following instructions to guide you through the process: Back-up your work before proceeding! Before creating the patch review the change list and make sure it contains all the changes that you applied and noting more. Modified files are automatically detected by svn but new files must be added explicitly with the svn add command: > svn add <filename> In case you need to revert to a previous version: > svn revert <filename> At this point you may examine the differences (the patch): > svn diff Alternatively, if you have a diff utility such as kompare: > svn diff kompare o - Once you are ready to create a patch simply make sure the output is redirected to the patch file: 4

5 > svn diff > ID1_ID2.patch Tip: Although grades will only apply your latest patch, the submission system supports multiple uploads. Use this feature often and make sure you upload patches of your current work even if you haven t completed the assignment. Finally, you should note that the graders are instructed to examine your code on lab computers only! We advise you to test your code on lab computers prior to submission, and in addition after submission to download your assignment, apply the patch, compile it, and make sure everything runs and works. 5