Evolution of the Unix File System Brad Schonhorst CS-623 Spring Semester 2006 Polytechnic University

Transcription

1 Evolution of the Unix File System Brad Schonhorst CS-623 Spring Semester 2006 Polytechnic University The Unix File System (UFS) has gone through many changes over the years and is still evolving to meet future demands. It has constantly been re-examined and has become the standard against which other files systems are compared. Different implementations of the Unix File System have sprung up in many operating systems, however, the bulk of the innovations are still happening within the BSD community. The Fast File System (FFS) solved many problems that began to surface as the original Bell Labs Unix file system saw increased deployment. Block fragments, cylinder groups and better layout policies allow the Fast File System to make much better use of its underlying hardware. A technique called Soft Updates was added to the FFS to improve performance of meta-data operations and recovery after system crashes. To completely eliminate non-operational recovery time after a system crash, the ability to run a file system check in the background was implemented. Most recently, the Unix File System has been overhauled to allow for much larger file system sizes. Original Unix File System - FS Many of the ideas developed for the original Unix File System (FS) are still relevant and in use today. When implementing FS, a disk is partitioned into one or more partitions or divisions of the total hard disk space. Each partition has its own file system. The original Unix File System divides the hard disk into 512 byte data blocks. Every file has a descriptor, called an inode which maintains information about the file. All the I-nodes along with the file system's superblock are located at the beginning of the file system. The super block maintains information about the file system including things like the total number of data blocks, the maximum number of files, file system status, key device parameters, and in the case of FS, a pointer to the free list. A file system's free list is represented by a linked list which contains the address of all the free blocks yet to be assigned for data storage. A file's inode is responsible for storing all the file's relevant data, other than its name which gets stored as an entry in its parent directory. Included in an inode entry are the type of file, a file's mode (access information including read, write, or execute), timestamps for a file's last access and last modification, number of links to the file, length of file in bytes, and integer representations of the user and group who own the file. Inodes in the Unix File System make use of indirect addressing of associated data blocks. A typical inode might contain a total of 13 physical block addresses. The first 10 are direct links or addresses to specific data blocks. If a file is larger than the space provided by 10 blocks, 3 additional levels of indirect addressing are stored in the final 3

2 address entries. The first level of indirect addressing, entry 11 in this case, points to a block which contains nothing but addresses to other data blocks where the file is stored. If the file is large enough to fill the original 10 blocks referenced by the inode and the additional blocks pointed to by the first level indirect block, it will spill into the second level indirect addresses stored in entry 12. The second level address points to a block containing links to other first level indirect data blocks, which in turn, each hold addresses to physical data blocks. A third level is provided in entry 13 which points to a block full of second level indirect blocks. The 512 byte choice for block sizes works well for small files but introduces significant latency when trying to access large files. Files larger than 512 bytes are stored across several data blocks. UFS imposes no restrictions about a file's data blocks being stored adjacently on the disk so a single file many be stored across many different cylinders, introducing the potential for additional seek times per data block. Larger blocks are a logical solution to this issue but if block size is set too large, must waste is incurred. Unix operating systems consist of many small files and each file, no matter how small will take up a minimum of one whole block. If a file only uses half of the set block size, the rest is inaccessible to the file system. To increase the FS's access times to the disk, the block size was increased to 1024 bytes which was shown to double the throughput of the file system. [7] Fast File System - FFS In 1984, ideas for further improving upon the original Unix File System were introduced with the Fast File System. The Fast File System (FFS) uses the original file system's underlying methods of handling data storage but adds a more sophisticated method of distributing the data based on the hardware storage device being used. Data block fragments are introduced to solve the block size problem. In addition, several other features were added including file locking, symbolic linking, long file names, user quotas, and the ability to rename files with a single system call. The layout policies of the original Unix file system seem to work well early in a file system's life but after some use the disks became fragmented. This leads to a dramatic increase in the time it takes to access a file. This is because FS tries to assign new data blocks sequentially on the drive which does a good job of minimizing seek times initially. The problem arises as files are modified or deleted and need to change the number of blocks they make use of. A deletion creates open blocks like holes in the linear layout of used data blocks. New data would be assigned to these blocks and end up spread out all over the disk. When a file needs to be expanded, the blocks directly after it may not be available and other non-sequential data blocks are used. To access a file that has been severely fragmented leads to costly seek times for each block of data. Once the disk begins to fragment files the problem compounds. The only solution is to regularly reorganize (defragment) the data on the drive which takes considerable resources or even downtime.

3 A major innovation introduced by the authors of FFS is to divide a hard disk up into cylinder groups. A cylinder is a representation of the tracks on each platter that are equidistant from the center. So, for example, each track that is 3rd on its platter is part of the same cylinder. A cylinder group can be made up of one or several cylinders, depending on the disk. FFS tries to assign new blocks of data to the same cylinder group as the rest of the file and in a rotationally optimal location. These steps help to minimize the amount of head movement and disk rotation time needed when accessing the file in the future. Also, by spreading out the location of new files across all the cylinder groups, fragmentation is greatly diminished. Fragmentation will still occur but it is localized to within the cylinder group where seek time penalties will be much less significant. Another problem the Fast File System addresses is FS's storage of inodes at the beginning of the disk. By keeping all the inodes together at the head of the disk, every file access requires a seek to the beginning of the disk to access the file's inode before returning to the data portion of the disk to find the actual file. The Fast File System replaces the free list with a bitmap to represent all available data blocks. A second bitmap is used to represent the availability of the I-nodes. Each cylinder group has its own bitmaps for the blocks and I-nodes it is assigned. This almost eliminates the seek times previously needed to move between data blocks and their I-nodes at the start of the drive. By localizing these bitmaps the entire block allocation process can be contained within a cylinder group and the free-list is completely eliminated. Each cylinder group contains a copy of the superblock. The superblock contains essential information describing the file system and because it is rarely updated, it will only be needed if the original is damaged. The superblock and the rest of a cylinder group's administrative information is stored at an offset from the beginning of the group. Each group uses a different offset (one additional track beyond the previous group's offset) so that all administrative data is never located on a single platter. [7] The FFS introduces an even larger block size, expanding the already increased 1024 byte block size to 4096 bytes. To offset the wasted disk space that comes with a larger block size, disk fragments were introduced. A disk fragment is created by dividing a block into smaller pieces for storing files that would not fill an entire block on their own. A block can be broken up into 2, 4, or 8 fragments but cannot be smaller than the size of a single sector bytes. Each fragment is addressable and its location and availability is stored in its cylinder group's block bitmap. With a block size of 4096 bytes, a 1000 byte file would leave 3096 bytes wasted. With the addition of fragments, the 1000 byte file can be stored in a 1024 byte fragment, reducing the amount of waste to only 24 bytes. This means the FFS can enjoy the fast throughput of larger block sizes without the wasted disk space that previously occurred. When a file stored in a fragment needs to be expanded to a size larger than one block, it is removed from the fragment and assigned its own full block. This keeps files that get changed often from being scattered across the disk as fragments, which would diminish the benefits of larger block sizes. To allow for this kind of relocation, the file system must never be completely full. FFS tries to keep a certain percentage (around 10%) of

4 free space in each cylinder group. [7] If less free space is available, throughput drops. With no free space throughput is cut in half. Once space is made by deleting files throughput will return to normal. Other Enhancements of FFS The Fast File System also introduced several new features to add further functionality to the file system. The rename system call was added to speed up the process of changing the name of an existing file or directory. To change a file name in the past required three system calls and a system crash during this process could leave the file with a temporary name. The rename call does the work with just the single system call and guarantees the existence of the target name. Support for long file names was added by assigning directories 512 byte chunks. These chunks store entries which map a long file name to its inode. Each entry contains 4 parts, the inode number, the size of the entry, the length of the file, and the file name. The maximum length of a file name stored in a directory entry is 255 characters. A file locking mechanism was added so that files could be locked by processes when they are being used. Programs can apply two different types of locks to a file, shared or exclusive but the two cannot exist on a single file at the same time. An exclusive lock will block any requests to lock the file. A shared lock allows several other programs to also apply a shared lock but will stop any other program from issuing an exclusive lock until the shared lock is removed. This shared locking technique keeps a process from exclusively locking down a file another process is currently accessing. The FFS also introduced the idea of symbolic links. Previously, only "hard" links were in use. Hard links map a file's name to an inode. There could be multiple hard links to the same file by allowing the different file names to point to the same inode. An inode records the number of links pointing to it and when it reaches zero, it is deallocated. Hard links cannot span file systems or machines and usually cannot reference directories. Symbolic links are files that store the Unix path to the target. This path is just added to the current program's search path and followed without the program even needing to be aware of the link. Since symbolic links do not point directly to the target's inode, they can cross file systems and reference directories. This also means that if the original target is deleted, the symbolic link will be broken. Disk quotas were developed and implemented with the FFS. Disk quotas allow administrators to limit the amount of space in blocks and number of inodes (often referred to as number of files) a single user consumes. Both the number of files and the number of blocks are assigned a 'hard' and 'soft' limit. The soft limit is the suggested limit and if a user goes over that number they will be warned but the program is not shut down. If a user's program exceeds the hard limit it is terminated. Soft Updates

5 In 1999, McKusick and Ganger introduced soft updates to the Unix Fast File System. Soft Updates provides a technique for ordering writes to ensure meta-data consistency. Before the addition of Soft Updates, the Fast File System used synchronous writes to maintain the proper sequence in which meta data was updated. Soft Updates has been shown to reduce the number of disk writes from 40-70% from a similarly active file system using synchronous writes while keeping meta-data consistent. [6] Meta-data refers to the administrative elements of a file system including directories, inodes, as well as the free inode and block bitmaps. Every time a new block is used, its bitmap must also be updated to record the change. When a new file is written, a free inode must be found from the inode bitmap. Then the inode must be initialized, given a free block for the start of the file and the directory entry entered. Only then can the file actually start to be written. If there was a system error or crash during this process, the file system would be left in an inconsistent state and the file systems check program (fsck) would need to be run at boot. On a large file system, this could take hours which is unacceptable for production machines. Soft Updates actually eliminates the need to run fsck after a crash. Update dependencies refer to the ordering of the writes to specific meta-data structures. Just like writing a new file, every write or delete within the file system involves altering several pieces of meta-data. These changes must occur in a specific order to ensure the consistency of the files system. Soft Updates maintains a record of these update dependencies while making changes to the file system. File systems implementing Soft Updates use write-back caching for meta-data changes. Write-back caching is the storing of new writes in a cache and waiting to actually perform the write until the the machine is less busy or a specified amount of time passes. While this does impose a slight delay before the data is written to disk, it allows Soft Updates to consider each piece of meta-data's dependencies and come up with the best order to write the changes out to disk. Each meta-data change has an associated dependency structure stored in a buffer used to track its dependencies and their current state. Meta-data that has no further dependencies is written to the disk. Meta-data that cannot yet be written is rolled back to its previous safe values until all its dependent writes are taken care of. Once their are no more writes dependent on the original value of the meta-data, its new value is rolled back and the disk is fully updated. Thus, no meta-data is ever written to disk unless all meta-data requiring its original value have been updated first. This means a file system will never exist on the disk in an inconsistent state. Soft Update's use of write-back caching provides a more efficient method of updating the disk than synchronous writes but can impose a delay which is especially noticeable when marking blocks and inodes as free. This is because deletion dependencies are passed to a separate background task which may delay execution for up 60 seconds. This task steps through all the inode and indirectly linked addresses and marks the blocks as free.

6 If the system crashes before this deletion process runs, the file system will report more used blocks and I-nodes than really exist. The addition of Soft Updates to a Unix file system includes an altered version of the fsck program to handle this situation. The unreferenced inodes and blocks are deleted rather than moved to the lost+found directory as they were in the past. The corresponding bitmaps can be referenced to be sure the I-nodes or blocks are not in use because Soft Updates guarantees that they will remain up to date. This will free up the wasted space that was reported as used. This initial decrease in available space is the only meta-data inconsistency that will occur to the file system after an abnormal shutdown. This is not a critical error so a file system utilizing Soft Updates can be mounted and used immediately after a crash without waiting on the fsck program. Managing the removal of directories proved to be a bit more difficult than other types of meta-data. The challenge is with a directory's ".." entry which points to its parent directory. Initially, Soft Updates tried to remove both the directory and its ".." reference at the same time but a system crash during this operation could leave a directory in place with no ".." entry. It was found to be better to leave the ".." entry until the directory has been fully unlinked, however, this introduced a delay from the time a directory is unlinked to when its ".." is finally removed due to the above described deletion process. This delay causes problems in applications that rely on the link count of a directory. To correct this, a new field is stored in kernel memory called "effnlink" which can be referenced rather than the "nlink" field in a directory's inode. This new counter will show users and applications the effective nlink count or the count that nlink will be at once outstanding operations complete. This gives the illusion that directory deletions are immediate. Alternatives to Soft Updates While the Unix file system with Soft Updates is currently available in the major BSD operating systems, most other OS's use a technique called Journaling to provide metadata protection to their file systems. Journaling file systems use write-ahead logging to record all changes to meta-data in a log or journal on the drive. The journal is written before actually making the updates active in the files system. This ensures a history of all meta-data operations should the files system crash before the meta-data changes have been made. After a crash, the file system can replay the journal and complete any outstanding meta-data operations. Soft Updates and Journaling file systems have been compared extensively[8] and both seem to offer advantages over the other in certain situations. Soft Updates does provide a slightly faster boot time after a crash. This is due to the time it takes a Journaling file system to apply its log. Soft Updates' ability to delay deletes has been shown to significantly improve performance on file systems used for delete intensive services. Journaling seems superior in systems that have lots of writes but only when there are few deletions. Snapshots

7 With the addition of Soft Updates, the ability to take live snapshots of a file system was introduced to the FreeBSD operating system. A snapshot is a copy of the file system at a specific point in time. In order to create a snapshot, the file system will need to be suspended briefly to ensure a consistent version is recorded. Before suspending the file system, the initial snapshot file is written by scanning the file system's block bitmaps to mark the snapshot's file block pointers as either 'not used' or 'not copied.' The 'not used' pointers represent blocks on the file system that are free. The 'not copied' pointers refer to blocks that are in use and have not yet been copied to the snapshot file. In order to allow snapshots to be made on an active file system, the system calls that write to the file system have been modified to include a 'gate.' These 'gates' keep track of how many processes are currently running the system call they are attached to. Additionally the gates are responsible for blocking system call requests when the file system needs to be halted for a snapshot to be written. This allows the creation of snapshots without actually unmounting the file system. Once the files system is suspended, the cylinder group's block bitmaps are rescanned to account for any changes between the time of the initial scan and the time it took to suspend the file system. Once these changes have been recorded, activity on the file system resumes and any processes blocked by a 'gate' pickup where they left off. By writing the bulk of the snapshot file before the system is suspended, the down time is minimal and not proportional to the size of the file system. [5] When a data block is to be changed, the file system checks with the snapshot to see if the data block has been recorded. If it is marked as 'not copied' the block gets copied to the snapshot before updating it. If the snapshot file is read, any pointers marked 'not copied' read directly from the file system. Several snapshots can be maintained. Snapshots use of hard links allow future snapshot files to only need space for data blocks that change. A newer snapshot links to the same data block as the older snapshot until the data is altered. Should an earlier snapshot be removed, the newer snapshot will absorb any mutual data blocks to maintain a complete picture of the file system. Currently up to 20 snapshots can be referenced by a file system's super block which allows the snapshots to survive reboots. This makes snapshots a great way to provide backups of files throughout the day. Background File System Consistency Checks In addition to providing backups of a file system, snapshots allow the verification of the integrity of a live file system. This eliminates downtime needed to run fsck, the file system consistency check tool. Soft Updates guarantees the consistency of the file system after a crash with the only exception being some lost space due to the delayed deletion process. To free up these resources, fsck is still necessary. Rather than run before mounting the file system, the regular start up process can take place and then fsck will run in its background mode.

8 First, fsck will take a snapshot of the file system. Then it will run as usual over the snapshot, scanning all allocated inodes to come up with a list of allocated blocks. Each directory from the first scan is verified to have the correct ".." entry and a link count to each inode is recorded. Any unreferenced directories are deleted. Finally, the list of referenced blocks and inodes gets compared against the running file system's inode and block bitmaps. They get updated to reflect the fsck results via five new system calls which allow the kernel to make the necessary changes to the live file system. Running fsck in the background consumes minimal amounts of CPU time and does not need to run at a high priority. When run on an idle file system, background fsck performs slightly slower than traditional fsck. On a 7.7 GB file system, background fsck takes a total time of 61.5 seconds to complete compared to 50.9 seconds to run traditional fsck. When the file system is active, as a server might be after a reboot, total time jumps ten fold to 591 seconds. [5] This jump in time is due to fsck yielding to other processes because its running at a lower priority (a nice value of four.) This is acceptable because of Soft Updates guarantees that the only inconsistencies with our file system will be wasted disk space and in most cases reclaiming this space will not need top priority. UFS2 In 2003, the Unix File System 2 (UFS2) was introduced as a replacement for UFS in FreeBSD. (FreeBSD documentation uses the term UFS to refer to FS + FFS + Soft Updates.) UFS2 adds 64 bit block pointers allowing the file system to break the Tera byte limit on file storage. Most hard disks today use a 512 byte sector size. With 64 bits available for addressing, file systems should now be able to grow to 2^64 * 512 bytes in size. To account for the larger addresses, the size of the inodes was increased to 256 bytes. Some additional features were added to UFS2 along with the increase in block pointer size. Various flag fields were extended and support for variable block sizes was added to allow big files to be written to blocks larger than the default. This allows for faster reads because there is less disk activity needed to access the file. Also, UFS2 now uses 'lazy' inode initialization which helps speed up the process of building a new file system. UFS2 fully supports both Soft Updates and the running of background fsck with snapshots. In Summary The Unix File System has seen many changes and been successfully adapted to suit the needs of its time. The file system's optimized use of its underlying hardware and disk fragments were introduced with the Fast File System. Soft Updates allow for more efficiently meta-data writes and recoveries. Snapshots allow administrators to get a picture of the file system at a given point in time and file system checks no longer need to occur before boot. The recent move to UFS2's larger bit counters will not only keep the Unix File System relevant but leading the way in file system innovation.

9 References [1] Burden, P. "The Unix File System" Retrieved April 2006 from [2] FreeBSD Documentation Project. "Large data storage in FreeBSD" Retrieved April 2006 from [3] FreeBSD Documentation Project. "FreeBSD Handbook" Retrieved April 2006 from [4] Kampe, M. "Introduction to BSD UNIX Volumes and File Structure" Retrieved April 2006 from [5] McKusick, M. Running "fsck" in the Background." Proceedings of the BSDCon [6] McKusick, M. and Ganger, G. "Soft Updates: A Technique for Eliminating Most Synchronous Writes in the Fast Filesystem." USENIX Annual Technical Conference [7] McKusick, M. K., Joy, W. N., Leffler, S. J. and Fabry, R. S. A Fast File System for UNIX, ACM Transactions on Computer Systems, 2,3 August [8] Seltzer, M. et al. "Journaling Versus Soft Updates: Asynchronous Meta-data Protection in File Systems." USENIX Annual Technical Conference [9] Van Gelderen, J. C. and BSD Documentation Project 2003 "UFS2 FAQ" Retrieved April 2006 from