1. Introduction. Traditionally, a high bandwidth file system comprises a supercomputer with disks connected

Size: px

Start display at page:

Download "1. Introduction. Traditionally, a high bandwidth file system comprises a supercomputer with disks connected"

Cory Patrick
5 years ago
Views:

1 1. Introduction Traditionally, a high bandwidth file system comprises a supercomputer with disks connected by a high speed backplane bus such as SCSI [3][4] or Fibre Channel [2][67][71]. These systems are limited in scalability by the speed of the bus and the shared CPU power that services the disks. More recent systems have disks attached to a network switch rather than a bus. The scalability of these newer designs is limited as well, because they employ traditional methods for their data structures such as superblocks and pointers for free blocks and inodes. Several of these new systems use logging to speed up access, but logging can backfire when the log has to be cleaned (logging is explained in Section 2.1.6). Other solutions require a dedicated metadata disk which could become a performance bottleneck. In addition, some of the new solutions are suitable for a specific class of data but they are not a general solution when high bandwidth, low cost, reliability and scalability are desired. What is a parallel file system? The largest component in disk access time is the time it takes to move the disk arm and heads to the desired sector where the requested data resides. Even though disks today have multiple platters and heads, and they all move together, only one head can be reading/writing at any given time, because of small fluctuation in the position of the sectors in each platter due to temperature variations. Therefore, if a file resides on a single disk, accessing the file may require multiple arm movements and head positioning operations, and hence, a large total access time. The solution then is to use multiple disks in parallel, stripe (spread) the file across many disks, and let all the arms move in parallel in their respective disks to read/write the file data for a much smaller total access time. 1

2 To find out where to place a specific file block on a disk, in traditional file systems, the location of a new data block 1 is selected out of a list of free blocks, and that location could be anywhere on the disk. To find an existing file block, the file system must look in other blocks (inode 2 blocks) that contain the list of block locations that belong to the specific file. Both of these operations required multiple accesses to the disk, and therefore slow file access. It is desirable to minimize the number of blocks that need to be examined before the actual data block is accessed. In Unix, for example, a super block and an inode block both need to be updated when a new block is assigned to a file. When such a file system grows, the super block becomes a hot spot and performance deteriorates. Such a system will not scale well. Such a system may not be consistent when a disk fails and special measures need to be taken to guarantee consistency (such as non volatile RAM). My work is an expansion of the ideas that were originally proposed and simulated by Miller and Katz for the RAMA parallel file system [49]. This file system is a set of computers, each with a disk, that are connected to other computers with disks and users over a network switch. The RAMA file system hashes the file name (fileid) and block offset and places the files on disks, at a physical location based on the hashed value. My contribution is building RAMA out of commodity computers (PCs). 1. A block is a unit of one or more consecutive sectors. Block and sector are standard file system terminology 2. A Unix concept. 2

3 I added dynamic parity protection to make this file system reliable. With the thinking that not all files need parity protection, I allow file owners to turn parity protection on or off on a per file basis at any time in the lifetime of a file, rather than the traditional always ON or NEVER, per physical block basis. I allow parity to be turned OFF when the system is busy, and ON again when it is not, which will benefit performance for data that is accessed in bursts. In addition, the file striping parameters are flexible and are set on a per file basis by the user at file creation time. I designed the data structures and access mechanism with scalability in mind and tested the scalability of the file system as I added disks to it. I examined the improvement to uneven data distribution on disks by placing overflow data and metadata at another location on the disk. RAMA runs in user mode and is designed to be portable. The fault tolerance feature requires no special hardware, uses commodity volatile RAM, and can recover from single disk errors. In some specific instances, it can recover double disk errors but this is not guaranteed. The file system capacity can expand by adding disks to the system. The expansion of the file system involves all disks in parallel and may require the reshuffling of all data blocks to other locations on other disks Main Contribution I built and improved on the RAMA file system as follows: Built a working RAMA file system using commodity components. Demonstrated that all disks perform at maximum speed in parallel. 3

4 Added reliability to the file system with a granularity of one file. Distributed files with user selected granularity on a per file basis over the file system disks. Demonstrated that the file system is scalable. Wrote portable software Issues The issues involving disk systems are as follows: performance, reliability, scalability, consistency and cost. An explanation of each of these issues is detailed in the following sections Performance The time it takes a file system to respond to its users is composed of several components: the time it takes to send the user request to the disk, the time it takes the disk to find the metadata which contains information about where the desired user data is located on the disk surface, the time it takes to get the data itself, and finally, the time it takes to return the data to the user. A parallel or multi-disk file system can work on several requests at a time. These several requests may be servicing one user or multiple users. In order to improve the speed performance of disk systems, I would need to improve any of the speed components mentioned above. The bigger the component (in time), the better the resulting improvement that should be achieved. Network performance is beyond the scope of my work; however, I will examine the most efficient way to use the given network so that the file system performs in the most favorable way. I will focus on the design 4

5 that will result in the fastest speed at which I find and read/write the metadata and the data itself. Striping parameters affect performance [17], and by allowing each file to chose the striping parameters at creation time, a file that needs to be written faster can be striped with wider stripes for fewer parity blocks to data blocks ratio, and a file which can tolerate slower writing can be written with more parity blocks. This organization keeps the average parity:data ratio constant but varies write performance on a per file basis Reliability Reliability is of major concern to disk system users. When no precautions are taken, a disk system failure may result in permanent loss of the information that was stored on the disk. Several methods are used to reduce data loss in case of failure such as periodic backup to a tape system or other cheaper devices, in which case lost data may be restored up to the time of the last backup. Another method is mirroring [13], in which all data is continuously duplicated on a second disk (or even a third) so that a single disk failure will cause no data loss. Another method is adding Hamming or other error correcting codes that require additional disks to hold the redundant data, but less than double the number of original disks. Yet another method is the addition of parity to the disk system. This method XORs all the data on the disk system and saves the result on another disk. If a disk fails, the missing data can be calculated from the parity disk. There are two ways to place the parity information on the disk system, either all on one disk, or distributed among all the disks. More details are given in Section The space overhead for the parity information is dependent on the width of the parity information, or how many disk data units 5

6 are protected by one unit of parity. In general, parity protects the disk system from a single fault, but in some cases, when multiple faults are completely independent (data and parity of first fault do not share any disks with a second fault), then multiple faults can be recovered. Otherwise, this thesis does not deal with multiple disk faults Scalability Scaling deals with the effect of adding capacity (disks) to the disk system and how it affects the performance of the file system. Some disk systems perform well when the total number of disks is relatively small, and others perform well when there are many disks. If a system scales well, it performs well at any size. A system usually scales well when there are no bottlenecks in its access paths Consistency Consistency deals with the relationship between the file data and the file metadata. If an inode is updated with file information and the disk system crashes before the file data is written to disk, then not only can the lost data not be recovered, worse yet, we may assume that the file data was actually updated before the crash, and that it contains valid information. In addition, if there are multiple copies of a data item, they should either be identical, or the difference between them identifiable (for example, reading old data block values before rewrites in order to update the parity block with the changes). A file system must be consistent; otherwise the data it supplies is not reliable Cost Many disk systems today include specialized hardware to boost performance. They may include specialized controllers, specialized busses, DMAs, high powered CPUs and more. 6

7 In addition, in multi-disk systems with parity protection for fault tolerance, the shared memory used for the write cache must be non volatile because it holds data which belongs to multiple disks, and therefore the loss of memory will constitute multiple losses in a single fault tolerant system. Thesis statement: This thesis demonstrates that a reliable, scalable file system can be built using commodity components. This file system stripes data and adds/excludes parity with the granularity of a file and therefore distributes the entire file system throughput to each file as needed. This indirectly controls the quality of Service of each individual file Organization of This Thesis Chapter 2 gives background on other file systems that came before RAMA. Chapter 3 describes the design of RAMA. Chapter 4 describes the development environment (hardware and software) where RAMA was developed and tested. Chapter 5 describes the software architecture. Chapter 6 presents the results that were obtained running the file system. Chapter 7 has my conclusions and ideas for future work. 7

Chapter 11: File System Implementation. Objectives

Chapter 11: File System Implementation Objectives To describe the details of implementing local file systems and directory structures To describe the implementation of remote file systems To discuss block