Lecture 14: Distributed File Systems CDK: Chapter 8 TVS: Chapter 11 Contents General principles Sun Network File System (NFS) Andrew File System (AFS) 18-Mar-11 COMP28112 Lecture 14 2 Basic File Service Architecture Directory service maps from names to unique file identifiers (e.g. inode numbers) Flat file service provides operations on files identified by these identifiers As directories are generally themselves files, the directory service uses the facilities of the flat file service in translating hierachic path names 18-Mar-11 COMP28112 Lecture 14 3 1
Idempotent versions of Operations Operations done by RPC Normal UNIX file operations rely on the system to remember position-in-the-file information for the process Such operations are not idempotent! Can create idempotent versions by making position part of read and write calls 18-Mar-11 COMP28112 Lecture 14 4 Stateless servers Making each interaction of a client with the server less dependent on previous interactions has advantages when servers and clients might reboot at any time, or messages might be lost So try to eliminate/minimise what the server remembers for the client 18-Mar-11 COMP28112 Lecture 14 5 Access checking On a simple UNIX system, access permissions can be checked when the file is opened and the process cannot tamper with the file descriptor it is then given With stateless interactions, each operation has to be checked so pass over userid, etc with each read/write (but still potential security problem!) 18-Mar-11 COMP28112 Lecture 14 6 2
Caching leads to problems For efficiency it is vital to using caching On a single machine this can be done without any effect on semantics of file operations In a distributed system client caching can lead to noticeable differences (see below) 18-Mar-11 COMP28112 Lecture 14 7 Sun NFS Initially developed for network of Unix machines in the 1980s Now a system-independent protocol Remote filestore is mounted in the client machine s hierachy NFS can be one of many file systems running under Virtual File System (VFS) 18-Mar-11 COMP28112 Lecture 14 8 NFS architecture Client computer Server computer UNIX system calls Application Local Application Virtual file system Remote Virtual file system UNIX file system Other file system NFS client NFS protocol NFS server UNIX file system 18-Mar-11 COMP28112 Lecture 14 9 3
Local and remote file systems accessible on an NFS client Server 1 (root) Client Server 2 (root) (root) export... vmunix usr nfs people Remote mount students x staff Remote mount users big jon bob... jim ann jane joe 18-Mar-11 COMP28112 Lecture 14 10 Automounter Could make client construct whole directory tree at boot time but this would be slow and quite possibly inefficient Automounter does it on demand when an empty mount point is referenced Simple read-only replication available client can probe several servers and use first to reply! 18-Mar-11 COMP28112 Lecture 14 11 Caching Server side caching works as normal and doesn t lead to any problems Client side caching has potential for problems because writes to a file do not affect cached versions Server has no knowledge of file sharing! Client has to use a timestamp based mechanism to validate cached blocks 18-Mar-11 COMP28112 Lecture 14 12 4
Using timestamps A cache entry is valid for a time t after it was last validated For files, t is 3-30 secs For directories, t is 30-60 secs Client also holds Tm (time last modified at the server) and can revalidate by checking this is the same as is held by the server 18-Mar-11 COMP28112 Lecture 14 13 AFS Developed at Carnegie-Mellon Uni (CMU) Wanted to maximise scalability by minimising client-server interactions Late 1980s Since incorporated into more advanced file systems (e.g. Coda which addresses replication and detached working issues) 18-Mar-11 COMP28112 Lecture 14 14 Basic principles of AFS Whole file serving files up to 64KB in size are transmitted between client and server Whole file caching served files are kept on client s local disk, and these are used rather than remote ones when available Quite large local caches e.g. 100 MB 18-Mar-11 COMP28112 Lecture 14 15 5
Andrew File System Workstations Venus User Servers Vice User Venus Network Vice Venus User 18-Mar-11 COMP28112 Lecture 14 16 File name space for clients of AFS Local / (root) Shared tmp bin... vmunix cmu bin Symbolic links 18-Mar-11 COMP28112 Lecture 14 17 Callback promises These provide a way for updating cached copies When a client writing to a file closes it, the modified file is returned to the server It makes an RPC (a callback) to each client which opened that file (and hasn t closed it) A client receiving this, marks the promise as cancelled so that subsequent opens don t use it 18-Mar-11 COMP28112 Lecture 14 18 6
And timestamps Venus also uses timestamps in case callbacks happen when the client is down, or the RPC goes missing So it renews callbacks before an open if time T (typically a few mins) has elapsed Note: callbacks imply servers are no longer stateless! 18-Mar-11 COMP28112 Lecture 14 19 Semantics of sharing This has changed a bit from UNIX model If have multiple writers to a file all but the last close will effectively be lost! It is a download/upload system, rather than a remote working version of UNIX which NFS nearly provides Specifically exclude using files for databases 18-Mar-11 COMP28112 Lecture 14 20 Based on observations Files are usually small ( < 10KB) Reads are about 6 * writes Sequential access common, random access rare Most files have only a single user Even shared files usually only 1 writer Files are referenced in bursts 18-Mar-11 COMP28112 Lecture 14 21 7
Conclusions NFS-4 has callbacks (above was NFS-3)! AFS and NFS both provide a useful service They are just two examples chosen because there are significant differences between them 18-Mar-11 COMP28112 Lecture 14 22 8