Distributed File System

Transcription

1 Distributed File System Project Report Surabhi Ghaisas ( ) Rakhi Agrawal ( ) Election Reddy ( ) Mugdha Bapat ( ) Mahendra Chavan( ) Mathew Kuriakose ( )

2 1 Introduction The project aims at providing a distributed file system that is scalable, transparent and location independent. We also take into account the fact that, nowadays the PCs, are equipped with enormously large hard disks. The distributed file provides the means of utilizing the hard disks of various machines in place. There is no need of dedicated servers. In an environment like educational institutions, lots of workstations with large disk space are available. So the distributed file system comprises of files that are stored on the hard disks of participating workstations across an interconnected network. We use a distributed look up mechanism to provide location independence. The distributed nature of look ups prevents the look up module from becoming bottleneck. Client side file caching on disk is employed, which allows the open request to be treated as a local file open. Read and write operations on an open file are directed to the cached copy, so that no network traffic is generated by such requests. If a cached file is modified, it is copied back to the custodian when the file closed. Concurrent reads and exclusive write policy is used in order to maintain the consistency while sharing. File locking enforces exclusive writes. The distributed file system is stateful. Servers keep track of which clients have which files. Hence, the only resource becoming orphan on client crashes is the record of files transferred by server to the crashed client. The goals stated above are realized based on the following practical assumptions: Files are small (i.e. entire file can be cached) Frequency of reads much more than those of writes Shared files are usually not written Disk space is plentiful Delay and reliability are taken into account while choosing the communication mechanisms. The communication over the network (between client and server) is in the form of RPC. The daemons and processes residing on the same node communicate though message passing. 2 Design Goals 2.1 Basic Design Figure 1 shows basic components in our distributed file system: 1. DFS Client APIs for distributed file system exposed to client processes.

3 2. DFS Server Routes local/remote file accesses. 3. Cache Manager File cache at each client node allowing fast file access and reducing network traffic. 4. Lock Manager Enforces sharing semantics. 5. Namespace Server Domain based namespace server performing name resolution. It has two levels namely RootDNS and DomainDNS. An example scenario is shown in Figure 2. Figure 1: Basic Components of DFS

4 Figure 2: An example scenario for namespace server Figure 3 shows the setup over which the distributed file system will operate. Virtual machines are implemented using KVM Linux. Linux on each logical node supports our distributed file system. 2.2 Server Side Figure 3 Though the server is stateful, it contains only the list of files transferred to the remote clients along with the identification of those clients, as part of the state. As a result, in event of client crash a negligible amount of memory is the only resource which gets orphaned. In case of the server crash, all the clients may still continue accessing the file cached on their local node. The clients who have cached the file for writing will be able to write the copy back to the server as soon as it is up (This feature is not implemented currently). 1. When a client node N requests to open a file F in write mode

5 a. The server asks lock manager to set the lock on F if it is available. b. If lock is available and set by the lock manager, i. The server makes the entry <N, F> in the table. ii. Then transfers F to N. 2. When a client node N requests to open a file F in read mode (This request reaches the server only when there is no cached copy of F on N) i. Server makes the entry <N, F> in the table. ii. Then transfers the F to N. 3. When a client node N requests to close the file F a. Server fetches back the updated copy of F from cache manager of N. b. It then, uses the list to find the nodes who have cached F and tells them to invalidate the cached copies of F. c. Then, it asks lock manager to release the lock on the F. The communication with remote client is carried out using RPC. To transfer the file to/from client a simple TCP based file transfer protocol is written. In practice, the RPC mechanism itself is used even for file transfer. But, this poses limit on maximum file size in the file system due to: The bound on maximum size of object that can be returned by RPC The main memory available to buffer the file before the RPC returns it in the form of object Such bounds don t come into picture due to the protocol. 2.3 Client Side DFS client is a library of calls that will be linked with the client process who wishes to use the distributed file system. DFS client requires interfaces with file servers on remote node (implemented using RPC), name server (implemented using RPC), local cache manager (implemented using message queues) and local lock manager (implemented using message queue).

6 The various operations supported by client are as follows: 1. Create File: a. Contact name server and add entry for the new file specifying path in DFS and absolute path in the distributed system. b. Create the file on local disk on the absolute path specified to name server c. Create a lock object for the new file 2. Open file: a. Get the absolute path for the file in distributed system by sending the logical path to name server. b. If the file is to be opened for write contact the file server to get the lock for the file. In case the file is already locked return the error status to requesting user. c. If we obtain the write lock or if the file is required for read, contact the local cache manager to check if the file is already in cache. d. If the file in cache and opened for read, get the file name for the local copy from cache manager and open it in read mode and return the handle to client process e. If the file is in cache and opened for write then cache manager will make a new copy in cache and return its name to DFS client on getting request from client. Open the file in append mode and return the handle to client. f. If the file is not in cache get the file from remote file server. g. Inform cache manager to add the new file in cache h. Go to step d. 3. Close file:

7 a. Contact cache manager to get information corresponding to the file handle such as the identity of the server where file belongs, name of the file in cache, name of the file in DFS and the mode in which file is opened. b. If the file is opened for write, then write back the copy in cache on remote server. And release the lock for the file. c. Close the file and inform cache manager that the client is done with using the cached copy. 4. Read file: a. Read the number of bytes specified by the user into buffer provided by the user from the current position of the file pointer. b. After the operation the file pointer will point to data following the currently read data. c. It will return the number of bytes read. If number of bytes read is less than the requested count then it indicates we have reached end of the file. 5. Write file: a. Write the number of bytes specified by the user from the buffer provided by the user to the current position of the file pointer. b. After the operation the file pointer will point to location following the currently written data. c. It will return the number of bytes written to the file. 6. Seek file:

8 a. This will take the current file pointer, offset count and move mode as arguments. b. Move mode can be current location, start of file or end of file. c. This sets the new location of file pointer at offset count from the move mode (i.e. from current position or start of end of file). 3 The Hard Issues 3.1File System Naming Namespace server module helps in providing location transparency and location independence. We have implemented domain based (distributed) namespace server. This module will be used by DFS Client module. We have implemented two level model (as shown in Figure 2): Level 1: RootDNS(one per DFS) Level 2: DomainDNS(one per domain) Root DNS maintains the list of all the files/folders created immediately under /. For each file/folder in this list, it keeps the addresses of the respective DomainDNSs on which they are created. RootDNS will not store the links and permissions for any of the files/folders. Below the root level there will be a number of DomainDNS servers. Every DomainDNS serves a number of nodes. Each DomainDNS server maintains a tree of all files/folders created by the nodes in its domain under /. The various operations supported by namespace server module are as follows: 1. Get the file location information This operation is invoked by DFS client to get the actual link information (ip:directory_path) of the requested file, provided the corresponding user has access permission for it. 2. Set the file location information This operation is invoked by DFS client when a user creates a new file. It sets the link for a file in the appropriate node in the directory structure (tree), provided the corresponding user has the required permission.

9 3. Remove the file link This operation is invoked by DFS client when a user deletes a file. It removes the link for a file from the directory structure (tree), provided the corresponding user has the required permission. 4. Make a directory This operation is invoked by DFS Client when a user creates a directory. It creates a directory in the tree at appropriate place, provided the corresponding user has the required permission. 5. Remove a directory This operation is invoked by DFS Client when a user removes a directory. It deletes the corresponding directory from the tree, provided the corresponding user has the required permission. 6. Get directory contents This operation is invoked by DFS Client when a user wants to see the contents of a directory. It displays names of all the files and directories contained within a given directory, provided the corresponding user has the required permission. 7. Grant permission This operation is invoked by DFS Client when a user gives other user access permission to a file/directory. It assigns/changes the permissions for a particular file/directory for a given user if the requesting user has the required permission. Following data structures have been maintained to implement above operations: struct perm_map{ char *user_id; char perm_bitmap; struct perm_map *next; }; In above data structure user id is user identifier and perm_bitmap denotes the permissions given to the user. Two lower bits of perm_bitmap (0 th bit for read access, 1 st bit for write access and both bits for full control delete permission) are used for setting the permissions. struct routing_table{ char *name; char *domain_id; struct routing_table *next; }; Above data structure provides mapping of various files/directories to the domain where they exist. struct link_map{ int link_id; char *link; BOOL is_active; struct link_map * next; };

10 Above data structure maintains actual link information (ip:directory_path) and node status ( is_active is true if node on which the file is present is alive otherwise false). 3.2 File Locking Lock manager is a daemon that keeps running on each node in the distributed system. It takes care of creating lock objects for files in the system residing on that node and locking and unlocking files as per requests from the file server. It uses hash table for faster search of lock objects in linked lists of locks. It communicates with the local DFS server and local DFS client using message queues. Lock manager also has the capability of maintaining the queue of clients on the file locks and automatically transfer the lock to the next client when the earlier client releases the lock. However, the current DFS server and client and are not making use of this capability. The various operations supported by lock manager are as follows: 1. Create a lock object for a file This will be used when local DFS client will create a new file in the distributed file system. 2. Put lock on a file This operation will fail if the file is already locked or no lock object has been created for file with the name provided. Failure status is returned to the DFS server. 3. Release lock on file. 4. Check the lock status for the file This operation returns the status of file lock (locked/ released) to the DFS server. 5. Destroy lock This will destroy the lock object for the file when it is deleted from the file system. 3.3 Caching In distributed file system files can be located on any node. In such a scenario, accessing files over network can result in a significant delay and network can become a bottleneck. File cache is implemented at each node to reduce the network overhead. The various operations supported by cache manager are as follows: 1. Add a new file to the cache Whenever a client accesses a file, its copy is saved in its local cache if it is not already cache resident. All future requests originating at that client for the cached file is fulfilled using corresponding cached copy.

11 2. Search for a file in the cache Any read/write request for a file by a client is routed via its local cache. 3. Invalidate a file in cache If any file is opened in write mode by a client then after file modification, all the cached copies of the corresponding file are invalidated by the server, having the actual file, to ensure cache coherence. 4. Deleting a file from cache A file is removed from cache in either of two cases, if at the time of file invalidation no client process is using that file or if the original file has been deleted. Following data structures, used to implement above operations, keep track of the cached files along with additional information such as number of client processes currently accessing a particular file, mode in which file has been opened by a client process, actual file location, etc. struct CacheFileDB{ char FileName[MAX_FILE_PATH_WITH_IP]; char LocalFileName[MAX_FILE_PATH_WITH_IP]; int flagperm; int clientcount; int flagupdate; In previous data structure, FileName is the absolute path of the file location. LocalFileName struct CacheFileDB *next; indicates the name by which the file has been cached. It is important to maintain a local file }; name while adding files in cache in order to differentiate between two files having same name but located on different nodes. We have taken care of a scenario where a file is being transferred to cache and at the same time another client process tries to access the same file. When looking for a file in cache, flag field flagperm is checked, if it is set it indicates presence of the corresponding file otherwise if the file entry is there but flag is reset, it indicates file in transit; accordingly an appropriate message is sent to the client. Another flag variable flagupdate and clientcount has been used to enforce file invalidation. struct FILE_HANDLES{ FILE * fd; char FileName[MAX_FILE_PATH_WITH_IP]; char LocalFileName[MAX_FILE_PATH_WITH_IP]; char ipaddress[16]; int mode; struct FILE_HANDLES * next; }; The above data structure maintains original file location information which is required to send back the updated file after a write operation. It also maintains the mode in which file has been opened by a particular client process to prevent any illegal operation on a file (such as client process trying to write in a file which has been opened in read mode).

12 Another significant feature is: whenever a client process requests to open a file in write mode (assuming file is already cached, in case it is not, first it will brought into cache), a separate copy of file is created which is then modified by the client process. This allows other processes on the same node to continue reading the cached copy concurrently. Also on reception of cache invalidate message the copy in cache is invalidated only when all the client processes who opened the file before receiving invalidate message are done with the file use. 3.4 Sharing Semantics The distributed file system has following sharing semantics: 1. Any number of readers located on same or different nodes of the distributed system can access the file simultaneously. 2. Only one writer can write to a file at a time. 3. If some processes are reading file then write access can be granted to a process. The clients who opened the file before the writer got the lock will continue seeing old copies till they are done with the use. 4. If some process wants to read a file that is locked for write it will be denied the file access if the old copy of file is not already present in the cache at the node. 5. Once the cache invalidate message is received by a node no new read request from the client on that node will be given access to old copy of file if the file is already present in cache since it may be being used by old clients. In any case any reader in the system sees the old copy or the updated copy of file but never the partially updated one. This takes care of the correctness of operation. 4 Conclusions Key features of the file system: 1. The distributed nature of name server avoids name resolution from becoming bottleneck. 2. Since name server is divided in two levels (domain DNS and root DNS), name resolution time is independent of location of the file. 3. File caching on disk reduces the network traffic. 4. Write back on close semantics with cache coherence. 5. Stateful server the only state it maintains per file is file lock status and ids of clients having cached copy of the file. Future extensions:

13 1. Location independence: Can be supported by the current design of name server. 2. Queuing of lock requests: Lock manager supports queuing of write requests and automatic transfer of lock to the next client in queue when lock is released but the DFS server and client are not making use of this feature in current implementation. 3. Load balancing module: Scheme 1: In case of our file system, the main source of network traffic is remote files opened for writing. Every time the file is opened for writing, it needs to be transferred to the cache of remote node, at time of closing it has to be restored back. The DFS server will keep a log of files which are being opened for writing. Whenever a request for opening the file F in write mode is made by node N, the server checks the frequency of such opens for file F requested by N. If this frequency crosses a predefined threshold, the decision of moving the file to N is made by server. The server tells node N to store the file on local disk and tells DNS server to change the link of F to its new location. Scheme 2: There will be a load balance daemon running on all nodes in the system. Periodically this daemon will collect the information per file residing on that node, from the local DFS client regarding which users accessed the file in last one period. If some particular remote node has made access requests exceeding the threshold the daemon can transfer file to that node by changing the link in the name server. All this process will be transparent to user as load balancing daemon will put lock on file while it is being transferred so that any new request will be denied. It will change the corresponding entry in name server and only then release the lock. 4. Handling server crash: In case of the server crash, all the clients can still continue accessing the file cached on their local node. The clients who have cached the file for writing will be able to write the copy back to the server as soon as it is up. References 1. Russel Sandberg, The Sun Network Filesystem: Design, Implementation and Experience, Sun Microsystems, Inc. Technical Report, (1985)

14 2. M. Satyanarayanan, A Survey of Distributed File Systems, (1989) 3. D. M. Dhamdhere, Operating System A Concept Based Approach 4. us/library/cc aspx 5. W.R. Stevens, Advanced Programming in the UNIX Environment

15 Test Case We will have two domain DNS servers domain1 and domain2 The client for domain1 will be client11. The client for domain2 will be client21 Configure the IP addresses properly in all the conf files (Put a new line at the end of the last line in the conf file) RUN the following processes on the appropriate VMs: root_dns domain1/domain_dns domain2/domain_dns file_server lockmanager rpc_server cachemanager client11/dfs_client client21/dfs_client user1 user2 client11: mkdir /A (user1) client11: ls / (user1) client21: mkdir /B (user2) client21: ls / (user2) client11: mkdir /B/D (user1) > permissin denied client21: grantpermission to user1 for /B client11: mkdir /B/D (user1) >success client11: grantpermission to user2 for /A client21: mkdir /A/C >permission denied client21: mkdir /A/C >success client21: grantpermission to user1 for /A/C

16 client11: addfile /A/C/user11.txt >success client11: addfile /B/D/user12.txt >success client21: grantpermission to user2 for /B/D client21:addfile /A/C/user21.txt >success client21:addfile /B/D/user22.txt >success client11: ls /A/C client11: openfile /A/C/user11.txt >success client11: openfile /B/D/user12.txt >success client21: openfile /A/C/user21.txt >success client21: openfile /B/D/user22.txt >success client11: rmdir /B/D > not empty client11:rmfile /B/D/user12.txt >success client11: ls /B/D client11: rmfile /B/D/user22.txt > permission denied client21:rmfile /B/D/user22.txt >success client11: rmdir /B/D >success client11: mkdir /A >already exists client21: openfile /A/xyz.txt >path not found client11: rmdir /A/X > path not found client21: mkdir /B/U2DIR > success client11:rmdir /B/U2DIR > permission denied stopdfs startdfs client11: ls / ls /A ls /A/C

17