zfs - A Reliable Distributed ObS-based File System

Transcription

1 zfs - A Reliable Distributed ObS-based File System Julian Satran and Avi Teperman IBM Research Laboratory in Haifa

2 zfs Background zfs is part of continued research on storage Started with Distributed Sharing Facility Continued with Antara Object Store Device ObS standard is under development by SNIA and is a formal T10 project zfs will move to future ObS zfs is an attempt to explore completely distributed File System based on Object Storage 2 IBM Research Laboratory in Haifa

3 zfs Goals Operate well on few or thousands of machines Built from off-the-shelf components with Object Storage Use the memory of all machines as a global cache Achieve almost linear scalability 3 IBM Research Laboratory in Haifa

4 How zfs Goals are Achieved Scalability is achieved by: Separating storage-space management from file management Storage-space management is encapsulated in the ObS Dynamically distributing file management No central server No single point of failure Machines can be dynamically added/removed from zfs 4 IBM Research Laboratory in Haifa

5 zfs Architecture 5 IBM Research Laboratory in Haifa

6 SAN File Systems Architecture Many computers connected to many storage devices by high speed network Clients can have non-mediated access to storage improved storage access 6 IBM Research Laboratory in Haifa

7 SAN File Systems Challenges Security and Protection Security vs. Protection Protection: buggy clients, inadvertent access, etc. Useful inside and outside glass house Security: intentional attempts at unauthorized access Essential outside the glass house SAN Security/Protection Today: At unit level Zoning/Fencing/Masking are hard to use and are done at LU level Too many actively used blocks to provide block level security Coordination overhead too high Assume only trusted clients 7 IBM Research Laboratory in Haifa

8 SAN File Systems Challenges Scalability Scalability is not an issue when volumes are partitioned among hosts However, Shared access is one of the touted benefits of a SAN For shared read-write access host must coordinate usage of blocks File systems must coordinate allocation of blocks to files Coordination can result in: False contention Between hosts allocating space from different logical units Additional communication It may be possible to piggyback some of the information 8 IBM Research Laboratory in Haifa

9 Object Store Security Credential Client Object Store Credential Authorization Req Security Admin Shared Secret All operations are secured by a credential Security achieved by cooperation of: Admin authenticates/authorizes clients and generates credentials. ObS -- validates credential that a client presents. Credential is cryptographically hardened ObS and admin share a secret Goals of Object Store security are: Increased protection/security At level of objects rather than LU Hosts do not access metadata directly Allow non-trusted clients to sit on SAN Allow shared access to storage without giving clients access to all data on volume 9 IBM Research Laboratory in Haifa

10 zfs Components Object Store Device (ObS) Assumes ObSs have fail over mechanism ObS is rather recent, and the Working Group is approaching a first version of the spec Lease Manager (LMGR) File Manager (FMGR) Transaction Manager (TMGR) Front-End/Cache (FE/Cache) No Single Point of Failure 10 IBM Research Laboratory in Haifa

11 zfs Components Object Store Device (ObS) Using ObS allows zfs to focus on File Management and Scalability Allocation of storage is done in the ObS Security is handled by the Security Admin and ObS Cooperative caching poses security challenge 11 IBM Research Laboratory in Haifa

12 zfs Components Lease Manager (LMGR) The need for lease manager stems from the following facts: Locking mechanism is required to control access to disks In SAN file systems clients can write directly to ObSs. zfs uses a locking discipline for concurrency control 12 IBM Research Laboratory in Haifa

13 zfs Components Lease Manager (LMGR) To reduce ObS s overhead the following mechanism is used: Each ObS is associated with one lease manager ObS maintains and grants to its LMGR one major lease LMGR grants object leases to the FMGRs requesting it FMGR grants range leases to the FEs requesting it 13 IBM Research Laboratory in Haifa

14 zfs Components Lease Manager (LMGR) The LMGR + FMGR operation distributed and safe lease. We prefer leases over locks to reduce state related issues that make recovery complex Leases incur the overhead of leases renewal negligible as leases a renewed periodically with one operation-per-renewing-machine 14 IBM Research Laboratory in Haifa

15 zfs Components File Manager (FMGR) Each file is managed by one file manager only Each lease request on the file is mediated by the FMGR The first client opening a file creates an instance of the file manager (in the current implementation this instance is local to the client) The FMGR interacts with the proper LMGR to get the object lease and grants range leases to the FE 15 IBM Research Laboratory in Haifa

16 zfs Components File Manager (FMGR) The FMGR keeps track of: Where (which FE) each file s extents reside Each file s leases If client X requests page and lease which resides on client Y the FMGR will direct FE/Cache on Y to send the requested page and lease directly to X 16 IBM Research Laboratory in Haifa

17 zfs Components File Manager (FMGR) File manager assignment is dynamic FE requests lease from local FMGR (fmgr i ) fmgr i requests object lease from LMGR. LMGR checks: If no other FMGR holds the object lease it grants the object lease to fmgr i if another FMGR (fmgr k ) has the object lease LMGR returns its address to fmgr i fmgr i instructs FE to request the lease from fmgr k 17 IBM Research Laboratory in Haifa

18 zfs Components Transaction Manager (TMGR) Meta data operations handle several objects To ensure file system consistency zfs implements them as distributed transactions All meta data operations are handled by the TMGR 18 IBM Research Laboratory in Haifa

19 zfs Components Front-End / Cache FE Runs on every client machine Presents to the application/user the standard file system API Provides access to zfs files and directories Cache Provides access to zfs data and metadata in local memory to other machines Since data is transferred from memory to memory, there is a security issue if one client is un-trusted Requires further research 19 IBM Research Laboratory in Haifa

20 zfs Architecture 20 IBM Research Laboratory in Haifa

21 zfs read() Operation 21 IBM Research Laboratory in Haifa

22 zfs write() Operation Only read leases 22 IBM Research Laboratory in Haifa

23 zfs write() Operation One write lease 23 IBM Research Laboratory in Haifa

24 zfs Failure Handling LMGR i failed Detected by all FMGRs that hold leases for objects of ObS i Each FMGR informs all FEs holding files on ObS i to flush their dirty data and release files FMGRs instantiate new LMGR i which tries to get the ObS i major lease Once the previous major lease expires, one LMGR i gets the major lease and all others are terminated Operation on ObS i resumes 24 IBM Research Laboratory in Haifa

25 zfs Failure Handling FMGR i failed Detected by all FEs that hold leases granted by FMGR i Each FE flushes all its dirty data to ObS i and release files FEs instantiate new local FMGR i When application requests a range lease for file F the local FMGR requests an object lease from the LMGR If another local FMGR already got the object lease, its address is passed back to the FE via the requesting FMGR The FE connects to the correct FMGR and operation continues All this is transparent to the application 25 IBM Research Laboratory in Haifa

26 zfs Future Research Distributed Transactions Handling meta data operations in a distributed scalable manner Add Security Mechanism Investigate how cooperative cache integrates with the security model of ObjectStore 26 IBM Research Laboratory in Haifa

27 Related Documents zfs Web Site DSF Web Site Publications zfs - A Scalable distributed File System using Object Disks O. Rodeh & A. Teperman; MSST03 Group Communication - still complex after all these years R. Golding & O. Rodeh; SRDS 2003 Object Store Based SAN File Systems J. Satran & A. Teperman; SSCCII IBM Research Laboratory in Haifa

28 Current Status zfs implemented on Linux Kernel Currently Except for TMGR all components work Initial results show ~20% improvement on read with cooperative cache 28 IBM Research Laboratory in Haifa

29 Related Work Lustre Clients Cluster control-system (MDS) Storage targets (programmable ObS) StorageTank Client MSD (different IP network) Storage: currently standard SCSI disks over SAN Integration with ObjectStore xfs and XFS xfs is similar to zfs with static distribution policy Implemented cooperative caching - M. D. Dahlin, C. J. Mather, R. Y. Wang, T. E. Anderson and D. A. Patterson, A Quantitative Analysis of Cache Policies for Scalable Network File System, Computer Science Division, University of California at Berkeley 29 IBM Research Laboratory in Haifa