OCFS2: Evolution from OCFS. Mark Fasheh Senior Software Developer Oracle Corporation

Transcription

1 OCFS2: Evolution from OCFS Mark Fasheh Senior Software Developer Oracle Corporation

2 Goals for OCFS2 Become a high performance general purpose Cluster FS Meta data caching Meta data journaling Cross node file data consistency Easy to administer, including operation as shared root fs Integrate well with the Linux Kernel Fix annoyances in OCFS Dis contiguous allocation of file data POSIX semantics for unlink DROP TABLESPACE INCLUDING CONTENTS deletes data files Multiple block sizes Cluster aware tools

3 Linux Kernel Integration OCFS did not integrate well with the Linux kernel 'Common' code layer meant to make OCFS2 cross kernel resulted in lots of duplicated infrastructure. Different coding style Many techniques deemed too dirty by kernel community OCFS2 integrates as tightly as possible Many design points taken from ext3 Take advantage of JBD subsystem No more abstraction layers OCFS2 is Linux only

4 Meta Data Journaling OCFS had limited journaling Relied on ordered sync writes of meta data operations to avoid journaling most operations Logical journaling with no caching records were written out to disk on each journaled operation OCFS2 second Linux file system to use JBD Simpler block based journaling subsystem shipped with the Linux kernel Well tested same journaling subsystem used in ext3 Caches transactions together before write out for improved performance Did not require clustering each node has it's own journal

5 Meta Data Caching OCFS did not cache inode meta data Relied on synchronous, ordered reads and writes Too slow for general purpose use Very inflexible when more complicated meta data operations are needed OCFS2 caches all meta data operations Local Linux file systems rely on buffer cache uptodate flag on buffer_head items. Write a clustered extension of uptodate check Associate buffers with their inodes as they are read and store their caching status on the memory inode structure

6 Flexibility In FS Configuration OCFS had a very inflexible disk format Static system area at beginning of disk could not easily grow Allowed maximum of 32 nodes some customers running 16 node clusters now Supported only one block size (512 bytes) Tiny block size allowed only 3 extent levels before having to grow to a tree! No method for discovering system features

7 Flexibility In FS Configuration OCFS2 configuration fixes many problems found with OCFS disk format Easily extensible, directory based method for finding system files allows any number of nodes (capped at 254) Support for up to 4k blocks Allows upwards of 200 extent records per disk inode Allows operation on high end boxes with a large minimum block size Reworked disk inode layout removes many annoyances Properly sized structure members Uses units in blocks (instead of bytes everywhere)

8 OCFS Directory Layout 255 file entries (1 sector each) File entry includes name + disk inode OCFS uses primitive dirnode directory layout Filename stored in disk inode! Makes hard links very difficult Wastes 256 bytes of disk inode that's half the OCFS inode Makes locking for name space operations versus file changes difficult and slow (must lock both under the same lock) 128K blocks means your smallest directory allocation is 128K Getting a directory listing requires reading at least a 128k block Cross directory rename requires that we copy a full block into another dirnode

9 OCFS2 Directory Layout OCFS2 Directories port of ext3 directory layout and code Well tested Paves the way for us to port the ext3 htree fast directory code Directories are normal files whose data contains an array of records: Inode number / File type / Name Inode blocks are not a part of the directory data Allows us to separate locking for name space operations from inode operations. Cross directory rename does not require a full copy of a disk inode Allows for trivial implementation of hard links Make POSIX style unlink easy and fast only name is removed from directory and disk inode is deallocated when last node closes it Unlinked (aka orphaned ) inodes are temporarily tracked in an orphan directory while in this state.

10 Space Management in OCFS Main (cluster) bitmap One large flat bitmap which could get very cumbersome to search Required lots of memory No meta data indicating where to find free blocks High contention every node went to it to allocate data Block allocators (inode and meta data blocks) Two files per allocator - a bitmap file and a block file Each bit in bitmap file represents a block in the block file Since they are regular files, extending one may require allocating meta data from another one (chicken and egg!)

11 OCFS2 Chain Allocators Alloc Inode } Group Descriptors Group descriptor contains a bitmap covering the group as well as certain accounting information (next group ptr, chain #) Uses allocation groups much like ext3's block groups Main bitmap formatted to cover entire volume at mkfs time. Units in clusters. Chains are self optimizing we can move empty groups to the beginning of their chain as we allocate from them. Suballocators use block size units but get their groups from the main bitmap as needed. Suballocators record suballoc file # and bit # for fast delete.

12 OCFS2 Local Alloc } Main Bitmap } Local Alloc Window One window per node, only use local alloc on smaller space allocations. Reduces lock contention on main bitmap. All bits in window are set on main bitmap, local alloc starts clean. As space is used, local alloc bits are set. Unused bits are returned to bitmap on shutdown, recovery, or on a window move.

13 Distributed Lock Manager Disk based locking in OCFS Slow - requires expensive disk I/O to take and drop cluster locks Locking information is contained in disk inode, making new lock types very difficult. DLM was built into the file system locking operations intermingled with meta data updates. DLM only supported synchronous operation OCFS2 uses VMS style lock manager Fully network based DLM is several orders of magnitude faster Optimized for file system use by removing un-needed features Flexible locking API allows us to create arbitrary lock types OCFS2 now implements locking on a files data (skipped for database operation) allowing for full data coherency, including shared mmap (shared writable mappings are not yet supported) Componentized - allows for a minimal user space locking module - DLMFS

14 OCFS2 CDSL Based on TRU64's mkcdsl utility Allows node local files Actual files are stored in.cluster directory in OCFS2 root Implemented by creating a symlink with special keys in it's target. File system readlink operation substitutes node / machine arch / uid / gid for keys Example: Shared Root We implemented a 14 node OCFS2 shared root cluster in our lab Used CDSL for node specific configuration files in /etc so that every node can have their own unique configuration information. Required no modification of system tools

15 File System Toolchain OCFS toolchain was primitive Didn't understand cluster locking Very limited set of tools OCFS2 has a full set of standard file system tools mkfs.ocfs2, mount.ocfs2, tunefs.ocfs2, fsck.ocfs2, debugfs.ocfs2 All are cluster enabled can detect whether other nodes have the file system mounted and will lock nodes out when making changes libocfs2 file system library modeled after libext2 libo2dlm exposes a very simple user space locking API Entire stack has only 1 configuration file ocfs2console GUI application to configure an OCFS2 cluster

16 Status Mounted a 14 node shared root cluster Same success rate as ext3 with LTP (POSIX compliance test) Most operations on OCFS2 are comparable to ext3 Retains the O_DIRECT performance of OCFS Currently included in Andrew Morton's -mm Linux kernel tree Node management subsystem is currently being abstracted further so other kernel cluster components can make use of it. Features to be added immediately following production release Endianness Read only operation

17 Q & A Q U E S T I O N S A N S W E R S

18