IndexFS: Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion

Transcription

1 IndexFS: Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion Kai Ren Qing Zheng, Swapnil Patil, Garth Gibson PARALLEL DATA LABORATORY Carnegie Mellon University

2 Why Scalable Metadata Service Many cluster file systems don t scale MDS well Single metadata server (e.g. Lustre/HDFS) Flat object name space (e.g. Amazon S3) Static partition namespace (e.g. Fed. HDFS/Lustre 2.4) Clients Apps Apps Metadata path: (stat, chmod, etc.) Metadata Service Data path: (read, write) Data server Data server Data server Data Data server Data server Data server Data server Data server server Data server Data server Data server Data server Data server Data server 2

3 Why Scalable Metadata Service Need a scalable, automatically load balanced, distributed metadata service IndexFS: A middleware solution that provides metadata performance scaling for existing file systems Throughput (K ops/sec)! 5,000" 500" 50" 5" 0" IndexFS-Lustre (32 clients run IndexFS)" Lustre (single server, 32 clients)" 100x Creation" Lookup" Deletion" 3

4 Motivation Outline! IndexFS: Distributed Metadata Service Architecture Namespace Distribution Hotspot Mitigation with Storm-free Caching Column-style Table and Bulk Insertion Summary 4

5 Middleware Design IndexFS is layered on top of original DFSs Provide a scalable metadata path for existing file systems such as HDFS, PVFS and Lustre Apps Apps Apps IndexFS IndexFS client client library Metadata path (stat, open) Data path (read, write) IndexFS IndexFS metadata IndexFS metadata server metadata server server Data path Underlying FSs (Lustre / HDFS) as object storage Data server Data server Data server Data Data server Data server Data server Data server Data server server Data server Data server Data server Metadata Server 5

6 IndexFS Directory Distribution Randomly assign directory to servers at creation Load balance small directory working set / home alice bob carol 4 Server 0 Server 1 / book Server 2 Server 3 6

7 Dynamically Split Large Directories Use GIGA+ for incremental growth [Patil11] Binary split a partition when its size exceeds a threshold until each server has the same work Load balance giant directories Binary split a large directory Server 1 /big1/.p 0 / _b1 _b4 _b6 50% files Server 2 /big1/.p 1 / _b2 _b3 _b5 50% files 7

8 Efficient Per-Node Metadata Table Represent directory / file as a key-value pair Key:<parent directory s inode number, filename> Value: attributes, file data or file pointer Stored in a key value store : LevelDB [Dean11] book home alice 5 2 / bob carol 4 Lexicographic order Key Value <0, home> 1, attributes <1, alice> 2, attributes <1, bob> 3, attributes <1, carol> 4, attributes <2, book> 5, attributes, small file data 8

9 Basic MDS Scalability: File Creation Workloads: clients create files in one directory Use NFS PRObE Kodiak (8-yr old LANL hardware) Aggregate file create throughput! (K creates/sec)! 1000" 800" 600" 400" 200" Linear" IndexFS-PVFS" Ceph(2006)" PVFS-RAM Disk" 0" 8" 16" 32" 64" 128" Cluster size (Number of servers)! 9

10 X Faster For HDFS & Lustre Run mdtest on IndexFS layered on top of HDFS in PRObE Kodiak and Lustre in LANL Smog clusters 10,000 " Throughput (K ops/sec)! 1,000 " 100 " 10 " 1 " IndexFS-HDFS (Total, 128 servers)" IndexFS-HDFS (Per-server)" HDFS (Single server)" 450x 3x Throughput (K ops/sec)! 5,000" 500" 50" 5" IndexFS-Lustre (Total, 32 servers)" IndexFS-Lustre (Per-server)" Lustre (Single server)" 100x 3x 0 " mknod" stat" remove" 0" mknod" stat" remove" 10

11 Motivation Outline! IndexFS: Distributed Metadata Service Architecture Namespace Distribution! Hotspot Mitigation with Storm-free Caching Column-style Table and Bulk Insertion Summary 11

12 Hot Spot: Server with Root Directories Every lookup starts with the root directory Path traversal needs to visit each ancestor Retrieve inode number and access permissions Early IndexFS was defeated by this hot spot / 0 Hot spot Server 0 Server 1 home 1 / alice carol bob book Server 2 Server 3 12

13 How To Mitigate Hot Spots? Client caches directory entry in many FSs Problem with invalidation callbacks at scale Cache invalidation waits for responses of all clients Clients may fail / may cause invalidation storm Many parallel file systems disable cache when busy Idea: let the clock do the invalidation 13

14 How To Mitigate Hot Spots? Client caches directory entry in many FSs Problem with invalidation callbacks at scale Idea: let the clock do the invalidation Clients use read-only cache for directory entries Servers remember only expiration deadline Assume clock synchronization within data center Directory mutations wait for expiration rmdir, rename, chmod can be slow 14

15 Timeout Based Lease Inspiration: Hot spots at top of tree has few changes / 0 home alice bob carol 4 book Explored two expiration time choices: tree depth: fixed duration K / depth rate based: L*read rate / (read rate+write rate) 15

16 Trace Replay Experiment IndexFS run on top of a 128-node PVFS cluster All metadata and data stored in PVFS as files Workload: Replay 1 million ops/server from LinkedIn trace Pre-create namespace in the empty file system One day trace:10m objects and 130M operations Distribution: 90% reads, 10% mutations Faction of Total Operations! 0.5" 0.4" 0.3" 0.2" 0.1" 0" Open" Readdir" Mknod" RenameF" Mkdir" Chmod" Remove" 16

17 Replay Result: Throughput Directory entry cache mitigates hot spots Throughput (K ops/sec)! 1000" 100" IndexFS+Fixed (100ms)" IndexFS+Tree (3/depth sec)" IndexFS+Rate (r/(r+w) sec)" PVFS+RAM Disk" IndexFS+NoCache" 1.5x 4x 10x 10" 8" 16" 32" 64" 128" Number of servers! 17

18 Replay Result: Chmod Latency Fixed duration lease gets the lowest tail latency Trade-off between tail latency and throughput Fraction of req. (CDF %) e1 1e2 3.6e2 6.4e2 1e3 3e3 5e3 1e4 3.2e4 1e5 2e5 Update latency (in microseconds) Fixed (100ms) Tree (3s/depth) Rate (r/(r+w)s) PVFS+RAM Disk 1e6 1e7 18

19 Motivation Outline! IndexFS: Distributed Metadata Service Architecture Namespace Distribution Hotspot Mitigation with Storm-free Caching! Column-style Table and Bulk Insertion Summary 19

20 Bulk File Creations Some applications want faster file creations e.g. HPC checkpoint applications IndexFS removes two main bottlenecks: LevelDB s background compaction Column-style storage schema Too many RPC round trips Bulk insertion 20

21 1 st Bottleneck: Compaction in LevelDB Slow insertion for storing all metadata and data file 75% of insertion disk traffic caused by compaction Compaction of larger values wastes more bandwidth Fraction of Disk Traffic Rewrite for Compaction First Write for LogCommit 100% 75% 50% 25% 0% 300B 4KB 64KB Value Size 21

22 Background: LevelDB Internal LevelDB (LSM Tree): Insert and update ops: Buffer and sort recent inserts/updates in memory Flush buffer to generate immutable sorted tables Perform compaction to reduce #tables for searching Want to avoid rewrites while having fast lookup A sorted list of (key, value) Memory (K, V) (K, V) Storage Sorted Table #1 Sorted Table #2 Sorted Table #n 22

23 Column-Style Metadata Schema Column-style approach: Metadata/small files appended to non-leveldb log files LevelDB stores only pointers to metadata Don t compact the metadata Delay space reclamation for deleted/over-written values since metadata are small Only compact pointers LevelDB No compaction (<0, home>, ptr) (<1, alice>, ptr) Log file #1 Log file #2 Log file #n 23

24 Speed Up Ingestion Rate Insert 30M entries sequentially or randomly Limit memory to 350 MB, using single SATA disk About 2 to 15 times faster insertion Throughput (K ops/sec)! 60" 50" 40" 30" 20" 10" 0" Sequential Insertion! 52" ColumnStyle" LevelDB" 28" 16" 6" 1.3" 0.37" 300B" 4KB" 64KB" Throughput (K ops/sec)! 60" 50" 40" 30" 20" 10" 0" Random Insertion! 52" ColumnStyle" LevelDB" 16" 10" 1" 1.3" 0.35" 300B" 4KB" 64KB" Value Size! Value Size! 24

25 Fix 2 nd Bottleneck: Bulk Insertion Extend client cache to support write-back No RPC overheads per metadata operation Avoids constant synchronization at servers Better utilization of the underlying bandwidth Assumptions Clients have access to backend storage nodes Only possible for clients to insert new subtrees No namespace conflicts Asynchronous error reporting is acceptable 25

26 Bulk Insertion Implementation File System Clients normal getattr/mknod IndexFS Server localized/batched mkdir/chmod under a subtree server metadata storage SSTables Bulk Insert SSTables Local lease protected namespace Shared Cluster File System Global namespace 26

27 Bulk Insertion: Factor Analysis Evaluation bulk insertion on top of PVFS Perform random insertion and stat in one directory Per-Node Throughput (K op/s) File Create 1.6x Random Getattr x x Flush size: 16KB->64KB 27

28 Summary IndexFS: a middleware approach to scale metadata path, portable to HDFS, Lustre, PVFS Scale out: partition tree on directory basis GIGA+ for incremental partition of giant directories. Read-only consistent caching without per-client state Scale up: optimize log-structured merge tree Column-style schema reduces compaction overhead Bulk insertion to avoid server coordination Code available:

29 Reference [Ren14] IndexFS: Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion. Kai Ren, Qing Zheng, Swapnil Patil and Garth Gibson. SC 2014 [Ren13] TableFS: Enhancing metadata efficiency in local file systems. Kai Ren and Garth Gibson. USENIX ATC 2013 [Welch13] Optimizing a hybrid ssd/hdd hpc storage system based on file size distributions. Brent Welch and Geoffrey Noer. 29th IEEE Conference on Massive Data Storage, [Meister12] A Study on Data Deduplication in HPC Storage Systems. Dirk Meister, Jurgen Kaiser, Andre Brinkmann, Toni Cortes, Michael Kuhn, Julian Kunkel. Supercomputing 2012 [Patil11] Scale and Concurrency in GIGA+: File System Directories with Millions of Files. Swapnil Patil and Garth Gibson. FAST 2011 [Dean11] LevelDB: A fast, lightweight key-value database library. Jeff Dean and Sanjay Ghemawat. [Meyer11] A Study of Practical De-duplication. Dutch T. Meyer, and William J. Bolosky. FAST 2011 [Ganger97] Embedded Inodes and Explicit Grouping: Exploiting Disk Bandwidth for Small Files. Gregory Ganger and Frans Kaashoek. Usenix ATC [O Neil96] The log-structured merge-tree (LSM-tree). Patrick ONeil and et al. Acta Informatica, 1996 [Rosenblum91] The design and implementation of a log-structured file system. Mendel Rosenblum and John Ousterhout. SOSP

30 Comparable Read Throughput Read after rand./seq. insertion of 4KB entries Random Read: read randomly over all entries Range Read: read randomly in 1% adjacent range No compaction, so no clustering for range read Operations/sec! 400" 300" 200" 100" Only use compaction for workload-specific prefetch 0" ColumnStyle" LevelDB" Seq Write, Rand Rand Write, Rand Read" Read" Random Reads! Operations/sec (K) ColumnStyle LevelDB Seq Write, Range Rand Write, Range Read Read Range Reads

31 Cluster Specification PRObE Kodiak CPU: AMD Opteron 252, Dual core, 2.6GHz Memory: 8GB Network: 1GE NIC Storage: Western Digital 1TB hard drive OS: Ubuntu Kernel x86-64 LANL Smog CPU: AMD Opteron 6136, 16-core, 2.4GHz Memory: 32GB Network: Torus 3D, bandwidth: 4.7GB/s Storage: Hardware RAID array, bandwidth 8GB/s OS: Cray Linux Lustre version: