Falcon: Scaling IO Performance in Multi-SSD Volumes. The George Washington University

Transcription

1 Falcon: Scaling IO Performance in Multi-SSD Volumes Pradeep Kumar H Howie Huang The George Washington University

2 SSDs in Big Data Applications Recent trends advocate using many SSDs for higher throughput in Graph Analytics Machine Learning Key-Value stores, etc. New techniques are taking advantage of high random IOPS of SSDs Fine grained IOs in graph processing [FAST 17] Doing random IOs in graph processing [ATC 16] Range scan in WiscKey is many parallel random IOs [FAST16] Increasing use of batched IO interfaces such as libaio in Linux 2

3 Programming Complexity Existing IO Model Computing Threads High Application-managed IO (1 application IO thread per-ssd) Userspace IO Buffer Application IO Threads User space Kernel Space Kernel-managed IO (many application IO threads) IO Stack IO Stack SSD SSD Volume (b) Kernel-managed 1 or more application IO thread per-volume SSD SSD (a) Application-managed 1 application IO thread per-ssd Low Kernel-managed IO (1 application IO thread) Low IO Performance Falcon (1 application IO thread) High 3

4 Outline 1 Overview and Problem Statement 2 Background and Block Layer Insufficiency 3 Falcon Architecture 4 Evaluation 4

5 Linux: IO Flow and IO States Direct IO bio bio SSD 1 start Block Layer Instance (SSD 1 ) Applications Volume Manager Instance split VFS SCSI Layer and Drivers Page Cache bio bio m SSD m 2 split Block Layer Instance (SSD m ) IO Phases: Plug, Unplug, Dispatch, Completion start IO States: start, split, merge, wait, ready, insert, dispatch, complete 1 Process Next Request 3 merge yes merge? 5 wait no tag no available? no yes enqueue to plug-list sort bio no unplug? yes classify 6 4 enqueue to software-queue ready Plug Phase Unplug Phase insert Free tag Dispatch Phase Complete IO 8 complete IRQ event completion Completion Phase dispatch IO to driver 7 dispatch 5

6 Linux: Mixes IO Batching and IO Serving Tasks Examples: Mixing batching with merge and tag allocation in plug phase Mixing classify with sort in unplug phase bio 1 bio 2 bio m Plug Phase (batch, merge, tag allocation) Thread-specific plug-list Root cause: Many tasks are tied to plug-list Not designed for multi-ssd volume Creates many Insufficiencies Lack of parallelism in IO processing Inefficient Merge and Sort Unpredictable blocking Unplug Phase (sort, classify) Software queues Software queues Dispatch Phase (dispatch) To SCSI Layer and Drivers Linux block layer control flow 6

7 Stack Latency (in usec) Insufficiency #1: Lack of Parallelism sda sdb sdc sdd sde sdf sdg sdh 8-SSD 8-SSD Volume Member Drives (avg) 1-SSD Increasing stack latency of member SSDs E.g., Stack Latency of sda is less than sdb Effect of sequential IO serving and round-robin dispatch IOs of last drive will acquire insert after IOs of every other drive gets dispatched 7

8 Latency (usec) Insufficiency #2: Inefficient Merge and Sort SSD Stack Latency 8-SSD Volume Device Latency 1% 8% 6% 4% 2% % 1-SSD 8-SSD Volume (a) Absolute latency (b) Percentage Stack Latency in 8-SSD volume is greater than 1-SSD system Stack latency is greater than device latency in 8-SSD volume Plug-list intermixes IOs destined to all member drives Search for a merge candidate even in unrelated IOs Larger sorting workload across SSDs 8

9 Normalized Throughput Queue Depth Insufficiency #3: Unpredictable Blocking SSD LSI 93-8i HBA 2-SSD Intel C62 SATA sda sdb sda+sdb Available tag Time (Sec) (a) IO performance Scaling (b) Tag usage in 2-SSD SATA volume If tag allocation fails, the IO thread blocks waiting for a free tag Uncertainty about active IO count in the pipeline Storage controller dependent Compromises the IO scalability in SATA controller connected SSD volume 9

10 Outline 1 Overview and Problem Statement 2 Background and Block Layer Insufficiency 3 Falcon Architecture 4 Evaluation 1

11 Falcon: Separates IO Batching from IO Serving Tasks Direct IO bio Applications VFS Page Cache bio 1 start 1 start bio 1 bio 2 bio m Batching Phase (batch) Thread-specific plug-list FML Falcon IO Management Layer (FML) Performs IO batching tasks Only Volume Manager Instance bio 1 bio m 2 split 2 split Classification Phase (classify) FBL FBL Falcon IO Management Layer (FML) FBL Instance (SSD 1 ) SCSI Layer and Drivers SSD 1 Falcon Threads FBL Instance (SSD m ) SSD m software queues Sort Phase (sort) Process Phase merge, tag allocation, dispatch 11 Software queues To SCSI Layer and Drivers Sort Phase (sort) Process Phase merge, tag allocation, dispatch Falcon Block Layer (FBL) performs IO Serving tasks only

12 Falcon: Feature Comparison with Linux Block Layer Features Linux 1-SSD Linux Multi-SSD Falcon Volume Parallel Processing NA Per-Drive Sort Neighbor Merge Dynamic Tag Management Well-suited for multi-ssd volume Improvements are applicable to 1-SSD system also 12

13 Falcon IO Management Layer IO batching in plug-list No processing, just batching Process Next IO Request bio (from Volume manager Layer) Classification Single pass operation No sorting Enabling parallel processing Creates Falcon threads per FBL Uniform unplug criteria 32 per-ssd, 256 for 8-SSD volume Lower criteria for low IO demand, and latency sensitive applications See the paper for more details no enqueue to plug-list unplug? yes Classify plug-list to temporary per-drive queues insert Batching Phase Move per-drive queues to software queues Spawn Falcon threads, if needed Classification Phase 13

14 Falcon Block Layer Sort Phase Per-Drive Sort bio (from FML layer) Process Phase Neighbor Merge Dynamic tag allocation Dispatch Sort software-queue Neighbor merge merge Sort Phase Process Phase Completion Phase Completion Phase Able to saturate a Samsung 95 Pro 512GB NVMe SSD 1375 MB/s (13% better than Linux) 14 Allocate tag ready Dispatch IO to driver dispatch To SCSI layer and Drivers Complete IO complete IRQ event completion

15 Normalized Throughput Queue Depth Dynamic Tag Management Allocate a tag only if a dispatch is required Bio-queue keeps bio objects yet to be dispatched Pressure point controls the active IO count in the pipeline 3 1-SSD 2-SSD 2 1 Linux Falcon (a) IO Throughput Scaling for SATA controller sda-linux sdb-linux sda-falcon sdb-falcon Time (sec) (b) Tag usage in 2-SSD SATA volume 15

16 Outline 1 Overview and Problem Statement 2 Background and Block Layer Insufficiency 3 Falcon Architecture 4 Evaluation 16

17 Evaluation Setup Falcon: 6 lines of C kernel code add Ubuntu 16.4 version with Kernel version 4.4., Blk-mq block layer 2 Intel Xeon CPU E GHz with 6 cores each 32GB DRAM 8 Samsung EVO 85 5 GB SSDs, connected using LSI SAS93-8i HBA 4KB Stripe size is used by default Raw volume, Ext4 and XFS file systems are evaluated Revised FIO is used as micro-benchmark 17

18 Throughput (MB/sec) Throughput (MB/sec) Ext4 File Random IO Linux Falcon IO thread count Linux Falcon IO thread count (a) Single file read throughput (b) Single file write throughput Ext4 has file inode lock issue 1.77x speedup for random read 1.59x speedup for random write 18

19 Throughput (MB/sec) Buffered Random Write Linux Falcon SSD count Buffer cache has just 1 thread per volume for flushing dirty pages 1.39x speedup for 4-SSD volume 1.59x speedup for 8-SSD volume 19

20 Speedup Graph Processing G-Store[SC 16] is used Semi-external graph analytics engine Configurable number of IO threads Linux setup, 1 and 8 IO threads Falcon : 1 IO thread 8-SSD volume, XFS filesystem Kronecker graph scale 28, edge factor 16 is used BFS, kcore: High random IO Connected component (CC), PageRank (PR): Sequential IO Linux 1 IO Thread Linux 8 IO Threads Falcon BFS kcore CC PR Graph Processing 4.12x speedup over Linux 1 IO thread setup 1.78x speedup over Linux 8 IO thread 2

21 Throughput (MB/sec) IO Trace Replay Trace Name Read (%) IO size range Size (GB) Type UM-Financial B KB Online transaction processing UM-Financial B KB 8.44 Online transaction processing UM-Websearch B KB Web Search UM-Websearch KB - 32KB Web Search FIU-Home 1 512B - 512KB Research group activities FIU-Mail KB - 4KB Mail Server FIU-Webuser KB - 128KB 3.94 Web User FIU-Web-vm KB - 4KB Webmail proxy/online course management Linux Falcon UM-Financial1 UM-Financial2 UM-Websearch1 UM-Webserach2 FIU-Home FIU-Mail FIU-Webuser FIU-Web-vm 1.67x better IO throughput on average 21

22 Conclusion Separating batching from IO serving tasks is the key for IO scalability in multi-ssd volume Falcon enforces per-drive processing Improves the IO stack performance Parallelizes the IO serving tasks across member SSDs Falcon improves the performance by 1.69x for various applications on 8-SSD volume Falcon achieves 1.13x throughput for an NVMe SSD 22

23 Thank You Falcon is open source now 23