pblk the OCSSD FTL Linux FAST Summit 18 Javier González Copyright 2018 CNEX Labs

Transcription

1 pblk the OCSSD FTL Linux FAST Summit 18 Javier González

2 Read Latency Read Latency with 0% Writes Random Read 4K Percentiles 2

3 Read Latency Read Latency with 20% Writes Random Read 4K + Random Write 4K 4ms! Signficant outliers! Worst-case 30X Percentiles 3

4 NAND capacity grows bigger Source: William Tidwell -The Harder Alternative Managing NAND capacity in the 3D age Capacity is overgrowing bandwidth Small form factors only aggravate the problem 4

5 Host-based QoS is best effort Increased capacity force schedulers to share resources that before could be dedicated. Applications optimize their internal I/O structures to help SSDs do a better job: - Log-structured databases - Journaled File Systems - Host Garbage Collection App 1 App 2 App 2 The SSD re-orders, -schedules, -maps I/Os based on hints and patterns There is no good host/device interface to do common QoS LUN 0 LUN 1 LUN 2 LUN 3 LUN 4 LUN 5 Traditional SSD LUN 6 LUN 7 LUN N-1 - Both sides fighting each other for QoS! 5

6 Open-Channel SSD Approach App 1 App 1 App 2 App 2 App 2 App 2 LUN 0 LUN 1 LUN 2 LUN 3 LUN 4 LUN 5 LUN 6 LUN 7 LUN N-1 LUN 0 LUN 1 LUN 2 LUN 3 LUN 4 LUN 5 LUN 6 LUN 7 LUN N-1 Traditional SSD Open-Channel SSD Tier 1 applications I/O patterns are: - well understood - heavily modified to be workload-optimized (OS support too) Tier X (>1) applications can be easily compartmentalize Tier 1 and Tier X applications are forced to coexist in order to maximize resource utilization 6

7 Open-Channel Comparison TRADITIONAL SSD Fully host-managed Open-Channel SSD (1.2) Host-driven Open-Channel SSD (2.0) 7

8 Open-Channel SSD 2.0 Host software is media-agnostic - Media abstracted by generic geometry - Wear-level indexes and thresholds Media-specific actions through feedback loop - Refresh data New NAND generations only need to integrate media-specific changes on each SSD generation Interface with the host remains the same - Advances as in NVMe: OCSSD 2.1, 2.2, etc. 8

9 Open-Channel SSD Identification: Expose the geometry of the SSD - Parallelism: # Channels, # LUNs, # Chunks, and number of LBAs within a chunk. - Media timings Read, write, and erase. - Write requirements Minimum write size and optimal write size 2. I/O submission: Richer I/O interfaces - Support for vector I/O (R/W/E) using scatter/gather address list - Support for NVMe read and write semantics (zoned devices) - Continuous access no maintenance windows Logical Block Address (LBA) Sector MSB Logical Block Address with Geometry Encoded Channel LUNs Chunk Sector MSB LSB LSB 3. Host / SSD communication: Richer admin interfaces NVMe Interface - Chunk states through Report Chunk command (get log page) LBA start address Write pointer (host guarantees to write sequentially within a chunk) Block State (Free, Open, Full, Bad) Wear Index Solid State Drive Media Controller LUNs Channel 0 Channel 1 - Active NAND management feedback loop using NVMe AER. Drive tells host to rewrite chunks when necessary. 9

10 LightNVM Architecture NVMe Device Driver - Detection of OCSSD - Implement support for commands LightNVM Subsystem - Core functionality - Target management (e.g., pblk) - Sysfs integration User Space Kernel Space Scalar Read/Write (optional) Geometry Vectored R/W/E LightNVM Subsystem NVMe Device Driver Application(s) File System pblk (3) (2) (1) High-level I/O Interface PPA Addressing - Block device using pblk - Application integration with liblightnvm Hardware Open-Channel SSD 10

11 pblk- Host-side Flash Translation Layer Multi-target support - I/O isolation Fully associative L2P table (4KB mapping) Host-side write buffer to guarantee read and writes Cost-based garbage collector, using valid sector count as metric Capacity based rate limiter. Function of user and GC I/O present in write buffer Scan-based L2P recovery. Scan metadata in closed lines and OOB on open lines sysfs interface for statistics and tuning 11

12 pblk: /drivers/lightnvm pblk: Responsibilities and Location LightNVM: /drivers/lightnvm & /drivers/nvme/host/lightnvm.c & /include/linux/lightnvm.h 12

13 Completion Path Submission Path pblk: I/O path User I/O threads generic_make_rq read GC I/O thread write Respect Media Constrains Ring Write Buffer 1. Reserve space in buffer 2. Copy user data 3. Save write context 4. Complete I/O to block layer Context User Data 1. Map buffer L2P in current line - Update L2P table on wrap-up 2. Map metadata for previous line 3. Map erase for next line 4. Submit I/O set Line 0 Line 1 Line 2 Line N P0 P1 P2 PN Configurable Mapping Strategy L2P Lookup Metadata 1. Update buffer pointers 2. Deal with W/E errors Open-Channel SSD 13

14 Dynamic Allocation Over-Provisioned Area Static Allocation pblk: Garbage Collection Cost-based recycling mode based on valid sector count - Wear-leveling being implemented (depending on 2.0) Naïve GC on (current implementation) - Requires rate-limiter to guarantee space for GC - Introduces write amplification - Unpredictable bandwidth (steady state) Host usable area Hot Data Hot Data Hot Data Over-Provisioned Area Cold Data Cold Data Cold Data Hot / Cold data separation (in-progress) Hot Data Cold Data - Improve write amplification Hot Data Cold Data - Predictable steady state (static / dynamic) - Use LUN bandwidth as a natural rate limiter - GC dedicated write buffer enable vector copy Two GC modes are available: - Move data using the host s CPU - Use vector copy command Move data directly in the controller New Data New Data New Data New Data New Data GC Data GC Data GC Data 14

15 Pblk - Status Fairly stable for its age targeting production in All basic functionality implemented Ongoing features - Hot / Cold data separation - RAILS: Trade write bandwidth and capacity for latency implemented by Heiner Litz - Wear-levelling - FTL log Generalization - Can it be useful for append-only file systems to manage random areas (e.g., metadata) - Convert into device mapper. Ideas? - Port pblk to user space. Ideas? Integrations - Implement data placement and scheduling into F2FS. Other proposals? 15

16 Active community Open-Channel SSD Ecosystem - Status - Multiple drives in development by commercial SSD vendors - Multiple contributions to open-source - Active research using Open-Channel SSDs Growing software stack - LightNVM subsystem since Linux kernel User-space library (liblightnvm) support from Linux kernel pblk host FTL available from Linux kernel Joint Development Framework (consortium) being formed in Apply industry input and standarize (CSP, NAND Vendors, SSD Vendors, Controller Vendors) - Result in form of 2.1, 3.0, something else? 16

17 Pblk the OCSSD FTL Linux FAST Summit 18 Javier González

18 L2P table - 4KB granularity (1GB per 1TB) Pre-populated bitmap encoding map (*) - Bitmap encodes bad blocks and metadata pblk: Data Placement - Save expensive calculations on fast path (+1 vs. division/modulus) - Trivial to change stripping strategy L2P mapping is decoupled from I/O scheduling - Simplifies adding new mapping strategies - Simplifies error handling - Does not necessarily affect disk format - Default: Stripe across channels and LUNs to optimize for throughput Metadata at beginning and ending of each line (*) missing patch for non power-of-2 NAND configurations 18

19 Goals pblk: I/O Scheduling - Fully utilize the bandwidth of the media 1 core (E5-2620, 2.4GHz) can move ~3.7GB/s (~1MIOPS) - Minimize impact of reaching steady state (i.e., user + GC) - Rate-limit user and GC I/O according to the device s capacity Single write thread - Submits user write I/Os as buffer entries are mapped - Submits write I/Os for previous line metadata Align with user data to minimize disturbances - Submits erase I/Os for next line Align with user data to minimize disturbances Distribute price of erasing across all lines 19

20 Per line metadata: - Distributed log across lines (user / GC) pblk: Recovery smeta - Mark line as open when it is allocated - Give line a sequence number - Create a reverse line list - Store the LUNs forming the line - Store active write LUNs emeta - Replicate smeta for consistency - Store updated bad block bitmap for line - Store L2P portion for line (lba list) - Store valid sector count (VSC) for all lines Per page metadata: - 16 bytes per 4KB - Store lba mapped to 4KB sector in OOB area (8 bytes) Recovery: Scan all lines and reconstruct L2P in order - first closed lines, then open lines 20

21 Monitor pblk s state through sysfs pblk: Debug, Tracing and Monitoring - /sys/class/nvme/nvme0/nvme0n1/lightnvm (static device information) - /sys/block/$pblk_block_dev/pblk/ Debug mode that allows sanity check on all command submission and internal state - CONFIG_NVM_DEBUG=y Implementing tracing points - Better tool integration - Less performance impact Implementing pblk tool - Equivalent to mkfs, but for a FTL - Allow sanity check, migration, recovery, etc. - Use liblightnvm 21

22 Multi-Tenant Workloads NVMe SSD pblk on OCSSD 2 Tenants (1W/1R) 4 Tenants (3W/1R) 8 Tenants (7W/1R) 22

23 Instantiate pblk using nvme-cli tool pblk: getting started - Example: sudo nvme lnvm create -d nvme0n1 -t pblk -n pblk0 -b 0 -e 127 f - Block device in /dev/pblk0 QEMU - OCSSD backend in QEMU. Simulates controller/media constrains - Repository: git@github.com:openchannelssd/qemu-nvme.git - Look at options in hw/block/nvme.c - nvme,drive=mynvme,serial=deadbeef,namespaces= 1,lver=1,lmetasize=16,ll2pmode=0,nlbaf=5,lba_inde x=3,mdts=10,lnum_lun=4,lnum_pln=2,lsec_size=40 96,lsecs_per_pg=4,lpgs_per_blk=512,ldebug=0 \ Available CNEX SDK for research and collaboration 23

24 Internals of an SSD Read/Write Solid-State Drive Host Interface Parallel Units Read/Write/Erase Transforms R/W/E to R/W Responsibilities Flash Translation Layer Media Error Handling Media Retention Management Media Controller Channel X Channel Y Read (50-100us) Write (1-10ms) Erase (3-15ms) Manage Media Constraints ECC, RAID, Retention Tens of Parallel Units! 24

25 Open-Channel SSD Benefits Allow software to innovate faster than hardware - Decouple placement and scheduling from media management - Workload-specific optimizations Rapid enablement of new NAND generations - Reuse FTL logic in hosts allows for faster time to market - Decoupled architectures is less error-prone Support a broad set of applications on shared hardware - Guarantee parallelism and I/O isolation - Do not require maintenance windows Vendor neutrality and supply chain diversity - Standardized specification supported by cloud and device vendors - Similar model to standard NVMe 25

26

27 27

28 28

29 Active community Status - Multiple drives in development by commercial SSD vendors - Multiple contributions to open-source - Active research using Open-Channel SSDs Growing software stack - LightNVM subsystem since Linux kernel User-space library (liblightnvm) support from Linux kernel pblk host FTL availiable from Linux kernel CNEX Microsoft strategic collaboration on Open-Channel SSDs announced at FMS Joint Development Framework (consortium) being formed in 2018

30 CNEX Labs, Inc. Teaming with NAND Flash manufacturers and industry leaders in storage and networking to deliver the next big innovation for solid-state-storage.