Exploiting the benefits of native programming access to NVM devices

Transcription

1 Exploiting the benefits of native programming access to NVM devices Ashish Batwara Principal Storage Architect Fusion-io

2 Traditional Storage Stack User space Application Kernel space Filesystem LBA Block Device Driver Hardware LBA view enforced by Storage Protocols (SCSI/SATA etc.) Device 2

3 Flash is Different From Disk Area Hard Disk Drives Flash Devices Logical to Physical Blocks Nearly 1:1 Mapping Remapped at every write Read/Write Performance Largely symmetrical Heavily asymmetrical. Sequential vs Random Performance Background operations An order of magnitude difference Rarely impact foreground Minimal difference Regular occurrence. If unmanaged - can impact foreground Wear out Largely unlimited writes Limited writes IOPS 100s to 1000s 100Ks to Millions Latency 10s ms 10s - 100s us TRIM Do not benefit Improves performance 3

4 Flash Translation Layer 101 Input Logical Block Address (LBA) Flash Translation Layer Output Commands to NAND flash 4

5 Flash in Traditional Storage Stack User space Application Kernel space Filesystem LBA Block Device Driver Hardware Flash Translation Layer LBA PBA Device 5

6 Virtual Storage Layer Fusion-io s host based FTL Virtual Storage Layer (VSL) Cut-thru architecture avoids traditional storage protocols Scales with multi-core Provide a large virtual address space HW/SW functional boundary defined as optimal for flash Traditional block access methods for compatibility New access methods, functionality and primitives natively supported by flash Host DRAM / Memory / Operating System and Application Memory Banks iomemory Virtualization Tables DATA TRANSFERS PCIe iodrive iomemory Data-Path Controller Channels Wide CPU and cores Virtual Storage Layer (VSL tm ) Commands 6

7 Flash Memory Evolution Native Access Traditional SSDs Flash as a drive Flash as a cache Flash with direct I/O semantics Flash with memory semantics Application Application Application Application Application Application Application OS Block I/O OS Block I/O OS Block I/O Open Source Extensions Direct-access I/O API Family Open Source Extensions Memory access API Family Host File System File System File System Block Layer SAS/SATA Host Block Layer Block Layer directfs native file system service directfs Remote Network RAID Controller Flash Layer VSL directcache VSL VSL VSL VSL Read/Write Read/Write Read/Write Read/Write Read/Write Load/Store 7

8 Direct-access I/O API family Native Access Application Host Remote Legacy SSDs Flash as a drive Flash as a cache direct I/O Primitives Sparse Addressing Application Application Atomic multi-block Operations Flash with direct I/O semantics Flash with memory semantics Application Application Application Application OS Block I/O OS Block I/O OS Block I/O File System File System File System Block Layer Block Layer SAS/SATA direct Key-Value Store Network RAID Controller Flash Layer Write PTRIM Exists, Range Exists Conditional Write Range Read Host NVM optimized with transactional semantics VSL Block Layer directcache VSL Open Source Extensions Direct-access I/O API Family directfs native file system service VSL Open Source Extensions MemorySemantics API Family directfs VSL VSL Read/Write Read/Write Read/Write Read/Write Read/Write Load/Store 8

9 Sparse Addressing 9

10 Excess work by conventional cache Conventional Block Cache Application Cache: HDD block -> Flash LBA Cache Hit Metadata, Persistence, Recovery logic etc. Cache Miss Flash FTL: LBA -> PBA Backend Store Block Device Two Translations Additional metadata, logic etc. 10

11 Sparse address mapping Mapped to sparse address space Mapped to sparse address space Backend store 11

12 VSL based cache VSL Based Cache Application VSL Based Cache: Minimal Fixed Metadata Cache Hit - Leverage on primitives VSL Sparse HDD LBA -> PBA Cache Miss Backend Store Block Device Fewer Translations Minimal additional metadata, logic etc. 12

13 Direct I/O - Atomic Operations Traditional Atomicity (with Hard Disks) Traditional Atomicity (with SSD) Atomicity in iomemory DBMS Atomicity Trans Log Applications DBMS Atomicity Trans Log Applications File System DBMS Applications Atomicity Atomicity Block IO Layer FileSystem Metadata Journaling, Copy-on-Write Block IO Layer FileSystem Metadata Journaling, Copy-on-Write Block IO Layer Generalized iomemory Layer Re-mapping Atomic- Operation Wear-Leveling Sector Read/Write Flash Translation Layer Sector Read/Write Page Read/Write Controller Block Erase Re-mapping Wear-Leveling Block Erase Page Write Page Read Solid State Disk NAND Flash Memory NAND Flash Memory iomemory Disk Drive 13

14 Transactional Block Interface Application issues call to transactional block interface WRITE ALL BLOCKS ATOMICALLY iov[0] iov[1] iov[2] iov[3] iov[4] Virtual Storage Layer 14

15 Transactional Block Interface Application issues call to transactional block interface WRITE ALL BLOCKS ATOMICALLY TRIM ALL BLOCKS ATOMICALLY LBA 7 + offset LBA 24 + offset LBA 42 + offset iov[0] iov[1] iov[2] iov[3] iov[4] LBA 68 + offset Virtual Storage Layer 15

16 Transactional Block Interface Application issues call to transactional block interface TRANSACTION ENVELOPS WRITE ALL BLOCKS ATOMICALLY TRIM ALL BLOCKS ATOMICALLY WRITE AND TRIM ATOMICALLY LBA 7 + offset LBA 7 + offset LBA 24 + offset LBA 24 + offset LBA 42 + offset iov[0] iov[1] iov[2] iov[3] iov[4] LBA 68 + offset iov[3] iov[4] Virtual Storage Layer 16

17 InnoDB (mysql) Double-Write Page CPage B Page A DB Server Pages copied into memory OS RAM Page CPage B Page A Two Writes iodrive Page C Page CPage B Page A Page A Double-write buffer Actual tablespace Page B 17

18 InnoDB (mysql) atomic-write Page CPage B Page A DB Server Less Writes Better Performance ACID Compliance OS Pages copied into memory RAM Page C Page B Page A iodrive VSL (Atomic- Write) Page C Page A Page B 18

19 Sysbench Performance With Atomic-Write MySQL extension for Atomic-Write 43% TRANSACTIONS/SEC INCREASE 2x ENDURANCE INCREASE Processor: Xeon 3.00GHz DRAM: 16GB DDR3 4x4GB DIMMs OS: Fedora 14 Linux kernel Sysbench config: 1 million inserts in 8, 2-million-entry tables, using 16 threads 19

20 Native raw performance comparison Significantly more functionality with NO additional performance impact 1U HP blade server with 16 GB RAM, 8 CPU cores - Intel(R) Xeon(R) CPU 3.00GHz with single 1.2 TB iodrive2 mono 20

21 TRIM TRIM a storage primitive to assist SSD FTLs in garbage collection Classic TRIM trim(lba) Hint to FTL to unmap LBA->PBA Improves wear leveling Improves write performance However only an advisory command 21

22 Persistent Trim and Exists Persistent TRIM (Virtual Address) Has all the positive properties of TRIM Improves wear leveling Improves write performance However well defined with respect to failures Deterministic return of zeros for read Survives power failures (transactional) EXISTS (Virtual Address) Query the existence of a particular element Enables sparse stores with well defined presence semantics 22

23 Conventional Key-Value Store Conventional KV store Application KV Store Key -> block mapping (overhead per key) block allocation Block Read/Write Metadata, persistence mechanisms, logging, recovery logic etc. VSL Dynamic provisioning, Block allocation, persistence mechanisms, Logging, recovery etc. 23

24 VSL based Direct Key-Value Store DirectKey-Value store Application Key Value API and Library Fixed zero metadata, leverages VSL Atomic write/delete, coordinated garbage collection VSL Dynamic provisioning, Block allocation, persistence mechanisms, Logging, recovery etc. 24

25 directkey-value Store API Overview KV Store Administration kv_create() kv_pool_create() kv_pool_delete() kv_open() kv_get_store_info() kv_get_key_info() kv_register_notification_handler() kv_close() kv_destroy() kv_get_pool_info() KV Store Data Operations kv_put() kv_batch_put() kv_get() kv_batch_get() kv_delete() kv_delete_all() kv_batch_delete() kv_begin() kv_next() kv_get_current() kv_exists() 25

26 Directkey-Value Store Sample Performance Sample Sample Performance - GET - GET Sample Performance - PUT - PUT GETs/s Threads 512B 4KB 16KB 64KB PUTs/s Threads 512B 4KB 16KB 64KB Significantly more functionality with NO additional performance impact Performance Performane relative ve to to iodrive iodrive Performance relative to to iodrive OPS/s B Key GET 1KB FIO READ OPS/s B Key PUT 1K-FIO WRITE Threads Threads 1U HP blade server with 16 GB RAM, 8 CPU cores - Intel(R) Xeon(R) CPU 3.00GHz with single 1.2 TB iodrive2 mono 26

27 Advantages of Native Flash Access 1. Helps accelerating applications 2. Eliminates redundant functionality 3. Leverages FTL mapping and sparse addressing 4. Optimizes garbage collection 5. Delivers transactional properties 6. Provides direct I/O as well as memory semantics. 27

28 Open Source Enablement and Standardization MySQL InnoDB extension (GPLv2) directfs native file system service (GPLv2) Standardization of primitives in T10 Current standards proposal Atomic Writes SBC-4 SPC-5 Atomic-Write SBC-4 SPC-5 Scattered writes, optionally atomic SBC-4 SPC-5 Gathered reads, optionally atomic 28

29 Open Source Enablement and Standardization Other T10 Proposal r0 Non-Volatile Memory Based Storage T10 Conference Call Friday, August 10, 10:00-12:00PDT See T10 meeting notice for details 29

30 Extended Memory Non-Persistence Path Fusion-io has developed a subsystem technology called Extended memory, which enables developers to take advantage of NAND Flash memory as an extension of DRAM. The idea is to move frequently accessed data pages to DRAM while rarely accessed data pages are transferred from DRAM to Flash. Thus, the overall available capacity for DRAM can indirectly be increased. Fusion-io said that the technology, which was created in collaboration with Princeton University researchers, allows software developers to simply assume that their entire data set is kept in-memory all the time as NAND is a much more cost-effective memory solution and can reach much greater capacities than DRAM. The Fusion iomemory architecture is uniquely suited to innovation like the Extended Memory subsystem, said Chris Mason, Fusion-io director of kernel engineering and principal author of the Btrfs file system for Linux, in a prepared statement. Since Fusion iomemory has moved beyond legacy disk-era protocols, we can integrate new features like the Extended Memory subsystem to truly advance application performance for enterprise computing in ways that are simply not possible with traditional SSDs. Developers can access the Extended Memory feature via Fusion-io's developer community. 30

31 Auto-Commit Memory: Cutting Latency by Eliminating Block I/O Auto-Commit Memory: Cutting Latency by Eliminating Block I/O Our recent demonstration of one billion IOPS showcased a new paradigm for storing data through Fusion-io Auto-Commit Memory (ACM). ACM isn t just about making NAND Flash storage devices go faster, although it does that too. It s about introducing a much simpler and faster way for an application to guarantee data persistence. When Simplicity Meets Speed For decades, the industry norm for persisting data has been the same an application manipulates data in memory, and when ready to persist the data, packages the data (sometimes called a transaction) for storage. At this point, the application asks the operating system to route the transaction through the kernel block I/O layer. The kernel block layer was built to work with traditional disks. In order to minimize the effect of slow rotational-disk seek times, application I/O is packaged into blocks with sizes matching hard disk sector sizes and sequenced for delivery to the disk drive. As Linux block maintainer (and Fusion-io chief architect) Jens Axboe points out, most real-world I/O patterns are dominated by small, random I/O requests, but are force-fit into 4k block I/Os sequenced by the block layer to match the characteristics of rotating disks. Note the number of steps in this pathway each one contributes to latency. Even more steps are introduced in this pathway when the block storage device is at the other end of a network, behind various bus adaptors, and controllers. As long as memory is volatile, this type of I/O pathway will be the norm. Addressing Both Halves of the Problem Block storage benchmarks such as throughput and IOPS are certainly important, but only address half of the problem. The other half of the problem is the work the application and kernel I/O subsystems must do to package and route data for storage. Applications can be accelerated by addressing either or both halves of this problem. However, note that, at some point, the overhead incurred by packaging and routing data through the kernel block storage subsystem will become the bottleneck. Breaking through that barrier was the goal of this technology demonstration. Give applications the software mechanisms to avoid this block storage packaging and routing latency and complexity. Let them spend more time processing data in memory, and less time packaging and waiting for that data to arrive at a block storage destination. Fusion-io does indeed make very fast block I/O devices. What's most exciting about last week s demonstration for us is looking beyond fast block I/O devices to show what is possible when you address this this fast device as memory rather than block storage. But, what if an application could designate a certain area within its memory space as persistent, and know that data in this area would maintain its state across system reboots? This application would no longer have the burden of following the multi-step block I/O pathway to persist that data. It would no longer need smaller transactions to be packaged into 4k blocks for storage. It would just place selected data meant for persistence in this designated memory area, and then continue using it through normal memory access semantics. If the application or system experienced a failure, the restarted application would find its data persisted in non-volatile memory exactly where it was left. To illustrate, how much faster could real-world databases go if the in-memory tail of their transaction logs had guaranteed persistence without waiting on synchronous block I/O? How much faster could real-world key-value stores (typical in NoSQL databases) go if their indices could be updated in non-volatile memory and not block while waiting on kernel I/O? That is the simplicity of Auto Commit Memory. Itreduces latency by eliminating entire steps in the persistence pathway. 31

32 Thank you! Ashish Batwara Fusion-io 32