Redrawing the Boundary Between So3ware and Storage for Fast Non- Vola;le Memories

Transcription

1 Redrawing the Boundary Between So3ware and Storage for Fast Non- Vola;le Memories Steven Swanson Director, Non- Vola;le System Laboratory Computer Science and Engineering University of California, San Diego 1

2 Humanity 9,000,000,000,000,000,000,000 processed 9 Zettabytes in Bytes 2008* Welcome to the Data Age! *hjp://hmi.ucsd.edu 2

3 Solid State Memories NAND flash Ubiquitous, cheap Sort of slow, idiosyncra;c Phase change, Spin torque MRAMs, etc. Not (yet) ready for prime ;me DRAM- like speed DRAM or flash- like density DRAM + BaJery + flash Fast, available today Reliability? 3

4 Integrating Them Into Systems PCIe- ajached NVMs Low Latency High bandwidth Flexible interface Flexible topology Scalable DDR- ajached NVMs Lowest Latency Highest bandwidth Rigid interface Restricted topology Limited scalability NVM NVM NVM PCIe CPU DDR DRAM NVM 4

5 Bandwidth Rela-ve to disk x à 2.4x/yr PCIe- Flash (2012) DDR Fast NVM (2016?) PCIe- PCM (2010) PCIe- PCM (2013?) Hard Drives (2006) PCIe- Flash (2007) 7200x à 2.4x/yr /Latency Rela-ve To Disk 5

6 Software Latency Will Dominate So:ware's Contribu-on to Latency 100% 80% 60% 40% 20% 0% 0.3% 19.3% 21.9% 10.4% 21.9% 70.0% 88.4% 94.09% 6

7 Software Energy Will Dominate So:ware's Contribu-on to Energy/IOP 100% 80% 60% 40% 20% 0% 0.4% Hard Drive (2006) 97.0% SSD (2012) 75.8% 78.6% PCIe- Flash (2012) PCIe- PCM (2013) 7

8 Software s New Role in Storage For Disks: Adding So3ware Could Make the System Faster For NVMs: Adding So3ware Makes It Slower Reduce so3ware interac;ons Remove disk- centric features Redesign IO so3ware (OS, FS, languages, and apps) 8

9 Two Case Studies Moneta: SSDs for Fast NVMs NV- Heaps: NVMS over DDR Database File System IO Stack Moneta Driver Programming Model File System IO Stack Block Driver NVM NVM NVM PCIe CPU DDR DRAM NVM 9

10 Moneta Internals: Performing a Write Host via PIO Request Queue Scoreboard Tag Status Registers Ring Control Transfer Buffers DMA Control Ring (4 GB/s) 16GB NVM 16GB NVM 16GB NVM 16GB NVM Host via DMA 10

11 The Moneta Prototype FPGA- based implementa;on DDR2 DRAM emulates PCM Configurable memory latency 48 ns reads, 150 ns writes 64GB across 8 controllers PCIe: 2 GB/s, full duplex 11

12 Optimizing Software for Moneta Op;miza;ons Remove IO scheduler Redesigned HW interface Remove locks SW latency drops from 13.4 us to 5us 62% reduc;on in latency Increased concurrency Bandwidth increases by up to 10x Latency (us) Moneta 0.1 Moneta 1.0 PCM Ring DMA Wait Interrupt Issue Copy Schedule OS/User 12

13 The App Gap Log Speedup vs DISK- RAID Moneta FusionIO SSD- RAID DISK- RAID XDD 4KB RW Btree HashTable Bio PTF We must close the App Gap for NVM success 13

14 Where is the App Gap (Part I)? 4KB Accesses Databases Sci Apps Web Apps File Servers File system IO Stack Driver Sustained MB/s Moneta fs Moneta Moneta

15 Eliminating FS and OS overheads Latency (us) Moneta 0.1 Moneta 1.0 PCM Ring DMA Wait Interrupt Issue Copy Schedule OS/User File System Separate 41% Protec;on is to protec;on support overheads protec;on mechanisms are almost and eliminated sharing from policy Move protec;on checks to hardware App FS IO Stack Allow applica;ons to access Moneta directly App Driver File system IO Stack Driver Moneta-D 1.0 App App Driver Perm Check 15

16 Some Progress on the App Gap 6 Speedup vs. Kernel Space Kernel Space User Space 0 16

17 Where is the App Gap (Part II)? FS IO Stack Logging/ Atomicity App Driver Moneta App Driver Perm Check Write Mul;- part ahead Atomic logging Writes e.g. ARIES Apps can create and Wasted commit so3ware write groups Write latency/power data resides in a Wasted redo log IO performance Extra IO opera;ons The applica;on can read and change the log contents 17

18 Bandwidth Comparison Sustained Bandwidth (MB/s) Access Size (KB) 2 to 3.8X improvement Normal Writes Normal Writes AtomicWrite WAL WAL Writes Writes 18

19 ARIES Write Ahead Logging for Databases on Disks ARIES provides useful database features Atomicity Durability High Concurrency Efficient recovery But it makes disk- centric design decisions Undo and redo logs to avoid random synchronous writes and provide IO scheduling flexibility Page- based storage management to match disk seman;cs We want to keep these Rethink these 19

20 ARIES Disk-Centric Design Decisions Design Decision Advantages Downsides No- force Eliminates synchronous random writes. Requires synchronous writes to the redo log. Steal Longer sequen;al writes. Requires an undo log. Pages Matches disk and OS page- based IO model. Page granularity is not always ideal. 20

21 MARS: Rethinking ARIES for Moneta No- force Steal Force updates on commit, but offload it to Moneta s memory controllers. Get rid of undo logging. Update redo log contents instead. Pages So3ware can choose object/page granularity. 21

22 ARIES vs MARS 4KB Swaps TX/sec Threads MARS ARIES 22

23 Software s Role in Moneta The basic Moneta hardware is (rela;vely) simple So3ware requirements drove the HW enhancements Reduce, Remove, Redesign Simplify the kernel IO stack Eliminate the OS and FS overheads for most accesses Rethink write- ahead logging for new memories 23

24 Two Case Studies Moneta: SSDs for Fast NVMs NV- Heaps: NVMS over DDR Database File System IO Stack Moneta Driver Programming Model File System IO Stack Block Driver NVM NVM NVM PCIe CPU DDR DRAM NVM 24

25 Disks and Files Are Passé DDR NVM is is a a new kind of Memory of disk - - Use VM conven-onal to access data file opera-ons - Create to classes access data - Use - Files pointers/references are bags of bytes - Leverage strong types - But make it persistent Applica-ons System System Calls Calls File File System System Block Block Driver Driver DDR NVM 25

26 The Dangers of Non-Volatile Programming Dlist Insert(Foo x, Dlist head) { Dlist r; r = new Dlist; r.data = x; r.next = head; r.prev = null; if (head) { head.prev = r; } return r; } What if the system crashes? What if x and head are in different files? 26

27 NV-Heaps: Safe, Fast, Persistent Objects Container for NV data structures Generic Self- contained Access via loads and stores Memory- like performance Familiar seman;cs Strong safety guarantees ACID transac;ons Pointer safety NV- Heap Transac-on Handling [ASPLOS 2011] 27

28 A Type System for NV-Heaps 1. Refine each type with a heap- index 2. Type rules ensure that the heaps match Dlist<h> Insert(Foo<h> x, Dlist<h> head) { atomic<h> { Dlist<h> r; r = new Dlist in heapof(r); r.data = x; r.next = head; r.prev = null in heapof(r); if (head) { head.prev = r; } } return r; } 28

29 Application: A Persistent Key-value Store X 8% slowdown Opera-ons/sec Berkelely DB - RAM Disk NV- heaps DRAM Memcached (vola;le) NV- heaps are much faster than the block- based systems NV- heaps make the cost of persistence very low 29

30 NV-Heaps Wrap Up The hardware is here already. NV- heaps can change how programmers think about persistent state. Careful system design can avoid the pisalls And the performance gains are huge 30

31 Other Efforts PCIe SSDs FusionIO et. al. Intel NVMExpress prototypes [FAST 12] Texas Memory Systems SSDs Atomic Writes Transac;onal Flash [OSDI 08] FusionIO AtomicWrite [HPCA 11] NV- Heaps Slow but safe Stasis [OSDI 06] Orthogonally persistent Java [SIGMOD 96] Many others Fast but unsafe Recoverable Virtual Memory [SOSP 93] Rio Vista [SOSP 97] Fast and safe Mnemosyne [ASPLOS 11] 31

32 Open Challenges 32

33 How Can We Elegantly Enrich Storage Semantics? Atomic Writes help but, Each applica;ons has its own needs Legacy system architectures Novel data structures Can we build flexible seman;cs for storage? Like shaders in graphics cards? 33

34 How do NVMs affect the data center? How should we organize a petabyte of NVMs? Centralized? Distributed? Hybrid? NVM performance with large- scale system overheads Replica;on Network latency Thick so3ware layers (e.g. OpenStack, HadoopFS) 34

35 Software Costs Will Remain High So:ware's Contribu-on to Latency PCIe and DDR performance will con;nue to improve. 100% 80% PCIe and DDR power efficiency will con;nue to improve. 60% 42.2% 40% Thick so3ware 19.3% stacks 21.9% will push so3ware 21.9% costs higher. 20% 10.4% 11.1% 11.8% 0.3% 0% Applica;ons will demand richer, more sophis;cated interfaces. 35

36 The App Gap Remains Log Speedup vs DISK- RAID Moneta FusionIO SSD- RAID DISK- RAID XDD 4KB RW Btree HashTable Bio PTF 36

37 Solid State Storage In the Data Age Harnessing the data we collect requires vastly faster storage systems Fast, non- vola;le memories can provide it, but NVMs will force so3ware s role to change So3ware will become a drag on IO performance Reduce, Remove, Redesign Leveraging NVMs quickly requires a coordinated research that spans system layers No aspect of the system is safe! 37

38 Thanks! 38

39 39

40 Ques;ons? 40