New Abstractions for Fast Non-Volatile Storage Joel Coburn, Adrian Caulfield, Laura Grupp, Ameen Akel, Steven Swanson Non-volatile Systems Laboratory Department of Computer Science and Engineering University of California, San Diego 1
How do you re-engineer a system where disk is as fast as DRAM? 2
Storage System Architecture of the Future IO access time nanoseconds! (read hit in L1, L2 caches) CPU Cache (SRAM) CPU ~1-10X latency PCM PCM 1X latency DRAM DRAM ~10000X latency Disk Memory Controller I/O Controller STTM STTM PCM CMOx PCM TAS-MRAM 3
System Stack for Disk User space Operating system Hardware Application Storage access, allocation, naming, organization, protection Disk 4
Yesterday s Interfaces for Tomorrow s Storage Random read of 4KB page Disk SSD Fast NVM System Call File System Hardware 0 1000 2000 3000 4000 5000 6000 7000 Latency (usec) 5
Amdahl s Law strikes again! Disk Random read of 4KB page SSD Fast NVM System Call File System Hardware 0 50 100 150 200 250 Latency (usec) 6
Why do we have file systems? Access data Persistence & consistency (safety) Holds data until it is explicitly deleted Recovers from system crashes Other stuff Needs to be fast Can be slow Structure/organization for data (directories) Allows sharing of data between processes, users, and machines Provides protection from data theft or destruction (security) Basic operations: create, delete, open 7
Criteria for New Abstractions Memory-like interface Rich, non-volatile data structures implemented in the same way that volatile data structures are implemented direct load/store access to memory Persistence and safety Data structures should be robust against application and system failure High performance Application level access latency should as close as possible to latency of the storage technology Leverage caching in the memory hierarchy 8
System Stack for Non-Volatile Memories Userspace Operating system Hardware Application NVTM Allocation & garbage collection Non-volatile memory allocation, mapping, and protection Non-volatile memory 9
The Old Way: Volatile Heap A program accesses a single heap in volatile memory Build rich, pointerbased data structures Serialize/de-serialize to transfer data to/from storage Memory (DRAM) Heap 0 1 1 0 1 Volatile Non-volatile 101100 101000 110111 Disk 10
The New Way: Non-Volatile Heaps We want to have multiple heaps in nonvolatile storage Appear in the application s address space just like memory Rich, pointer-based data structures live directly in storage Challenge: provide safety guarantees Non-Volatile Storage (i.e. PCM) Heap A Heap B Heap C 11
Accessing Non-Volatile Heaps: Transactions Used by databases to make guarantees about storage Now also used to manage concurrent access to volatile memory in multi-core systems Definition: a sequence of operations that is atomic and durable Give us guarantees against Failures Concurrency issues 12
Transactions: How they work 1. Begin transaction 2. Perform operations Record the state of any data that is read or written in a log to be played back later if the transaction fails 3. If no conflicts, then commit (makes permanent) Conflict: data was changed by another transaction during execution 4. If conflicts, then abort and rollback (undo) Replay the logs to restore old values 5. End transaction 13
A Simple Example We have a linked list in non-volatile storage We want to insert a node in sorted order Requirement: If we crash, our list must remain in a good state Either the node is added or it is not No wild pointers, missing links, etc. 14
A Simple Example Insert(Node *node) { Node *curr, *prev; atomic { curr = head; prev = NULL; while (curr->value < node->value && curr->next!= NULL) { prev = curr; curr = curr->next; } prev->next = node; node->next = curr; } } prev head node Heap curr NVHeap 2 4 3 Volatile Non-volatile Write Log Read Log Transaction Commit Begin Read Write 2 3 2 4 15
A Simple Example Insert(Node *node) { Node *curr, *prev; atomic { curr = head; prev = NULL; while (curr->value < node->value && curr->next!= NULL) { prev = curr; curr = curr->next; } prev->next = node; node->next = curr; } } prev head node Heap curr NVHeap 2 4 3 Volatile Non-volatile Write Log Read Log Transaction Abort Write 2 2 4 16
Conclusion New non-volatile memory technologies have the potential to revolutionize computing DRAM-like performance (both sequential and random access) + density + persistence Re-designing these abstractions will unleash the full potential of these technologies! 17