Hardware Undo+Redo Logging. Matheus Ogleari Ethan Miller Jishen Zhao CRSS Retreat 2018 May 16, 2018

Transcription

1 Hardware Undo+Redo Logging Matheus Ogleari Ethan Miller Jishen Zhao CRSS Retreat 2018 May 16, 2018

2 Typical Memory and Storage Hierarchy: Memory Fast access to working data Storage Data persistence Persistent Memory: Fast memory interface + persistence 2

3 Here! Persistent Memory is Coming! Hardware NonVolatile Random Access Memories (NVRAMs) DRAM w/ Ultracapacitor Batterybacked DRAM NV-DIMM 3D XPoint Persistent Memory Aware Database DDR3 Compatible MRAM NVMM file systems are not strongly consi Software Persistent-memory-aware BPFS, PMFS, Ext4-DAX, system SCMFS, software Aerie None of them provide strong metadata and data cons Persistent Memory File Systems Persistent Memory Support in OS File system Metadata atomicity Data atomicity Mmap Atomicity [1] BPFS Yes Yes [2] No PMFS Yes No No Ext4-DAX Yes No No SCMFS No No No Aerie Yes No No NOVA Yes Yes Yes Persistent Memory Support Over Fabric [1] Each msync() commits updates atomically. [2] In BPFS, write times are not updated atomically with respect to the write itself. 3

4 System Throughput Here! Persistent Memory is Coming! but unlocking its full potential isn t easy Persistence Logging, checkpointing, copy-on-write, etc Traditionally property of storage systems Now must maintain in the memory system Native system, no persistence Performance Gap Flash SSD Memory, w/ persistence [Zhao +, MICRO 13] 4

5 Opportunity Hardware Undo + Redo Logging Contribution 1: Relax write order control with undo+redo logging Contribution 2: Leverage cache policies to efficiently enable undo+redo logging in hardware 5

6 Persistence Requirement in Cache-Memory Hierarchy Update persistent memory with a transaction Tx_begin do some reads do some computation Uncacheable_Rlog( addr(c), new_val(c) ) memory_barrier write C Tx_commit Crash Core Core L1 NVRAM LLC L1 Root A Core Core L1 NVRAM LLC L1 Root A Micro-ops: store C 1 store C 2... C 1 B C D B C D Log_C 6

7 Preview of Undo+Redo Logging Benefits Tx begin Undo logging only Undo logging of store A 1 Uncacheable Cacheable Tx commit Logging Write A Ulog_A 1 Ulog_A 2 Ulog_A N store A 1 store A 1 store A N clwb A 1..A N Time Redo logging only Write A consists of N store instructions Redo logging of the transaction Memory_barrier Tx commit Logging Write A Logging Write A Rlog_A 1 Rlog_A 2 Rlog_A N store A 1 store A 1 store A N Undo+redo logging Rlog_A 1 Rlog_A 2 Rlog_A N Ulog_A 1 Ulog_A 2 Ulog_A N store A 1 store A 1 store A N Tx commit Time Time 7

8 Benefits of Undo + Redo Logging Micro-ops Crash A 1 Corrupted! store A 1 store A 2 store A N A A B C Transactions: T A, T B, T C A B C CPU Caches Redo logging RLog_A Version 1 Version 2 NVRAM Memory barrier Undo logging ULog_A No force A copy of the old value, Can undo the changes made by partially completed transactions 8

9 Benefits of Undo + Redo logging Crash store A 1 store A 2 store A N Transactions: T A, T B, T C A B C CPU Caches Redo logging Undo logging No force Steal A B C Version 1 NVRAM RLog_A A 1 ULog_A Uncacheable Version 2 9

10 Undo+Redo Logging Benefits Tx begin Undo logging only Undo logging of store A 1 Uncacheable Cacheable Tx commit Logging Write A Ulog_A 1 Ulog_A 2 Ulog_A N store A 1 store A 1 store A N clwb A 1..A N Time Redo logging only Write A consists of N store instructions Redo logging of the transaction Memory_barrier Tx commit Logging Write A Logging Write A Rlog_A 1 Rlog_A 2 Rlog_A N store A 1 store A 1 store A N Undo+redo logging Rlog_A 1 Rlog_A 2 Rlog_A N Ulog_A 1 Ulog_A 2 Ulog_A N store A 1 store A 1 store A N Tx commit Time Time 10

11 Inefficiency of Software Logging in Persistent Memory Increased memory traffic Extra instructions in the CPU pipeline Conservative cache flushes Risks with multithreading? Transaction T A : Tx_begin do some reads do some computation Uncacheable_log( addr(a), new_val(a), old_val(a) ) write A clwb // can be // delayed? Tx_commit Micro-ops: store log_a 1 store log_a 2... Microops: load A 1 load A 2 store log_a 1 store log_a 2... A 1 Cache flush A A Core cache Processor Core cache Shared Caches Memory Controllers undo redo NVRAM ULog_B RLog_B ULog_ A RLog_ RLog_C A ULog_C 11

12 Better Transaction Throughput Performance Cost of Increased Memory Traffic 100% 80% 60% 40% 20% 0% Ideal Performance Performance Gap 70% Native memory, No persistence in NVRAM Persistent memory logging [Zhao +, MICRO 13] 12

13 What Can We Leverage from Hardware? Mechanism: Hardware Logging (HWL) store A 1 New Value Old Value Write-back Write-allocate Core Processor hit L1 Caches Core miss Last-level Cache (LLC) New Value Old Value Uncacheable_log( addr(a), new_val(a), old_val(a) ) Main Memory 13

14 Undo+Redo Logging: Rides Along with CPU Caching Naturally maintains the order between log and data Volatile Nonvolatile Core L1 Caches Core Last-level Cache (LLC) NVRAM store A 1 TxID, ThreadID Log record Log buffer (or simply WCB) Bypass caches Uncacheable Undo+Redo Log (Circular Buffer) Transaction T A Tx_begin do some reads do some computation write A clwb Tx_commit 14

15 How About Cache Flushes? Mechanism: Force Write Back (FWB) Supports Multithreading Volatile Nonvolatile fwb dirty data Already There The log is a circular buffer When is the best time for cache flushes? Cache flushes frequency depends on log size and log update speed! CPU Caches Transaction T A Tx_begin do some reads do some computation write A clwb Tx_commit 15

16 Commit the Transaction Design principles Hardware Logging (HWL) implements undo+redo in hardware Force Write Back (FWB) decoupled from transaction execution Transaction T A Tx_begin do some reads do some computation write A clwb Tx_commit 16

17 Software and Hardware Cost Software support Transaction interface Tx_begin do some reads do some computation Write A Tx_commit Log_create() Log_truncate() NVRAM Hardware overhead Major Components Logic Type Size Transaction ID register Flip-flop 1 byte per HW thread Log head and tail registers Flip-flop 16 bytes Fwb cache tag bit SRAM 1 bit per cache line 17

18 Better Key Performance Results Ideal performance No-pers SW-redo-clwb SW-undo-clwb Our-design : undo+redo Transaction Throughput 100% 89% 80% 56% 60% 40% 20% 0% Avg of five WHISPER micro-benchmarks Better NVRAM Write Traffic Avg of five micro-benchmarks WHISPER Processor configuration: Core i7, 22nm, 4-core, 2.5GHz, 2 threads/core Other results: energy consumption, instruction increase, IPC, sensitivity studies, etc. 18

19 Summary Contribution 1: Relax write order control with undo+redo logging Contribution 2: Leverage cache policies to efficiently enable undo+redo logging in hardware Key points Rethink the way traditional software logging is done Exploit opportunities in existing hardware can naturally support data persistence 19

20 Ongoing Work and Collaboration Western Digital Internship 20

21 Hardware Undo+Redo Logging Matheus Ogleari Ethan Miller Jishen Zhao CRSS Retreat 2018 May 16, 2018