Request-Oriented Durable Write Caching for Application Performance appeared in USENIX ATC '15 Jinkyu Jeong Sungkyunkwan University
Introduction Volatile DRAM cache is ineffective for write Writes are dominant I/Os [FAST 09, FAST 10, FAST 14] Non-volatile write cache (NVWC) provides Fast response for write w/o loss of durability NVWC candidates: Flash 3D Xpoint PCM MRAM GB/$ [Bhadkamkar et al., FAST 09] BORG: Block-reORGanization for self-optimizing storage systems [Koller et al., FAST 10] I/O deduplication: Utilizing content similarity to improve I/O performance [Harter et al., FAST 14] Analysis of HDFS under HBase: a Facebook messages case study NV-DRAM Performance 2/35
Non-volatile Write Cache Usage Simple caching policy Application P1 P2 P3 Write Write Write Operating System NVWC Blindly caching all writes Backing Storage Lazily writing back to storage 3/35
Non-volatile Write Cache Usage Simple caching policy P1 P2 P3 No consideration for Write Write Write application performance Operating System Application NVWC Blindly caching all writes Backing Storage Lazily writing back to storage 4/35
Impact on Application Performance Illustrative experiment P1 PostgreSQL RDBMS P2 P3 TPC-C workload Write Write Write Operating System NVWC Backing Storage 32MB NV-DRAM or 4GB Flash SSD 2 HDDs (Data/Log) 5/35
Impact on Application Performance Experimental result * System perf. Marginal gain 2.1X 1.7X - ~ 2.1X improved * Application perf. - ~ 50% degraded Performance drop by 50%! 6/35
What s the Problem? Criticality-agnostic contention 7/35
Criticality-Agnostic Contention Different write criticality Client Request Response Application Application performance P1 P2 P3 Operating System NVWC Backing Storage 8/35
Criticality-Agnostic Contention Different write criticality Client Request Response Application P1 P2 P3 Background process/thread Critical Non-critical Operating System NVWC Backing Storage 9/35
Criticality-Agnostic Contention Different write criticality Client Request Response Application P1 P2 P3 Critical Non-critical * Contentions - Capacity contention - Bandwidth contention Operating System NVWC Backing Storage 10/35
Criticality-Agnostic Contention Capacity contention Client Request Response Application P1 P2 P3 Critical Non-critical Operating System Frequent write stalls Backing Storage Bounded writeback throughput 11/35
Criticality-Agnostic Contention Bandwidth contention Client Request Response Application P1 P2 P3 Critical Non-critical C NCNCNCNCNCWBWBWBWB Head Excessive queueing delay Sufficient free blocks Backing Storage 12/35
Our Approach Request-oriented caching policy Client Application * Definitions CP P1 NCP P2 NCP P3 Critical Non-critical Operating System - Critical process (CP): a process handling request - Critical write: a write awaited by a critical proc. NVWC Backing Storage Caching critical writes only Sync I/O Critical I/O Async I/O 13/35
Challenge How to accurately detect critical writes Types of critical write Sync. writes from critical processes Dependency-induced critical writes Process dependency-induced I/O dependency-induced 14/35
Dependency Problem Process dependency NCP Process Dep. Lock B1 Wake B3 B4 B5 CP Wait for B1 15/35
Dependency Problem I/O dependency NCP CP I/O Dep. B1 B2 Sync Wait for B2 Complete B3 B4 B5 * Example scenarios: - CP fsync() to a block under writeback issued by NCP - CP tries to overwrite fs journal buffer under writeback 16/35
Critical Write Detection Critical process identification Application-guided identification 17/35
Critical Process Identification Application-guided identification Client 1 Client 2 Application CP CP NCP NCPNCP API Operating System NVWC Backing Storage 18/35
Critical Write Detection Critical process identification Application-guided identification Dependency resolution Criticality inheritance protocols Process criticality inheritance I/O criticality inheritance Blocking object tracking 19/35
Criticality Inheritance Protocols Process criticality inheritance NCP Lock B1 Inherit Wake B2 B3 B4 CP 20/35
Criticality Inheritance Protocols I/O criticality inheritance NCP B1 B2 Discard B3 B4 B5 CP Sync Reissue B2 Complete Key issue: caching the dependent write outstanding to disk w/o side effects 21/35
Criticality Inheritance Protocols Blocking object tracking Handling cascading dependencies NCP B1 Lock Inherit Wake B2 B3 B4 CP Reissue B1 Wake 22/35
Evaluation Implementation on Linux 3.13 w/ FlashCache 3.1 Application studies PostgreSQL database Client 1 Client 2 Backend1 Backend2 Check pointer Log writer Writer Redis key-value store Client 1,2,3, Master Snap shotter Log rewriter 23/35
Evaluation Experimental setup PostgreSQL / Redis FlashCache 4GB ramdisk / 256GB SSD 10K RPM HDD x2 1Gbps TPC-C / YCSB Server Machine Client Machine 24/35
Evaluation Experimental setup PostgreSQL / Redis * Caching policies - ALL (default) FlashCache No discretion 4GB ramdisk / 256GB SSD 10K RPM HDD x2 1Gbps TPC-C / YCSB Server Machine Client Machine 25/35
Evaluation Experimental setup PostgreSQL / Redis * Caching policies - ALL (default) - SYNC FlashCache Sync. writes 4GB ramdisk / 256GB SSD 10K RPM HDD x2 Async. writes 1Gbps TPC-C / YCSB Server Machine Client Machine 26/35
Evaluation Experimental setup PostgreSQL / Redis * Caching policies - ALL (default) - SYNC - CP FlashCache CP sync. writes 4GB ramdisk / 256GB SSD Rest of writes 10K RPM HDD x2 1Gbps TPC-C / YCSB Server Machine Client Machine 27/35
Evaluation Experimental setup PostgreSQL / Redis FlashCache * Caching policies - ALL (default) - SYNC - CP - CP+PI + Process criticality inheritance 4GB ramdisk / 256GB SSD 10K RPM HDD x2 Rest of writes 1Gbps TPC-C / YCSB Server Machine Client Machine 28/35
Evaluation Experimental setup PostgreSQL / Redis FlashCache * Caching policies - ALL (default) - SYNC - CP - CP+PI - CP+PI+IOI + I/O criticality inheritance 4GB ramdisk / 256GB SSD 10K RPM HDD x2 Rest of writes 1Gbps TPC-C / YCSB Server Machine Client Machine 29/35
Evaluation Experimental setup Trx log writes PostgreSQL / Redis FlashCache 4GB ramdisk / 256GB SSD 10K RPM HDD x2 Rest of writes 1Gbps * Caching policies - ALL (default) - SYNC - CP - CP+PI - CP+PI+IOI - WAL (PostgreSQL) TPC-C / YCSB Server Machine Client Machine 30/35
PostgreSQL Performance TPC-C workload w/ ramdisk 80% Same performance w/ 72% less cached writes Our scheme resolves capacity contention & runtime dependencies Scarce Sufficient 31/35
PostgreSQL Performance TPC-C workload w/ SSD 2.2X Our scheme resolves bandwidth contention & runtime dependencies Sufficient Sufficient 32/35
Redis Performance Update-heavy workload w/ 16GB SSD 47% better throughput Improved tail latency 13X better @ 99.9 th %ile (50ms vs. 649ms) Our scheme improves request throughput & request latency 33/35
Conclusion Key observation Each write has different performance-criticality Request-oriented caching policy Solely utilizes NVWC for application performance Improves performance while reducing cached writes Future work Criticality-aware I/O management without NVWC Application to user-interactive environments 34/35
Q&A Thank you 35/35