SHRD: Improving Spatial Locality in Flash Storage Accesses by Sequentializing in Host and Randomizing in Device

Transcription

1 SHRD: Improving Spatial Locality in Flash Storage Accesses by Sequentializing in Host and Randomizing in Device Hyukjoong Kim 1, Dongkun Shin 1, Yun Ho Jeong 2 and Kyung Ho Kim 2 1 Samsung Electronics 2 Presented at FAST 17 1

2 Bandwidth (MB/s) Random write is still slow at SSD random write (4KB) sequential write (512KB) emmc 5.0 (Odroid-XU3) UFS (Galaxy S6) SATA SSD (Intel 525) SATA SSD (Samsung 850 Pro) NVMe SSD (Intel 750) 2

3 Why is RW slower than SW? 1. Request Handling Overhead 2. Garbage Collection Overhead 3. Mapping Table Handling Overhead 3

4 Why is RW slower than SW? 1. Request Handling Overhead Sequential write Large, few requests Random write Small, many requests Packed command e.g. emmc Interrupt coalescing e.g. NVMe, SATA NCQ Vectored I/O e.g. OpenChannel SSD [FAST 17] 4

5 Why is RW slower than SW? 2. Garbage Collection Overhead Blocks with SW Invalid I I I Valid V V V Blocks with RW RW generates hot/cold-mixed blocks Dispersed invalid pages high GC overhead I V I V V I V I Hot/cold separation Stores hot and cold data into different blocks Incremental GC / bggc can hide GC latency 5

6 Why is RW slower than SW? 3. Mapping Table Handling Overhead Page-level mapping FTL shows good performance on RW Requires a large DRAM to maintain fine-grained mapping table 4 byte per 4 KB 8 GB DRAM for 8 TB storage Demand loading FTL (DFTL [ASPLOS 08]) Uses a small map cache with on-demand map loading Random writes invoke frequent map loading/unloading 6

7 Demand-loading FTL (DFTL) Write LPN 768 Map caching scheme can show good performance by utilizing temporal & spatial locality Page level map load/unload One map page contains multiple contiguous mapping entries Vulnerable to random workload low temporal & spatial locality high map miss rate high map loading overhead LRU list loads map pages map blocks map page NAND Flash Chips DRAM Map Cache map page unloads data blocks map page data OOB 7

8 Previous Solution: LFS Generate only sequential writes out-of-place append-only write scheme Invalid Invalid Append logging storage space Problems reclaiming log space (cleaning overhead) Filesystem needs to copy valid page host-to-device data transfer Large metadata, wandering tree problem Fragmented read operation FS Storage read copy write LPN: Cleaning requires host-to-device data transfer operations 8

9 Can we remove copy overhead? SSD maintains a page-level mapping table Address remapping Can change the logical address of a written data by modifying mapping table AnViL [FAST 15], SHARE [SIGMOD 16] Can reclaim log space with address remapping FS Storage read copy write LPN: < Copying at filesystem > remap request FS Storage LPN: 32 remap 128 < Remapping at storage > Logical Physical Mapping Table 9

10 Which layer? File System or Block Layer Our solution is Append logging on Block Layer Append logging on log area temporarily Remap to the original location Can utilize legacy filesystems (e.g. EXT4) Simpler metadata management Faster sequential read performance Legacy Applications Legacy Filesystems Append-logging D/D Storage 10

11 SHRD (Sequentializing in Host, Randomizing in Device) Sequentializing in Host Host OS writes random requests sequentially at log area Randomizing in Device SSD modifies the mapping table to change the logical address Normal area Log area (reserved) (1) Sequentializing (2) Randomizing 11

12 SHRD Example: write multiple small random writes Logical address 0 Logging 1024 Normal area (FS visible) single large sequential write 1024 Log area (FS invisible) olpn: original LPN tlpn: temporal LPN Host redirection table olpn tlpn physical address Device mapping table LPN PPN NAND flash

13 SHRD Example: read redirection olpn: original LPN tlpn: temporal LPN Logical address 0 Read 32 redirect to Host redirection table olpn tlpn physical address Device mapping table LPN PPN NAND flash

14 SHRD Example: remap olpn: original LPN tlpn: temporal LPN Logical address 0 remap Host redirection table olpn tlpn physical address Device mapping table LPN PPN NAND flash

15 Can we really reduce map loading overhead? Remap modifies the mapping entries of sequentialized pages A time-ordered access scheme olpn tlpn olpn inherits the original random pattern low spatial locality olpn-ordered map access The mapping table is olpn-indexed Can increase spatial locality time-ordered access 8 map loads remapping sequence tlpn olpn-ordered access 5 map loads 15

16 Map miss ratio utilization of parallel unit The effect of request reordering 15% 10% 5% stg_0 proj_0 stg_0 proj_0 improve parallelism reduce map miss 100% 80% 60% 40% 20% 0% NONE 2MB 4MB 8MB 16MB 32MB 64MB Reordering window size 0% 16

17 SHRD (Sequentializing in Host, Randomizing in Device) File system, Applications Device Driver map reclaim read() map insert write() Randomizer Redirection Table Sequentializer SSD (SHRD-FTL) Map Cache remap() read( ) twrite() Map Blocks Data Blocks RWLB Blocks 17

18 SHRD (Sequentializing in Host, Randomizing in Device) File system, Applications Device Driver Randomizer Redirection Table Sequentializer Sequentializer Gathers random write requests, sequentially logs into temporal location Redirection table Maintains redirection table between temporal address and original address Randomizer Sends remap command to storage device and reclaims temporal location 18

19 SHRD (Sequentializing in Host, Randomizing in Device) SHRD-FTL Receives twrite and remap command from host OS twrite Write command with two addresses, temporal/original address The data must be stored into separate physical blocks called RWLB remap Restores the data written at temporal location into original address Changes mapping table from temporal address to original address Corresponding RWLB blocks will be transferred into data blocks remap (tlpn, olpn) SSD (SHRD-FTL) Map Cache twrite (olpn, tlpn) Map Blocks Data Blocks RWLB Blocks 19

20 Special commands: twrite & remap twrite (olpn[n], tlpn_start, n, data) Write command sends two addresses, (tlpn, olpn) olpn is stored at the OOB area of physical page used for power-off-recovery / GC Packed command with multiple RW requests remap (olpn[m], tlpn[m], m) m = # of remapping entries per remap command olpn-sorted entries Improving spatial locality Changes mapping table from tlpn to olpn tlpn : PPN olpn : PPN 20

21 Command Sequence: Sequentializing in Host a b c d I/O scheduler twrite header olpn tlpn twrite data mapping a b c d SHRD driver SATA write command (OOB) Completion SATA write command (OOB) Completion SATA Interface seq_ptr rand_ptr 1024 a SSD Device Normal area b c d Log area 1024 data spare NAND a 32 21

22 Command Sequence: Randomizing in Device I/O scheduler remap olpn tlpn SHRD driver SATA write command (OOB) Change mapping table SATA Interface SSD Device rand_ptr seq_ptr Normal area 1024 a b c d valid log area Log area 22

23 GC & Power Off Recovery (POR) Reverse map in out-of-band (OOB) area SSD stores corresponding LPN in OOB area Reverse map is used for GC & recovery GC: change the mapping table of victim valid page Recovery: recover the mapping table of active blocks Physical page layout Data OOB ECC LPN 23

24 GC & Power Off Recovery (POR) Store olpn at the OOB area of RWLB RWLB blocks must be excluded from choosing victim until entire data stored in the blocks are remapped Non-remapped data will be auto-remapped at POR by scanning the OOB area of RWLB blocks twrite Data RWLB Block olpn after remap moved into data block can be victim of GC OOB before remap POR will scan OOB do auto-remap 24

25 Implementation SHRD D/D is implemented in Linux kernel Additional kernel module at SCSI D/D layer Host redirection table: about 1 MB for 64 MB log area Prototype SSD device Modified the firmware of a commercial SATA3 SSD device (Samsung 843) DFTL & SHRD-FTL are implemented Map cache size is configurable 25

26 bandwidth (MB/s) map load per page IO RW Performance According to cache DFTL SHRD 30x faster than DFTL at tiny size of cache 128KB 256KB 512KB 1MB 2MB 4MB 8MB fully Map cache size loaded Better performance than DFTL By reducing map loading/unloading overhead SHRD shows steady performance regardless of cache size fio random write test (32GB space, 4KB write) DFTL SHRD 128KB 256KB 512KB 1MB 2MB 4MB 8MB fully Map cache size loaded 26

27 throughput (normalized to DFTL) Performance on Real Benchmarks small random write dominant workloads sequential write workload CMT caches about 5% of entire workload space read/flush dominant workload tpcc YCSB postmark fileserver varmail Better performance at all workloads Small gains at sequential or read dominant workload still better than DFTL 27

28 bandwidth (MB/s) SHRD gains at EXT4 vs. F2FS low utilization high utilization F2FS (SSR) w/ DFTL EXT4 w/ DFTL time(s) EXT4 shows bad performance on random write Performance of F2FS decreases due to SSR at high utilization 28

29 bandwidth (MB/s) SHRD gains at EXT4 vs. F2FS low utilization ext4 improvement high utilization F2FS (SSR) w/ DFTL F2FS (SSR) w/ SHRD EXT4 w/ DFTL EXT4 w/ SHRD F2FS improvement time(s) SHRD improves both EXT4 and F2FS SHRD improves the bandwidth of aged F2FS EXT4 shows similar performance as F2FS by using SHRD 29

30 bandwidth (MB/s) bandwidth (MB/s) SHRD gains at EXT4 vs. F2FS F2FS (SSR) w/ DFTL F2FS (SSR) w/ SHRD EXT4 w/ DFTL EXT4 w/ SHRD EXT4 w/ SHRD F2FS (SSR) w/ SHRD time(s) Sequential read performance of EXT4 is much better The out-of-place scheme of F2FS scatters the data blocks of a file 0 Sequential read Random read 30

31 bandwidth (MB/s) Latency Comparison Remap command can delay read operations Several remapping entries are batched into a single command # of remapping entries per command can control the maximum latency of following I/O operations read write fio mixed workload remap period (32GB area, 4KB random read/write mixed) DFTL SHRD 31

32 Visualizing address accessing pattern: postmark seqeuntialized write remap command access < without SHRD > < with SHRD >

33 Conclusion SHRD is an address reshaping technique transforms RW into SW at the block D/D restores the original addresses without copy operations Solves POR / GC issues of address remapping SHRD improves 30x better performance at a small map cache reduces DRAM drastically 33

34 Thank you. 34