Unblinding the OS to Optimize User-Perceived Flash SSD Latency

Transcription

1 Unblinding the OS to Optimize User-Perceived Flash SSD Latency Woong Shin *, Jaehyun Park **, Heon Y. Yeom * * Seoul National University ** Arizona State University USENIX HotStorage 2016 Jun. 21, 2016

2 OS I/O Path Optimizations: Reducing S/W Overheads Application Simplified I/O Path Storage Device OS I/O Path Issue Side Completion Side Overhead from 13 us down to 0.2 ~ 2.5 us Under 20us Technology Memory technology with ultra low (nanoseconds), predictable latencies. Block CPU (poll) while waiting for the I/O to complete

3 To Block the CPU (sync) or to Yield the CPU (async) Issue Side Completion Side OS I/O Path CPU Deferred Processing Hard IRQ (i.e., read: 20us ~ 150us) Higher latency & High Variability Cannot use polling: Wastes CPU cycles Harms system parallelism H/W context switch := 4 us ~ 5 us OS involved switch := 7 us ~ 8 us Impact of Modern SSDs More contexts on a CPU core Larger scheduling delays

4 Impact of Modern SSDs 700,000 IOPS NVM-e SSD Bandwidth: 4kB x 700 kiops = 2.8 GB/s Latency: 1 sec / 700 kiops = 1.42 µs?!?

5 Impact of Modern SSDs 700,000 IOPS NVM-e SSD Single NAND die: apprx. 14,285 8kB IOPS (i.e, 70us read latency) Requires more than 49 NAND dies to achieve 700,000 IOPS Large DRAM Powerful Controllers Multi-channel Multi-way High die count NAND array

6 Impact of Modern SSDs 700,000 IOPS NVM-e SSD Single NAND die: apprx. 14,285 8kB IOPS (i.e, 70us read latency) Requires more than 49 NAND dies to achieve 700,000 IOPS High count of I/O contexts (threads, state machines) Large Powerful required DRAM for Controllers high IOPS Multi-channel Multi-way High die count NAND array

7 Higher Context Multiplexing Cost Scheduling Delays Context To Context CPU Core Scheduler Time sharing a CPU core (Sync. I/O) CPU CPU NAND die count >>> CPU core count i.e., Four PCI-e 3.0 4lane slots in a chassis, more SSDs Redundancy, more capacity... Context To Context CPU Core Worker thread With multiple I/O contexts (Async. I/O) Large DRAM Powerful Controllers Multi-channel Multi-way High die count NAND array

8 The OS is Blind: Conservative Strategies Issue Side Completion Side OS I/O Path DRAM SSD Controller

9 The OS is Blind: Conservative Strategies Issue Side Completion Side OS I/O Path Deferred Processing Hard IRQ 20us ~ 10ms Higher latency, Higher Variance Externally Experienced Latency R W R R W R R R R E W E R SSD Controller DRAM NAND dies Physical Destination within the SSD

10 This Work: Unblinding the OS Issue Side Completion Side OS I/O Path Deferred Processing Hard IRQ SSD Latency H/W context switch := 4 us ~ 5 us OS involved switch := 7 us ~ 8 us More contexts on a CPU core (higher)

11 This Work: Unblinding the OS Issue Side Completion Side OS I/O Path Deferred Processing Hard IRQ SSD Latency OS SSD Latency I/O destination

12 This Work: Unblinding the OS Issue Side Completion Side OS I/O Path Deferred Processing Hard IRQ Latency Exposed Knowledge A simplified model A B C Unpredictable I/O destination x I/O type DRAM NAND NAND SmallR/W Small READ Else SSD Latency OS Better Interaction Extended Interface SSD Latency Internal Knowledge A B C Unpredictable DRAM NAND SmallR/W Small READ I/O destination x I/O type NAND Else

13 Latency OS I/O Path Issue Side This Work: Unblinding the OS SSD Latency Completion Side Exploiting SSD Internal Information Exposed Knowledge A simplified model SmallR/W Small READ Towards a SSD A B C Unpredictable I/O destination x I/O type Else OS Better Interaction Extended Interface SSD Deferred Processing Hard IRQ New Optimization Opportunities Latency Internal Knowledge A B C Unpredictable DRAM NAND SmallR/W Small READ I/O destination x I/O type NAND Else

14 Exploiting SSD Internal Information OS SSD The Unpredictable DRAM SSD SSD Controller Multi-channel Multi-way High die count NAND array

15 Exploiting SSD Internal Information Decomposition & Classification I/O request Classification OS Destination Decomposition DRAM NAND Small R Small W Large R Large W Cache hit Buffer hit N/A N/A Cache miss Buffer miss Interleaved Interleaved reads writes Unpredictable No benefit No benefit SSD DRAM SSD Controller Multi-channel Multi-way High die count NAND array

16 Exploiting SSD Internal Information Decomposition & Classification I/O request Classification OS Small R Small W SSD Destination Decomposition DRAM NAND Cache hit Cache miss Buffer hit Buffer miss Unpredictable DRAM SSD Controller Multi-channel Multi-way High die count NAND array

17 Exploiting SSD Internal Information Decomposition & Classification I/O request Classification OS Small R Small W SSD Destination Decomposition DRAM NAND Cache hit Cache miss Buffer hit Buffer miss Unpredictable DRAM SSD Controller Multi-channel Multi-way High die count NAND array

18 Mitigating the Impact of Scheduling Delays Issue Side Completion Side OS I/O Path Deferred Processing Hard IRQ NAND Read I/O request Classification OS Small R Small W SSD Destination Decomposition DRAM NAND Cache hit Cache miss Buffer hit Buffer miss Unpredictable

19 Mitigating the Impact of Scheduling Delays Issue Side Completion Side Issue Side Completion Side OS I/O Path Deferred Processing Hard IRQ NAND Read Buffer Hits I/O request Classification OS Small R Small W SSD Destination Decomposition DRAM NAND Cache hit Cache miss Buffer hit Buffer miss Unpredictable

20 Accurate Latency Prediction: Remaining I/O Time Issue Side OS I/O Path Classification Small Read Latency Predictor I/O processing & Queuing delays Flash I/O + ECC + DMA transfer Only for NAND reads Remaining I/O time Total SSD I/O time

21 Precompletions: Overlapping I/O & Scheduling Delay Issue Side Completion Side Busy wait for flag (waiting on L1) OS I/O Path Classification Small Read Deferred Processing Hard IRQ Latency Predictor Precompletion Wait period Precompletion IRQ Precompletion window Actual Completion (Flag update: No IRQ) I/O processing & Queuing delays Flash I/O + ECC + DMA transfer Only for NAND reads Remaining I/O time Total SSD I/O time

22 OS & SSD Interaction: Simple Behavioral Models I/O request Classification OS Small R Small W SSD Destination Decomposition DRAM NAND Cache hit Cache miss Buffer hit Buffer miss Unpredictable

23 In-band Communication Channel Piggybacked Information Previous I/O completion! I/O request OS I/O Path I/O Classifier Current Blocks Blocks to consume Buffer Monitor Buffer Monitor

24 Implementation & Evaluation Host system Application FIO DDR3 DRAM 512MB x 2 OS: Linux Custom Block Driver NVM-e like I/F FTL NAND Controller 128GB NAND Module x 1 (4ch, 4way each) External PCI-e Gen 2 four lane connection (Operating in Gen2 one lane) MSI IRQ PCI-e external adaptor Zync-7000 FPGA + ARM Cortex A9 Dual core

25 Implementation & Evaluation Limitation Single I/O Depth Host system Application Unoptimized FPGA FIONAND Controller (Higher Latencies) OS: Linux Fixed latency Custom Slow DMA transfers Block (low freq. bus) PCI-e Gen2 one lane Driver External PCI-e Gen 2 four lane connection (Operating in Gen2 one lane) PCI-e external adaptor DDR3 DRAM 512MB x 2 128GB NAND NVM-e like Module x 1 I/F Latency FTL Prediction (4ch, 4way each) NAND Controller MSI IRQ Impact of precompletion Zync-7000 FPGA + ARM Cortex A9 Dual core

26 Predicting SSD Latency (Small NAND Read) Flash: NAND I/O + ECC Prediction: three value moving average DMA: device to host transfer (4kB) Low variance Low Error

27 The Impact of Precompletion fio I/O thread vs background threads Non-NAND Avg. latency: Measured AVG latency 352us (NAND latency)

28 The Impact of Precompletion fio I/O thread vs background threads I/O vs CPU requires Priority boost for Polling (priority degradataion) Polling damages system parallelism Non-NAND Avg. latency: Measured AVG latency 352us (NAND latency)

29 The Impact of Precompletion fio I/O thread vs background threads Overshoot harms parallelism (busy wait) Undershoot will expose scheduling delays Non-NAND Avg. latency: Measured AVG latency 352us (NAND latency) Precompletion requires an adequate precompletion window

30 The Impact of Precompletion fio I/O thread vs background threads Approx. 20% degradation of system parallelism IRQ vs Precompletion vscpu: 7.16 us gain vsio: 7.52 us gain With no degradation in system parallelism Non-NAND Avg. latency: Measured AVG latency 352us (NAND latency)

31 Summary Unblinding the OS - Cross layer optimization Achieved a partially predictable SSD decomposition / classification Exploit SSD internal information - Remaining I/O time Protecting SSD proprietary internals Abstracted behavioral models Mitigating scheduling delays Exploiting predictability of certain I/O requests Pre-completion - Projection (1 I/O depth vs other threads) IRQ Polling This work Latency Bad Good Good Parallelism Good Bad Good

32 Future Work Future Implementation & Evaluation Full blown SSD Projection (1 I/O depth this work) à Simulation (varying tech latency & etc) à Real implementation Cross layer optimization More models More use cases More backend technologies rather than flash

33