Linux Storage System Bottleneck Exploration

Transcription

1 Linux Storage System Bottleneck Exploration Bean Huo / Zoltan Szubbocsev Beanhuo@micron.com / zszubbocsev@micron.com 215 Micron Technology, Inc. All rights reserved. Information, products, and/or specifications are subject to change without notice. All information is provided on an AS IS basis without warranties of any kind. Statements regarding products, including regarding their features, availability, functionality, or compatibility, are provided for informational purposes only and do not modify the warranty, if any, applicable to any product. Drawings may not be to scale. Micron, the Micron logo, and all other Micron trademarks are the property of Micron Technology, Inc. All other trademarks are the property of their respective owners.

2 Outline Introduction Introduction to Linux storage stack analysis methodology emmc stack analysis UFS and NVMe stack analysis comparison Summary Micron Technology, Inc. Micron Confidential October 25, 217

3 Introduction

4 About us Work for Micron Part of the embedded business unit Focusing on embedded storage SW Based in Munich part of the embedded system architecture and engineering group Areas we work on, embedded file systems, emmc, UFS, NVMe for embedded Micron Technology, Inc. Micron Confidential October 25, 217

5 Goals of our project Quantify storage system overhead in embedded systems for emmc, UFS and NVMe Understand how much of the physical speed is realized at user level Get an idea on how much Linux storage stack impacts the overall user space performance Quantify NVMe storage stack improvements over UFS storage stack Does NVMe provide a better user level speed in embedded systems? If we put two equivalent UFS and NVMe devices in a system which would be better and how much? How much improvements does it provide over a UFS stack? Micron Technology, Inc. Micron Confidential October 25, 217

6 Introduction to emmc, UFS and NVMe All three are solid state drive technologies, below in chronological order Consisting of NAND chips plus controller and firmware emmc (embedded multi-media card) HS4 and Maximum speed can reach to 4 MB/s UFS (universal flash storage) UFS Gear3 728 MB/s per lane NVMe (non-volatile memory express) Gen3 1MB/s per lane Micron Technology, Inc. Micron Confidential October 25, 217

7 Introduction to Linux storage stack analysis methodology

8 Methodology Workload generation Fio tool Single and multi-threaded workloads (1, 8 thread) 4KB random read, write 128 KB sequential read, write Direct IO and sync IO The utilities of tracing Ftrace Tracer: function_graph trace_printk() add tracing point Blktrace Cannot trace VFS-FS layer blk_add_trace_msg() Micron Technology, Inc. Micron Confidential October 25, 217

9 Methodology Applications Page cache BIOs 1 Write/read data VFS/File System Block (Request Q, I/O schedule, plug/unplug) Direct I/O BIOs Request end Latencies broken down to 4 sections VFS-FS latency: user space submission to block layer receiving a BIO Block layer submission to storage device submission 2 SCSI Stack Requests Host Bus Driver (Peek request->bounce copy->dma mapping) Bounce buffer post / DMA un-mapping HW IRQ handling 4 HW transfer, storage submission to completion interrupt Request post, completion interrupt to block layer completion Enqueue Tasks Data Transfer 3 Storage Devices Add trace point Latency Micron Technology, Inc. Micron Confidential October 25, 217

10 emmc stack analysis Linux Storage System Analysis for e.mmc With Command Queuing freely download from

11 Speciation Of emmc Target Platforms Xilinx Zynq Zed board Jenson TX1 NVIDIA Board CPU type ARM Cortex A9 ARM Cortex-A57 MPcore Core number 2 4 L1 cache 32KB L1 I-cache and 32KB L1 D- caches with parity per core 48KB L1 I-cache per core; 32KB L1 D-cache per core L2 cache 512KB (Unified Cache) 2MB (Unified Cache) CPU frequency 667MHz 1,73GHz DDR type 512MB DDR3 (32bit) 4GB LPDDR4 (64 bit) DDR frequency 533MHz 16MHz CQE Hardware CQ engine software simulation CQ emmc 32GB@HS4 8GB@HS4 Block FS Ext4 Ext Micron Technology, Inc. Micron Confidential October 25, 217

12 Latency (us) Latency (us) emmc system overhead on the Xilinx Zed KB data I/O request system overhead on the Zed KB data I/O request system overhead on the Zed % 75% 69% 63% 78% 37% 22% 31% 8% direct seq_read direct_seq_write sync_seq_read sync_seq_write % 65% 88% 82% 34% 35% 18% 12% direct_random_read direct_random_write sync_random_read sync_random_write HW transfer VFS_FS Block_MMc request post HW transfer VFS_FS Block_MMc request post On the Zed board, the emmc performance is dominated by software overhead, rather than device Micron Technology, Inc. Micron Confidential October 25, 217

13 Latency (us) Latency (us) emmc system overhead on the TX KB data I/O request system overhead on TX1 61% 39% 88% 12% 65% 35% 67% direct_seq_read direct_seq_write sync_seq_read sync_seq_write HW transfer VFS_FS Block_MMc request post 33% KB data I/O request system overhead on TX1 74% 73% 81% 76% 27% 26% 24% 19% direct_random_read direct_random_write sync_random_read sync_random_write HW transfer VFS_FS Block_MMc request post The performance with 128KB chunk size is impacted by 12-39% system overhead. With 4KB chunk size, we observe 73-81% system overhead Micron Technology, Inc. Micron Confidential October 25, 217

14 Summary Direct I/O has better performance than Sync I/O due to lack of memory copy operation and page cache usage. System overhead is observed to be a significant contributor to I/O duration and even in higher end system with small chunk accesses it is significant. Even for high speed hardware platform, the conventional Linux software stack can not fully exploit the speed provided by high speed emmc devices Micron Technology, Inc. Micron Confidential October 25, 217

15 UFS and NVMe stack analysis comparison

16 Speciation Of UFS/NVMe Target Platform CPU # Hikey96 A73 x 4 + A53 x 4(BigLittle) OS Android, Linux kernel IO Type File System IO Tool IO Trace Tool Direct IO ext4 fio Blktrace / ftrace UFS Lanes Micron Technology, Inc. Micron Confidential October 25, 217 NVMe Density 128GB 128GB Phy / link Interface M-phy Gear 3 PCIe Gen2 Queues 1 8 Queue depth UFS supports to 256, but UFS host capacity only can support per queue INTs 1 1(Sharing)

17 UFS/NVMe SW Stack Application R/W/ VFS / File System BIO Block(blk-mq) NVMe has more compact/simpler SW stack and provides shorter code paths and lower overhead. NVMe stack is newer optimized for managed NAND devices NVMe Driver (Multi queue) PCIe host Driver SCSI top layer SCSI Mid layer (scis-mq) SCSI low layer UFS Host Driver Nvme can better support parallelism due to advanced multi-queue capability NVMe Device UFS Device Micron Technology, Inc. Micron Confidential October 25, 217

18 us us 4KB Random HW/SW latency comparison(per thread) 12 4KB Random write 16 4KB Random Read % % 28% 4 71% % 72% 2 26% 29% 6% 75% 4% 25% UFS NVME UFS NVME 1T 8T Thread count 2 69% 7% UFS NVME UFS NVME 1T 8T Thread count HW SW HW SW System overhead is significant in both UFS and NVMe ranging from 6-75% of total IO time for random write and 1-37% from random read Micron Technology, Inc. Micron Confidential October 25, 217

19 us us 128KB sequential HW/SW latency comparision (per thread) 128KB Sequential write 128KB Sequential Read 6 5 1% % % 45% 8% 9% 74% 54% 92% UFS NVME UFS NVME 1T 8T Thread count % 87% 7% 67% 19% 93% 81% UFS NVME UFS NVME 1T 8T Thread count HW SW HW SW System overhead on sequential large chunk writes in ranging between 26-45% with single thread and between 8-1% with multiple threads Sequential reads experience overhead between 13-33% with single threads and 7-19% with multiple threads Micron Technology, Inc. Micron Confidential October 25, 217

20 US US System overhead comparison with 4KB 4KB Random write 4KB Random read 9 1% 6, 1% T 66% Thread count UFS NVME 1% 8T 8% 5, 4, 3, 2, 1,, 1T 79% Thread count UFS NVME 1% 8T 96% NVMe shows a significantly lower system overhead vs UFS in small chunk accesses The difference decreases with increase in number of threads Micron Technology, Inc. Micron Confidential October 25, 217

21 us us System overhead comparison with 128KB 128KB Sequential Write 128KB Sequential Read % 1T 58% Thread count UFS NVME 1% 8T 41% % 1T 74% Thread count 1% 8T 62% UFS NVME NVMe shows a significantly lower system overhead vs UFS in small chunk accesses Micron Technology, Inc. Micron Confidential October 25, 217

22 MB/s Estimated system level performance comparison with typical device access times IOPS 128KB Sequential W/R Comparison(single thread) 4KB Random W/R IOPS comparison (single thread) ,1 357,14 25% 578,7 53,2 13% % % seq write NVMe UFS seq read 2 Random write NVMe UFS Random read Assuming identical NVMe and UFS devices, graph represents performance differences Sequential write access time:15 usec, Sequential read access time:124 usec Random write access time: 28 usec, Random read access time: 8 usec NVMe shows 25% benefits in sequential write and 13% benefit in sequential read. 15% improvement in random write and 8% improvement in random read Micron Technology, Inc. Micron Confidential October 25, 217

23 UFS queue limitation Number of outstanding tasks in UFS queue samples From sample 2 to sample Task count Task count Micron Technology, Inc. Micron Confidential October 25,

24 Number of outstanding tasks in NVMe queue 3ms 5ms samples Task count Task count NVME queue limitation(single queue) From sample 1 to sample Micron Technology, Inc. Micron Confidential October 25,

25 Normalized MB/s UFS/NVMe performance comparison in high parallel level 1,2 4KB RandomRead Normalized Comparison Between UFS and NVMe 1,,8,6,4,2, Thread count UFS NVMe Micron Technology, Inc. Micron Confidential October 25, 217

26 Summary We observe storage system overhead that eats into underlying storage device bandwidth. Overhead is more significant with small chunk IO accesses This overhead is expected to be more significant with faster storage devices NVMe shows improved performance due to leaner storage system up to 25% in certain cases NVMe provides a richer queuing infrastructure, this has an observable benefit in high thread count Micron Technology, Inc. Micron Confidential October 25, 217

27 Q&A

28