An Experimental Study of Redundant Array of Independent SSDs and Filesystems

Transcription

1 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems An Experimental Study of Redundant Array of Independent SSDs and Filesystems Yuxuan Xing 1, Ya Feng 1, Songping Yu 1, Zhengguo Chen 1, Fang Liu 1, Nong Xiao State Key Laboratory of High Performance Computing 2 School of Data and Computer Science National University of Defense Technology, NUDT Sun yat-sen University Changsha, China Guangzhou, China Abstract Solid state disks (SSDs) become more and more popular in personal devices and data centers Flash chips can be packaged in Hard disk drive (HDD) form factors and provide the same interface as HDDs This character makes SSDs easily replace HDDs in existing storage systems PCIebased SSD can provide a higher I/O performance, but it is still a little expensive This paper studies the feasibility of Redundant Arrays of Independent SSDs (RAIS) with different filesystems We comprehensively analyze the performance of RAIS constructed by SATA SSD and PCIe SSD individually We investigate different RAIS configurations (RAIS0, 5, 6) and filesystems under various I/O access patterns Finally, we illustrate our serval key findings and recommendations for building RAIS Keywords-Solid state disks; storage systems; Redundant Arrays of Independent SSDs; I/O performance; filesystems I INTRODUCTION Hard disk drives (HDDs) are the leading media in storage systems for decades The capacity of HDDs is continuous growing and the price-per-byte is reducing However, the I/O performance improves slowly due to its mechanical characteristic that is in the absence of parallelism Unfortunately, this state will not change for a long time In recent years, new storage media (such as Flash, PCM, FeRAM, etc) have sprung up in our sight Flash memory has been deployed widely in modern storage system due to its advantageous properties, including non-volatility, high I/O performance, low power consumption and mature technology Flash memory-based solid state disks (SSDs), especially made by NAND Flash, are becoming another leading media in storage systems International prestigious companies such as EMC, NetApp, IBM, have introduced their SSD-based storage products to both enterprise and consumer markets for years Some research institutes also take advantage of SSDs to accelerate I/O performance of the compute nodes in TianHe supercomputer [1] Facebook, Google, Baidu and Alibaba said they have deployed many SSDs in their data centers to fulfill the need for a highly reliable and available storage system, as well as high I/O throughput [2] Since David Patterson [3] first described the way to construct redundant arrays of independent disks (RAID) in 1988, RAID is the dominant technology in enterprise storage systems Although SSDs can replace HDD easily in the array, it also needs technical ability to build a highly reliable and high-performance redundant arrays of independent SSDs (RAIS) [4] RAIS has its own design challenges due to some typical characteristics of SSDs Besides, filesystem also plays an important role in the storage system since it is a key part of I/O stack Therefore, it is important to comprehensively evaluate the performance impact on the design of both RAIS and file system at the same time In our study, we used two different kinds of SSDs (depicted in Table 1) to construct three types of RAIS: RAIS0, RAIS5 and RAIS6 We choose these RAIS configurations because they are the most widely used schemas in the actual environments We also choose four filesystems to test them with RAIS: XFS, ext4, F2FS and btrfs [5][6][7] Here are the salient contributions of our work: We make a detailed analysis of single raw-device s performance by varying I/O access patterns and queue depth, and check their I/O behaviors under four filesystems We deeply explore the I/O characteristics of RAIS0, RAIS5 and RAIS6, which are based on two different types of SSD individually We conduct a comprehensive comparison of RAIS schemas under different filesystems, and reveal some design insights for constructing high costeffective RAIS Table 1: Storage device characteristics examined in our study Label Model Type Interface Capacity Erase (GB) (#) Samsung 850 EVO MLC SATA K Fusion-io iodrive2 MLC PCIe K The rest of this paper is organized as follows We first present an overview of the SSD, RAIS and filesystem in Section 2 We describe the methodology and testbed in section3 And section4 analyzes the I/O performance in detail Section5 introduces some related work We conclude in section6 II BACKGROUND A Solid State Disk Flash memory is a non-volatile storage medium based on semiconductor technology It can be classified into two types: NAND and NOR NAND flash memory has been widely used for massive storage devices over NOR flash because of its large capacity and low cost relative to NOR flash memory Figure 1 shows a schematic of a flash package It is composed of one or more dies And each die has a separate chip enable and ready/busy signals so that one of the dies can accept commands and data while the other is carrying out another operation [8] Thanks to the above features, NAND /16 $ IEEE DOI /HPCC-SmartCity-DSS

2 flash based solid state storage device (SSD) can provide lower latency and higher parallelism than HDD However, SSD has some drawbacks due to its own physical characteristics A block has a limited number of erasing times and its bit error rate rises sharply when it is close to the upper endurance cycles It also has the write amplification problem due to its out-of-place update mechanism Flash Translation Layer (FTL) can hide internal details in SSD and provide a normal access pattern like HDD to the application The FTL consists of address mapping, garbage collection, wear leveling, and managing bad blocks, and it is a key component of SSD Block0 Block4094 Plane0 Die0 Block1 Block4095 Plane1 Chip Package Serial Connection Block4096 Block8190 Plane0 Die1 Figure 1 Flash Internals Block4097 Block8191 B Redundant Array of Independent SSDs As individual SSD has a limitation in the aspects of capacity, performance and reliability, mass storage systems mainly adopt RAID technology to satisfy practical requirements We refer this as the Redundant Array of Independent SSDs (RAIS) NetApp has introduced a storage system based all flash memory arrays in 2013 [9] EMC also provided similar products and storage management solutions [10] RAIS replaces HDD with SSD and is very similar to RAID As RAIS-0, 5, 6 are more common in storage systems, we present them as follows: RAIS0 consists of striping, and doesn t have any parity Thus if any SSD fails, data loss will occur RAIS5 consists of block-level striping with distributed parity It can allow one SSD failure without data loss RAIS6 consists of block-level striping with double distributed parity RAIS6 can allow two SSDs failure at the same time while no data is lost However, when any data in a stripe updates, RAIS5 and RAIS6 must update the parity meanwhile This cause an inevitable performance loss, and also has a detrimental influence on the lifespan of SSD C Filesystem Filesystem plays an important role in the I/O stack Filesystem allocates space in a granular manner, usually multiple physical units on the device It is responsible for Plane1 organizing files and directories, and keeping track of the storage media usage and the mapping from files to allocated storage space There are many types of filesystem and it is very critical to choose an appropriate filesystem for a better I/O performance In our experiments, we select four filesystems to draw a comprehensive comparison: XFS, ext4, f2fs and btrfs XFS is a high performance 64-bit journaling file system and many Linux distributions use it as the default file system XFS is based on allocation groups for its design and enables extreme scalability of IO threads and file system bandwidth when spanning multiple physical storage devices The ext4 or fourth extended filesystem is a journaling filesystem for Linux, and it is designed to extend storage limits and add other performance improvements F2FS or Flash-Friendly File System is a flash file system initially developed by Samsung Electronics which takes into account the characteristics of NAND flash memory-based storage devices Btrfs (B-tree file system) is initially designed at Oracle Corporation for use in Linux based on the copy-on-write (COW) principle It is intended to address the lack of pooling, snapshots, checksums, and integral multi-device spanning in Linux file systems III METHODOLOGY AND SYSTEMS TESTBED Experiments are conducted on a single server, it contains 2 Intel Xeon E CPU, 4 GB memory and 7 PCI Express 30 16(8) slots on the motherboard The operating system is Centos65 and its kernel is We totally use 6 Samsung SSDs and 4 Fusion-io SSDs in our tests And we denote the Samsung and Fusion-io device as SSD- S/RAIS-S and SSD-F/RAIS-F in the rest of this study, respectively SSD-S is connected to the server with LSI MegaRAID SAS i Controller Card This card can connect at most eight external data lines and each data line can provide 12 Gb/s bandwidth As for SSD-F, we combine them with the MD module built-in Linux kernel Fio [10] is a widely used tools for benchmark and stress/hardware verification It has support for 19 different types of I/O engines, I/O priorities, rate I/O, forked or threaded jobs, and much more It can work on block devices as well as files We use it to measure the I/O performance of RAIS with/without filesystems We begin our experiment on the single SSD-S and SSD-F Then we perform detailed research on RAIS And we test the I/O characteristics of raw device and under different filesystems We repeat each experiment three times for all test case and acquire the mean value as the final result 43

3 Figure 2 Sequential I/O Throughput of Individual SSD IV EXPERIMENT RESULT We analyze the performance of both the raw devices and under filesystem in our experiments The first section focuses on the I/O characteristics of individual devices This provides a baseline performance for RAIS-S and RAIS-F In section B, we evaluate the I/O features of RAIS0, RAIS5 and RAIS6 We present several key findings and give some advice for constructing cost-effective RAIS in section C And before each of our tests, we erase the devices in order to obtain more accurate results We use libaio engine, one thread number in the entire experiments A Individual SSD Performance Performance of Raw Device We present the results on individual SSD-S and SSD-F using sequential I/O requests in Figure 2 In each test, we changed the queue depth with different request size Figure 2-1 and figure 2-2 show the sequential read performance of SSD-S and SSD-F, and in the figure 2-3 and figure 2-4, we present the sequential write performance The data show that the read/write bandwidth increases with queue depth and request size When the queue depth is greater than 4, I/O bandwidth grows very slowly And if the queue depth is smaller than 4, it is in favor of large I/O request to get a high bandwidth SSD needs a number of I/O requests in the queue to develop its parallelism, and when the I/O requests is enough, the bandwidth also reaches the upper limit Besides, SSD-F performs better than SSD-S both read and write bandwidth just due to its higher parallelism from more flash chips And the maximum sequential read/write performance is around 525/338 MB/s for SSD-S, SSD-F is around 1500/595 MB/s Figure 3 shows the I/O characteristics of SSD-S and SSD-F for 4KB random access patterns with respect to varying the queue depth for each case The random read/write performance improves with the queue depth Interestingly, the performance of random read exceeds random write when the queue depth reaches 16 for SSD-S, and SSD-F s performance of random write is always better than read It needs a higher queue depth to achieve the peak for random read/write IOPS in relative to sequential read/write bandwidth And the maximum random read/write performance is around 99000/81000 IOPS for SSD-S, SSD-F is around / IOPS As a result, PCIe-based SSD-F is better than SATA- -based SSD-S in all aspects Performance of SSD under Filesystems We mounted the XFS/ext4/f2fs/btrfs on the raw device and tested sequential read/write performance under the condition of 64KB I/O request and 64 queue depth For random read/write IOPS, we use 4KB IO request and 64 queue depth We present the final results in Figure 4 And the minuscule d stands for Direct I/O, c means Cached I/O In figure 4-1, there is no obvious gap among the filesystems as for sequential read For sequential write, ext4 performs best, while F2Fs is the worst The sequential read performance is generally small between Direct I/O and Cached I/O But Cached I/O is better for sequential write under all filesystems, because the data is written in DRAM firstly and flushed to disk later Some data maybe become invalid in this period of time SSD-F is a little different than SSD-S as can be seen in Figure 4-2 The difference among 44

4 Figure 3 Random I/O throughput of individual SSD Figure 4 I/O Throughput of Individual SSD under Filesystem the four filesystems is very small for the sequence read and write except for the write performance of F2FS And Direct I/O performs better in sequential read under all filesystems The results in figure 4-3 and 4-4 is very different It has an important effect on performance that we select Direct I/O or Cached I/O Obviously, Direct I/O is better for random read But for random write, XFS/ext4/f2fs prefer Cached I/O as to SSD-S while the same is XFS/ext4 for SSD-F B RAIS Performance In this section we present experimental results on RAIS constituted by SSD-S and SSD-F We used 4 SSD-S for the RAIS0-S configuration, 5 SSD-S for RAIS5-S, and 6 SSD-S for RAIS6-S As for SSD-F, RAIS0-F consists of 2 SSD-F, RAIS5-F is 3, and RAIS6-F is 4 Table 2 shows the details about the RAIS configuration Performance of RAIS We present the sequential read bandwidth for RAIS0, RAIS5 and RAIS6 in Figure 5 Figure 5-1, 5-2, 5-3 are the results about SSD-S, and Figure 5-4, 5-5, 5-6 are for SSD-F The trend is similar to individual device in figure 2-1, 2-2 The read performance increases with queue depth and request size for all RAIS And the maximum bandwidth can reach the sum performance of all devices in RAIS except for RAIS5-F For example, there is 6 devices in RAIS6-S and the sequential read performance is nearly 3070 MB/s as much as 6 times of the single device It is abnormal that RAIS0-F and RAIS6-F can achieve almost 3000/5800 MB/s while RAIS5-F is only 2580 MB/s This is very strange and probably is caused by the design of MD Table 2 RAIS Configuration RAIS0/5/6-S RAIS0/5/6-F Tools LSI MegaRAID SAS i Linux MD Num of device 4/5/6 2/3/4 Stripe size 64KB 64KB Others Directed IO Write Through No Read Ahead none We configure LSI MegaRAID Controller with the police-directed IO, Write Through and No Read Ahead for a better performance We have tested 256KB, 512KB and 1MB stripe size and their behavior is similar to 64KB stripe size 45

5 Figure 5 Sequential Read Throughput of RAIS The I/O characteristic of sequential write is quite different from each other as can be seen in Figure 6 RAIS0- S, RAIS5-S and RAIS6-S achieve the same maximum sequential write bandwidth as they all have four data disks (RAIS5-S has five data disks and one of them stores parity in each stripe while RAIS6-S has six data disks and two of them store parity) More interestingly, the peak value is a little higher than 4 times as much single device (we have tested many times and the reason may come from the optimization of LSI MegaRAID controller) As opposed to RAIS-S, RAIS-F must consumes the memory and CPU in the host in order to manage I/O request This could be an additional overhead in the I/O stack In Figure 6-4, RAIS0-F has the similar trends as RAIS0-S, and the peak value is nearly twice as much one SSD-F But the maximum sequential write bandwidth of RAIS5 is just 970 MB/s and RAIS6 is 760 MB/s This may be caused by the parity computation in each stripe RAIS6 needs two parities so that it has a lower performance The random read/write performance is not linear growth with the number of device as shown in Figure 7 RAIS-S and RAIS-F have the similar trends in random read/write I/O characteristics RAIS0 is the best, and RAIS5 is better than RAIS6 in this tests For random read, there is no obvious difference among them But RAIS0 achieves a significantly higher performance than RAIS5 and RAIS6 as to random write The IOPS of RAIS-S is greater than single SSD-S for random read, while RAIS-F is almost the same as an individual SSD-F And the random read/write IOPS of RAIS0-S/F is in the same way It is not surprising that RAIS5/6 achieves a very low performance, each random write will result in two read and write operations every time In addition, this will lead to frequently garbage collection (GC) which has a bad influence on the SSD performance Figure 6 Sequential Write Throughput of RAIS 46

6 Figure 7 Random I/O Throughput of RAIS Performance of RAIS under Filesystems We present the sequential read/write performance under four different filesystems for all RAIS schemas in Figure 8 It is clear that Direct I/O is better than Cached I/O for sequential read/write requests, except for btrfs in write requests And the performance of RAIS-S drops a little relative to the raw devices, while RAIS-F is not As can be seen in Figure 8-1 and 8-4, there is little difference among different filesystems for the read/write bandwidth except for f2fs F2fs has the worst performance in write requests In figure 8-3, XFS and ext4 have a better performance for both read and write As for RAIS6-F, ext4 and f2fs has a higher read bandwidth and all filesystems performance is similar for write requests The situation for random read/write is more different and complicated in Figure 9 Random read prefers Direct I/O while random write is in favor of Cached I/O And f2fs stands out among the four filesystems in case of the random write requests As for random read requests, ext4 have a little better performance Figure 8 Sequential I/O Throughput of RAIS under Filesystem 47

7 Figure 9 Random I/O Throughput of RAIS under Filesystem C Disscusion In this section, we discuss several our key findings: 1 It seems that we can use RAIS constructed with SATA SSDs to reach the same performance as PCIe SSD And if you need both large capacity and high performance, perhaps the RAIS plan is better For example, Intel 750s PCIe 12TB SSD is about 1160$, and we need 5 Samsung 850 SATA 256GB SSDs and a LSI MegaRAID i controller to get the approximate capacity and performance And the total cost is almost 805$ However, if you just want a higher performance, it is better for you to choose PCIe SSD with proper capacity (Intel 750s PCIe 400GB SSD is only 433$ with the equal performance) Because the RAID controller also costs a lot, and if we use Linux MD to manage SATA SSDs, the highest bandwidth we can achieve is only 1400MB/s due to the limitation of DMI We don't take the energy consumption of SATA/PCIe SSD into account just because almost 80%-90% energy consumption is on the CPU 2 It is better to merge IO requests to be integer times of stripe size and apply for a queue depth at least 4 so that can fully develop the parallelism of SSD or RAIS as shown in Figure2, 5 and 6 3 Sequential read performance can reach the sum of its all devices in RAIS0/5/6, while sequential write performance only is the sum of valid data devices in RAIS0/5/6 For example, there are 3 devices in RAIS5 schema and each device has a 500/400MB/s sequential read/write bandwidth, thus the total sequential read/write bandwidth of this RAIS5 can achieve 1500/800MB/s separately As for random read/write IOPS, it is distinctly lower than the sum of single devices 4 As can be seen in section B, the RAID controller has little impact on the performance thanks to its own computation resource and memory While soft RAID using Linux MD has a bad influence on the performance of devices especially for RAIS5 and RAIS6 5 We also give some preferences of different RAIS schemas and individual SSDs in Table 3 ( * means that it is almost the same we use different filesystems and Direct/Cached I/O) And we must pick an appropriate filesystem on the basis of the real load characteristics It is better for sequential read and random read if we use Direct I/O As to random write, Cached I/O has more advantages But the situation for sequential write is much changeable, it is better to make a choice in accordance with the actual environment Table 3 Analysis of Filesystems and I/O schemas seqential_read seqential_write random_read random_write SSD-S * xfs/ext4 xfs/ext4/f2fs f2fs cached IO cached IO SSD-F xfs/ext4/btrfs ext4/f2fs xfs cached IO RAIS0- xfs/ext4/btrfs ext4 f2fs S cached IO RAIS0- xfs/ext4/btrfs xfs/ext4/f2fs f2fs F cached IO RAIS5- xfs/ext4/btrfs xfs/ext4/f2fs f2fs S cached IO RAIS5- F ext4/f2fs xfs/f2fs cached IO ext4/f2fs f2fs cached IO RAIS6- S xfs/ext4/f2fs ext4/btrfs ext4 f2fs cached IO RAIS6- ext4/f2fs xfs/ext4/f2fs f2fs * F cached IO 48

8 V RELATED WORK There are several other related work about SSD and RAIS Youngjae Kim [11] analyze the RAIS performance using commercially-off-the-shelf SSDs, and investigate RAIS configurations under a variety of I/O access patterns However, the SSD -they used to construct RAIS is based on SATA- interface, and the sequential read/write bandwidth is no more than 350 MB/s SSD based on SATA- is widespread which can provide around 500MB/s read/write bandwidth A higher bandwidth may bring some new challenges in RAIS Besides, it does not include the study of the I/O performance of RAIS under different filesystems Another work from Intel exposed all platform bottlenecks that may interfere with delivering full SSD performance to applications [12] They identified data copies, uncacheable MMIO reads, interrupt processing, and context switches to be the primary contributors of I/O processing cost And they found typical reference platforms is surprisingly robust Yimo Du [13] introduced deduplication in RAIS construction to improve the read/write performance And it also proposes some optimization which effectively reduces the number of parity update times and extends SSD s lifespan effectively But this work is based on an analytical modeling and simulation approach Asim Kadav [14] proposed Diff-RAID, a new RAID variant that distributes parity unevenly across SSDs to create age disparities within arrays Diff-RAID can make a balance between the high BER of old SSDs and the low BER of young SSDs As a result, Diff-RAID will provide a higher reliability for RAIS However, it is difficult for RAID controller to monitor and make appropriate adjustments dynamically VI CONCLUSION AND FUTURE WORK We conducted a comprehensive empirical work on SSDs and RAIS configurations in terms of I/O performance in this paper We first analyze individual, single SSD-S/SSD-F performance to obtain the individual baseline performance of each device And in the second step, we construct RAIS0, RAIS5 and RAIS6 with SSD-S and SSD-F respectively and analyze the I/O performance under different filesystems and I/O schemas In addition, we discuss I/O characteristics among different RAIS Finally, we make a conclusion based on our findings and give some advice for building costeffective RAIS Our study is still in progress As the garbage collection (GC) has an important influence on SSDs, we will explore the timing of GC and the I/O characteristics during GC Besides, our results are based on benchmark tools and we will perform performance tests with real applications in the future ACKNOWLEDGMENT We are grateful to our anonymous reviewers for their suggestions to improve this paper This work is supported by the National High-Tech Research and Development Projects (863) and the National Natural Science Foundation of China under Grant Nos 2015AA015305, , , REFERENCES [1] Yang X J, et al The TianHe-1A supercomputer: its hardware and software[j] Journal of Computer Science and Technology, 2011, 26(3): [2] Jian Ouyang, et al Active SSD design for energy-efficiency improvement of web-scale data analysis In the Proceedings of the 2013 International Symposium on Low Power Electronics and Design 2013: [3] Patterson D A, et al A case for redundant arrays of inexpensive disks (RAID)[M] ACM, 1988 [4] Du Y, et al CD-RAIS: Constrained dynamic striping in redundant array of independent SSDs[C] Proceedings of the 16 th IEEE Cluster Computing (CLUSTER), Madrid: IEEE, 2014: [5] Mathur A, et al The new ext4 filesystem: current status and future plans[c] Proceedings of the Linux symposium 2007, 2: [6] C Lee, et al F2FS: A New File System for Flash Storage Proceedings of the 13 th USENIX Conference on File and Storage Technologies (FAST 15) Santa Clara, CA, USA, PP , February 16-19, 2015 [7] O Rodeh, et al BTRFS: The Linux B-Tree Filesystem ACM Transactions on Storage Vol 9, No 3, Aticle 9, August 2013 [8] Agrawal N, et al Design Tradeoffs for SSD Performance[C] Proceedings of the 2008 USENIX Conference on USENIX Annual Technical Conference 2008: [9] NetApp Netapp EF540 flash array [R] NetApp enterprise: NetApp 2013: 1~4 [10] EMC Vnx5500-f unified storage flash array [R] EMC enterprise: EMC 2014: 1~8 [11] Axboe, J Flexible IO Tester (fio) [12] Kim Y, etc An empirical study of redundant array of independent solid-state drives (RAIS)[J] Proceeding of the 2015 IEEE International Conference on Cluster Computing, 2015, 18(2): [13] Foong A P, et al Towards SSD-Ready Enterprise Platforms[C] Proceedings of the 36 th International Conference on Very Large Data Bases (VLDB) Singapore, 2010: [14] Du Y, et al R-Dedup: Content Aware Redundancy Management for SSD-Based RAID Systems[C] Proceedinds of the 43 rd International Conference on Parallel Processing (ICPP), Minneapolis: IEEE, 2014: [15] Balakrishnan M, Kadav A, Prabhakaran V, et al Differential RAID: rethinking RAID for SSD reliability[j] ACM Transactions on Storage (TOS), 2010, 6(2): 4 49