PSA: Performance and Space-Aware Data Layout for Hybrid Parallel File Systems

Transcription

1 PSA: Performance and Space-Aware Data Layout for Hybrid Parallel File Systems Shuibing He, Yan Liu, Xian-He Sun Department of Computer Science Illinois Institute of Technology

2 I/O Becomes the Bottleneck for Computer Speed Gap Between CPU and Hard Drive [1] Selected 212 INCITE applications at Argonne Leadership Computing Facility of ANL [2] Project On-line Data (TB) Off-line Data (TB) Combustion in Reactive Gases 1 17 CO2 Absorption 5 15 Seismic Hazard Analysis 6 1 Climate Science 2 75 Energy Storage Materials 1 1 Stress Corrosion Cracking Nuclear Structure and Reactions 6 3 Reactor Thermal Hydraulic 1 1 Modeling Laser-Plasma Interactions 6 6 Vaporizing Droplets in a Turbulent Flow 2 4 [1] [2] R. Latham, R. Ross, B. Welch, and K. Antypas, "Parallel I/O in Practice," Tutorial of the International Conference for High Performance Computing, Networking, Storage and Analysis,

3 High Performance I/O Solutions 1.Parallel File Systems 2. Solid State Disks 3 Bandwith (MB/sec) Request Size (KB) SSD-seq SSD-ran HDD-seq HDD-ran

4 Hybrid Parallel File Systems Hybrid PFS Clients Network File servers HDD HDD HDD HDD HDD SSD SSD SSD HServers SServers Consists of HServers and SServers The performance depends on an efficient data layout scheme 4

5 Issues of Traditional Data Layout Schemes What is data layout? An algorithm defining a file's data distribution across available servers. Two challenges The storage performance of each server is different, which may lead to load imbalance among servers. The storage space of each server is different, which may make SServers quickly run out of space 5

6 Traditional Fixed-Size Stripe Layout Scheme R R1 R2 R3 R4 R5 P P1 P Each server has a fixed-size stripe HServer1 HServer2 SServer1 Designed for homogeneous PFSs, suitable for certain I/O patterns

7 Problem for Fixed-Size Data Layout R R1 R2 R3 R4 R5 HServer HServer SServer Time t t1 t2 t3 t4 t5 t6 Load-imbalance among file servers in hybrid PFSs 7

8 Performance-Aware Layout Scheme R R1 R2 R3 R4 R5 P P1 P HServer 1 HServer 2 SServer1 PA: HServers with smaller stripe size, SServers with larger stripe size SSDs quickly run out of space, remaining requests are only served by the low-speed HServers 8

9 Our solution: Performance and Space-Aware Data Layout R R1 R2 R3 R4 R5 P P1 P HServer1 HServer2 SServer1 Idea: Assign HServers with larger file stripes and SServers with smaller stripes More requests are served by the hybrid servers More hybrid requests Less pure requests 9

10 An Example of I/O Time Comparison Layout # hybrid requests I/O time per hybrid request # pure requests I/O time per pure request Overall I/O time PA 6 2T 14 8T 124T PSA 2 4T 8T 1

11 Challenges of Determining the Stripe Sizes Server performance can be impacted by I/O patterns. Server performance can also be impacted by storage media. Different stripe sizes lead to various ratio of hybrid requests over pure requests, which can impact the overall system performance. 11

12 Data Access Cost Model The cost is defined as the overall I/O time of each data access in hybrid PFSs. Mainly includes two parts Table 1. Parameters (short in Pars) in cost analysis model. Network time: T NET =T E +T X Storage time: T SROR =T S +T R/W Two types of requests Hybrid requests Requests served by all servers Pure requests Requests served by HServers 12

13 Cost Formulas for Hybrid PFSs Table 2. Cost formulas for hybrid requests on all HServers and SServers Table 3. Cost formulas for pure requests on all HServers More details about constructing the cost model can be found in our paper. 13

14 Optimal Stripe Size for File Servers Algorithm Overhead The calculations are simply arithmetic operations and run offline Computational overhead for executing this algorithm is acceptable 14

15 The Procedure of the PSA Scheme Pre-estimation Estimate parameters PFS parameters: Layout determination App I/O Traces, profile data,... File stripe size determination algorithm in Algorithm 1 with cost model Optimal stripe sizes: h and s Data placement Distribute file data 15

16 Experimental Setup 65-nodes SUN Fire Linux Cluster CPU Quad-Core AMD Opteron(tm) Processor 2376 * 2 Memory 4 * 2GB, DDR2 333MHz Network 1 Gbps Ethernet HDD: Seagate SATA II 25GB, 72RPM Storage SSD: OCZ PCI-E X4 1GB OS Linux kernel File system OrangeFS Nodes Eight computing nodes, four HServers, four SServers Data placement schemes Default scheme (DEF) Performance-aware scheme (PA) The proposed performance and space-aware scheme (PSA) Benchmark: IOR 16 processes access a shared 16GB file,each HServer has 1GB storage space. send sequential/random read/write requests request size is 512KB 16

17 Different Type of I/O Operations Read Write DEF PA PSA DEF PA PSA I/O Throughput (MB/Sec) Sequential Random I/O Throughput (MB/Sec) Sequential Random For different operations, PSA uses different stripe sizes <12KB, 8KB> and <12KB, 8KB> for sequential and random reads <116KB, 12KB> and <12KB, 8KB> for sequential and random writes PSA has optimal I/O performance compared to the other data layout schemes PSA's read performance exceeds its write performance because SSDs performs better for read operations than write 17

18 Different Number of Processes Read Write DEF PA PSA DEF PA PSA I/O Throughput (MB/Sec) Number of Processes I/O Throughput (MB/Sec) Number of Processes PSA has the best performance among the three schemes. As the number of processes increases, the performance of the hybrid PFS decreases because more processes lead to setious I/O contention in HServers. PSA has high scalability in terms of number of processes. 18

19 Different Request Sizes Request size is 128K DEF PA PSA Request size is 124K DEF PA PSA I/O Throughput (MB/Sec) Sequential read Random read Sequential write Random write I/O Throughput (MB/Sec) Sequential read Random read Sequential write Random write PSA can improve the read and write performance. As the request size increases, PSA achieves more improvements, because large requests benefit more from the hybrid file servers than the pure HServers. 19

20 Different Server Configurations 5HServers : 3 SServers DEF PA PSA 3HServers : 5 SServers DEF PA PSA I/O Throughput (MB/Sec) Sequential read Random read Sequential write Random write I/O Throughput (MB/Sec) Sequential read Random read Sequential write Random write PSA has the similar performance advantages. Read and write performance improvement increases as the number of SServers increases, because the I/O performance of hybrid requests benefits more from more SServers. 2

21 Conclusions A model to evaluate the I/O time of a request on hybrid file servers and pure file servers A performance and space-aware scheme to determine the appropriate file stripe sizes for HServer and SServers in hybrid PFSs Implement the prototype of the PSA scheme under OrangeFS. Experiment results illustrate that PSA can significantly improve I/O performance. 21

22 Thank you very much! Shuibing He, Yan Liu, Xian-He Sun Department of Computer Science Illinois Institute of Technology