Storage Hierarchy Management for Scientific Computing

Transcription

1 Storage Hierarchy Management for Scientific Computing by Ethan Leo Miller Sc. B. (Brown University) 1987 M.S. (University of California at Berkeley) 1990 A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the GRADUATE DIVISION of the UNIVERSITY of CALIFORNIA at BERKELEY Committee in charge: Professor Randy H. Katz, Chair Professor Michael Stonebraker Professor Phillip Colella 1995

2 Storage Hierarchy Management for Scientific Computing Copyright 1995 by Ethan Leo Miller All rights reserved

3 Abstract Scientific computation has always been one of the driving forces behind the design of computer systems. As a result, many advances in CPU architecture were first developed for high-speed supercomputer systems, keeping them among the fastest computers in the world. However, little research has been done in storing the vast quantities of data that scientists manipulate on these powerful computers. This thesis first characterizes scientists usage of a multi-terabyte tertiary storage system attached to a highspeed computer. The analysis finds that the number of files and average file size have both increased by several orders of magnitude since The study also finds that integration of tertiary storage with secondary storage is critical. Many of the accesses to files stored on tape could have easily been avoided had scientists seen a unified view of the mass storage hierarchy instead of the two separate views of the system studied. This finding was a major motivation of the design of the RAMA file system. The remainder of the thesis describes the design and simulation of a massively parallel processor (MPP) file system that is simple, easy to use, and integrates well with tertiary storage. MPPs are increasingly commonly used for scientific computation, yet their file systems require great attention to detail to get acceptable performance. Worse, a program that performs well on one machine may perform poorly on a similar machine with a slightly different file system. RAMA solves this problem by pseudo-randomly distributing data to a disk attached to each processor, making performance independent of program usage patterns. It does this without sacrificing the high performance that scientific users demand, as shown by simulations comparing the performance of RAMA and a striped file system on both real and synthetic benchmarks. Additionally, RAMA can be easily integrated with tertiary storage systems, providing a unified view of the file system spanning both disk and tape systems. RAMA s ease of use and simplicity of design make it an ideal choice for the massively parallel computers used by the scientific community.

4 Table of Contents CHAPTER 1. Introduction Thesis Statement Dissertation Outline... 2 CHAPTER 2. Background and Related Work Storage Devices Magnetic Disk Magnetic Tape Optical Disk and Tape Other Storage Technologies Robotic Access to Storage Media File System Concepts Berkeley Fast File System Log-Structured File System Mass Storage Systems Long-Term Reference Patterns and File Migration Algorithms Existing Mass Storage Systems for Scientific Computing Massively Parallel File Systems File System-Independent Parallel I/O Improvements Bridge Concurrent File System Vesta CM-5 File System (sfs) Uniprocessor File Systems Used by MPPs Other Parallel File Systems Conclusions CHAPTER 3. File Migration NCAR System Configuration Hardware Configuration System Software Applications Tracing Methods Trace Collection Trace Format Observations Trace Statistics Latency to First Byte MSS Usage Patterns Interreference Intervals File Reference Patterns File and Directory Sizes File Migration Algorithms Conclusions iii

5 CHAPTER 4. RAMA: a Parallel File System File System Design File System Information Data Placement in RAMA File Block Placement Intrinsic Metadata Placement File System Operation Access to File Blocks on Disk Disk Storage Management Tertiary Storage and Rama Implementation Issues Hashing Algorithm Interconnection Network Congestion Data integrity and availability Disk Storage Utilization Conclusions CHAPTER 5. Simulation Methodology Simulator Design Simulator Implementation Disk Model Network Model Multiprocessor CPU Model File System Simulation Applications Simulated Strided Sequential Access LU Matrix Decomposition Global Climate Modeling Small File Access Conclusions CHAPTER 6. Sensitivity of the RAMA Design to Changes in Technology, Design and Usage Technological Parameters Network Performance Network Bandwidth Network Message Latency Disk Performance Disk Track Density Disk Rotational Speed Disk Head Positioning Latency CPU Speed and Memory Size Design Parameters Data Distribution on Disk Scalability and Multiple Disks Per Node Network Configuration Small File Performance Future Performance Conclusions iv

6 CHAPTER 7. A Performance Comparison of RAMA and Striped File Systems Application Performance Strided Access Raw Bandwidth Experiments Strided Access with Computation Matrix Decomposition Global Climate Modeling Disk Utilization Spatial Distribution of Disk Requests Temporal Distribution of Disk Requests File System Data Distribution Network Utilization Network Utilization Under Striped File Systems Network Utilization Under RAMA Conclusions CHAPTER 8. Conclusions Summary Future Work Future Research in Tertiary Storage Future Research on RAMA and Parallel File Systems Summary Bibliography v

7 List of Figures CHAPTER Figure 1-1. Thesis methodology CHAPTER Figure 2-1. The storage pyramid Figure 2-2. Components of a typical disk drive Figure 2-3. Longitudinal and helical scan tape technologies Figure 2-4. Inode and indirect blocks in the Berkeley Fast File System Figure 2-5. Sequential file access patterns in parallel programs CHAPTER Figure 3-1. The NCAR network Figure 3-2. Latency to the first byte for various MSS devices Figure 3-3. Daily variations in MSS access rates Figure 3-4. Weekly variations in MSS access rates Figure 3-5. Long-term variations in MSS transfer rates Figure 3-6. MSS interreference intervals Figure 3-7. File reference count distribution Figure 3-8. MSS file interreference intervals Figure 3-9. Size distribution of files transferred between the MSS and the Cray Figure MSS static file size distribution Figure Distribution of data and files by directory size CHAPTER Figure 4-1. Typical hardware running the RAMA file system Figure 4-2. Typical hardware running conventional MPP file systems Figure 4-3. A RAMA disk line and line descriptor Figure 4-4. Intrinsic metadata placement options Figure 4-5. A file read in RAMA Figure 4-6. A file write in RAMA Figure 4-7. Disk line reorganization Figure 4-8. Scheme for insuring consistency after a RAMA crash CHAPTER Figure 5-1. Threads and resources in the RAMA simulator Figure 5-2. Sequence of operations for a single disk request Figure 5-3. Disk seek time curve for ST31200N Figure 5-4. Mesh network topology Figure 5-5. Star network topology Figure 5-6. Actions simulated for a read or write request Figure 5-7. Algorithm for the strided sequential access application Figure 5-8. Out-of-core LU decomposition vi

8 Figure 5-9. Block-cyclic layout of data for GATOR CHAPTER Figure 6-1. Effects of varying link bandwidth in a mesh interconnection network on RAMA read performance Figure 6-2. Effects of varying link bandwidth in a mesh interconnection network on RAMA write performance Figure 6-3. Effects of varying link bandwidth in a star interconnection network on RAMA performance Figure 6-4. Effects of varying network message latency on RAMA performance Figure 6-5. Components of disk latency in a single RAMA disk request Figure 6-6. Effects of varying disk track capacity on RAMA performance Figure 6-7. Effects of varying disk rotational speed on RAMA performance Figure 6-8. The effects of average and maximum disk head seek time on RAMA performance Figure 6-9. Acceleration and travel components of disk read/write head seek time Figure Matrix decomposition performance with varying memory sizes Figure Effects of varying the amount of consecutive file data stored per disk Figure RAMA performance with varying numbers of disks per MPP node Figure RAMA read performance for small files Figure Projected RAMA performance using future technologies CHAPTER Figure 7-1. Node sequential and iteration sequential request ordering Figure 7-2. Comparison of raw bandwidth in RAMA and striped file systems Figure 7-3. Interaction of stripe size and application stride Figure 7-4. Performance of a synthetic workload doing write-dominated strided access under RAMA and striped file systems Figure 7-5. I/O done by matrix decomposition on a 64 processor MPP Figure 7-6. Execution time for LU decomposition under RAMA and striped file systems Figure 7-7. Execution time for LU decomposition with an alternate data layout Figure 7-8. GATOR performance under striping and RAMA Figure 7-9. Sequential request distribution in a striped file system Figure Poor distribution of requests to striped disks for LU decomposition Figure Distribution of requests to RAMA disks during the read of a 32 GB file Figure Distribution of requests to RAMA disks for LU decomposition Figure Temporal distribution of MPP file request streams to a striped file system Figure Effectiveness of the RAMA hash function at eliminating poor temporal distribution Figure Interconnection network utilization under a striped file system Figure Interconnection network utilization under RAMA CHAPTER vii

9 List of Tables CHAPTER CHAPTER Table 2-1. Trends in disk technology Table 2-2. Characteristics of various tertiary storage technologies CHAPTER Table 3-1. Information in a single trace record Table 3-2. Overall NCAR trace statistics Table 3-3. NCAR MSS access errors Table 3-4. NCAR MSS overall statistics CHAPTER CHAPTER Table 5-1. Parameters for the simulator s disk model Table 5-2. Simulation parameters for the interconnection network CHAPTER Table 6-1. RAMA s expected load on various network configurations Table 6-2. Parameters for simulated future disks CHAPTER CHAPTER viii