Computer Science Section. Computational and Information Systems Laboratory National Center for Atmospheric Research

Transcription

1 Computer Science Section Computational and Information Systems Laboratory National Center for Atmospheric Research

2 My work in the context of TDD/CSS/ReSET Polynya new research computing environment Polynya prototype Building a cluster using existing hardware Deploying and optimizing Lustre parallel file system Workflow support Lustre on RAM disks Provisioning of resources Performance analysis Future work 2 of 15

3 Polynya (Russian: полынья) areas of sea in Arctic or Antarctic regions which remain unfrozen for much of the year Wikipedia Polynya system s mission: Platform for CISL Technology Development Support NCAR Computational Science Evaluation of new products 3 of 15

4 Use existing hardware to build a prototype of the future environment Explore architectural options Maximize performance of nodes and services Gain experience deploying and managing a heterogeneous system Develop practices that can be applied to the full Polynya system 4 of 15

5 admin and login nodes 8 GPU-accelerated nodes 3 large-memory nodes Lustre file system on spinning disks Lustre on RAM disks compilers, MPI, user environment internal and external authentication Xen-based VM hosting Accelerated Compute Nodes Configurable Fat Nodes RAM Disk Parallel Platform Evaluation Node Docks Scratch Disk 5 of 15

6 Metadata Server (MDS) Metadata Target MDT Lustre clients Sun Fire X GB SAS Object Storage Server (OSS) Sun Fire X GB SATA Object Storage Targets OST OST OST OST OST 6 of 15

7 18 TB of usable /scratch space Hard to predict performance Lots of Benchmarking to determine highest performing configuration Tiered Architecture of Lustre OSS Lustre RAIDs Spinning disks Selected configuration: Lustre OSTs are RAID-5 arrays of disks 5 RAID arrays, 9 disks in each RAID Sequential order of disks in RAIDs 64 KB RAID chunk size Single global hot-spare drive 4 1Gbps Ethernet connections 7 of 15

8 Dell PowerEdge R910 1 with 512GB RAM 1 with 1TB RAM X 2 Challenges: Understanding complexity Hardware aspects (NUMA architecture) Kernel, rd module implementation & performance Tiered architecture with RAIDs for small RAM disks Lustre configuration Making right choices from performance perspective Disk size, pre-allocation Number of Lustre OSTs, stripe size and count Finely tuned memory usage Node crashes if all of the memory is used 8 of 15

9 Independent variables: Pre-allocation: yes/no RAM disk size: GB Number of Lustre OSTs: 1 8 Number of I/O streams: 1 32 Benchmarks: sgp_dd, STREAM, IOR Local client tests only (avoiding network bottleneck) Highest aggregate performance configuration: 64 GB disks, pre-allocated, 8 Lustre OSTs, stripe count 8 9 of 15

10 Workflow 1: Lustre Dedicated large-memory nodes Workflow 2: Lustre Dedicated large-memory nodes Managed at administrator level Allan Espinosa s work: Integrated scheduling Schedule data storage on a dedicated RAM disk Schedule data transfers with scheduled Lustre/RAM storage 10 of 15

11 Memory pre-allocation is time-consuming on creation but highly desirable 11 of 15

12 I/O scalability with 8-stripes/writer and pre-allocation: 12 of 15

13 Lustre/disk* 5 OSTs over 1Gpbs network: 110 MB/s (network is the bottleneck currently) Lustre/RAM** Single OST Across all 8 OSTs Write, MB/s 2,880 3,045 Read, MB/s 10,627 8,627 Allan s work: GridFTP benchmarking Kraken to Polynya Lustre/RAM: ~10 MB/s (requires additional tuning) Frost to Polynya Lustre/RAM: ~130 MB/s * 5 Lustre OSTs, 5 stripes/writer, single client, IOR benchmark ** 64 GB RAM disks, 8 Lustre OSTs, 8 I/O streams, IOR benchmark 13 of 15

14 Deploy Lustre/RAM on multiple nodes Benchmark Lustre/RAM with high-bandwidth interconnect Explore further Lustre shrinking/growing capabilities Integrate with data-intensive workflows Enhance scripts to enforce memory limits Enhance Lustre/RAM migration to disk and tape Explore Lustre 2.0 capabilities, i.e. using storage pools to provide single namespace Integrate new equipment and move from the prototype to the complete research systems environment 14 of 15

15 Thank you! Questions? Dmitry Duplyakin University of Colorado at Boulder 15 of 15