HPC Growing Pains. IT Lessons Learned from the Biomedical Data Deluge

Transcription

1 HPC Growing Pains IT Lessons Learned from the Biomedical Data Deluge John L. Wofford Center for Computational Biology & Bioinformatics Columbia University

2 What is? Internationally recognized biomedical computing center. Broad range of computational biomedical and biology research, from biophysics to genomics. More than 15 Labs and nearly 200 faculty, staff, students and postdocs across multiple campuses and 8 departments. IT staff of 8, covering everything from desktop to HPC & datacenter. Affiliates

3 HPC Growing Pains Over the past 3 years we have grown our HPC resources by an order of magnitude, driven largely by genomic data and processing demands. Before (2008) After (2012) Total CPU-cores ~500 ~4500 Largest cluster: CPU 400 core ~4000 core Largest cluster: Memory 800 GB 8 TB Annual CPU-hr 2 M CPU-hrs >50 M CPU-hrs Average Daily Active Users Storage capacity 30 TB ~ 1PB Data Center space 800 sq.ft sq.ft.

4 Outline IT Lessons Learned from the Biomedical Data Deluge I. Intro: Biomedical data growth II. Storage challenges II.1. Performance II.2. Capacity II.3. Data integrity III. Conclusions

5 What data deluge? A few stock facts: With a collective 269 petabytes of data, education was among the U.S. economy s top 10 sectors storing the largest amount of data in 2009, according to a McKinsey Global Institute survey. The world will generate 1.8 zettabytes of data this year alone, according to IDC s 2011 Digital Universe survey. Worldwide data volume is growing a minimum of 59% annually, a Gartner report estimates, outrunning Kryder s law for disk capacity per cost growth. Biomedical data primarily driven by gene sequencing is growing dramatically faster than industry average and Kryder s law....not only does that data need to be stored, it needs to be heavily analyzed, demanding both performance and capacity.

6 Sequence data production rates

7 From Kahn S.D. On the future of genomic data. Science 2011;331:

8 3000 C2B2 data growth, June 08 - March 12 Data Usage (TB) Raw capacity Raw requirement (50% overhead) Usage trend 750 Usage (logical) Industry trend (59% annual growth) Months since June 08

9 The challenge: Design a system that can: 1. Perform well enough to analyze the data (i.e. stand up to a top500 supercomputer); 2. Scale from from TeraBytes to PetaBytes (without having to constantly rebuild); 3. Protect important data (from crashes, users and floods);

10 Performance For parallel processes you need parallel file access. We have 4000 CPUs working around the clock on analyzing data; we want to keep them all fed with data all of the time. Our workload is random and not well behaved. It s notoriously difficult to design for this kind of workload. Ideally, we want a solution that can be flexible as our workloads change. The more disks we have spinning in parallel, the better the performance. We re going to need a lot of disks. Using a rough heuristic, 1 disk per compute would mean ~500 disks. But, to make that many disks useful, we re going to need a lot of processing and network capabilities.

11 Traditional NAS Architecture Single NAS head with multiple disk arrays Netapp, Bluearc, EMC... Pro Support: Time-tested architecture with many major, competing vendors. Capacity scaling: relatively easy on modern NAS. Con Performance scaling: is difficult & unpredictable. Management: Storage pools must be managed and tuned. Reliability: NAS head provides single failure point. Traditional NAS Single controller, many arrays NAS Head Processes network file requests (NFS, CIFS, etc). Manage SAN pools. CPU Cache Network SAN or Direct Attached Interconnect (FC,, ) Disk arrays JBODs or RAID... SAN Storage SAN SSDStorage SAN SSDStorage SSD Disk Disk Disk Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,...

12 Clustered NAS architecture Single filesystem distributed across many s. Isilon, Panasas, Gluster,... Pro Capacity scaling: new s automatically integrate. Performance scaling: new s add CPU, Cache and network performance. Reliability: most architectures can survive multiple failures. Con Support: Relatively new technology. Few vendors (but rapidly growing). High speed backend network (transfers data between s). Clustered NAS A single filesystem is presented by multiple s which each process filesystem requests. Clustered NAS CPU Cache Network Disk pool Disk Disk Disk Clustered NAS CPU Cache Network Disk pool Disk Disk Disk Clustered NAS CPU Cache Network Disk pool Disk Disk Disk Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,... Storage client * Cluster, Server, Desktop,...

13 Clustered NAS doesn t solve everything... In late 2009 we had scaled to where we thought we should be, but our system was unresponsive, with constant, very high load. More puzzling, was that the load didn t seem to have anything correlation with the network, disk throughput, or even load on the compute cluster.

14 What we found (With the help of good analytics)

15 Namespace reads Namespace operations consume CPU, and waste I/O capabilities. It s common in biomedical data to have thousands, or millions of small files on a project. We have ~500M files, with an average filesize of less than 8 kb. Many genome databases are directory structures of flat files that get indexed by filename (NCBI, for instance, hosts > 20k files in their databases). Our system was thrashing, and we weren t getting a lot of I/O... the 40% namespace reads were killing our performance. Distribution of Protocol Operations Write 15% Other 13% Read 32% Namespace Read 40%

16 SSD Accelerated Namespace Sold Sate Disks provide fast seek time for namespace reads. Namespace reads are seek intensive (not throughput intensive). SSDs generally have > 40x seek time of spinning disks. We were able to spread our filesystem metadata over SSDs on our s, dramatically increasing namespace performance and decreasing system load. We experienced an immediate, ~8x increase in filesystem responsiveness, and overall system performance increase. Filesystem metadata SSD Accelerated Namespace Namespace data live on SSDs (very low seek time), Ordinary data lives on ordinary disks. SSD Enabled Node Metadata SSD Data SSD Enabled Node Metadata SSD Disk Data SSD Enabled Node Metadata SSD Disk Data Disk Disk Disk Disk

17 Capacity How we scaled a single filesystem from 168 TB to 1 PB 1. While the High-performance Clustered NAS naturally scales capacity as well as performance, it s an expensive way to build capacity. 2. We don t want to have the big filesystem and the fast filesystem. We want everyone to see the same files from everywhere. 3. High-performance systems benefit from uniform hardware. Capacity scaling benefits from being able to use the latest, biggest, densest disks. 4. In fact, typically a clustered NAS is made of entirely uniform hardware, so how do you update without time consuming data migrations?

18 Multi-tiered Clustered NAS Single filesystem distributed across many pools of s. Multiple pools of s share a single filesystem namespace. Different pools can have different performance/capacity, allowing for independent scaling of capacity and performance. New pools can be added, and old removed, allowing seamless upgrades. All pools are active, so s configured for large capacity can still serve data to lowdemand devices. Rule based data migration Multi-tiered Clustered NAS multiple pools, single namespace High-speed pool compute clusters, sequencers, etc.... SSD-accelerated SSD-accelerated SSD SSD-accelerated SSD SSD SSD High-capacity pool infrastructure servers, desktops, etc.... Nearline Nearline SSD Nearline SSD SSD Shared backend network High-Performance Compute Clusters Blade chassis Compute Compute Data generating research equipment (e.g. gene sequencers) Server Infrastructure Server Desktops & Workstations

19 Evolution of a filesystem Q3 1. Merged pools 2. New s (SSD) Cap: 672 TB 2010 Q4 Swap original s for new s Cap: 648 TB 2011 Capacity upgrade (denser s) Cap: 984 TB Raw capacity Q1 new s (separate cluster) 496 TB Requirement new s Cap: 276 TB 2008 inception Cap: 168 TB

20 Caveat: A single namespace has challenges Since the namespace doesn t refresh, it tends to grow and grow. We currently have > 500 Million files in our filesystem. While it s nice to have all of your files in one space, it takes a lot of effort to keep it organized. In initial deployment, spent roughly 30x longer planning filesystem structure than deploying hardware. If you plan your filesystem poorly, it could take a long time to relocate or remove all of those files.

21 Data Integrity How do you protect large-scale, important data from users, glitches & floods? 1. Users: Oops! I didn t mean to delete that! 2. Glitches: Error$mounting:$mount:$ wrong$fs$type,$bad$option,$bad$ superblock$on$/dev/ Floods: Who installed the water main over the data center?!

22 vs. Disk-to-Disk is dead. Right? backup D-t-D NAS File server File server Fast Easy to maintain Reliable Backup server Library Backup server Cheap Low-power Long shelf life Disk-to-disk backup Primary NAS Secondary NAS

23 Neither option is ideal on its own. Using only tape is impractical. LTO5 can write 1TB in ~ 6hr, or 200 TB in 50 days. You can split across drives, but this becomes a management nightmare. Using only disk is cost prohibitive. A complete disk system vs. a complete tape system. Plus: you need to keep disk powered. there s no easy (or safe) way to archive disks. it has to grow faster than primary (if you want historical archives).

24 Our middle ground Snapshots + replication + tape = protection from: Users: frequent snapshots on the source provide easy oops recovery to the user. Glitches: replication provides shortterm rapid recovery. Added snapshots extend replication archives to the mid-term (~6 mo.). Floods: backup provides cheap, reliable archival of data, for largescale disaster recovery (or important files from 06). Leaving backup windows flexible keeps tape manageable. Primary cluster Easy user recovery with short-term snapshots Replication cluster Live replication of critical data, with historical snapshots Replication Replication Replication Library Offsite DR Archival... Storage Storage SSD Storage SSD SSD Frequent replication (daily) Data backup path Disk Disk Disk Lazy archival (~ 6 mo.) 1 copy per year Long-term archival

25 The big picture: Multi-tiered scale-out architecture from HPC Infrastructure to the Desktop High-Performance Compute Clusters Blade chassis Compute Compute Blade chassis Compute Compute 10 Gbps Aggregation network Rule based data migration Primary Storage Cluster multiple pools, single filesystem High-speed pool compute clusters, sequencers, etc.... SSD-accelerated SSD-accelerated SSD SSD-accelerated SSD SSD SSD Nearline pool infrastructure servers, desktops, etc.... Data generating research equipment (e.g. gene sequencers) Server Infrastructure Physical servers Nearline Nearline SSD Nearline SSD SSD Virtualization Infrastructure Backup Infrastructure Long-term archival Library Offsite DR Archival Replication cluster Live replication of critical data Replication Replication Replication Desktops & Workstations

26 The big picture: Multi-tiered scale-out architecture from HPC Infrastructure to the Desktop High-Performance Compute Clusters Blade chassis Compute Compute Blade chassis Compute Compute 10 Gbps Aggregation network Rule based data migration Primary Storage Cluster multiple pools, single filesystem High-speed pool compute clusters, sequencers, etc.... SSD-accelerated SSD-accelerated SSD SSD-accelerated SSD SSD SSD Nearline pool infrastructure servers, desktops, etc.... Data generating research equipment (e.g. gene sequencers) Server Infrastructure Physical servers Nearline Nearline SSD Nearline SSD SSD Virtualization Infrastructure Backup Infrastructure Long-term archival Library Offsite DR Archival Replication cluster Live replication of critical data Replication Replication Replication Desktops & Workstations

27 The big picture: Multi-tiered scale-out architecture from HPC Infrastructure to the Desktop High-Performance Compute Clusters Blade chassis Compute Compute Blade chassis Compute Compute 10 Gbps Aggregation network Rule based data migration Primary Storage Cluster multiple pools, single filesystem High-speed pool compute clusters, sequencers, etc.... SSD-accelerated SSD-accelerated SSD SSD-accelerated SSD SSD SSD Nearline pool infrastructure servers, desktops, etc.... Data generating research equipment (e.g. gene sequencers) Server Infrastructure Physical servers Nearline Nearline SSD Nearline SSD SSD Virtualization Infrastructure Backup Infrastructure Long-term archival Library Offsite DR Archival Replication cluster Live replication of critical data Replication Replication Replication Desktops & Workstations

28 Conclusion Putting it all together. With Clustered NAS and SSD acceleration, we re regularly seeing filesystem throughput in excess of 10 Gbps and IOPs well over 500k without an issue. So far we ve managed to stay ahead of our data-growth curve with multi-tiered. We plan to at least double capacity in the next 6-12 months with no major architectural changes. With combination of snapshots, disk-to-disk replication and tape, we re getting daily backups of all important data as well as longterm archivals. Thank you! Questions?