A Converged HPC & Big Data System for Nontraditional and HPC Research

Transcription

1 A Converged HPC & Big Data System for Nontraditional and HPC Research Nick Nystrom Sr. Director of Research Overview March 15, Pittsburgh Supercomputing Center

2 2 Bridges, a new kind of supercomputer at PSC, is designed to converge HPC and Big Data, empower new research communities, bring desktop convenience to high-end computing, expand remote access, and help researchers to work more intuitively. Funded by a $17.2M NSF award, Bridges emphasizes usability, flexibility, and interactivity Available at no charge for open research and course support and by arrangement to industry Popular programming languages and applications: R, MATLAB, Python, Hadoop, Spark, 846 compute nodes containing 128GB (800), 3TB (42), and 12TB (4) of RAM each; 96 GPUs Dedicated nodes for persistent databases, web servers, and distributed services The first deployment of the Intel Omni-Path Architecture fabric

3 Acquisition and operation of Bridges are made possible by the National Science Foundation through award #ACI ($17.2M): Bridges: From Communities and Data to Workflows and Insight delivered Bridges All trademarks, service marks, trade names, trade dress, product names, and logos appearing herein are the property of their respective owners. 3

4 For open research: Accessing Bridges Allocations on Bridges are available at no charge through XSEDE, For industrial affiliates, Pennsylvania-based researchers, and others to foster discovery and innovation and broaden participation in data-intensive computing: Up to 10% of Bridges capacity is available on a discretionary basis. Contact Nick Nystrom, Bridges PI, at nystrom@psc.edu. 4

5 Motivating Use Cases Data-intensive applications & workflows Gateways the power of HPC without the programming Shared data collections & related analysis tools Cross-domain analytics } Deep learning Graph analytics, machine learning, genome sequence assembly, and other large-memory applications Scaling beyond the laptop Scaling research to teams and collaborations In-memory databases Optimization & parameter sweeps Distributed & service-oriented architectures Data assimilation from large instruments and Internet data Leveraging an extensive collection of interoperating software Research areas that haven t used HPC New approaches to traditional HPC fields (e.g., machine learning) Coupling applications in novel ways Leveraging large memory, GPUs, and high bandwidth 5

6 Objectives Overview Deliver HPC to new users and research communities. Bring the power of HPC to Big Data. Streamline access and provide burst capability to campuses. Approach Unify compute and storage on a single, high-performance fabric to unify HPC and Big Data. Embrace heterogeneity to enable expression of uniquely efficient application frameworks. Very large coherent shared memory 12TB, 3TB, and 128GB of RAM per server supports productivity, transparent scaling, and important applications in (for example) data analytics, statistics, the life sciences, engineering, and machine learning. State-of-the-art GPUs support deep learning and acceleration of a broad range of applications. Implement a uniquely flexible environment featuring interactivity, gateways, databases, distributed services, high-productivity programming languages and frameworks, and virtualization and containers. 6

7 Interactivity Interactivity is the feature most frequently requested by nontraditional HPC communities. Interactivity provides immediate feedback for doing exploratory data analytics and testing hypotheses. Bridges offers interactivity through a combination of shared and dedicated resources to maximize availability while accommodating different needs. 7

8 Gateways and Tools for Building Them Gateways provide easy-to-use access to Bridges HPC and data resources, allowing users to launch jobs, orchestrate complex workflows, and manage data from their browsers. Provide HPC-as-a-Service Extensive use of VMs, databases, and distributed services Galaxy (PSU, Johns Hopkins) The Causal Web (Pitt, CMU) GenePattern (Broad Institute) 8

9 Virtualization and Containers Virtual Machines (VMs) enable flexibility, security, customization, reproducibility, ease of use, and interoperability with other services. User demand is for custom database and web server installations to develop data-intensive, distributed applications and containers for custom software stacks and portability. Bridges leverages OpenStack to provision resources, between interactive, batch, Hadoop, and VM uses. 9

10 High-Productivity Programming Supporting languages that communities already use is vital for them to apply HPC to their research questions. 10

11 Spark, Hadoop & Related Approaches Bridges large memory offers unique benefits to Spark applications. Bridges enables workflows that integrate Spark/Hadoop, HPC, GPU, and/or large shared-memory components. 11

12 Conceptual Architecture Users, XSEDE, campuses, instruments Web Server nodes Database nodes Data Transfer nodes Login nodes Parallel File System Intel Omni-Path Architecture fabric Management nodes ESM Nodes 12TB RAM 4 nodes LSM Nodes 3TB RAM 42 nodes RSM Nodes 128GB RAM 800 nodes, 48 with GPUs 12

13 Bridges Virtual Tour: 20 Storage Building Blocks, implementing the parallel Pylon storage system (10 PB usable) 4 MDS nodes 2 front-end nodes 2 boot nodes 8 management nodes 6 core Intel OPA edge switches: fully interconnected, 2 links per switch Intel OPA cables 4 HPE Integrity Superdome X (12TB) compute nodes each with 2 OPA IB gateway nodes 42 HPE ProLiant DL580 (3 TB) compute nodes 12 HPE ProLiant DL380 database nodes 6 HPE ProLiant DL360 web server nodes 20 leaf Intel OPA edge switches Purpose-built Intel Omni-Path Architecture topology for data-intensive HPC 32 RSM nodes, each with 2 NVIDIA Tesla P100 GPUs 800 HPE Apollo 2000 (128GB) compute nodes 16 RSM nodes, each with 2 NVIDIA Tesla K80 GPUs 13

14 Node Types Type RAM Phase n CPU / GPU / other Server 14 ESM 12TB b Intel Xeon E v3 (18c, 2.3/3.1 GHz, 45MB LLC) HPE Integrity 12TB c Intel Xeon E v4 (22c, 2.2/3.3 GHz, 55MB LLC) Superdome X 3TB b Intel Xeon E v3 (16c, 2.2/3.2 GHz, 40 MB LLC) LSM 3TB c Intel Xeon E v4 (20c, 2.1/3.0 GHz, 50 MB LLC) RSM 128GB b Intel Xeon E v3 (14c, 2.3/3.3 GHz, 35MB LLC) HPE ProLiant DL580 RSM-GPU 128GB b Intel Xeon E v3 + 2 NVIDIA Tesla K80 HPE Apollo GB c Intel Xeon E v4 (16c, 2.1/3.0 GHz, 40MB LLC) + 2 NVIDIA Tesla P100 DB-s 128GB b Intel Xeon E v3 + SSD HPE ProLiant DL360 DB-h 128GB b Intel Xeon E v3 + HDDs HPE ProLiant DL380 Web 128GB b Intel Xeon E v3 HPE ProLiant DL360 Other a 128GB b Intel Xeon E v3 Gateway 64GB b Intel Xeon E v3 (14c, 2.0/3.0 GHz, 35MB LLC) 64GB c Intel Xeon E v3 128GB b Intel Xeon E v3 (12c, 2.5/3.3 GHz, 30 MB LLC) Storage 256GB c Intel Xeon E v4 (14c, 2.4/3.3 GHz, 35 MB LLC) Total TB 908 a. Other nodes = front end (2) + management/log (8) + boot (4) + MDS (4) b. DDR c. DDR HPE ProLiant DL360, HPE ProLiant DL380 HPE ProLiant DL380 Supermicro X10DRi

15 System Capacities Compute nodes only, per node type RSM RSM GPU LSM ESM Total Peak fp64 (Tf/s) ,346 RAM (TB) Local disk (TB) 6, ,328 Parallel file system: usable space (PB) 10 15

16 Intel Omni-Path Architecture (OPA) Bridges is the first production deployment of Omni-Path Omni-Path connects all nodes and the shared filesystem, providing Bridges and its users with: 100 Gbps line speed per port; 25 GB/s bidirectional bandwidth per port Measured 0.93μs latency, GB/s/dir 160M MPI messages per second 48-port edge switch reduces interconnect complexity and cost HPC performance, reliability, and QoS OFA-compliant applications supported without modification Early access to this new, important, forward-looking technology Bridges deploys OPA in a two-tier island (leaf-spine) topology developed by PSC for cost-effective, data-intensive HPC 16

17 Data Management Pylon: A large, central, high-performance storage system 10 PB usable Visible to all compute nodes Currently implemented as two complementary file systems: /pylon1: Lustre, targeted for $SCRATCH use; non-wiped directories available by request /pylon2: SLASH2, targeted for large datasets, community repositories, and distributed clients Distributed (node-local) storage Enhance application portability Improve overall system performance Improve performance consistency to the shared filesystem Aggregate 7.2 PB (6 PB non-gpu RSM: Hadoop, Spark) 17

18 Database and Web Server Nodes Dedicated database nodes power persistent relational and NoSQL databases Support data management and data-driven workflows SSDs for high IOPs; HDDs for high capacity (examples) Dedicated web server nodes Enable distributed, service-oriented architectures High-bandwidth connections to XSEDE and the Internet 18

19 GPU Nodes Bridges GPUs are accelerating both deep learning and simulation codes Phase 1: 16 nodes, each with: 2 NVIDIA Tesla K80 GPUs (32 total) 2 Intel Xeon E v3 (14c, 2.3/3.3 GHz) 128GB DDR RAM Phase 2: +32 nodes, each with: 2 NVIDIA Tesla P100 GPUs (64 total) 2 Intel Xeon E v4 (16c, 2.1/3.0 GHz) 128GB DDR RAM Kepler architecture 2496 CUDA cores (128/SM) 7.08B transistors on 561mm 2 die (28nm) 2 24 GB GDDR5; GB/s 562 MHz base 876 MHz boost 2.91 Tf/s (64b), 8.73 Tf/s (32b) Pascal architecture 3584 CUDA cores (64/SM) 15.3B transistors on 610mm 2 die (16nm) 16GB CoWoS HBM2 at 720 GB/s w/ ECC 1126 MHz base 1303 MHz boost 4.7 Tf/s (64b), 9.3 Tf/s (32b), 18.7 Tf/s (16b) Page migration engine improves unified memory 64 P100 GPUs 600 Tf/s (32b) 19

20 20 Examples of Research that Bridges is Enabing

21 The Causal Web Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence Browser-based UI Prepare and upload data Run causal Internet discovery algorithms Visualize results Web node VM Apache Tomcat Messaging Database node VM Authentication Data mgmt. Provenance Other DBs Execute causal discovery algorithms Omni-Path Pylon filesystem TCGA fmri Algorithm execution: FGS continuous and discrete, GFCI, and other algorithms, building on TETRAD LSM Node (3TB) Memoryresident datasets ESM Node (12TB) 21

22 Galaxy Galaxy provides a powerful and flexible platform for creating and running reproducible scientific workflows, mostly focused on bioinformatics applications, but also adopted by other domains. Some specialize, e.g., metagenomics or proteomics Galaxy Main at TACC now routes large-memory Trinity assembly jobs to Bridges A second, Docker-based Galaxy instance at PSC servers a local user community 22

23 Improved SNP Detection in Metagenomic Populations Wenxuan Zhong, Xin Xing, and Ping Ma, Univ. of Georgia Assembled 378 Gigabase pairs (Gbp) of gut microbial DNA from normal and diabetic patients Massive metagenome assembly took only 16 hours using an MPI-based metagenome assembler, Ray, on 20 Bridges RM nodes connected by Omni-Path Identified 2,480 species-level clusters in assembled sequence data Ran MetaGen clustering software across 10 RM nodes, clustering 500,000 contiguous sequences in only 14 hours Currently testing new likelihood-based statistical method for fast and accurate SNP detection from these metagenomic sequencing data The team is now using Bridges to test a new statistical method on the sequence data to identify critical differences in gut microbes associated with diabetes. Environmental Shotgun Sequencing (ESS). (A) Sampling from habitat; (B) filtering particles, typically by size; (C) DNA extraction and lysis; (D) cloning and library; (E) sequence the clones; (F) sequence assembly. By John C. Wooley, Adam Godzik, Iddo Friedberg - CC BY 2.5, 23

24 Characterizing Diverse Microbial Ecosystems from Terabase-Scale Metagenomic Data Brian Couger, Oklahoma State University Assembled 11 metagenomes sampled from diverse sources, comprising over 3 trillion bases of sequence data Including a recent massive assembly of 1.6 Tbp of metagenomic data from an oil sands tailings pond, a bioremediation target Excellent performance of MPI-based Ray assembler on 90 RM nodes completed assembly in only 4.25 days Analysis of assembled data in progress to characterize organisms present in these diverse environments and identify new microbial phyla Oil sands tailings pond. By NASA Earth Observatory. Public Domain, 24

25 Creation of World s Largest k-mer Database Rachid Ounit and Chris Mason, Cornell University Created a database of 153 billion species-specific nucleotide sequences (k-mers) Analyzed entire NCBI Reference Sequence (RefSeq) archive containing over 15k species to create the k-mer database Required a massive in-memory hash table Computation took 24 days and 4.8 TB of RAM on a 12 TB node of Bridges Allows rapid identification and classification of species in metagenomics samples By Madprime - Own work, CC BY-SA 3.0, 25

26 Applying Deep Learning to Connectomics Goal: Explore the potential of deep learning to automate segmentation of high-resolution scanning electron microscope (SEM) images of brain tissue and the tracing of neurons through 3D volumes to automate generation of the connectome, a comprehensive map of neural connections. Motivation: This project builds on an ongoing collaboration between PSC, Harvard University, and the Allen Institute for Brain Science, through which we have access to high-quality raw and labeled data. The SEM data volume for mouse cortex imaging is ~3TB/day, and data processing is currently human-intensive. Forthcoming camera systems will increase data bandwidth by a factor of 65. Images courtesy Florian Engert, David Hildebrand, and their students at the Center for Brain Science, Harvard. Datasets: Zebrafish larva ( ), mouse,. Data volume: mouse brain 430mm 3 ~1.4 exabytes. Collaborators: Ishtar Nyawĩra, Iris Qian, Annie Zhang, Joel Welling, John Urbanic, Art Wetzel, Nick Nystrom 26

27 Some of the Deep Learning Projects Using Bridges Deep Learning of Game Strategies for RoboCup, Manuela Veloso (CMU) Automatic Evaluation of Scientific Writing, Diane Litman (U. of Pittsburgh) Image Classification Applied in Economic Studies, Param Singh (CMU) Exploring Stability, Cost, and Performance in Adversarial Deep Learning, Matt Fredrikson (CMU) Enabling Robust Image Understanding Using Deep Learning, Adriana Kovashka (U. of Pittsburgh) Preparing Grounds to Launch All-US Students Kaggle Competition on Drug Prediction, Gil Alterovitz (Harvard Medical School/Boston Children s Hospital) Deep Learning the Gene Regulatory Code, Shaun Mahony (Penn State) Automatic Building of Speech Recognizers for Non-Experts, Florian Metze (CMU) Development of a Hybrid Computational Approach for Macroscale Simulation of Exciton Diffusion in Polymer Thin Films, Based on Combined Machine Learning, Quantum- Classical Simulations and Master Equation Techniques, Peter Rossky (Rice U.) Developing Large-Scale Distributed Deep Learning Methods for Protein Bioinformatics, Junbo Xu (Toyota Technological Institute at Chicago) Education Allocation for the Course Unstructured Data & Big Data: Acquisition to Analysis, Dokyun Lee (CMU) Deciphering Cellular Signaling System by Deep Mining a Comprehensive Genomic Compendium, Xinghua Lu (U. of Pittsburgh) Quantifying California Current Plankton Using Machine Learning, Mark Ohman (Scripps Institution of Oceanography) Petuum, a Distributed System for High- Performance Machine Learning, Eric Xing (CMU) Automatic Pain Assessment, Michael Reale (SUNY Polytechnic Institute) Learning to Parse Images and Videos, Deva Ramanan (CMU) Deep Recurrent Models for Fine-Grained Recognition, Michael Lam (Oregon State) 27

28 Thermal Hydraulics of Next-Generation Gas-Cooled Nuclear Reactors PI Mark Kimber, Texas A&M University 28 Completed 3 Large Eddy Simulations of turbulent thermal mixing of helium coolant in the lower plenum of the Gen IV High- Temperature Gas-cooled Reactor (HTGR) Performed using OpenFOAM, a open-source CFD code Simulations are for scaled-down versions of representative sections of the HTGR lower plenum, each containing 7 support posts and 6 high-reynolds number jets in a cross-flow Each simulation involves a high-quality block-structured mesh with 16+ million cells, and was run on 20 regular Bridges nodes (560 cores) for ~10 million time steps (~1 second of simulated time) Helps to understand the turbulent thermal mixing in great detail, as well as temperature distributions and hotspots on support posts Anirban Jana (PSC) is co-pi and computational lead for this DOE project Acknowledgements: DOE-NEUP (Grant nos. DE-NE ) NSF-XSEDE (Grant no. CTS160002) Distribution of vorticity in a representative unit cell of the HTGR lower plenum containing 7 support posts and 6 jets in a crossflow (right to left). Turbulent mixing of core coolant jets in the lower plenum can cause temperature fluctuations and thermal fatigue of the support posts.

29 Simulations of Nanodisc Systems Jifey Qi and Wonpil Im, Lehigh University Using NAMD on Bridges NVIDIA P100 GPU nodes to simulate a number of nanodisc systems to be used as examples for the Nanodisc Builder module in CHARMM-GUI For a system size of 263,888 atoms, the simulation speed on one P100 node (6.25 ns/day) is faster than on three dual-cpu nodes (4.09 ns/day) CHARMM-GUI provides a web-based graphical user interface to generate various molecular simulation systems and input files (for CHARMM, NAMD, GROMACS, AMBER, GENESIS, LAMMPS, Desmond, OpenMM, and CHARMM/OpenMM) to facilitate and standardize the usage of common and advanced simulation techniques A nanodisc system showing two helical proteins warping around a discoidal lipid bilayer. 29

30 Reconstructing Large Historical Social Networks Christopher Warren, Carnegie Mellon University Six Degrees of Francis Bacon: Employs statistical graph learning and interactive web interfaces to reconstruct and communicate the social networks of Britain from approximately Analyzes the 58,625 biographical entries of the Oxford Dictionary of National Biography (ODNB), a 62M-word corpus produced by 10,000 scholars. Employs a modified Poisson Graphical Lasso method, confidence estimation, chronological filter and probabilistic techniques for name disambiguation, and expert validation. The methods are applicable to other historical and contemporary societies and other document collections. Six Degrees of Francis Bacon, Additional resources: Using Supercomputers to Illuminate the Renaissance: Warren, C. N., et al. (2016). "Six Degrees of Francis Bacon: A Statistical Method for Reconstructing Large Historical Social Networks." Digital Humaties Quarterly 10(3). Methods and R code: 30

31 Investigating Economic Impacts of Images and Natural Language in E-commerce Dokyun Lee, CMU Tepper School of Business Security and uncertain quality create challenges for sharing economies Lee et al. studied the impact of high-quality, verified photos for Airbnb hosts 17,000 properties over 4 months Used Bridges GPU nodes and large R jobs Difference-in-Difference (DD) analysis showed that on average, rooms with verified photos are booked 9% more often Separating effects of photo verification from photo quality and room reviews indicates that high photo quality results in $2,455 of additional yearly earnings They found asymmetric spillover effects: on the neighborhood level, there appears to be higher overall demand if more rooms have verified photos 31

32 Leveraging Bridges to Support Anton 2 2 nd -generation molecular dynamics supercomputer in production at PSC Performs MD simulations 100 faster than other resources Allows users to run simulations 4 faster and larger than the previous 1 st -generation Anton system at PSC Allocations on 1 st -generation Anton have resulted in over 140 publications so far Bridges Supports Anton 2 Users Anton 2 has specific resources for analysis of large MD trajectories; simulations require pre-equilibrated MD systems The Anton 2 filesystem will be mounted on Bridges with Omni-Path to facilitate in situ analysis of large trajectories 2016 D.E. Shaw Research. Used with permission D.E. Shaw Research. Used with permission. 32

33 Thank You Questions? 33