A Converged HPC & Big Data Architecture in Production

Transcription

1 A Converged HPC & Big Data Architecture in Production Nick Nystrom Sr. Director of Research & Bridges PI/PD HP-CAST Invited Keynote November 11, 2016 Salt Lake City 2016 Pittsburgh Supercomputing Center

2 Bridges, a new kind of supercomputer at PSC, is designed to empower new research communities, converge HPC and High Performance Data Analytics (HPDA), bring desktop convenience to high-end computing, expand campus access, and help researchers to work more intuitively. Bridges entered production in July, Funded by a $17.2M NSF award, Bridges emphasizes usability, flexibility, and interactivity Available at no charge for open research and course support Popular programming languages and applications: R, MATLAB, Python, Hadoop, Spark, 846 compute nodes: 128GB (800), 3TB (42) & 12TB (4) of RAM each; 96 GPUs (64 P100, 32 K80) Dedicated nodes for persistent databases, web servers, and distributed services The first deployment of Intel s Omni-Path Architecture fabric

3 Acquisition and operation of Bridges are made possible by the $17.2M National Science Foundation award #ACI : Bridges: From Communities and Data to Workflows and Insight is delivering Bridges All trademarks, service marks, trade names, trade dress, product names, and logos appearing herein are the property of their respective owners. 3

4 Gaining Access to Bridges Bridges is allocated through XSEDE: Startup Allocation Can request anytime; requires only a brief proposal/abstract Up to 50,000 core-hours on RSM and GPU (128GB) nodes, XSEDE calls these 1,000 TB-hours on LSM (3TB) and ESM (12TB) nodes, Service Units (SUs) and a user-specified (in GB) amount of storage on Pylon Can request XSEDE ECSS (Extended Collaborative Support Service) Research Allocation (XRAC) Appropriate for larger requests; can request ECSS Community allocations can support gateways Can be up to millions to tens of millions of SUs Quarterly submission windows: March 15 April 15, June 15 July 15, September 15 October 15, December 15 January 15 Educational Allocations To support use of Bridges for courses Up to 10% of Bridges SUs are available on a discretionary basis to industrial affiliates, Pennsylvania-based researchers, and others to foster discovery and innovation and broaden participation in data-intensive computing. 4

5 Data Infrastructure for the National Advanced Cyberinfrastructure Ecosystem Bridges is a new resource on XSEDE and can interoperate with other XSEDE resources, Advanced Cyberinfrastructure (ACI) projects, campuses, and instruments nationwide. Examples: Reconstructing brain circuits from highresolution electron microscopy Carnegie Mellon University s Gates Center for Computer Science 5 High-throughput genome sequencers Social networks and the Internet Data Infrastructure Building Blocks (DIBBs) Data Exacell (DXC) Integrating Geospatial Capabilities into HUBzero Building a Scalable Infrastructure for Data- Driven Discovery & Innovation in Education Other DIBBs projects Other ACI projects Temple University s new Science, Education, and Research Center

6 Motivating Use Cases (examples) Data-intensive applications & workflows Gateways the power of HPC without the programming Shared data collections & related analysis tools Cross-domain analytics Deep learning Graph analytics, machine learning, genome sequence assembly, and other large-memory applications Scaling beyond the laptop Scaling research to teams and collaborations In-memory databases Optimization & parameter sweeps Distributed & service-oriented architectures Data assimilation from large instruments and Internet data Leveraging an extensive collection of interoperating software Converging HPC and Big Data Research areas that haven t used HPC New approaches to traditional HPC fields, for example, machine learning Coupling applications in novel ways Leveraging large memory, GPUs, and high bandwidth 6

7 Objectives and Approach Bring HPC to nontraditional users and research communities. Apply HPC effectively to big data. Bridge to campuses to streamline access and provide cloud-like burst capability. Leveraging PSC s expertise with shared memory, Bridges will feature 3 tiers of large, coherent shared-memory nodes: 12TB, 3TB, and 128GB. Bridges implements a uniquely flexible environment featuring interactivity, gateways, databases, distributed (web) services, high-productivity programming languages and frameworks, and virtualization, and campus bridging. 7

8 Interactivity Interactivity is the feature most frequently requested by nontraditional HPC communities. Interactivity provides immediate feedback for doing exploratory data analytics and testing hypotheses. Bridges offers interactivity through a combination of shared and dedicated resources to maximize availability while accommodating different needs. 8

9 Gateways and Tools for Building Them Gateways provide easy-to-use access to Bridges HPC and data resources, allowing users to launch jobs, orchestrate complex workflows, and manage data from their browsers. - Provide HPC Software-as-a-Service - Extensive use of VMs, databases, and distributed services Galaxy Col*Fusion portal for the systematic accumulation, integration, and utilization of historical data, from Interactive pipeline creation in GenePattern (Broad Institute) 9

10 Virtualization and Containers Virtual Machines (VMs) enable flexibility, security, customization, reproducibility, ease of use, and interoperability with other services. User demand is for custom database and web server installations to develop data-intensive, distributed applications and containers for custom software stacks and portability. Bridges leverages OpenStack to provision resources, between interactive, batch, Hadoop, and VM uses. 10

11 High-Productivity Programming Supporting languages that communities already use is vital for them to apply HPC to their research questions. 11

12 Spark, Hadoop & Related Approaches Bridges large memory is great for Spark! Bridges enables workflows that integrate Spark/Hadoop, HPC, GPU, and/or large shared-memory components. 12

13 Campus Bridging Federated identity management Burst offload Through a pilot project with Temple University, the Bridges project will explore new ways to transition data and computing seamlessly between campus and XSEDE resources. Federated identity management will allow users to use their local credentials for single sign-on to remote resources, facilitating data transfers between Bridges and Temple s local storage systems. Burst offload will enable cloud-like offloading of jobs from Temple to Bridges and vice versa during periods of unusually heavy load. 13

14 20 Storage Building Blocks, implementing the parallel Pylon storage system (10 PB usable) 4 MDS nodes 2 front-end nodes 2 boot nodes 8 management nodes 6 core Intel OPA edge switches: fully interconnected, 2 links per switch Intel OPA cables 4 HPE Integrity Superdome X (12TB) compute nodes each with 2 OPA IB gateway nodes 42 HPE ProLiant DL580 (3TB) compute nodes 12 HPE ProLiant DL380 database nodes 6 HPE ProLiant DL360 web server nodes 20 leaf Intel OPA edge switches 32 RSM nodes, each with 2 NVIDIA Tesla P100 GPUs 800 HPE Apollo 2000 (128GB) compute nodes 16 RSM nodes, each with 2 NVIDIA Tesla K80 GPUs Purpose-built Intel Omni-Path Architecture topology for data-intensive HPC 14 Bridges Virtual Tour:

15 Type RAM a Ph n CPU / GPU / other Server ESM LSM 12 TB 3TB Intel Xeon E v3 (18c, 2.3/3.1 GHz, 45MB LLC) HPE Integrity Intel Xeon E v4 (22c, 2.2/3.3 GHz, 55MB LLC) Superdome X Intel Xeon E v3 (16c, 2.2/3.2 GHz, 40 MB LLC) Intel Xeon E v4 (20c, 2.1/3.0 GHz, 50MB LLC) RSM 128 GB Intel Xeon E v3 (14c, 2.3/3.3 GHz, 35MB LLC) RSM-GPU 128 GB Intel Xeon E v3 + 2 NVIDIA Tesla K Intel Xeon E v4 (16c, 2.1/3.0 GHz, 40MB LLC) + 2 NVIDIA Tesla P100 HPE ProLiant DL580 HPE Apollo 2000 DB-s 6 2 Intel Xeon E v3 + SSD HPE ProLiant DL GB 1 DB-h 6 2 Intel Xeon E v3 + HDDs HPE ProLiant DL380 Web 128 GB Intel Xeon E v3 HPE ProLiant DL360 Other b 128 GB Intel Xeon E v3 Gateway Storage 128 GB 128 GB Total 282 TB 908 Node Types HPE ProLiant DL360, HPE ProLiant DL380 2 Intel Xeon E v3 HPE ProLiant DL Intel Xeon E v3 (12c, 2.5/3.3 GHz, 30MB LLC) Intel Xeon E v4 (14c, 2.4/3.3 GHz, 35MB LLC) Supermicro X10DRi a. RAM in Bridges Intel Xeon v3 nodes is DDR4-2133, and RAM in Bridges Intel Xeon v4 nodes is DDR b. Other nodes = front end (2) + management/log (8) + boot (4) + MDS (4) 15

16 System Capacities Phase 1 Phase 2 Technical Upgrade Compute Nodes Peak fp64 (Tf/s) RSM RSM GPU LSM ESM Ph. 1 Total RSM GPU LSM ESM Ph. 2 Total Total RAM (TB) HDD (TB) Phase 1 Phase 2 Total Storage Pylon PB, raw PB, usable Local PB, raw Excludes utility nodes (database, web server, front-end, management, storage, etc. 16

17 GPU Nodes Bridges GPUs are intended to accelerate both deep learning and simulation codes Phase 1: 16 nodes, each with: 2 NVIDIA Tesla K80 GPUs (32 total) 2 Intel Xeon E v3 (14c, 2.3/3.3 GHz) 128GB DDR RAM Phase 2: +32 nodes, each with: 2 NVIDIA Tesla P100 GPUs (64 total) 2 Intel Xeon E v4 (16c, 2.1/3.0 GHz) 128GB DDR RAM 17

18 NVIDIA Tesla K80 Kepler architecture 2496 CUDA cores (128/SM) 7.08B transistors on 561mm 2 die (28nm) 2 24 GB GDDR5; GB/s 562 MHz base 876 MHz boost 2.91 Tf/s (64b), 8.73 Tf/s 32b Bridges: 32 K80 GPUs 279 Tf/s (32b) 18

19 NVIDIA Tesla P100 Pascal architecture 3584 CUDA cores (64/SM) 15.3B transistors on 610mm 2 die (16nm) 16GB CoWoS HBM2 at 720 GB/s w/ ECC 4.7 Tf/s (64b), (9.3) Tf/s 32b, (18.7) Tf/s 16b Page migration engine improves unified memory Bridges: 64 P100 GPUs 600 Tf/s (32b) From Bridges results forthcoming. 19

20 Database and Web Server Nodes Dedicated database nodes will power persistent relational and NoSQL databases Support data management and data-driven workflows SSDs for high IOPs; HDDs for high capacity (examples) Dedicated web server nodes Enable distributed, service-oriented architectures High-bandwidth connections to XSEDE and the Internet 20

21 Data Management Pylon: A large, central, high-performance storage system 10 PB usable Visible to all compute nodes Currently implemented as two complementary file systems: /pylon1: Lustre, targeted for $SCRATCH use; non-wiped directories available by request /pylon2: SLASH2, targeted for large datasets, community repositories, and distributed clients Distributed (node-local) storage Enhance application portability Improve overall system performance Improve performance consistency to the shared filesystem Aggregate 7.2 PB (6 PB non-gpu RSM: Hadoop, Spark) 21

22 Intel Omni-Path Architecture (OPA) Bridges is the first production deployment of Omni-Path Omni-Path connects all nodes and the shared filesystem, providing Bridges and its users with: 100 Gbps line speed per port; 25 GB/s bidirectional bandwidth per port Measured 0.93μs latency, GB/s/dir 160M MPI messages per second 48-port edge switch reduces interconnect complexity and cost HPC performance, reliability, and QoS OFA-compliant applications supported without modification Early access to this new, important, forward-looking technology Bridges deploys OPA in a two-tier island (leaf-spine) topology developed by PSC for cost-effective, data-intensive HPC 22

23 Omni-Path Architecture User Group (OPUG) OPUG meeting at SC16 November 17, 1:30-3:00p, Room 155-A Opportunity for OPA early adopters to share experiences Led by PSC Bridges is the first large-scale deployment of Omni-Path Initial conversations on advanced interconnect technology date back to 2007 Sign up at PSC OPUG contacts Dave Moses J. Ray Scott Nick Nystrom 23

24 MIDAS held its first Public Health Hackathon at PSC The Challenge: Develop applications to visualize public health data Participants: 12 teams from across the country and India competed Given early access to Bridges nodes for analytics Sponsored by Intel Omni-Path and Dell First Place Entry: SPEW VIEW created by Shannon Ghallager and Lee Richardson at Carnegie Mellon University. Full application available at 24

25 Example: Causal Discovery Portal Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence Browser-based UI Prepare and upload data Run causal Internet discovery algorithms Visualize results Web node VM Apache Tomcat Messaging Database node VM Authentication Data mgmt. Provenance Other DBs Execute causal discovery algorithms Omni-Path Pylon filesystem TCGA fmri Analytics: FGS, IMaGES, and other algorithms, building on TETRAD LSM Node (3TB) Memoryresident datasets ESM Node (12TB) 25

26 De Novo Assembly of the Sumatran Rhino Genome James Denvir and Swanthana Rekulapa, Marshall University First assembled the 1 gigabase Narcissus flycatcher (Ficedula narcissina) genome On the users local resources, they could only assemble ⅓ of the data due to memory limitations, taking 16 hours Assembly of all data on a 3TB node of Bridges required 1.5 TB of memory and only 6.6 hours, which was at least 3-4 faster than anticipated based on execution times elsewhere Narcissus Flycatcher (Ficedula narcissina) in Osaka, Japan, Kuribo, CC BY-SA issus_flycatcher-cropped.jpg Then assembled the 3 gigabase Sumatran rhino (Dicerorhinus sumatrensis) genome Required 1.9 TB of memory and completed in only 11 hours, again 3-4 faster than anticipated Sumatran Rhinoceroses at the Cincinnati Zoo & Botanical Garden, Charles W. Hardin, CC BY ons/3/33/sumatran_rhino_2.jpg 26

27 Improved SNP Detection in Metagenomic Populations Wenxuan Zhong, Xin Xing, and Ping Ma, University of Georgia Assembled 378 Gigabase pairs (Gbp) of gut microbial DNA from normal and diabetic patients Massive metagenome assembly took only 16 hours using an MPI-based metagenome assembler, Ray, on 20 Bridges RM nodes connected by Omni-Path Identified 2480 species-level clusters in assembled sequence data Ran MetaGen clustering software across 10 RM nodes, clustering 500,000 contiguous sequences in only 14 hours Currently testing new likelihood-based statistical method for fast and accurate SNP detection from these metagenomic sequencing data The team is now using Bridges to test a new statistical method on the sequence data to identify critical differences in gut microbes associated with diabetes. Environmental Shotgun Sequencing (ESS). (A) Sampling from habitat; (B) filtering particles, typically by size; (C) DNA extraction and lysis; (D) cloning and library; (E) sequence the clones; (F) sequence assembly. By John C. Wooley, Adam Godzik, Iddo Friedberg - bi , CC BY 2.5, 27

28 Characterizing Diverse Microbial Ecosystems From Terabase-Scale Metagenomic Data Brian Couger, Oklahoma State University Assembled 11 metagenomes sampled from diverse sources, comprising over 3 trillion bases of sequence data Including a recent massive assembly of 1.6 Tbp of metagenomic data from an oil sands tailings pond, a bioremediation target Excellent performance of MPI-based Ray assembler on 90 RM nodes completed assembly in only 4.25 days Analysis of assembled data in progress to characterize organisms present in these diverse environments and identify new microbial phyla Oil sands tailings pond. By NASA Earth Observatory Public Domain, 28

29 Investigating Economic Impacts of Images and Natural Language in E-commerce Dokyun Lee, CMU Tepper School of Business Security and uncertain quality create challenges for sharing economies Lee et al. studied the impact of highquality, verified photos for Airbnb hosts 17,000 properties over 4 months Used Bridges GPU nodes Security and uncertain quality create challenges for sharing economies Difference-in-Difference (DD) analysis showed that on average, rooms with verified photos are booked 9% more often Separating effects of photo verification from photo quality and room reviews indicates that high photo quality results in $2,455 of additional yearly earnings They found asymmetric spillover effects: on the neighborhood level, there appears to be higher overall demand if more rooms have verified photos 29

30 Connectionist Temporal Classification Florian Metze, Hari Parthasarathi, Yajie Miao, and Dong Yu, Carnegie Mellon University A revived approach to sequence classification that works well with speech at the phoneme, syllable, character or word level Bridges is being used to train the Eesen Toolkit part of the Speech Recognition Virtual Kitchen, a webbased resource that allows users to check out VMs with complete deep neural network (DNN) environments. 30

31 Thermal Hydraulics of Next-Generation Gas-Cooled Nuclear Reactors PI Mark Kimber, Texas A&M University Completed 3 Large Eddy Simulations of turbulent thermal mixing of helium coolant in the lower plenum of the Gen IV High- Temperature Gas-cooled Reactor (HTGR) Performed using OpenFOAM, a open-source CFD code Simulations are for scaled-down versions of representative sections of the HTGR lower plenum, each containing 7 support posts and 6 high-reynolds number jets in a cross-flow Each simulation involves a high-quality block-structured mesh with 16+ million cells, and was run on 20 regular Bridges nodes (560 cores) for ~10 million time steps (~1 second of simulated time) Helps to understand the turbulent thermal mixing in great detail, as well as temperature distributions and hotspots on support posts Anirban Jana (PSC) is co-pi and computational lead for this DOE project Distribution of vorticity in a representative unit cell of the HTGR lower plenum containing 7 support posts and 6 jets in a crossflow (right to left). Turbulent mixing of core coolant jets in the lower plenum can cause temperature fluctuations and thermal fatigue of the support posts. Acknowledgements: DOE-NEUP (Grant nos. DE- NE ) NSF-XSEDE (Grant no. CTS160002) 31

32 Mechanisms of Chromosomal Rearrangements Jose Ranz and Edwin Solares, UC Irvine Assembled Drosophila willistroni genome four ways with various assemblers Mix of large memory and regular memory nodes on Bridges enables testing various assemblers and sequencing technologies to obtain the highest-quality genome High-quality assemblies are critical to understanding chromosomal rearrangements, which are pivotal in natural adaptation Bridges high-core, high-memory nodes saved the researchers at least a month compared to other resources Assembled Anopheles arabiensis genome andtranscriptomes Again, using multiple sequencing data types and assembly software Flexibility of Bridges key to supporting diverse needs of this project Drosophila willistoni Adult Male. By Mrs. Sarah L. Martin - Plate XVI, J.T. Patterson and G.B. Mainland. The Drosophilidae of Mexico. University of Texas Publications, 4445:9-101, 1944., Public Domain, = Anopheles gambiae mosquito By James D. Gathany - The Public Health Image Library, ID#444, Public Domain, =

33 Simulations of Nanodisc Systems Jifey Qi and Wonpil Im, Lehigh University Using NAMD on Bridges NVIDIA P100 GPU nodes to simulate a number of nanodisc systems to be used as examples for the Nanodisc Builder module in CHARMM-GUI For a system size of 263,888 atoms, the simulation speed on one P100 node (6.25 ns/day) is faster than on three dual-cpu nodes (4.09 ns/day) CHARMM-GUI provides a web-based graphical user interface to generate various molecular simulation systems and input files (for CHARMM, NAMD, GROMACS, AMBER, GENESIS, LAMMPS, Desmond, OpenMM, and CHARMM/OpenMM) to facilitate and standardize the usage of common and advanced simulation techniques A nanodisc system showing two helical proteins warping around a discoidal lipid bilayer. 33

34 Integration of Membrane Proteins into the Cell Membrane Michiel Niesen and Thomas Miller, Caltech Calculated the integration efficiencies for protein sequences by means of their own coarse-grained (CG) model, running ensembles of ~ simulations in parallel As of 9/13/16, they ve calculated membrane protein integration efficiencies for 16 proteins on Bridges The calculated integration efficiencies correlate well with experimentally-observed expression levels for a series of three homologs of the membrane protein TatC Schematic representation of integration. Ribosome (violet), nascent loops (cyan), nascent transmembrane domains (red), lateral gate (green), lipids (light gray). Experimentally observed expression levels for a series of three homologs of the membrane protein TatC (left) correlate well with the calculated integration efficiency (right). 34

35 For Additional Information Website: Bridges PI: Nick Nystrom Co-PIs: Michael J. Levine Ralph Roskies J Ray Scott Project Manager: Robin Scibek 35

36 Thank you. Questions? 36

37 37 Additional Content

38 Interactions of Outer Membrane Proteins and Lipids Wonpil Im, Lehigh University The outer membrane (OM) of Gram-negative bacteria separates the periplasm from the external environment and functions as a selective barrier that prevents the entry of toxic molecules such as antibiotics and bile salts into the bacteria, which is crucial for survival of bacteria in diverse and hostile environments, but causes significant threats to the public health. The OM is an asymmetric bilayer composed of phospholipids in the periplasmic leaflet and lipopolysaccharides (LPS) in the external leaflet, along with β-barrel OM proteins (OMPs) and lipidated periplasmic lipoproteins. The picture shows a crowding OM model constructed using the coarse-grained Martini force field. The OMPs covers 70% of the membrane with the top leaflet lipid being LPS and the bottom leaflet lipid being POPE, POPG, and Cardiolipin. Simulations are underway using NAMD on Bridges to elucidate interesting behaviors of OMPs and lipids, such as protein clustering and anomalous lipid diffusion in a crowding environment. The system consists of ~1.2 million atoms with a box size of 991Å 991Å 135Å, and thus requires the computer power of Bridges to reach a reasonable sampling. Coarse-grained simulations of crowding bacterial outer membranes. The system consists of 114 OMPF trimer (~4.7%, in blue), 282 OMPA (~11.6%, in red), 2034 LPS (~83.7%) in the top layer, and 3629 POPE (~75.1%), 963 POPG (~19.9%), and 240 Cardiolipin (~5.0%) in the bottom layer. O-antigens and other lipid atoms are white. 38

39 Molecular Dynamics Simulation of YidC Protein J Hettige, K Immadisetty, and M Moradi, University of Arkansas First simulated the atomic model of protein insertase YidC (embedded in an explicit membrane) for 200ns. On the users local resources, these simulations would have taken ~66 days. On Bridges, these simulations only took ~10 days using 4 regular shared memory nodes. Then repeated the simulations after adding a loop, which was missing in the original crystal structure. The fast simulations allowed for repeating the simulations in a feasible time with a more complete model of YidC. The comparison of the two sets of simulations provided evidence for the hypothesis that the C2 loop stabilizes the protein, particularly in the C1 region, which is known to play an important role in protein insertion mechanism. C1 region RMSF (Å) Membrane-embedded YidC protein YidC without C2 with C2 C1 C Residue lipids missing C2 loop Root mean square fluctuation (RMSF) of different residues of YidC protein obtained from 200-ns long simulations show the missing C2 loop can stabilize the functionally important C1 region of YidC protein. 39

40 Theoretical Studies on Physical Chemical Properties Yi-Gui Wang, Southern Connecticut State University Investigated large systems with very detailed basis sets MP2/ G(2d,p)// B3LYP/6-31+G* Locally, they could only run at HF/3-21G, which is inadequate Thoroughly evaluated the N- selective reaction with Marcus theory and O-selective reaction with association effect They report that Bridges speed allows them to address the large number of intermediates and to run the many jobs that are needed 40