CSCS Site Update. HPC Advisory Council Workshop Colin McMurtrie, Associate Director and Head of HPC Operations.

Transcription

1 CSCS Site Update HPC Advisory Council Workshop 2018 Colin McMurtrie, Associate Director and Head of HPC Operations. 9th April 2018

2 The Swiss National Supercomputing Centre Driving innovation in computational research in Switzerland Established in 1991 as a unit of ETH Zurich >90 motivated staff from ~15 nations Now with a division in Zurich working on Scientific Software and Libraries (aka SSL) Develops and operates the key supercomputing capabilities required to solve important problems to science and/or society Leads the national strategy for High-Performance Computing and Networking (HPCN) that was passed by Swiss Parliament in 2009 Has a dedicated User Laboratory for supercomputing since 2011 Research infrastructure funded by the ETH Domain on a programmatic basis ~1000 users, 200 projects CSCS Site Update - HPCACW

3 water is sent t hrough a furt her heat exchanger, connected to a second cooling circuit. This mid-temperat ure circuit cools t he air in t he housings for t he comput ers and hard drives of lesser Flexible Facilities Infrastructure energy densit y, which can t herefore be cooled wit h water t hat is less cold. This means t hat wit h one pumping operat ion, cold water is supplied t o two circuits to cool two types of systems. The cold water pipe is designed to cool supercomput ers of up to 14 megawatt s on t he first cooling circuit. The second circuit can cool a furt her 7 megawatt s of comput ers. The more t he second circuit is used, t he higher t he waste heat absorbed by t he water and so t he more useful it is to t he local indust rial works who will be able t o use it. Before t he lake water ret urns to t he lake, it passes t hrough a st illing basin which can hold 120 cubic metres. The basin collect s t he water and makes sure t hat it flows freely down t he ret urn pipe back to t he lake at a constant pressure and wit h no need for furt her power to be used. On t he cont rary, t he plan is to use t he energy generated as it falls to produce elect ricit y. That is why connect ions for a microt urbine have been provided in the pumping station. Flexible Facilities Infrastructure is important since we cannot be certain about future system requirements CSCS Data Centre provides: was power/cooling: barely able 12 t o accommodat MW e t he inst allat ion of new comput ers. upgradable to 25 MW Only a t rapdoor indicates t he existence of t he pumping st at ion below the surface of Parco Ciani to visitors. (Picture: CSCS) Free cooling with water from lake Lugano Current Power Usage Effectiveness (PUE) = 1.2 So as not to affect t he ecological balance of t he lake, t he water going back into t he lake must never exceed 25 degrees Celsius. To ensure t hat t his is always t he case, a back-mixing funnel has been fitted which will add cold water if necessary. I The suct ion st rainers for t he lake water pipe, just before t hey were lowered 45 metres into Lake Lugano. (Picture: CSCS) CSCS Site Update - HPCACW The water pipe (green) stretches 2.8 km accross t he cit y to connect t he lake (right ) wit h t he comput ing centre. On it s way it crosses under t he Casserate river t wice.

4 Flagship Supercomputer Piz Daint Cray XC40 / Cray XC50 Operational since April 2013 Extension + upgrade to hybrid in late 2013 Upgrade to new GPU in 2016 Compute nodes dual-socket nodes with Intel Xeon CPU and NVIDIA Tesla P100 GPU dual-socked nodes with Intel Xeon CPUs Total system memory 521 TB RAM Peak Performance Hybrid partition 25.3 Petaflop/s Multicore partition 1.7 Petaflop/s Measured Linpack performance of Pflop/s Most powerful petascale supercomputer in the Top10 of the Green500 CSCS Site Update - HPCACW

5 Evolution of the Flagship System - Piz Daint 5,272 hybrid nodes (CrayXC30) Nvidia Tesla K20x 16x PCIe 2.0 Intel Xeon E (SB) 6GB GDDR5 32GB DDR3 No multi-core partition Cray Aries interconnect 16x PCIe 3.0 Dragonfly topology ~33TB/s bisection bandwidth Fully provisioned for 28 cabinets Cray Sonexion Lustre File System 2.7PB Snx1600 Slurm WLM Slurm + ALPS 5320 hybrid nodes (Cray XC50) Nvidia Tesla P100 16x PCIe 3.0 Intel Xeon E v3 (HSW) 16GB HBM2 64GB DDR multi-core nodes (Cray XC40) Dual socket Intel Xeon E v4 (BDW) 64GB and 128GB DDR4 Cray Aries interconnect 16x PCIe 3.0 ~36TB/s bisection bandwidth Public IP routing to CSCS network Sonexion Lustre file system 9.6PB Snx PB Snx1600 External GPFS on selected nodes Slurm WLM Native Slurm (no ALPS) CSCS Site Update - HPCACW

6 Piz Daint A Consolidated HPC Environment Pre- & postprocessing Data Mover Data Warp Machine Learning Deep Learning Support for Docker Computing Visualisation Data Analysis CSCS Site Update - HPCACW

7 Data Centre Ecosystem Internet Access (via SWITCHlan;100 Gbit/s) CSCS LAN Consolidated HPC Environment On-site Cloud IaaS Dedicated Customer Systems/Platforms Data Centre Network (IB, Ethernet) Site-wide Storage CSCS Site Update - HPCACW TSM + Tape Library

8 Infrastructure-as-a-Service (IaaS) IaaS is a service model that delivers computer infrastructure on an outsourced basis to support enterprise operations. Typically, IaaS provides hardware, storage, servers and data center space or network components; it may also include software, Source: Technopedia.com Legend: You = Platform Provider Other = Infrastructure Provider CSCS Site Update - HPCACW

9 IaaS - Why Use It? Enables the hosting of Domainspecific portals that are managed by external entities Separation of concerns means we don t need to get involved with the details of what powers the Portal(s) Dynamic Provisioning possible But the Web Services themselves need to be scalable in such an env. Challenges: Infrastructure provider has no visibility on what is happening within the Platform(s) NGINX RDBMS (Postgresql or Mysql) NGINX Reverse Proxy Server (SSL + Caching) OIDC Extension (lua) Django REST Django ORM + Business logic IaaS VM Infrastructure Example of a Web Service Arrows denote functional dependency OIDC REST API OIDC Service (Mitreconnect based, Java) Mysql DB Identity Service Log collection and monitoring services CSCS Site Update - HPCACW

10 OpenStack IaaS Architecture Summary CSCS Site Update - HPCACW

11 Production OpenStack Environment - Pollux Director node Controller nodes New system for generic Cloud Services 30 new servers Directors, Controllers, Compute Dedicated network 2 x 48 port 40 Gbit/s switches integrated into the CSCS network Storage ~30 TB usable internal CEPH storage External Swift-on-GPFS storage RedHat OpenStack Platform 11 (RHOSP11) Integrated with CSCS AAI via KeyCloak (RHSSO) FireWall rules configured for Internet-facing services Now hosting production platforms for third-party projects Adding more HW Augmenting the Cloud offerings (work on-going) KeyCloak/RHSSO Compute nodes Storage nodes SWIFT (IBM Spectrum Scale) 40 Gb/s SAN Storage CSCS Site Update - HPCACW

12 Bringing Cloud Technologies closer to Piz Daint Docker Containers and Shifter Production workflows are using Docker Containers with Shifter on Piz Daint: 1. Build and test containers with Docker on your Laptop Convenience for the user 2. Run securely and with high-performance on Piz Daint using Shifter Native GPU and MPI performance Improves parallel file system performance for some applications (e.g. Spark) Current use cases: LHConCray Data Analytics frameworks (e.g. Spark) HBP Neurorobotics Platform >5000 container launches per day Others coming watch this space CSCS Site Update - HPCACW

13 Bringing it all together 1. IaaS relies on REST APIs to offer Services to Platforms We collectively term these services Infrastructure Services 2. OpenStack is one way to provide IaaS and this can be done with satellite clusters 3. For Piz Daint we need other mechanisms to provide the necessary Infrastructure Services APIs Work underway in this area 4. IaaS opens the door for Interactive Supercomputing There are known Use Cases coming from various communities Implies some policy-level changes (e.g. job preemption or node sharing for some queues) 5. This does NOT mean we will do away with our usual operations The User Lab will remain our main core business These new services are a way to augment our capability and open doors to new communities CSCS Site Update - HPCACW

14 Q&A Contact: