Leonhard: a new cluster for Big Data at ETH

Transcription

1 Leonhard: a new cluster for Big Data at ETH Bernd Rinn, Head of Scientific IT Services Olivier Byrde, Group leader High Performance Computing Bernd Rinn & Olivier Byrde

2 Agenda Welcome address by Rui Brandao, Director of IT Services (10 ) HPC & Big Data strategy of IT Services (10 ) Update on Euler III expansion (10 ) Overview of the Leonhard cluster (10 ) Leonhard Open vs. Leonhard Med (10 ) Timeline and prices (10 ) Q & A (30 ) Bernd Rinn & Olivier Byrde

3 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

4 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

5 Scientific IT vs. Commodity IT Bernd Rinn & Olivier Byrde

6 Scientific IT Services (SIS) SIS is a section of IT Services created 4 years ago to offer scientific computing support and services to researchers at ETH Our staff (about 30) are all experts in scientific computing algorithms, data analysis, visualization, software development, HPC system management, data management, etc.; many have a background in a scientific domain Our mission is to take care of scientists IT problems, so that they can focus on their research Our customers can pay for these services either on a project basis or via a subscription for expert services; we provide computing power to researchers who are investing in the central clusters of ETH Bernd Rinn & Olivier Byrde

7 Scientific IT Services (cont.) High- Performance Computing Scientific Software Data Analysis and Management Consulting & Training High-performance computing and Big Data Analytics Writing, porting and optimizing scientific software Building data processing pipelines and data management solutions Triaging computing needs in research / for research projects Bernd Rinn & Olivier Byrde

8 Services / Groups of SIS High Performance Computing The HPC group is responsible for the procurement, installation and operation of the central HPC and Big Data clusters of ETH and provides support Scientific Software and Data Management The SSDM group develops and maintains bespoke scientific software according to the needs of our customers, including software for data acquisition, management and analysis Data Analysis and Management The Research Informatics (RI) group builds data pipelines and data management solutions, offers consulting services, and provides training on scientific computing topics (e.g. parallel programming, Python, Spark, research data management) Bernd Rinn & Olivier Byrde

9 What impacts our goals for HPC & Big Data Our goals are defined by Strategic goals set by ETH Council, ETH Exec. Board, and IT Services The (constantly evolving) needs of our existing customers The wishes of prospective customers However, these goals are subject to many constraints Available budget and manpower Technical (space, power and cooling; technological evolution; security) Legal (WTO procurement rules; Swiss data protection laws) Time (customers deadlines; delivery and installation time for new hardware) Our goals are shaped by these needs and constraints Bernd Rinn & Olivier Byrde

10 Optimization under boundary conditions We work closely with all stakeholders to determine which goals we can fulfill within the imposed constraints Some services may be better provided by colleagues, e.g. Local IT Support (ISG) or another ITS section, e.g. S4D or SD Swiss National Supercomputing Centre (CSCS) We must focus on goals which have the highest impact for ETH Goals that benefit multiple research groups and many ETH researchers Goals of strategic importance Bernd Rinn & Olivier Byrde

11 From Brutus to Euler and Leonhard Brutus (decommissioned last month) was a multi-purpose cluster Standard compute nodes optimized for energy efficiency (due to power & cooling limitations) GPGPU nodes Its duties are being taken over by Euler and the new Leonhard cluster Euler is focusing on high-performance / high-throughput computing (HPC / HTC) Leonhard will be the platform of choice for special applications, especially GPGPUs Euler III and Leonhard are extending into new areas Euler III provides very high single-core performance for serial applications Leonhard Open provides fast, scalable and cost-effective storage for Big Data applications Leonhard Med provides secure environment (including built-in encryption) for computing on sensitive biomedical data Bernd Rinn & Olivier Byrde

12 Performance growth Storage growth 1500 Leonhard 5000 Leonhard Peak performance [TF] Euler Brutus Storage capacity [TB] Euler Brutus Bernd Rinn & Olivier Byrde

13 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

14 Motivation for Euler III Previous Euler expansions have been aimed at large parallel jobs Compute nodes with 24 cores, connected together via high-speed InfiniBand network It turns out that many Euler users (still) run single-core jobs They do not need InfiniBand, and would rather have smaller, faster CPUs In 2016, these single-core jobs represented 84% of all jobs submitted on Euler 17% of all CPU time used on Euler These users have seen little performance improvements since Euler I Let s do something different for them! Bernd Rinn & Olivier Byrde

15 And now for something completely different Euler III is based on Hewlett-Packard s Moonshot platform It is a radical change from the compute nodes that we have used so far In fact it is so radical that Euler III is the largest system of its kind in the world J Each compute node is an HPE m710x server featuring A quad-core Intel Xeon E3-1585Lv5 Skylake GHz 32 GB of DDR4 RAM clocked at 2133 MHz A 256-GB NVMe SSD 10G Ethernet network That s more computing power than an old Brutus node, in 1/10 th the volume! Bernd Rinn & Olivier Byrde

16 Battery Chassis connector Power button Intel Xeon E3-1585Lv5 (under heat sink) m.2 NVMe SSD (4x) Memory modules (4x) m.2 SATA SSD (1x) Photos courtesy of Hewlett-Packard Enterprise Bernd Rinn & Olivier Byrde

17 Photo by O. Byrde, ETH Zurich Bernd Rinn & Olivier Byrde

18 Photo by O. Byrde, ETH Zurich Bernd Rinn & Olivier Byrde

19 Current status of Euler III In total Euler III contains 1215 nodes / 4860 cores / 9720 threads That is enough to handle all single-core jobs currently running on Euler It was opened to selected beta users one month ago Their feedback regarding performance has been very positive The hardware has proven extremely reliable (1 minor incident per week on average) Beta testing will now focus on the new software environment (CentOS 7 and LSF 10) that we plan to introduce on Euler as a whole in a few months Please let us know if you would like to give it a try Bernd Rinn & Olivier Byrde

20 Future Euler expansions & upgrades We are already working on Euler IV The type and number of nodes will be defined in the coming months Please let us know what you need, so that we can take it into account Installation expected in September 2017 We are also working on a massive storage upgrade New storage system with petabytes of high-speed storage Installation expected in June 2017 Bernd Rinn & Olivier Byrde

21 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

22 New Leonhard cluster for Big Data applications at ETH The main features of Leonhard are: Multiple petabytes of fast parallel storage, scaling to tens of petabytes Different compute nodes for specific use cases (high throughput, large memory, GPUs, etc.) Enhanced security for sensitive applications (e.g. personalized medicine) Innovative tools and methods to analyze these data (Hadoop, R, Spark, TensorFlow, etc.) Leonhard was approved by the Executive Board of ETH in September 2016 The Executive Board also provided partial funding for the medical part The rest is financed by shareholders and by IT Services Bernd Rinn & Olivier Byrde

23 Infrastructure A secure computer room has been built just for Leonhard It contains 22 racks and 12 heat exchangers arranged in a so-called warm island Flexible cooling supports low- to high-density systems using kw per rack Ideal for the mix of storage, compute and GPU nodes needed for Big Data The whole electrical infrastructure is protected against power interruptions (UPS) The room is locked and accessible to authorized personnel only Bernd Rinn & Olivier Byrde

24 Warm island Picture courtesy of Modulan GmbH Bernd Rinn & Olivier Byrde

25 Photo by O. Byrde, ETH Zurich Bernd Rinn & Olivier Byrde

26 Storage systems Core storage Redundant NFS servers for system management, applications and users homes Usable capacity of 70 TB, stored entirely on SSD Built-in encryption Big Data storage Parallel file system based on IBM Spectrum Scale Advanced Edition (formerly GPFS) Initial capacity of 3.5 PB split between two separate storage systems ( PB) Expandable to 20 PB ( PB) on short notice, and to tens of PB within a few months Performance of ~10 GB/s per system; will grow to GB/s as we expand capacity Built-in encryption Bernd Rinn & Olivier Byrde

27 Data protection Hardware protection The storage systems of Leonhard use double-parity RAID to protect against disk failure They use redundant file servers configured for high availability (HA) operation File system protection All file systems support snapshots to protect against accidental file modification / deletion Snapshot frequency and retention time to be defined according to customers needs Backup System state, applications and users homes are backed up daily to a remote system Big Data storage is not backed up (tape backup may be an option; this will need to be discussed and tested together with the Storage Group of IT Services) Until then, critical data on Leonhard Med can be duplicated on Leonhard Open Bernd Rinn & Olivier Byrde

28 Compute nodes Standard / large-memory nodes 2 x 18-core Intel Xeon CPUs 128 / 512 GB RAM GPU nodes (on order; not delivered yet) 2 x 10-core Intel Xeon CPUs 256 GB RAM 8 x Nvidia GTX-1080 GPUs Different types of nodes will be added later as needed Very large memory, Xeon Phi, Hadoop, etc. Bernd Rinn & Olivier Byrde

29 Software Same software stack as Euler Same OS, admin tools, batch system, compilers, libraries and applications Allows us to exploit synergies between the two clusters Makes life easier to people using both clusters Some software (e.g. CUDA library for Nvidia GPUs) will be usable only on Leonhard Bernd Rinn & Olivier Byrde

30 Access restricted to shareholders Unlike Euler, which is open to all users of ETH, Leonhard will be accessible to shareholders only Shareholders can grant access to collaborators outside their group as needed SIS will create temporary accounts on demand for groups who would like to test Leonhard before becoming shareholders Bernd Rinn & Olivier Byrde

31 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

32 Leonhard is actually made of two clusters Leonhard Open, for the analysis of open (public) research data Same security level as Euler (firewall, login, NETHZ user & group file permissions) File system encryption supported but not mandatory Leonhard Med, for the analysis of sensitive biomedical research data Access restricted to people officially authorized to handle these data Enhanced security to protect against both theft and inadvertent leak of sensitive data Compliant with Swiss law and other regulations for medical data, such as the US Health Insurance Portability and Accountability Act ( Security policies defined by a committee of legal, medical and IT experts Bernd Rinn & Olivier Byrde

33 Leonhard Open vs. Leonhard Med Leonhard Open Leonhard Med Intended for Open Big Data applications Sensitive Big Data applications from biomedical research Designed for Open to Shareholders and their collaborators Specific groups and users on project basis Open to Security Standard for academic cluster, similar to Euler Enhanced, policy based and strictly enforced Security Data encryption Supported Mandatory for certain applications File encryption Storage capacity 2 PB initially, expandable to >10 PB 1.5 PB initially, expandable to >10 PB Storage capacity Bernd Rinn & Olivier Byrde

34 Schematic view Enhanced security Login node Key server Login node Open storage Open compute nodes Med compute nodes Med storage One-way data flow (replication) Bernd Rinn & Olivier Byrde

35 Flexible architecture To simplify hardware maintenance and system administration, Leonhard Open and Leonhard Med are based on identical components One type of server for admin, login and compute nodes The partition between the two clusters is not fixed Compute nodes can be moved from one cluster to the other as needed For security reasons, this cannot be done on-the-fly (nodes must be drained and erased, networks must be reconfigured, OS must be reinstalled, etc.) You can thus buy a share in Leonhard Open and convert it to Leonhard Med (or vice versa) later if needed for technical or compliance reasons Bernd Rinn & Olivier Byrde

36 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

37 Timeline Leonhard Open is being installed right now Beta testing will start in late February or early March Normal operation is expected to start in April Leonhard Med is open to alpha users since December 2016 Beta testing will start in March Normal operation will start when security and compliance issues have been defined, implemented and verified Bernd Rinn & Olivier Byrde

38 Current Euler price list Compute nodes (valid for 4 years) Standard node (24 cores, 64 GB RAM, InfiniBand FDR) Large-memory node (24 cores, 256 GB RAM, InfiniBand FDR) Storage (valid for 4 years) High-performance work storage, per TB Long-term project storage, per TB Future high-performance storage (mid-2017), per TB Bernd Rinn & Olivier Byrde

39 Preliminary Leonhard price list (subject to change) Compute nodes (valid for 4 years) Standard node (36 cores, 128 GB RAM, InfiniBand EDR) Large-memory node (36 cores, 512 GB, IB EDR) GPU node (20 cores, 256 GB, 8 x GTX-1080, IB EDR) Big Data storage (valid for 4 years) First 20 TB Each additional block of 20 TB, up to 100 TB Each additional block of 50 TB, up to 500 TB Larger capacities Storage is only offered in combination with compute nodes (and vice versa) Minimum investment: 1 compute node and 20 TB of storage Bernd Rinn & Olivier Byrde

40 Availability of Leonhard nodes Standard and large-memory nodes are available now in limited quantities We can order additional ones at any time (delivery time: 6-8 weeks) World-wide RAM prices have increased a lot; this may affect the price of our next order The first batch of GPU nodes are sold out All the GPU nodes that we ordered have already been sold to shareholders According to WTO rules, we need to issue a call for tender to order additional nodes We are presently working on it this process will take several months If you have special wishes (GPU model, number of GPUs per node), please let us know Bernd Rinn & Olivier Byrde

41 Agenda Welcome address by Rui Brandao HPC & Big Data strategy of IT Services Update on Euler III expansion Overview of the Leonhard cluster Leonhard Open vs. Leonhard Med Timeline and prices Q & A Bernd Rinn & Olivier Byrde

42 Useful links General information IT Services Scientific IT Services Wiki Main page Euler Leonhard Support IT Services support / servicedesk@id.ethz.ch Cluster support cluster-support@id.ethz.ch Bernd Rinn & Olivier Byrde