Elastic HPC on the MOC

Transcription

1 Elastic HPC on the MOC Evan Weinberg, Boston University Rajul Kumar, Northeastern University Christopher Hill, Massachusetts Institute of Technology (and the efforts of many others) The Massachusetts Open Cloud (MOC) December 6, 2016

2 HPC in a Cloud Environment? An HPC environment may have Full bi-sectional bandwidth network Accelerators such as FPGA, GPUs Some Cloud environments have these. An Cloud environment may have Fault tolerant environments Higher level programming models Some HPC environments have these.

3 HPC in a Cloud Environment? An HPC environment may have Full bi-sectional bandwidth network Accelerators such as FPGA, GPUs Some Cloud environments have these. An Cloud environment may have Fault tolerant environments Higher level programming models Some HPC environments have these. We believe there is a convergence between HPC and the Cloud.

4 HPC in a Cloud Environment? An HPC environment may have Full bi-sectional bandwidth network Accelerators such as FPGA, GPUs Some Cloud environments have these. An Cloud environment may have Fault tolerant environments Higher level programming models Some HPC environments have these. We believe there is a convergence between HPC and the Cloud. What s different: HPC has an infinite demand, uses batch schedulers to maximize utilization. The Cloud never wants to turn away customers and always meets peek demand: low utilization.

5 HPC as part of a Cloud Environment Benefits HPC can fill idle cycles on the Cloud HPC becomes a bigger market HPC can be brought to the masses

6 HPC as part of a Cloud Environment Benefits HPC can fill idle cycles on the Cloud HPC becomes a bigger market HPC can be brought to the masses Depends on Elasticity!

7 HPC: Issues & Efforts What are some issues? Virtualized clusters aren t ideal for large distributed HPC computing yet.

8 HPC: Issues & Efforts What are some issues? Virtualized clusters aren t ideal for large distributed HPC computing yet. What efforts can be made now? Project 1: Checkpointing and migrating virtualized workloads from the Open Science Grid.

9 HPC: Issues & Efforts What are some issues? Virtualized clusters aren t ideal for large distributed HPC computing yet. What efforts can be made now? Project 1: Checkpointing and migrating virtualized workloads from the Open Science Grid. Project 2: Dynamically growing and shrinking a bare metal HPC cluster to boost Cloud utilization.

10 Current Cluster Technology: Slurm Slurm is open-source, modular, scalable Slurm is a workload manager designed for Linux clusters of all sizes.

11 Current Cluster Technology: Slurm Slurm is open-source, modular, scalable Slurm is Professionally Deployed Slurm is a workload manager designed for Linux clusters of all sizes. MGHPCC: 15 MW, 90,000 square feet + can grow Slurm is used at engage1.

12 Current Cluster Technology: Slurm Slurm is open-source, modular, scalable Slurm is Professionally Deployed Slurm is a workload manager designed for Linux clusters of all sizes. MGHPCC: 15 MW, 90,000 square feet + can grow Slurm is used at engage1. Slurm is traditionally static. What is needed to make it elastic?

13 Static Slurm Configuration Controller Physical Cluster Slurm controller manages scheduling, node state. Slurm compute nodes runs executables as directed. Compute

14 Static Slurm Configuration Controller Physical Cluster slurm.conf Slurm controller One ring to rule them all manages scheduling, node state. Slurm compute nodes runs executables as directed. Compute

15 Static Slurm Configuration Controller Physical Cluster Compute Slurm controller manages scheduling, node state. Slurm compute nodes runs executables as directed. slurm.conf maintains the initial state of all nodes in the cluster.

16 Elastic Slurm Configuration Elastic Cluster Controller Daemon VM Compute slurm.conf Contains the hostname and initial state of all possible nodes Virtual or physical. What do we need to be flexible? Need an extra daemon VM. Provisioning Cluster Management Network Management File System Management [See talk by RK]

17 Use Cases Controller Elastic Cluster Daemon VM These mechanisms for elastic HPC are very generic. Compute A specific application driver decides how, when to reprovision.

18 Use Cases Controller Elastic Cluster Daemon VM These mechanisms for elastic HPC are very generic. Driver Compute A specific application driver decides how, when to reprovision.

19 Use Cases Controller Elastic Cluster Daemon VM These mechanisms for elastic HPC are very generic. Driver Compute A specific application driver decides how, when to reprovision.

20 Checkpointing and migrating virtualized workloads from the Open Science Grid EW, RK, CH Engage1 s Slurm cluster supports backfilled HTC workloads from the These can need about an hour. But if higher priority work comes along OSG work gets killed and resubmitted! This keeps utilization up, but no work gets done.

21 Elastic Virtual OSG in Slurm (MOC-Kaizen) VMs run OSG work Daemon VM Driver

22 Elastic Virtual OSG in Slurm (MOC-Kaizen) High Priority work comes in suspend OSG VMs Daemon VM Driver

23 Elastic Virtual OSG in Slurm (MOC-Kaizen) Start High Priority work Daemon VM Driver

24 Elastic Virtual OSG in Slurm (MOC-Kaizen) Start High Priority work Possibility one: High priority work finishes Daemon VM Driver

25 Elastic Virtual OSG in Slurm (MOC-Kaizen) Remove high priority work resources Daemon VM Driver

26 Elastic Virtual OSG in Slurm (MOC-Kaizen) Resume OSG work Daemon VM Driver

27 Elastic Virtual OSG in Slurm (MOC-Kaizen) Start High Priority work Possibility two: New resources become available Daemon VM Driver

28 Elastic Virtual OSG in Slurm (MOC-Kaizen) Prepare to migrate OSG VM Daemon VM Driver

29 Elastic Virtual OSG in Slurm (MOC-Kaizen) Daemon VM Resume OSG work Driver

30 Elastic Virtual OSG in Slurm (MOC-Kaizen) Utilization is meaningfully up transparently to the end user. Daemon VM Driver

31 Dynamically growing and shrinking a bare metal HPC cluster to boost Cloud utilization (as part of BMI) Apoorve Mohan (NEU), Ravi Gudimetla (NEU), Sourabh Bollapragada (NEU), Ata Turk (BU), Jason Hennesey (BU), Sahil Tikale (BU), EW, RK, Emine Ugur Kaynar (BU), Peter Desnoyers (NEU), Gene Cooperman (NEU), Chris Hill (MIT), Orran Krieger (BU)

32 Elastic Bare Metal HPC & Cloud in BMI Elastic Cluster Controller Daemon VM Driver Compute RUCS is the driver for how and when to reprovision. [See BMI talk, Ravi + Apoorve] Uses Elastic Slurm to manage HPC capable resources.

33 HPC & Cloud: Thoughts & Conclusions We believe there is a convergence between the Cloud and HPC. As a part of the Cloud, HPC can be brought to the masses. Elasticity is an essential step in this direction. We have identified a base set of mechanisms for any Elastic HPC deployment. We can start realizing this today.