Rapid Bare-Metal Provisioning and Image Management Service. Presenters: Apoorve Mohan Ravi Santosh Gudimetla (Northeastern University)

Transcription

1 Rapid Bare-Metal ing and Image Management Service Presenters: Apoorve Mohan Ravi Santosh Gudimetla (Northeastern University)

2 : Bare Metal Imaging Service Motivation: Even with HIL using existing provisioning systems is time consuming. When users release back their nodes, they will lose all of their customizations. As as result, our users were hugging their hardware. We built the to: Rapid ing. Pause/Resume the state of node.

3 Why Not Existing ing Systems? Well-known ing Systems: Foreman, MaaS, Ironic etc. All of them install or copy state into local disk.. Problems: Slow, network congestion, lose the state of disk. Heroic Engineering Efforts: Improves the local disk installation time. Lack image management(like snapshotting the current configuration state of node). Fundamental questions: Why local disk installation or copying? Why not network mounted systems and preserve the disk state?

4 Are Network Mounted Systems Really Bad? All the previous efforts state: Network overhead. Application performance degradation. Are these problems for real? Let s find out...

5 Network Overhead of Big Data Application

6 ~ 170 MB Read while provisioning for a 10 GB image Network Overhead of Big Data Application Critical if we want to avoid Boot Storms. When provisioning 100 s of nodes concurrently.

7 Network Overhead of Big Data Application ~ 170 MB Read while provisioning for a 10 GB image ~ 3KB/s data read after initial boot For a 10GB installation, local disk installation equals one month of run on related image.

8 Network Overhead of Big Data Application ~ 170 MB Read while provisioning for a 10 GB image ~ 3KB/s data read after initial boot For a 10GB installation, local disk installation equals one month of run on related image. Critical if we want to avoid Boot Storms if we want to provision hundreds of nodes.

9 OpenStack Operations Performance VM Creation VM Deletion Local Disk Installation Network Mounted System using Less than 3% Performance degradation

10 MySQL Operations Performance Local Disk Installation Network Mounted System using Less than 5% Performance degradation.

11 ing/re-ing Times Comparison For Single Big Data Node Foreman or Re- Re- ~ 11 Minutes ~ 5 Minutes 30 Seconds ~ 25 Minutes

12 ing Times Comparison For Single Big Data Node Foreman Node Power Cycle

13 ing Times Comparison For Single Big Data Node Foreman Power-on Self-test (POST)

14 ing Times Comparison For Single Big Data Node Foreman PXE Request

15 ing Times Comparison For Single Big Data Node Foreman Kernel Download & Local Disk Installation

16 ing Times Comparison For Single Big Data Node Foreman Node Power Cycle

17 ing Times Comparison For Single Big Data Node Foreman Power-on Self-test (POST) & PXE Request

18 ing Times Comparison For Single Big Data Node Foreman Booting OS from Local DisK

19 ing Times Comparison For Single Big Data Node Foreman Big Data Application Installation

20 ing Times Comparison For Single Big Data Node Foreman Big Data Application Configuration

21 ing Times Comparison For Single Big Data Node Foreman ~ 25 Minutes

22 ing Times Comparison For Single Big Data Node Foreman Foreman Re- ~ 25 Minutes

23 ing Times Comparison For Single Big Data Node Foreman or Re- ~ 25 Minutes Node Power Cycle

24 ing Times Comparison For Single Big Data Node Foreman or Re- ~ 25 Minutes Power-on Self-test (POST) & PXE Request

25 ing Times Comparison For Single Big Data Node Foreman or Re- ~ 25 Minutes OS Chain Booting (ipxe)

26 ing Times Comparison For Single Big Data Node Foreman or Re- ~ 25 Minutes Big Data Application Installation and Configuration

27 ing Times Comparison For Single Big Data Node Foreman or Re- ~ 25 Minutes ~ 11 Minutes

28 Foreman or Re- Re- ing Times Comparison For Single Big Data Node ~ 11 Minutes ~ 5 Minutes 30 Seconds ~ 25 Minutes Big Data Installation overhead removed ( ).

29 ing Times Comparison For Single Big Data Node Foreman Re- ~ 25 Minutes Re- ~ 5 Minutes 30 Seconds

30 ing Times Comparison For Single Big Data Node Foreman Re- ~ 25 Minutes Re- ~ 5 Minutes 30 Seconds Reduces ing/re-ing Times. POST ( ) dominates provisioning time.

31 ing Times Comparison For Single Big Data Node Foreman Re- ~ 25 Minutes Re- ~ 5 Minutes 30 Seconds Without POST ( ) Re- time ~ 2 Minutes. ~ 2 Minutes to spin up Virtual machines in AWS.

32 How Works? HIL CEPH Service

33 How Works? HIL USER Service CEPH

34 1. Reserve Nodes How Works? HIL USER CEPH Service Reserved Servers

35 1. Reserve Nodes How Works? 2. Reserved Node HIL USER CEPH Service Reserved Servers

36 1. Reserve Nodes How Works? 2. Reserved Node 3. Clone Golden Image HIL USER CEPH Service CEPH Interface Reserved Servers Cloned Images

37 1. Reserve Nodes How Works? 2. Reserved Node 3. Clone Golden Image HIL USER CEPH Service CEPH Interface iscsi Gateway Reserved Servers 4. Expose Cloned Image as iscsi Target Cloned Images

38 1. Reserve Nodes How Works? 2. Reserved Node 3. Clone Golden Image HIL USER CEPH Service CEPH Interface iscsi Gateway Reserved Servers DHCP TFTP 5. Configure to PXE boot 4. Expose Cloned Image as iscsi Target Cloned Images

39 1. Reserve Nodes How Works? 2. Reserved Node 3. Clone Golden Image HIL USER CEPH Service HIL Interface CEPH Interface Reserved Servers 6. Attach Nodes to ing Network iscsi Gateway DHCP TFTP 5. Configure to PXE boot 4. Expose Cloned Image as iscsi Target Cloned Images

40 1. Reserve Nodes How Works? 2. Reserved Node 3. Clone Golden Image HIL USER CEPH 7. PXE Boot Reserved Nodes Service HIL Interface CEPH Interface Reserved Servers 6. Attach Nodes to ing Network iscsi Gateway DHCP TFTP 5. Configure to PXE boot 4. Expose Cloned Image as iscsi Target Cloned Images

41 Project Status And Future Directions PROJECT STATUS: Functional Software with Exposed API s. Used by Big Data and Secure Cloud Research Teams at MOC. Proof-of-Concept for Functional OpenStack and Slurm Clusters. FUTURE DIRECTIONS: Suspend/Resume Functionality for Bare-Metal Systems. Removing Power-on Self Start (POST) Overhead while re-ing. Exploring Dynamic Scheduling Policies for improving Data Center Utilization. Improving Cache-Buffer Utilization on iscsi server using OverlayFS. Scalability evaluation and optimization.

42 Summary Has Minimal Performance Overhead. Rapid Bare-Metal ing and Image Management. Reduces ing Times. Greatly reduces network traffic while provisioning.