PRP Distributed Kubernetes Cluster

Size: px

Start display at page:

Download "PRP Distributed Kubernetes Cluster"

Carmella Robinson
5 years ago
Views:

1 PRP Distributed Kubernetes Cluster John Graham (UCSD) Dmitry Mishin (SDSC) Shawfeng Dong (SLAC) Workshop on Applying Advanced AI Workflows in Astronomy and Microscopy UCSC Silicon Valley Campus, Santa Clara, CA September 11, 2018

2 Pacific Research Platform NSF CC*DNI DIBBs award $6M, 10/1/2015 9/30/2020 Image Source: John Hess (CENIC) PI: Larry Smarr (UCSD & Calit2) Co-PIs: Frank Wuerthwein (UCSD) Camille Crittenden (CITRIS) Thomas DeFanti (Calit2) Philip Papadopoulos (SDSC)

CHASE-CI (Cognitive Hardware and Software Ecosystem Community Infrastructure) "build a cloud of hundreds of affordable Graphics Processing Units (GPUs), networked together with a variety of neural

3 CHASE-CI (Cognitive Hardware and Software Ecosystem Community Infrastructure) "build a cloud of hundreds of affordable Graphics Processing Units (GPUs), networked together with a variety of neural network machines to facilitate development of next generation cognitive computing." NSF CI-New award $1.2M, 10/1/2017 9/30/2020 PI: Larry Smarr (UCSD & Calit2) Co-PIs: Tajana Rosing (UCSD) Ilkay Altintas (SDSC) Thomas DeFanti (Calit2) Kenneth Kreutz-Delgado (UCSD)

4 FIONA8 GPU Nodes ~$15K Dual 8-core CPUs, 128GB RAM, 6x 480GB SATA SSDs, 2x NVMe bays, 40G + dual 10G ports, TPM Distributed over PRP and Orchestrated with Kubernetes

5 Kubernetes Kubernetes is a containerorchestration platform for automating deployment, scaling and management of containerized applications across clusters of hosts Originally designed by Google. Its development and design are heavily influenced by Google's Borg system

Pods Pods are the basic building block of Kubernetes Pods are the smallest deployable units that are created and managed in Kubernetes A Pod is a group

6 Pods Pods are the basic building block of Kubernetes Pods are the smallest deployable units that are created and managed in Kubernetes A Pod is a group of one or more deeply coupled containers o Shared network o Shared storage o Options to run the containers Each pod is assigned a unique, dynamic IP address

7 Other Kubernetes Units Job: creates one or more pods and ensures that a specified number of them successfully terminate. Service: an interface to a set of replicated pods, which acts as a loadbalancer ReplicaSet: ensures that the number of desired pods "replicas" are running at any time Deployment: declarative way to describe the desired state of the application (pods, replica sets) DaemonSet: runs a POD on every node in a cluster StatefulSet: clustered application o Startup/shutdown ordering o Stable hostname and storage o Clustered MySQL, Zookeeper, Elasticsearch, Cassandra Volume plugins: ConfigMaps, Secrets, HostPath, rook (ceph), nfs, iscsi, gitrepo,

8 Grafana View of GPUs in the Cluster

9 Rook: Cloud-Native Storage Orchestrator for Kubernetes

10 Containerized Ceph Storage

11 Rook Deployment over PRP

12 Grafana View of Storage in the Cluster

13 PRP Kubernetes Portal

14 PRP Kubernetes Quick Start

15 Pod Manifest (for creating a pod from the tensorflow image) apiversion: v1 kind: Pod metadata: name: gpu-pod spec: containers: - name: gpu-container image: tensorflow/tensorflow:latest-gpu args: ["sleep", "infinity"] resources: limits: nvidia.com/gpu: 1

16 Create a pod in your namespace from the manifest: $ kubectl create -n ucsc -f tensorflow.yaml pod/gpu-pod created List all pods in your namespace: $ kubectl get pods -n ucsc NAME READY STATUS RESTARTS AGE gpu-pod 1/1 Running 0 2m Start an interactive shell in the running pod: $ kubectl exec -it gpu-pod -n ucsc -- /bin/bash root@gpu-pod:/notebooks# nvidia-smi Sun Sep 9 05:29: NVIDIA-SMI Driver Version: GPU Name Persistence-M Bus-Id Disp.A Volatile Uncorr. ECC Fan Temp Perf Pwr:Usage/Cap Memory-Usage GPU-Util Compute M. ===============================+======================+====================== 0 Tesla K40c Off :41:00.0 Off 0 23% 35C P8 22W / 235W 0MiB / 11441MiB 0% Default root@gpu-pod:/notebooks# python

17 You can even start a Jupyter Notebook server in the running pod: root@gpu-pod:/notebooks# jupyter notebook --allow-root On another terminal, forward local port 8888 to port 8888 on the pod: $ kubectl port-forward gpu-pod 8888:8888 -n ucsc Load in your browser:

18 NOTE: when you are done, don't forget to delete the pod! $ kubectl delete -n ucsc -f tensorflow.yaml ### or $ kubectl delete pod gpu-pod -n ucsc

19 Pod Manifest (for creating a pod with Rook block volume attached) apiversion: v1 kind: PersistentVolumeClaim metadata: name: tf-pvc spec: storageclassname: rook-block accessmodes: - ReadWriteOnce resources: requests: storage: 100Gi --- apiversion: v1 kind: Pod metadata: name: tf-gpu-with-rook-block spec: containers: - name: tf-gpu image: tensorflow/tensorflow:latest-gpu args: ["sleep", "infinity"] resources: limits: nvidia.com/gpu: 1 volumemounts: - mountpath: /data name: tf-pvc restartpolicy: Never volumes: - name: tf-pvc persistentvolumeclaim: claimname: tf-pvc

20 Create a pod in your namespace from the manifest: $ kubectl create -n ucsc -f tf-rook-block.yaml persistentvolumeclaim/tf-pvc created pod/tf-gpu-with-rook-block created List all pods in your namespace: $ kubectl get pods -n ucsc NAME READY STATUS RESTARTS AGE tf-gpu-with-rook-block 1/1 Running 0 16m List all Persistent Volume Claims in your namespace: $ kubectl get pvc -n ucsc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE tf-pvc Bound pvc Gi RWO rook-block 13m Start an interactive shell in the running pod: $ kubectl exec -it tf-gpu-with-rook-block -n ucsc -- /bin/bash root@gpu-pod:/notebooks# df -h /data Filesystem Size Used Avail Use% Mounted on /dev/rbd1 100G 33M 100G 1% /data root@gpu-pod:/notebooks# exit

21 JupyterHub

23 23

Introduction to Kubernetes Storage Primitives for Stateful Workloads

September 12, 2017 Introduction to Kubernetes Storage Primitives for Stateful Workloads Saad Ali Google @the_saad_ali Chris Duchesne {code} @ChrisDuchesne Agenda Presentation Quick intro to Kubernetes