Kubernetes Storage: Current Capabilities and Future Opportunities. September 25, 2018 Saad Ali & Nikhil Kasinadhuni Google

Size: px

Start display at page:

Download "Kubernetes Storage: Current Capabilities and Future Opportunities. September 25, 2018 Saad Ali & Nikhil Kasinadhuni Google"

Annabelle Hodge
5 years ago
Views:

1 Kubernetes Storage: Current Capabilities and Future Opportunities September 25, 2018 Saad Ali & Nikhil Kasinadhuni Google

2 Agenda Google & Kubernetes Kubernetes Volume Subsystem Container Storage Interface (CSI) Untapped Opportunities Q&A

3 Google & Kubernetes

4 Google is living a few years in the future and sends the rest of us messages, -- Doug Cutting, Hadoop founder, 2013 WWGD?

5 Humble Beginnings

6 Humble Beginnings Google File Sys tem

8 Compute Storage and Databases Networking Compute Engine App Engine Container Engine Storage Bigtable Datastore Virtual Network Load Balancing CDN Container Registry Functions SQL Spanner Persistent Disk Interconnect DNS Networking Big Data Machine Learning Identity & Security BigQuery Dataflow Dataproc Machine Learning Vision API Speech API IAM Resource Manager Security Scanner Key Management Service Datalab Pub/Sub Genomics Natural Language API Translation API Jobs API BeyondCorp Data Loss Prevention Identity -Aware Proxy Security Key Enforcement

9 Cattle Not Pets

11 Compute Storage and Databases Networking Compute Engine App Engine Container Engine Storage Bigtable Datastore Virtual Network Load Balancing CDN Container Registry Functions SQL Spanner Persistent Disk Interconnect DNS Networking Big Data Machine Learning Identity & Security BigQuery Dataflow Dataproc Machine Learning Vision API Speech API IAM Resource Manager Security Scanner Key Management Service Datalab Pub/Sub Genomics Natural Language API Translation API Jobs API BeyondCorp Data Loss Prevention Identity -Aware Proxy Security Key Enforcement

14 Kubernetes Storage Layer

15 What do these words mean and how do they fit together? Persistent Volume Claims Remote File Flex Driver Block Persistent Volumes CSI Stateless Storage Classes Ephemeral Local Out-of-tree Dynamic Provisioning In-tree Volume Object Stateful Plugin

16 Kubernetes Principle Workload Portability

17 Kubernetes: Workload Portability Kubernetes Goal Abstract away cluster details Decouple apps from infras tructure To enable users to Write once, run anywhere (workload porta bility!) Avoid vendor lock-in

18 Kubernetes: Workload Portability App 1 App 2 App 3 App 4 Kubernetes Cluster Node 1 Kernel/OS Node 2 Kernel/OS Node 3 Kernel/OS Hardware Hardware Hardware

19 Kubernetes: Workload Portability App 1 App 2 App 3 App 4 Kubernetes Cluster GCE Instance 1 Kernel/OS GCE Instance 2 Kernel/OS GCE Instance 3 Kernel/OS Hardware Hardware Hardware

20 Kubernetes: Workload Portability App 1 App 2 App 3 App 4 Kubernetes Cluster EC2 Instance 1 Kernel/OS EC2 Instance 2 Kernel/OS EC2 Instance 3 Kernel/OS Hardware Hardware Hardware

21 Kubernetes: Workload Portability App 1 App 2 App 3 App 4 Kubernetes Cluster Bare Metal 1 Kernel/OS Bare Metal 2 Kernel/OS Bare Metal 3 Kernel/OS Hardware Hardware Hardware

22 Kubernetes: Workload Portability apiversion: App 1 kind: ReplicaSet App 2 App 3 App 4 metadata: name: frontend spec: Kubernetes Cluster replicas: 2 template: Node 1 Node 2 Node 3 spec: Kernel/OS containers: Kernel/OS Kernel/OS - name: php-redis image: gcr.io/google_samples/gb-frontend:v3 Hardware Hardware Hardware

23 Kubernetes: Workload Portability App 1 App 2 App 3 App 4 Kubernetes Cluster Node 1 Frontend Pod Replica 1 Kernel/OS Node 2 Frontend Pod Replica 2 Kernel/OS Node 3 Kernel/OS Hardware Hardware Hardware

24 Problem with Containers and State What about stateful apps? Pod and ReplicaSet abstract compute and memory. 1. Containers are ephemeral: no way to persist state Container termination/crashes result in loss of data Can t run stateful applications 2. Containers can t share data between each other. Content Manager File Puller Pod Consumers Web Server

25 Challenges with Abstracting Storage So many different types of storage Object Stores AWS S3, GCE GCS, etc. SQL Databases MySQL, SQL Server, Postgres, etc. NoSQL Databases MongoDB, ElasticSearch, etc. Pub Sub Systems Apache Kafka, Google Pub/Sub, AWS SNS, etc. Time series databases InfluxDB, Graphite, etc. File Storage NFS, SMB, etc. Block Storage GCE PD, AWS EBS, iscsi, Fibre Channel, etc. File on Block Storage And more! What do we focus on?

26 What do we focus on? In s cope: File Stora ge NFS, SMB, etc. Block Storage GCE PD, AWS EBS, iscsi, Fibre Channel, etc. File on Block Stora ge Out of scope: Object Stores AWS S3, GCE GCS, etc. SQL Databases MySQL, SQL Server, Postgres, etc. NoSQL Databases MongoDB, ElasticSearch, etc. Pub Sub Systems Apache Kafka, Google Pub/Sub, AWS SNS, etc. Time series databases InfluxDB, Graphite, etc. etc.

27 What do we focus on? In s cope: File Stora ge NFS, SMB, etc. Block Storage Data Path Standardized (Posix, SCSI) GCE PD, AWS EBS, iscsi, Fibre Channel, etc. File on Block Stora ge Out of scope: Object Stores AWS S3, GCE GCS, etc. SQL Databases MySQL, SQL Path Server, Postgres, etc. NoSQL Databases Not Standardized, MongoDB, ElasticSearch, etc. Pub Sub Systems yet Apache Kafka, Google Pub/Sub, AWS SNS, etc. Time series databases InfluxDB, Graphite, etc. etc.

28 Kubernetes Volume Plugins A way to reference block device or mounted filesystem (possibly with some data in it) Accessible by all containers in pod Volume plugins specify How volume is setup in pod Medium that backs it Lifetime of volume is same as the pod or longer Content Manager File Puller Pod Consumers Web Server

29 Kubernetes Volume Plugins Kubernetes has many volume plugins Remote Storage GCE Persistent Disk AWS Elas tic Block Store Azure File Storage Azure Da ta Dis k Dell EMC ScaleIO iscsi Flocker NFS vsphere Glus terfs Ceph File and RBD Cinder Quobyte Volume FibreChannel VMware Photon PD Ephemeral Storage Em ptydir Expose Kubernetes API Secret ConfigMap DownwardAPI Local Host path Local Persistent Volume (Beta) Out-of-Tree Flex (exec a binary) CSI (Beta) Other

30 Ephemeral Storage Temp scratch file space from host machine Da ta exis ts only for lifecycle of pod. Can only be referenced in-line in pod definition not via PV/ PVC. Content Manager File Puller Consumers Web Server Volume Plugin: EmptyDir EmptyDir Pod

31 Ephemeral Storage Temp scratch file space from host machine Da ta exis ts only for lifecycle of pod. Can only be referenced in-line in pod definition not via PV/ PVC. Volume Plugin: EmptyDir apiversion: v1 kind: Pod metadata: name: test-pod spec: containers: - image: k8s.gcr.io/container1 name: container1 volumemounts: - mountpath: /shared name: shared-scratch-space - image: k8s.gcr.io/container2 name: container2 volumemounts: - mountpath: /shared name: shared-scratch-space volumes: - name: shared-scratch-space emptydir: {}

32 Ephemeral Storage Built on top of EmptyDir: Secret Volume ConfigMa p Volum e Downwa rdapi Volum e Populate Kubernetes API as files in to an EmptyDir

33 Kubernetes Principle Meet the user where they are

34 Ephemeral Storage Built on top of EmptyDir: Secret Volume ConfigMa p Volum e Downwa rdapi Volum e Populate Kubernetes API as files in to an EmptyDir

35 Remote Storage Data persists beyond lifecycle of any pod Referenced in pod either in-line or via PV/ PVC Examples : GCE Persistent Disk AWS Ela s tic Block Store Azure Da ta Dis k iscsi NFS Glus terfs Cinder Ceph File and RBD And m ore!

36 Remote Storage Kubernetes will automatically: Attach volume to node Mount volume to pod apiversion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcepersistentdisk: pdname: panda-disk fstype: ext4 containers: - name: sleepycontainer image: gcr.io/google_containers/busybox command: - sleep - "6000" volumemounts: - name: data mountpath: /data readonly: false

Remote Storage Kubernetes will automatically: Attach volume to node Mount volume to pod apiversion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcepersistentdisk:

37 Remote Storage Kubernetes will automatically: Attach volume to node Mount volume to pod apiversion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcepersistentdisk: pdname: panda-disk fstype: ext4 containers: - name: sleepycontainer image: gcr.io/google_containers/busybox command: - sleep - "6000" volumemounts: - name: data mountpath: /data readonly: false

38 Kubernetes Principle Workload Portability

39 Remote Storage Pod yaml is no longer portable across clusters!! apiversion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcepersistentdisk: pdname: panda-disk fstype: ext4 containers: - name: sleepycontainer image: gcr.io/google_containers/busybox command: - sleep - "6000" volumemounts: - name: data mountpath: /data readonly: false

40 Persistent Volumes & Persistent Volume Claims Pers is tentvolume and Pers is tentvolumeclaim Abs traction Decouples storage implementation from storage consumption

41 PersistentVolume apiversion: v1 kind: PersistentVolume metadata: name : mypv1 spec: accessmodes: - ReadWriteOnce capacity: storage: 10Gi persistentvolumereclaimpolicy: Retain gcepersistentdisk: fstype: ext4 pdname: panda-disk apiversion: v1 kind: PersistentVolume metadata: name : mypv2 spec: accessmodes: - ReadWriteOnce capacity: storage: 100Gi persistentvolumereclaimpolicy: Retain gcepersistentdisk: fstype: ext4 pdname: panda-disk2

42 PersistentVolumeClaim apiversion: v1 kind: PersistentVolumeClaim metadata: name: mypvc namespace: testns spec: accessmodes: - ReadWriteOnce resources: requests: storage: 100Gi

43 PV to PVC Binding $ kubectl create -f pv.yaml persistentvolume "pv1" created persistentvolume "pv2" created $ kubectl get pv NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE pv1 10Gi RWO Available 1m pv2 100Gi RWO Available 1m $ kubectl create -f pvc.yaml persistentvolumeclaim "mypvc" created $ kubectl get pv NAME CAPACITY ACCESSMODES STATUS CLAIM REASON AGE pv1 10Gi RWO Available 3m pv2 100Gi RWO Bound testns/mypvc 3m

44 Remote Storage Volume referenced via PVC Pod YAML is portable across clusters again!! apiversion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data gcepersistentdisk: pdname: panda-disk fstype: ext4 containers: - name: sleepycontainer volumes: - name: data persistentvolumeclaim: claimname: mypvc image: gcr.io/google_containers/busybox command: - sleep - "6000" volumemounts: - name: data mountpath: /data readonly: false

45 Dynamic Provisioning Clus ter admin pre-provis ioning PVs is painful and was teful. Dynamic provis ioning creates new volumes on-demand (when requested by user). Eliminates need for clus ter adminis trators to pre-provision storage.

46 Dynamic Provisioning Dynamic provisioning enabled by creating StorageClass. StorageClass defines the parameters used during crea tion. StorageClass parameters opaque to Kubernetes so storage providers can expose any number of custom parameters for the cluster admin to use. kind: StorageClass apiversion: storage.k8s.io/v1 metadata: name: slow provisioner: kubernetes.io/gce-pd parameters: type: pd-standard -- kind: StorageClass apiversion: storage.k8s.io/v1 metadata: name: fast provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd

47 Dynamic Provisioning Users consume storage the same way: PVC Selecting a s tora ge cla s s in PVC triggers dyna mic provis ioning apiversion: v1 kind: PersistentVolumeClaim metadata: name: mypvc namespace: testns spec: accessmodes: - ReadWriteOnce resources: requests: storage: 100Gi storageclassname: fast

48 Dynamic Provisioning $ kubectl create -f storage_class.yaml storageclass "fast" created $ kubectl create -f pvc.yaml persistentvolumeclaim "mypvc" created $ kubectl get pvc --all-namespaces NAMESPACE NAME STATUS VOLUME CAPACITY ACCESSMODES AGE testns mypvc Bound pvc-331d7407-fe18-11e6-b7cd-42010a8000cd 100Gi RWO 6s $ kubectl get pv pvc-331d7407-fe18-11e6-b7cd-42010a8000cd NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE pvc-331d7407-fe18-11e6-b7cd-42010a8000cd 100Gi RWO Delete Bound testns/mypvc 13m

49 Dynamic Provisioning Volume referenced via PVC apiversion: v1 kind: Pod metadata: name: sleepypod spec: volumes: - name: data persistentvolumeclaim: claimname: mypvc containers: - name: sleepycontainer image: gcr.io/google_containers/busybox command: - sleep - "6000" volumemounts: - name: data mountpath: /data readonly: false

50 Hostpath Volumes Expose a directory on the host machine to pod What happens if your pod is moved to a different node? Don't use hostpath (unless you know what you are doing)!!

51 Local Persistent Volumes Expose a local block or file as a PersistentVolume Reduced dura bility Useful for building distributed storage systems Useful for high performance caching Kubernetes takes care of data gravity Referenced via PV/ PVC s o workload portability is maintained

52 In-Tree Volum e Plugins Kubernetes In-tree Volume Plugins are awesome =) Powerful abs traction for file and block s torage Automate provisioning, attaching, mounting, and more! Storage portability via PV/ PVC/ StorageClas s objects

53 In-Tree Volum e Plugins Kubernetes In-tree Volume Plugins are painful =( Painful for Kubernetes Developers Testing and maintaining external code Bugs in volume plugins affect critical Kubernetes components Volume plugins get full privileges of kubernetes components (kubelet and kube-controllermanager) Pa inful for Stora ge Vendors Dependent on Kubernetes releases Source code forced to be open source

54 Out-of-Tree Volum e Plugins Container Storage Interface (CSI) - Beta in v1.10; Targeting GA in v1.13 Follows in the s teps of CRI and CNI Collaboration with other cluster orchestration systems CSI makes Kubernetes volume layer truly extensible Plugins may be containerized Flex Volumes Legacy attempt at out-of-tree Exec based Deploym ent difficult Doesn't support clusters with no master access

55 Untapped Opportunities

56 Application Portability Legacy Software Local Execution Jurisdictional / PII Warehouse Europe Branch US Ecommerce site Catalog, ERP Factory IT policy Secure records Augmented Services Edge / IoT bursting On-Prem Storage ML Big Query

57 Snapshot Portability

58 Unified Observability

59 Uniform Management

60 The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it." - Mark Weiser, The Computer for the 21st Century

61 Questions? Get Involved! Container Storage Interface Community github.com/ conta iner-s torage-interfa ce/ com m unity Meeting every week, Wednesdays at 9 AM (PT) container-storage-interface-community@googlegroups.com Kubernetes Storage Special-Interest-Group (SIG) github.com/kubernetes/community/tree/master/sig-storage Meeting every 2 weeks, Thursdays at 9 AM (PST) kubernetes-s ig-s torage@googlegroups.com

Introduction to Kubernetes Storage Primitives for Stateful Workloads

September 12, 2017 Introduction to Kubernetes Storage Primitives for Stateful Workloads Saad Ali Google @the_saad_ali Chris Duchesne {code} @ChrisDuchesne Agenda Presentation Quick intro to Kubernetes