Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 CNA2080BU Deep Dive: How to Deploy and Operationalize Kubernetes Cornelia Davis, Pivotal Nathan Ness Technical Product Manager, #VMworld #CNA2080BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #CNA2080BU CONFIDENTIAL 2

3 Agenda 1 What is the need? 2 Introducing the toolchain 3 Pivotal Container Service (PKS) 4 PKS Day 1 5 PKS Day 2 #CNA2080BU CONFIDENTIAL 3

4 The Need for Operationalizing Kubernetes VMworld 2017 Content: Not for publication

5 Companies Have Many Ways to Package and Run Their Workloads in the Cloud MICROSERVICES CONTAINERS DATA SERVICES BATCHES MONOLITHIC APPLICATIONS EVENT-DRIVEN FUNCTIONS #CNA2080BU CONFIDENTIAL 5

6 Workloads that Might Be Suitable for Kubernetes Those: Requiring Persistence MongoDB, CouchDB, Couchbase, Elastic Search, Managed as a cluster nodes need to communicate with one another often with the help of service meshes such as Istio or Linkerd Spark, Elastic Search Needing new architectural primitives Misc things like multiple ports, etc. #CNA2080BU CONFIDENTIAL 6

7 Serving up Kubernetes Dial-tone kubectl Responsible for the workloads running in K8s Master Master Kubernetes Routing manage Responsible for the K8s cluster(s) themselves #CNA2080BU CONFIDENTIAL 7

8 Operational Challenges with Any Platform Day 1 - Build Multi-cloud Provide a reliable and smooth experience for any cloud. Open APIs Allow platform operations from different toolsets and the creation of CD pipelines. Consistency Provide a consistent setup experience, across different cloud environment configurations. Setup time How long does it take to setup a real world working environment? Think hours, not weeks. Day 2 - Operate Patches Patching platform components with thousands of apps running should feel normal. Scaling Seamlessly scale platform components to accommodate changing demand. Upgrades How do you roll out new versions of the platform with the lights on? Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits? #CNA2080BU CONFIDENTIAL 8

9 Kubernetes - Especially Hard to Operationalize VMworld 2017 High Availability. No out-of-the-box fault-tolerance for the cluster components themselves (masters, workers and nodes). Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn t provide a mechanism to scale s, Masters & VMs. Health checks and healing. The Kubernetes cluster only does routine health checks for the health of workloads running on Nodes. Content: Not for publication Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on? #CNA2080BU CONFIDENTIAL 9

10 Introducing BOSH

11 Powered by BOSH Pivotal container service ops BOSH BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. Packaging w/ embedded OS Server provisioning on any IaaS Software deployment across availability zones Health monitoring (server AND processes) Self-healing w/ Resurrector Storage management Rolling upgrades via canaries Easy scaling of clusters #CNA2080BU CONFIDENTIAL 11

12 Powered by BOSH Pivotal container service ops Master Master BOSH Kubernetes Packaging w/ embedded OS Server provisioning on any IaaS Software deployment across availability zones Health monitoring (server AND processes) Self-healing w/ Resurrector Storage management Rolling upgrades via canaries Easy scaling of clusters #CNA2080BU CONFIDENTIAL 12

13 Primary BOSH Entities Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment Relationship to the underlying infrastructure BOSH cloud config #CNA2080BU CONFIDENTIAL 13

14 The Workflow Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release STEP 2: Install and Manage Kubernetes A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment Relationship to the underlying infrastructure STEP 1: Install and configure BOSH BOSH cloud config #CNA2080BU CONFIDENTIAL 14

15 Pivotal Container Service (PKS)

16 Project Kubo Uniform way to instantiate, deploy, and manage highly available Kubernetes clusters. On any cloud. Launched by Pivotal & Google Feb 2017, Donated to Cloud Foundry Foundation June 2017 VMworld 2017 Day 1 Build Deploy Kubernetes cluster via BOSH Day 2 Operate Self-healing VMs and monitoring via BOSH Elastic scaling for clusters Rolling upgrades to latest Kubernetes release High-availability and multi-az support Content: Not for publication #CNA2080BU CONFIDENTIAL 16

17 Kubo Provides Specification of K8S Components Kubo Release Release templates Manifest This forms the Open Core of Pivotal Container Service (PKS) bosh deploy Master Master BOSH Kubernetes #CNA2080BU CONFIDENTIAL 17

18 Provides the control plane for provisioning and managing Kubo releases Joint development effort between Pivotal, VMWare and Google Kubernetes Dial Tone: Health management Aggregated Metrics and Logging Autoscaling Persistence interface Control Plane: Provisioning Engine Self-service Clusters Software Update Automation Load balancing Networking Multi-tenancy #CNA2080BU CONFIDENTIAL 18

19 PKS Leverages the Power of BOSH Kubo Release Release templates Manifest PKS BOSH 19

20 Kubernetes Cluster Day 1 Deploy

21 Starting with a BOSH Deployment... Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment #CNA2080BU CONFIDENTIAL 21

22 Deploying a Kubernetes Cluster with Cloud Foundry BOSH Deploy my K8s Deployment Packages Blobs Source Jobs Manifest DB Blobs Health Monitor BOSH Director Message Bus BOSH Master VMs Target VM Target VM Target VM vsphere #CNA2080BU CONFIDENTIAL 22

23 Kubernetes Cluster Day 2 Operationalize

24 Day 2: Operationalize 1 Managing Health 2 Scaling 3 Upgrade

25 K8s Cluster Health: Processes are Monitored Health Monitor Responses: pager monitoring Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 25

26 K8s Cluster Health: Processes are Monitored Health Monitor Responses: pager monitoring Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 26

27 K8s Cluster Health: Processes are Monitored Health Monitor Responses: pager monitoring Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 27

28 K8s Cluster Health: VMs are Monitored Desired State Health Monitor Responses: pager monitoring ressurector Actual State BOSH Director Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 28

29 K8s Cluster Health: VMs are Monitored Desired State Health Monitor Responses: pager monitoring ressurector Actual State BOSH Director Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 29

30 K8s Cluster Health: VMs are Monitored Desired State Health Monitor Responses: pager monitoring ressurector Actual State BOSH Director Message Bus CPI AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 30

31 Day 2: Operationalize 1 Managing Health 2 Scaling 3 Upgrade #CNA2080BU CONFIDENTIAL 31

32 Primary BOSH Entities Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment Relationship to the underlying infrastructure BOSH cloud config #CNA2080BU CONFIDENTIAL 32

33 instance_groups: - name: - name: master instances: 3 instances: 2 networks: networks: - name: &network-name ((deployments_network)) Manifest azs: [z1] jobs: - name: release: kubo- properties: : require_ssl: false peer_require_ssl: false stemcell: trusty vm_type: common persistent_disk_type: name: *network-name azs: [z1] jobs: - name: cloud-provider release: kubo properties: {} - name: kubernetes-api release: kubo properties: - name: worker instances: 3 networks: - name: *network-name azs: [z1] jobs: - name: docker release: docker properties:... - name: kubeconfig release: kubo admin-username: admin admin-password: ((kubo-admin-password)) properties:... - name: kubeconfig release: kubo properties: stemcell: trusty vm_type: master... - name: kubelet release: kubo properties:... - name: kubernetes-proxy release: kubo properties:... stemcell: trusty vm_type: worker persistent_disk_type:

34 instance_groups: - name: - name: master instances: 3 instances: 2 networks: networks: - name: &network-name ((deployments_network)) azs: [z1] jobs: - name: release: kubo- properties: : require_ssl: false peer_require_ssl: false stemcell: trusty vm_type: common persistent_disk_type: name: *network-name azs: [z1] jobs: - name: cloud-provider release: kubo properties: {} - name: kubernetes-api release: kubo properties: Scaling is a matter of changing the number of instances and telling BOSH to make it so - name: worker instances: 3 networks: - name: *network-name azs: [z1] jobs: - name: docker release: docker properties:... - name: kubeconfig release: kubo admin-username: admin admin-password: ((kubo-admin-password)) properties:... - name: kubeconfig release: kubo properties: stemcell: trusty vm_type: master... - name: kubelet release: kubo properties:... - name: kubernetes-proxy release: kubo properties:... stemcell: trusty vm_type: worker persistent_disk_type:

35 Day 2: Operationalize 1 Managing Health 2 Scaling 3 Upgrade

36 K8s Cluster Upgrade: Canary Deployments Manifest update: canaries: 1 max_in_flight: 1 serial: true canary_watch_time: update_watch_time: VMworld 2017 Content: Not for publication #CNA2080BU CONFIDENTIAL 36

37 K8s Cluster Upgrade: Canary Deployments EXAMPLE: # OF CANARIES: 2 MAX IN FLIGHT: 2 CANARIES V1.0 V1.1 #CNA2080BU CONFIDENTIAL 37

38 K8s Cluster Upgrade: Canary Deployments EXAMPLE: # OF CANARIES: 2 MAX IN FLIGHT: 2 V1.1 Once failed, Canary VMs are kept V1.2 for troubleshooting purposes. #CNA2080BU CONFIDENTIAL 38

39 Operationalizing at Scale

40 Supporting Kubernetes Needs at Scale Kubo Release Release templates Manifest PKS Service Broker BOSH 40

41 Supporting Kubernetes Needs at Scale create cluster (with upgrade policy) Thousands Kubo Release Release templates Manifest PKS Service Broker BOSH manage Ones 41

42 Let Us Show You

43 #CNA2080BU CONFIDENTIAL 43

44 PaaS Control Plane NSX-T Integration API-Server Scheduler NSX Container Plugin Kubernetes Adapter CloudFoundry Adapter Mesos Adapter Libnetwork Adapter NCM Infra NSX Container Plugin (NCP) for integrating with Kubernetes NSX Features for K8s PODs IP address per container / POD Container Network Routed (BGP) & NATed mode NSX Manager API Client Microsegmentation via K8s Network Policy or native NSX APIs (mapping K8s labels to NSX tags) Network & Security automation created as part of app deployment Multi-tenant network topologies VMworld 2017 NSX topology for K8s / CF Content: Not for publication Proj: foo Proj: bar

45 vrealize Ops, vrealize Log Insight For Comprehensive Visibility VMware vrealize Operations Capacity, Performance and Configuration Management Structured Data Metrics Alerts Events Launch in Context Events Virtual Applications VMware vrealize Log Insight Log analytics, aggregation, and search Logs Unstructured Data Messages or distribution

46 46 vrealize Ops Managing Kubernetes Clusters K8S Summary Nodes, Pods, etc. K8S Topology - Health K8S Pods - Health

47 47 vrealize Ops Kubernetes Integration Details K8S Alerts K8S Alerts K8S Pod Relationship to Components

48 Introducing Wavefront By VMware SaaS-Based Metrics Monitoring and Analytics Platform Iterate & Troubleshoot Issues Trend & Alert on Anomalies UI and API Backend Advanced Analytics Engine Metrics Collection and Storage Visualize Metrics at Scale Self-Service Metrics Analytics for All Engineering & Business 4

49 Real-time insight into Docker containers and orchestration systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS Wavefront Container Monitoring Suite Amazon ECS App Containers Docker Host Container Metric Collector Docker Swarm Docker Host Docker Host Docker Cluster

50 50 Registry Enterprise-grade Private Registry Need Harbor screenshot user management & access control role-based access control AD/LDAP integration security vulnerability scanning content trust - image signing policy based image replication audit and logs restful API lightweight & easy deployment open-source under Apache 2 license

51 51 Registry Content Trust, When Enabled Un-signed Images Can t Be Pulled

52 52 Registry Image Vulnerability Scanning Details

53 Analytics Logging Operations Automation Monitoring Security Container Registry VMworld 2017 vsphere master VMware PKS Kubernetes on BOSH (Kubo) worker NSX BOSH master vsan worker GCP Service Broker Content: Not for publication Physical Infrastructure

54

55