CNA2080BU Deep Dive: How to Deploy and Operationalize Kubernetes Cornelia Davis, Pivotal Nathan Ness Technical Product Manager, CNABU @nvpnathan #VMworld #CNA2080BU
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #CNA2080BU CONFIDENTIAL 2
Agenda 1 What is the need? 2 Introducing the toolchain 3 Pivotal Container Service (PKS) 4 PKS Day 1 5 PKS Day 2 #CNA2080BU CONFIDENTIAL 3
The Need for Operationalizing Kubernetes VMworld 2017 Content: Not for publication
Companies Have Many Ways to Package and Run Their Workloads in the Cloud MICROSERVICES CONTAINERS DATA SERVICES BATCHES MONOLITHIC APPLICATIONS EVENT-DRIVEN FUNCTIONS #CNA2080BU CONFIDENTIAL 5
Workloads that Might Be Suitable for Kubernetes Those: Requiring Persistence MongoDB, CouchDB, Couchbase, Elastic Search, Managed as a cluster nodes need to communicate with one another often with the help of service meshes such as Istio or Linkerd Spark, Elastic Search Needing new architectural primitives Misc things like multiple ports, etc. #CNA2080BU CONFIDENTIAL 6
Serving up Kubernetes Dial-tone kubectl Responsible for the workloads running in K8s Master Master Kubernetes Routing manage Responsible for the K8s cluster(s) themselves #CNA2080BU CONFIDENTIAL 7
Operational Challenges with Any Platform Day 1 - Build Multi-cloud Provide a reliable and smooth experience for any cloud. Open APIs Allow platform operations from different toolsets and the creation of CD pipelines. Consistency Provide a consistent setup experience, across different cloud environment configurations. Setup time How long does it take to setup a real world working environment? Think hours, not weeks. Day 2 - Operate Patches Patching platform components with thousands of apps running should feel normal. Scaling Seamlessly scale platform components to accommodate changing demand. Upgrades How do you roll out new versions of the platform with the lights on? Operating Effort Operating a platform should require very few resources and minimum manual intervention. Otherwise, is it really providing operational benefits? #CNA2080BU CONFIDENTIAL 8
Kubernetes - Especially Hard to Operationalize VMworld 2017 High Availability. No out-of-the-box fault-tolerance for the cluster components themselves (masters, workers and nodes). Scaling. Kubernetes clusters handle scaling the pod/service within the Nodes, but doesn t provide a mechanism to scale s, Masters & VMs. Health checks and healing. The Kubernetes cluster only does routine health checks for the health of workloads running on Nodes. Content: Not for publication Upgrades. Rolling upgrades on a large fleet of clusters is hard. Who manages the system it runs on? #CNA2080BU CONFIDENTIAL 9
Introducing BOSH
Powered by BOSH Pivotal container service ops BOSH BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems. Packaging w/ embedded OS Server provisioning on any IaaS Software deployment across availability zones Health monitoring (server AND processes) Self-healing w/ Resurrector Storage management Rolling upgrades via canaries Easy scaling of clusters #CNA2080BU CONFIDENTIAL 11
Powered by BOSH Pivotal container service ops Master Master BOSH Kubernetes Packaging w/ embedded OS Server provisioning on any IaaS Software deployment across availability zones Health monitoring (server AND processes) Self-healing w/ Resurrector Storage management Rolling upgrades via canaries Easy scaling of clusters #CNA2080BU CONFIDENTIAL 12
Primary BOSH Entities Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment Relationship to the underlying infrastructure BOSH cloud config #CNA2080BU CONFIDENTIAL 13
The Workflow Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release STEP 2: Install and Manage Kubernetes A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment Relationship to the underlying infrastructure STEP 1: Install and configure BOSH BOSH cloud config #CNA2080BU CONFIDENTIAL 14
Pivotal Container Service (PKS)
Project Kubo Uniform way to instantiate, deploy, and manage highly available Kubernetes clusters. On any cloud. Launched by Pivotal & Google Feb 2017, Donated to Cloud Foundry Foundation June 2017 VMworld 2017 Day 1 Build Deploy Kubernetes cluster via BOSH Day 2 Operate Self-healing VMs and monitoring via BOSH Elastic scaling for clusters Rolling upgrades to latest Kubernetes release High-availability and multi-az support Content: Not for publication #CNA2080BU CONFIDENTIAL 16
Kubo Provides Specification of K8S Components Kubo Release Release templates Manifest This forms the Open Core of Pivotal Container Service (PKS) bosh deploy Master Master BOSH Kubernetes #CNA2080BU CONFIDENTIAL 17
Provides the control plane for provisioning and managing Kubo releases Joint development effort between Pivotal, VMWare and Google Kubernetes Dial Tone: Health management Aggregated Metrics and Logging Autoscaling Persistence interface Control Plane: Provisioning Engine Self-service Clusters Software Update Automation Load balancing Networking Multi-tenancy #CNA2080BU CONFIDENTIAL 18
PKS Leverages the Power of BOSH Kubo Release Release templates Manifest PKS BOSH 19
Kubernetes Cluster Day 1 Deploy
Starting with a BOSH Deployment... Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment #CNA2080BU CONFIDENTIAL 21
Deploying a Kubernetes Cluster with Cloud Foundry BOSH Deploy my K8s Deployment Packages Blobs Source Jobs Manifest DB Blobs Health Monitor BOSH Director Message Bus BOSH Master VMs Target VM Target VM Target VM vsphere #CNA2080BU CONFIDENTIAL 22
Kubernetes Cluster Day 2 Operationalize
Day 2: Operationalize 1 Managing Health 2 Scaling 3 Upgrade
K8s Cluster Health: Processes are Monitored Health Monitor Responses: pager email monitoring Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 25
K8s Cluster Health: Processes are Monitored Health Monitor Responses: pager email monitoring Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 26
K8s Cluster Health: Processes are Monitored Health Monitor Responses: pager email monitoring Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 27
K8s Cluster Health: VMs are Monitored Desired State Health Monitor Responses: pager email monitoring ressurector Actual State BOSH Director Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 28
K8s Cluster Health: VMs are Monitored Desired State Health Monitor Responses: pager email monitoring ressurector Actual State BOSH Director Message Bus AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 29
K8s Cluster Health: VMs are Monitored Desired State Health Monitor Responses: pager email monitoring ressurector Actual State BOSH Director Message Bus CPI AGENT AGENT AGENT Master BOSH vsphere #CNA2080BU CONFIDENTIAL 30
Day 2: Operationalize 1 Managing Health 2 Scaling 3 Upgrade #CNA2080BU CONFIDENTIAL 31
Primary BOSH Entities Master Master BOSH Kubernetes The definition of each of the nodes in the cluster, including: The The definition bits installed of each on of a the node nodes (packages) in the cluster, The processes including: started on a node (jobs) Parameterized The bits installed on a node (packages) The processes started on a node (jobs) BOSH release A declaration of the desired state of the cluster: Assembly of the components from BOSH releases (relationships, dependencies) Parameter values BOSH deployment Relationship to the underlying infrastructure BOSH cloud config #CNA2080BU CONFIDENTIAL 32
instance_groups: - name: - name: master instances: 3 instances: 2 networks: networks: - name: &network-name ((deployments_network)) Manifest azs: [z1] jobs: - name: release: kubo- properties: : require_ssl: false peer_require_ssl: false stemcell: trusty vm_type: common persistent_disk_type: 5120 - name: *network-name azs: [z1] jobs: - name: cloud-provider release: kubo properties: {} - name: kubernetes-api release: kubo properties: - name: worker instances: 3 networks: - name: *network-name azs: [z1] jobs: - name: docker release: docker properties:... - name: kubeconfig release: kubo admin-username: admin admin-password: ((kubo-admin-password)) properties:... - name: kubeconfig release: kubo properties:...... stemcell: trusty vm_type: master... - name: kubelet release: kubo properties:... - name: kubernetes-proxy release: kubo properties:... stemcell: trusty vm_type: worker persistent_disk_type: 10240 33
instance_groups: - name: - name: master instances: 3 instances: 2 networks: networks: - name: &network-name ((deployments_network)) azs: [z1] jobs: - name: release: kubo- properties: : require_ssl: false peer_require_ssl: false stemcell: trusty vm_type: common persistent_disk_type: 5120 - name: *network-name azs: [z1] jobs: - name: cloud-provider release: kubo properties: {} - name: kubernetes-api release: kubo properties: Scaling is a matter of changing the number of instances and telling BOSH to make it so - name: worker instances: 3 networks: - name: *network-name azs: [z1] jobs: - name: docker release: docker properties:... - name: kubeconfig release: kubo admin-username: admin admin-password: ((kubo-admin-password)) properties:... - name: kubeconfig release: kubo properties:...... stemcell: trusty vm_type: master... - name: kubelet release: kubo properties:... - name: kubernetes-proxy release: kubo properties:... stemcell: trusty vm_type: worker persistent_disk_type: 10240 34
Day 2: Operationalize 1 Managing Health 2 Scaling 3 Upgrade
K8s Cluster Upgrade: Canary Deployments Manifest update: canaries: 1 max_in_flight: 1 serial: true canary_watch_time: 10000-300000 update_watch_time: 10000-300000 VMworld 2017 Content: Not for publication #CNA2080BU CONFIDENTIAL 36
K8s Cluster Upgrade: Canary Deployments EXAMPLE: # OF CANARIES: 2 MAX IN FLIGHT: 2 CANARIES V1.0 V1.1 #CNA2080BU CONFIDENTIAL 37
K8s Cluster Upgrade: Canary Deployments EXAMPLE: # OF CANARIES: 2 MAX IN FLIGHT: 2 V1.1 Once failed, Canary VMs are kept V1.2 for troubleshooting purposes. #CNA2080BU CONFIDENTIAL 38
Operationalizing at Scale
Supporting Kubernetes Needs at Scale Kubo Release Release templates Manifest PKS Service Broker BOSH 40
Supporting Kubernetes Needs at Scale https://thenewstack.io/comcast-1500-developers-working-cloud-foundry create cluster (with upgrade policy) Thousands Kubo Release Release templates Manifest PKS Service Broker BOSH manage Ones 41
Let Us Show You
#CNA2080BU CONFIDENTIAL 43
PaaS Control Plane NSX-T Integration API-Server Scheduler NSX Container Plugin Kubernetes Adapter CloudFoundry Adapter Mesos Adapter Libnetwork Adapter NCM Infra NSX Container Plugin (NCP) for integrating with Kubernetes NSX Features for K8s PODs IP address per container / POD Container Network Routed (BGP) & NATed mode NSX Manager API Client Microsegmentation via K8s Network Policy or native NSX APIs (mapping K8s labels to NSX tags) Network & Security automation created as part of app deployment Multi-tenant network topologies VMworld 2017 NSX topology for K8s / CF Content: Not for publication Proj: foo Proj: bar
vrealize Ops, vrealize Log Insight For Comprehensive Visibility VMware vrealize Operations Capacity, Performance and Configuration Management Structured Data Metrics Alerts Events Launch in Context Events Virtual Applications VMware vrealize Log Insight Log analytics, aggregation, and search Logs Unstructured Data Messages or distribution
46 vrealize Ops Managing Kubernetes Clusters K8S Summary Nodes, Pods, etc. K8S Topology - Health K8S Pods - Health
47 vrealize Ops Kubernetes Integration Details K8S Alerts K8S Alerts K8S Pod Relationship to Components
Introducing Wavefront By VMware SaaS-Based Metrics Monitoring and Analytics Platform Iterate & Troubleshoot Issues Trend & Alert on Anomalies UI and API Backend Advanced Analytics Engine Metrics Collection and Storage Visualize Metrics at Scale Self-Service Metrics Analytics for All Engineering & Business 4
Real-time insight into Docker containers and orchestration systems Kubernetes, Pivotal Cloud Foundry, Amazon ECS Wavefront Container Monitoring Suite Amazon ECS App Containers Docker Host Container Metric Collector Docker Swarm Docker Host Docker Host Docker Cluster
50 Registry Enterprise-grade Private Registry Need Harbor screenshot user management & access control role-based access control AD/LDAP integration security vulnerability scanning content trust - image signing policy based image replication audit and logs restful API lightweight & easy deployment open-source under Apache 2 license
51 Registry Content Trust, When Enabled Un-signed Images Can t Be Pulled
52 Registry Image Vulnerability Scanning Details
Analytics Logging Operations Automation Monitoring Security Container Registry VMworld 2017 vsphere master VMware PKS Kubernetes on BOSH (Kubo) worker NSX BOSH master vsan worker GCP Service Broker Content: Not for publication Physical Infrastructure