Kubernetes on Openstack

Size: px

Start display at page:

Download "Kubernetes on Openstack"

Clifford Williamson
5 years ago
Views:

1 Kubernetes on Openstack Saverio Proto Lugano 2018 SWITCH 1

2 Infrastructure & Data Services Your added value Our customers Our offer 2018 SWITCH 2

usage and charging model, no up-front investment Simple administration; integrated into the academic network of

3 Your added value SWITCH made Swiss made Swiss law and data location Scalable storage and computing power on demand with immediate availability in accordance to the need of and controlled by the institutions Flexible usage and charging model, no up-front investment Simple administration; integrated into the academic network of SWITCH; security and identity services included Support for academic use cases Created together with you 2018 SWITCH 3

4 Our customers Higher education Cantonal universities ETH domain with research institutions Universities of applied sciences Universities of teacher education University-related organizations Spin-Offs Research institutions elearning Center University hospitals 2018 SWITCH 4

5 Our offer SWITCHengines Virtual Private Cloud (VPC) SCALE-UP (Project)* * For developing academic services with 9 universities, as part of the Scientific Information projects mandated by swissuniversities SWITCH 5

6 SWITCHengines Customer tailored computing and storage performance for universities, research and teaching further developed in the SCALE-UP project mandated by swissuniversities. Customers Universities Research institutions elearning Center University hospitals Spin-Offs Services SWITCHengines (IaaS) Virtual Private Cloud (VPC) SCALE-UP (academic project) Your benefits Your data in Switzerland Integrated network and security Support for academic use cases Simple administration and billing Created together with you 2018 SWITCH 6

7 Saverio Proto Openstack Cloud Engineer at SWITCH. He works in NRENs since 2011, first in Italy and then in Switzerland. He has significant experience in running critical infrastructures using Open Source software. He works together with the Infrastructure & Data team, to deliver an Openstack based cloud to the Swiss Universities SWITCH 7

8 Kubernetes (K8s): what is it? Container orchestration. Open Source system to deploy containerized applications. What you should already know: Docker, build docker containers Run your application in a container What Kubernentes will do for me? Manage the lifecycle of containers Schedule the containers to the hosts Attach storage volumes to the containers Setup networking 2018 SWITCH 8

9 Kubernetes on Openstack K8s can interact with the Openstack API Use keystone for authentication Create volumes Create load balancers Set routes in Neutron routers 2018 SWITCH 9

10 Deploy K8s on Openstack Forked from This ansible playbook will create Openstack instances, and will install Kubernetes on them. export KEY=keyname export IMAGE="Ubuntu Xenial (SWITCHengines)" export NETWORK=k8s ( read REAME for complete variables list) ansible-playbook site.yaml 2018 SWITCH 10

11 Start! kubectl get nodes NAME STATUS AGE VERSION k8s-1 Ready 3m v1.9.4 k8s-2 Ready 3m v1.9.4 k8s-3 Ready 3m v1.9.4 k8s-master Ready 4m v SWITCH 11

12 Client config The playbook returns an admin.conf that looks like this: apiversion: v1 clusters: - cluster: certificate-authority-data: [..CUT..] server: name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: [..CUT..] client-key-data: [..CUT..] 2018 SWITCH 12

13 Client config kubectl is the command line tool ~/.kube/config is the config file Supports Openstack Keystone authentication kubectl is able to read usual Openstack OS_ env vars kubectl config set-credentials \ osuser --auth-provider=openstack source ~/openrc 2018 SWITCH 13

14 Client config Create a context to use the Openstack User kubectl config set-context \ --cluster=kubernetes \ --user=osuser osuser@kubernetes Lets use the context kubectl config use-context osuser@kubernetes 2018 SWITCH 14

15 Keystone authentication for k8s 1. Token issue Openstack Keystone Kubectl client 4. Token validation with keystone 2. Request with bearer token WebHook KubeApi server 3. Calling WebHook for token validation 2018 SWITCH 15

16 Configure the webhook kube-apiserver --authentication-token-webhook-config-file=webhook.kubeconfig.yaml --- apiversion: v1 clusters: - cluster: insecure-skip-tls-verify: true server: name: webhook contexts: - context: cluster: webhook user: webhook name: webhook current-context: webhook kind: Config preferences: {} users: - name: webhook 2018 SWITCH 16

17 Start the webhook container It is just a container: More recent source code can be found here: make image-k8s-keystone-auth 2018 SWITCH 17

18 Authorization via RBAC Keystone tested for Authentication Default RBAC for Kubernetes implements Authorization Keystone projects are mapped to Kubernetes Groups 2018 SWITCH 18

19 K8s Master interaction with Openstack /etc/kubernetes/cloud-config [Global] auth-url = username = saverio.proto@switch.ch password = ****** tenant-name = saverio.proto@switch.ch region = ZH [BlockStorage] trust-device-path = false bs-version = v2 [Route] router-id = 3a6cd142-91cd-4a04-9a86-a73455a0155d [LoadBalancer] lb-version = v2 floating-network-id = 3cc83f7d b-ba17-f3510c7902e8 subnet-id = 5e18c72c bd84-ec54cf SWITCH 19

20 Interaction with Openstack /etc/kubernetes/manifests/kube-controller-manager.yaml spec: containers: - command: - kube-controller-manager - --cloud-provider=openstack - --cloud-config=/etc/kubernetes/cloud-config 2018 SWITCH 20

21 Kubernetes Neutron Networking I changed my the way of doing Networking after the Openstack summit in Sydney Use --network-plugin=kubenet b /kubernetes-on-openstack-the-technical-details 2018 SWITCH 21

gateway is the neutron router VM aka k8s node Pod network assigned by the

22 Kubernetes Neutron Networking neutron router Has static routes to reach Pod Networks Default gateway is the neutron router VM aka k8s node Default gateway is the neutron router VM aka k8s node Pod network assigned by the k8s master Pod network assigned by the k8s master Pod Pod Pod Pod 2018 SWITCH 22

23 Persistent Storage Docker containers run into Openstack instances Persistent storage is cinder volumes K8s will attach the Cinder Volume to the right Openstack instance were the docker container is scheduled volumes: - name: mysql-persistent-storage cinder: volumeid: <uuid> fstype: ext SWITCH 23

24 Persistent storage This method is not practical we have to manage Cinder volumes on the side Export UUID Solution Define a Storage Class 2018 SWITCH 24

25 Storage Class kind: StorageClass apiversion: storage.k8s.io/v1 metadata: name: default annotations: storageclass.beta.kubernetes.io/isdefault-class: "true" provisioner: kubernetes.io/cinder parameters: type: fast availability: nova 2018 SWITCH 25

26 Persistent Storage pvc kind: PersistentVolumeClaim apiversion: v1 metadata: name: mysql-pvc annotations: volume.beta.kubernetes.io/storage-class: default spec: accessmodes: - ReadWriteOnce resources: requests: storage: 30Gi 2018 SWITCH 26

27 Persistent Storage with pvc volumes: - name: mysql-persistent-storage persistentvolumeclaim: claimname: mysql-pvc 2018 SWITCH 27

28 Openstack LBaaSv2 with K8s I suggest not to use it, at least in Openstack Newton Both DNAT and SNAT are necessary you lose the client IP information IPv6 support status is unknown to me It is a pure L4 loadbalancer, cannot terminate TLS connections or rewrite HTTP headers 2018 SWITCH 28

29 Openstack LBaaSv2 Svc FloatingIP configured here Both DNAT and SNAT are necessary here. neutron LBaaS neutron router Openstack project network VM aka k8s node Default gateway is the neutron router VM aka k8s node Default gateway is the neutron router Pod network assigned by the k8s master Pod network assigned by the k8s master Pod Pod Pod Pod 2018 SWITCH 29

30 Kubernetes NGINX Ingress An API object that manages external access to the services in a cluster, typically HTTP. Ingress can provide load balancing, TLS termination and name-based virtual hosting. At least two known implementations Nginx based HAProxy 2018 SWITCH 30

master VM aka k8s node VM aka k8s node Default network namespace (docker net=host) Kubernetes-nginx-ingress

31 Kubernetes NGINX Ingress It is a special privileged Docker container running NGINX On the k8s master (s) or dedicated VM Has access to external IP addresses and cluster IP addresses Kubernetes cluster network VM k8s master VM aka k8s node VM aka k8s node Default network namespace (docker net=host) Kubernetes-nginx-ingress Pod network assigned by the k8s master Pod network assigned by the k8s master Pod Pod Pod Pod 2018 SWITCH 31

32 How I use the Ingress? 2018 SWITCH 32

33 Getting involved Slack channel #sig-openstack Join via Follow our cloud blog: Get in touch with me 2018 SWITCH 33

34 2018 SWITCH 34 Thank you! Questions?

VMware Integrated OpenStack with Kubernetes Getting Started Guide. VMware Integrated OpenStack 4.0

VMware Integrated OpenStack with Kubernetes Getting Started Guide VMware Integrated OpenStack 4.0 VMware Integrated OpenStack with Kubernetes Getting Started Guide You can find the most up-to-date technical