Kubernetes Love at first sight?

Transcription

1 Kubernetes Love at first sight? 15, February 2018 Joost Hofman (Lead Albert Heijn IT Online) Milo van der zee (Senior Heijn IT Online)

2 Agenda Kubernetes Why at AH? How? Questions Relational problems Is it real love?

3 Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications.

4 Kubernetes - Searches

5 Kubernetes Service POD POD 1 n 1 n

6 Kubernetes Operator / Developer API Server Kubernetes Master Controller Manager Scheduler ETCD Users Kubelet cadvisor kube-proxy Kubelet cadvisor kube-proxy Pod Pod Pod Pod Up to 5000 Pod Pod Pod Pod Kubernetes Node Plugin Network - Calico Kubernetes Node

7 Kubernetes $ nodes NAME STATUS ROLES AGE VERSION k8snode2098 Ready,SchedulingDisabled master 12d v1.8.4+coreos.0 k8snode2099 Ready,SchedulingDisabled master 12d v1.8.4+coreos.0 k8snode2100 Ready,SchedulingDisabled master 12d v1.8.4+coreos.0 k8snode2101 Ready node 12d v1.8.4+coreos.0 k8snode2102 Ready node 12d v1.8.4+coreos.0 k8snode2103 Ready node 12d v1.8.4+coreos.0 k8snode2104 Ready node 12d v1.8.4+coreos.0 k8snode2105 Ready node 12d v1.8.4+coreos.0 k8snode2107 Ready node 12d v1.8.4+coreos.0 k8snode2108 Ready node 12d v1.8.4+coreos.0 k8snode2109 Ready node 12d v1.8.4+coreos.0 k8snode2110 Ready node 12d v1.8.4+coreos.0 k8snode2111 Ready node 12d v1.8.4+coreos.0

8 Kubernetes $ pods -o wide NAME READY STATUS IP NODE shoppinglist-widget q7c1x 1/1 Running k8snode1657 subscription-service-8cc4c97fb-dh9zz 1/1 Running k8snode1656 subscription-service-8cc4c97fb-t7wrj 1/1 Running k8snode1651 taxonomy-neo4j-neo4j-core-0 1/1 Running k8snode1814 taxonomy-neo4j-neo4j-core-1 1/1 Running k8snode1651 taxonomy-neo4j-neo4j-core-2 1/1 Running k8snode1813 taxonomy-service-7b4fb7f8d5-c6mvb 1/1 Running k8snode1813 taxonomy-service-7b4fb7f8d5-h2hjk 1/1 Running k8snode1655 gateway r22 1/1 Running k8snode1814 gateway lqzk 1/1 Running k8snode1655 gateway fkt9k 1/1 Running k8snode1654 gateway ls9pv 1/1 Running k8snode1813

9 Pods kubectl describe pod api gateway # kubectl -n online-prd describe pod gateway r22 Name: gateway r22 Namespace: online-prd Node: k8snode1814/ Start Time: Wed, 14 Feb :12: Labels: name=gateway Status: Running IP: Containers: gateway: Image: regisry-docker.online.ah.nl:443/ah-open-api-gateway:0.1.2 Port: <none>

10 Service - kubectl describe svc api gateway # kubectl describe svc gateway Name: gateway Namespace: online-prd Labels: run=gateway Annotations: kubectl.kubernetes.io/last-appliedconfiguration={"apiversion":"v1","kind":"service","metadata":{"annotations":{},"labels":{"ru n":"gateway"},"name":"gateway","namespace":"online-prd"},"spec":{"ports":... Selector: run=gateway Type: ClusterIP IP: Port: <unset> 8080/TCP TargetPort: 8080/TCP Endpoints: :8080, :8080, : more... Session Affinity: None Events: <none>

11 api service iptables -A KUBE-SERVICES -d /32 -p tcp -m tcp --dport 443 -j SVC-JFMNS -A SVC-JFMNS --mode random --probability j KUBE-SEP-JPX2Q -A SVC-JFMNS --mode random --probability j KUBE-SEP-KUJYT -A SVC-JFMNS --mode random --probability 0.5 -j KUBE-SEP-HTGFR -A SVC-JFMNS --mode random -j KUBE-SEP-JP5GT -A SEP-JPX2Q -p tcp -m recent -j DNAT --to-destination :6443

12 Albert Heijn? 2015 Monolith Binary coupling Scalability problems Growth issues CI/CD impossible Downtime NOW and future Scalable Decoupling Rolling updates Services CI/CD to the max Isolation of code Zero downtime Technology agnostic

13 Albert Heijn? Scalable architecture and technology on a modern, scalable, automated platform Containers - Fully automated - Within minutes Container management platform Virtual hardware - Semi-automated - Within weeks Virtualization Commodity hardware - Manual - Within months

14 On Premise VS Cloud No cloud options in 2016 and 2017

15 How?

16 How?

17 A HTTP call to appietoday.nl Users Nginx - Ingress Frontend (service) Loadbalancer Frontend (pod) API Gateway (service) API Gateway (pod) IDP (service) API (service) IDP (pod) API (pod)

18 Our setup? Frontend API Gateway Services Platform Continuous delivery 25+ services Continuous delivery Automated from development to production Authorization Authentication Throttling Routing 5 Clusters 40+ nodes 650+ Docker containers Automate platform deployment with Ansible

19 Relational problems: Communication.

20 Relational problems: Storage. On premise Storage Host path NFS vsphere volumes

21 Relational problems: Storage. On premise Storage GlusterFS

22 Relational problems: Postgres on Gluster. pg_restore: [archiver (db)] Error from TOC entry 53398; TABLE DATA l1aaux_sci sdmcleod pg_restore: [archiver (db)] COPY failed for table "l1aaux_sci": ERROR: unexpected data beyond EOF in block 9391 of relation base/16386/17043 HINT: This has been seen to occur with buggy kernels; consider updating your system. CONTEXT: COPY l1aaux_sci, line : " \N :04: :04: " pg_restore: [archiver (db)] Error from TOC entry 53399; TABLE DATA l1afts_dbl sdmcleod pg_restore: [archiver (db)] COPY failed for table "l1afts_dbl": ERROR: unexpected data beyond EOF in block of relation base/16386/17068 HINT: This has been seen to occur with buggy kernels; consider updating your system.

23 Relational problems: Postgres on Gluster. postgres source code: src/backend/storage/buffer/bufmgr.c /* * We get here only in the corner case where we are trying to extend * the relation but we found a pre-existing buffer marked BM_VALID. * This can happen because mdread doesn't complain about reads beyond * EOF (when zero_damaged_pages is ON) and so a previous attempt to * read a block beyond EOF could have left a "valid" zero-filled * buffer. Unfortunately, we have also seen this case occurring * because of buggy Linux kernels that sometimes return an * lseek(seek_end) result that doesn't account for a recent write. In * that situation, the pre-existing buffer would contain valid data * that we don't want to overwrite. Since the legitimate case should * always have left a zero-filled buffer, complain if not PageIsNew. */ bufblock = islocalbuf? LocalBufHdrGetBlock(bufHdr) : BufHdrGetBlock(bufHdr); if (!PageIsNew((Page) bufblock)) ereport(error, (errmsg("unexpected data beyond EOF in block %u of relation %s", blocknum, relpath(smgr->smgr_rnode, forknum)), errhint("this has been seen to occur with buggy kernels; consider updating your system.")));

24 Relational problems: Communication. Nodes can t reach each other anymore KubeProxy can t reach API iptables are broken Network interface changes Subnet Flannel and Docker mismatch (magicly)

25 Relational problems: Communication. Nodes can t reach each other anymore Migration from Flannel to Calico resulted in a small downtime but a very stable network afterwards Created a Network test DaemonSet, as our own relation therapist

26 Relational problems: Communication. ~]# bridge fdb grep cali 33:33:00:00:00:01 dev calif8b8ce32fae self permanent 01:00:5e:00:00:01 dev calif8b8ce32fae self permanent... ~]# ip -d link show calif8b8ce32fae 8: <BROADCAST,MULTICAST,UP,LOWER_UP> state UP mode DEFAULT link/ether 7e:3f:ee:5e:d4:ed brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 veth addrgenmode eui64 [prd-node1:pnlmv17y@k8snode1650 ~]$ route -n grep cali UH 0 calif8b8ce32fae UH 0 cali2b5d60cd0be UH 0 cali9fa8da UH 0 cali4c2e295795a UH 0 cali5c975203c3b

27 Relational problems: Communication. ~]$ ip addr... 13: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN qlen 1 link/ipip brd inet /32 scope global tunl0 valid_lft forever preferred_lft forever... [pnlmv17y@k8snode2101 ~]$ route -n grep tunl UG tunl UG tunl UG tunl UG tunl0

28 Relational problems: Communication. Not much knowledge about Calico... And that is a good thing. It just works. We know a lot more about Flannel and that also says enough...

29 Relation problems: Containers drop

30 Relation problems: Communication. Kubernetes Master Network Test (Pod) DS Kube DNS (pod) Network Test (Pod) DS Network Test (Pod) DS Kube DNS (service) Kube DNS (pod) Kubernetes Node Kubernetes Node

31 Kubernetes gives more benefits than doubts on premise A lot of open source tools around Helm packages Fast delivery of software Auto healing Very very stable (Only got called out of bed once at night in 2017) Happy developers Enabler for DevOps Etc..

32 Open source tools that boosts our relationship

33 Projects that boosts our relationship Easily deploying production-ready Kubernetes clusters. Kubespray saved months of work setting up Kubernetes on premise.

34 Projects that boosts our relationship Package manager for Kubernetes Helm makes upgrading and maintaining our applications predictable and super easy.

35 Love Joost Milo

36 Questions?