Cisco Virtual Update Container networking. Hans Donnerborg, Lars Granberg, Maj 2018

Transcription

1 Cisco Virtual Update Container networking Hans Donnerborg, Lars Granberg, Maj 2018

2 Why ACI for Application Container Platforms Turnkey solution for node and container connectivity Flexible policy: Native platform policy API and ACI policies Hardware-accelerated: Integrated load balancing Fast, easy, secure and scalable networking for your Application Container Platform Visibility: Live statistics in APIC per container and health metrics Enhanced Multitenancy and unified networking for containers, VMs, bare metal

3 Challenges running Kubernetes in production

4 Segmentation Secure K8s infrastructure: network isolation for kube-system and other infrastructure related objects (i.e. heapster, hawkular, etc.) Network isolation between namespaces Controlling access between Kubernetes services and external services

5 Communications outside of the Cluster Non-Cluster endpoints communicating with Cluster: Exposing external services Scaling-out ingress controllers Cluster endpoints communicating with non-cluster endpoints: POD access to external services and endpoints

6 Storage Access from Nodes Kubernetes nodes need 10/25GE line rate access to centralized storage, from node (maybe also from PODs): iscsi, NFS, GlusterFS, CEPH, etc. HyperFlex

7 Operations Skills gap between network and Kubernetes admins Visibility and governance of network policies Simplified Network Operations

8 The ACI CNI Plugin provides solutions to these challenges

9 Cisco ACI Integration with Containers / PaaS Key Benefits Unified networking: Containers, VMs, and bare-metal Micro-services load balancing integrated in fabric for HA / performance EXT Secure multi-tenancy and seamless integration of Kubernetes network policies and ACI policies OpFlex Node OVS OpFlex Node OVS Visibility: Live statistics in APIC per container and health metrics

10 Virtual Machines vs Containers Townhouses/Communities vs Apartments VMs are like houses Containers are like apartments Services infra Installation Services infra Installation Services infra Installation Services Infrastructure Installation VMs emulate everything, including hardware Containers isolate processes, users and filesystems

11 Linux Kernel Foundational Blocks for Containers chroot: change of root directory for the current running process and its children. A program that is run in chroot cannot access files outside the designated directory tree. Namespaces: lightweight process isolation. Enable a process (or several processes) to have different views of the system than other processes. cgroups (Control Groups): Resource Management solution providing a generic process-grouping framework. Cgroups limit, account for, and isolate the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.

12 Container-based architecture Common elements Container Platforms Orchestration Kubernetes Nomad Runtime Infrastructure

13 Docker was originally created by dotcloud (a PaaS provider) in Docker was released in March 2013 as an open source project, using LXC as its execution environment.

14 In version 0.9, Docker switched from LXC to their own execution driver, called libcontainer.

15 In 2015, Docker donated the libcontainer project to the OCI. This formed the basis of RunC, which is now the universal container runtime (Docker 1.11 uses RunC).

16 Docker Market Acceptance

17 Docker Container Fundamentals How it looks with Docker: Simplified architecture for Windows and Linux bins/libs bins/libs bins/libs bins/libs bins/libs bins/libs engine engine 10 Server 2016 Kernel 1 2 Kernel Bare-metal or Virtual 1 Docker for Windows may use Windows Containers or Hyper-V Containers 2 Docker for Mac uses HyperKit VM to run

18 Docker consists of three main components: Push Pull Docker Engine the actual app running on the host. Docker Image the libraries and files a container is built from Docker Hub repository and registry to store Docker Images for the Docker Engine.

19 Eksempel HDONNERB-M-R35E:hans-sqlc hdonnerb$ more Dockerfile # Download base image with latest ubuntu FROM ubuntu:latest # Contact info MAINTAINER Hans Donnerborg "hans@donnerborg.net" ENV REFRESHED_AT # Install RUN apt-get update RUN apt-get install -y mysql-client # Install various network tools RUN apt-get install -y iproute2 RUN apt-get install -y dnsutils RUN apt-get install -y net-tools RUN apt-get install -y iputils-ping RUN apt-get install -y tcpdump HDONNERB-M-R35E:hans-sqlc hdonnerb$ HDONNERB-M-R35E:hans-sqlc hdonnerb$ docker tag hans-sqlc hdonnerb/hans-sqlc HDONNERB-M-R35E:hans-sqlc hdonnerb$ docker push hdonnerb/hans-sqlc:latest

20 Docker Networking

21 After installing Docker, a new Linux bridge called docker0 gets installed on the host. jlage@jlage-svt:~$ ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu qdisc noqueue state UNKNOWN group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet /8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:1a:64:21:bb:68 brd ff:ff:ff:ff:ff:ff inet /24 brd scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::21a:64ff:fe21:bb68/64 scope link valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:11:5d:f5:c5 brd ff:ff:ff:ff:ff:ff inet /16 scope global docker0 valid_lft forever preferred_lft forever inet6 fe80::42:11ff:fe5d:f5c5/64 scope link valid_lft forever preferred_lft forever jlage@jlage-svt:~$

22 Basic Docker Networking with Linux Bridge By default Docker configures /16 on the Docker0 Bridge. When a container is created, the runtime engine will connect its virtual Ethernet interface to the docker0 bridge. An IP Address is assigned to the container, and docker0 Bridge is the default gateway.

23 Basic Docker Networking with Linux Bridge Containers on the same host communicate within the same subnet via docker0 Linux Bridge. External communication uses the host TCP/IP stack and Docker automatically configures NAT. Containers can also consumed Host ports to expose services from the container.

24 Containers can also share the same namespace It is also possible for two or more containers to share the same net namespace. This allows containers to communicate with one another via localhost

25 Kubernetes & Docker Kubernetes uses Docker to execute/run the containers Kubernetes adds, on top of Docker, all the intelligence and features of an orchestrator 25

26 Kubernetes Architecture At a very high level, Kubernetes has the following main components: One or more Master Nodes One or more Worker Nodes Distributed key-value store, like etcd. 26

27 Kubernetes PODs A is a group of one or more containers with shared storage and network, and a specification for how to run the containers. A models an application-specific logical host - it contains one or more application containers which are relatively tightly coupled in a precontainer world, they would have executed on the same physical or virtual machine. Containers within a share an IP address and port space, and can find each other via localhost Network Namespace (POD veth and IP address) Hostname (for the POD) cgroup (cpu, mem) nginx cgroup (cpu, mem) confd A POD with two containers, nginx and confd.

28 Kubernetes - Pod A is the scheduling unit in Kubernetes. It is a logical collection of one or more containers which are always scheduled together. The set of containers composed together in a share an IP. [sedk@kube-master ~]$ kubectl get --namespace=guestbook -o wide NAME READY STATUS RESTARTS AGE IP NODE frontend lq7j 1/1 Running 0 1d kube-node1.ciscolab.dk frontend dnwp0 1/1 Running 0 6h kube-node3.ciscolab.dk frontend w964t 1/1 Running 0 6h kube-node1.ciscolab.dk redis-master c79mb 1/1 Running 0 1d kube-node3.ciscolab.dk redis-slave z10x 1/1 Running 0 1d kube-node3.ciscolab.dk redis-slave dtl7m 1/1 Running 0 1d kube-node1.ciscolab.dk [sedk@kube-master ~]$ 28

29 Kubernetes Deployment Deployments are a collection of s providing the same service You describe the desired state in a Deployment object, and the Deployment controller will change the actual state to the desired state at a controlled rate for you For example you can create a deployment that declare you need to have 3 copies of your front-end. [sedk@kube-master ~]$ kubectl get deployment --namespace=guestbook NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE frontend h redis-master h redis-slave h [sedk@kube-master ~]$ 29

30 Kubernetes Scale Deployment ~]$ kubectl --namespace=guestbook scale deployment frontend --replicas=10 deployment "frontend" scaled ~]$ kubectl get --namespace=guestbook -o wide NAME READY STATUS RESTARTS AGE IP NODE frontend sv 1/1 Running 0 6h kube-node2.ciscolab.dk frontend g3mq 1/1 Running 0 16s kube-node3.ciscolab.dk frontend fpds 1/1 Running 0 16s kube-node1.ciscolab.dk frontend nmtq 1/1 Running 0 6h kube-node3.ciscolab.dk frontend znt 1/1 Running 0 16s kube-node2.ciscolab.dk frontend mz7 1/1 Running 0 16s kube-node3.ciscolab.dk frontend bl8tp 1/1 Running 0 6h kube-node1.ciscolab.dk frontend ccz13 1/1 Running 0 16s kube-node2.ciscolab.dk frontend lg20q 1/1 Running 0 16s kube-node1.ciscolab.dk frontend w3f01 1/1 Running 0 16s kube-node2.ciscolab.dk 30

31 Kubernetes Services A service tells the rest of the Kubernetes environment (including other s and Deployments) what services your application provides. While s come and go, the service IP addresses and ports remain the same. Kubernetes automatically load balance the load across the replicas in the deployment that you expose through a Service Other applications can find your service through Kurbernetes service discovery. Every time a service is create a DNS entry is added to kube-dns NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE frontend :32140/TCP 6h redis-master <none> 6379/TCP 6h redis-slave <none> 6379/TCP 6h [sedk@kube-master ~]$ 31

32 Kubernetes DNS nslookup frontend.guestbook.svc.cluster.local. Server: Address: #53 Name: frontend.guestbook.svc.cluster.local Address: nslookup redis-master.guestbook.svc.cluster.local. Server: Address: #53 Name: redis-master.guestbook.svc.cluster.local Address: nslookup redis-slave.guestbook.svc.cluster.local. Server: Address: #53 Name: redis-slave.guestbook.svc.cluster.local Address:

33 Namespace Guestbook Deployment: Frontend 1 Container Container Container Ext-Service :80 Service :80 1 Container Deployment: redis-master Service :6379 Service : Container Container Deployment: redis-slave

34 Goal #1: leverage Kubernetes APIs, transparent for devops administrators Security Policies within and between PODs are defined using Kubernetes Network Security API. Kubernetes policies implemented on OVS by APIC, visible to fabric administrator. Kubernetes Services: the service IP is automatically managed by ACI + Opflex + OVS as an anycast service Kubernetes External Services (LoadBalancer) can be implemented by ACI PBR or an external LB.

35 Goal #2: facilitate policy and connectivity to other workloads ACI admin can define EPGs and contracts that are exposed to Kubernetes users Kubernetes administrator can annotate objects, such as namespaces, deployments or even specific s to map them into endpoint groups Enable Kubernetes Deployments/PODs to access other resources without bottlenecks and with conforming policy

36 VMM domain helps to bridge the gap between Kubernetes admin and network operations

37 Kubernetes Load Balancing with ACI - LoadBalancer APIC provisions a distributed load balancer on on the leafs using PBR --- apiversion: v1 kind: Service metadata: name: frontend labels: app: guestbook tier: frontend spec: type: LoadBalancer ports: - port: 80 selector: app: guestbook tier: frontend ACI Leaf K8s-master OpFlex OVS K8s-node-01 OpFlex ACI Leaf OVS WAN K8s-node-02 OpFlex OVS POD POD POD POD POD POD POD POD POD POD POD POD

38 Kubernetes Load Balancing with ACI - LoadBalancer --- apiversion: v1 kind: Service metadata: name: frontend labels: app: guestbook tier: frontend spec: type: LoadBalancer ports: - port: 80 selector: app: guestbook tier: frontend Distributed Hardwarebased North-South Load Balancing ACI Leaf K8s-master OpFlex OVS K8s-node-01 OpFlex ACI Leaf OVS WAN K8s-node-02 OpFlex OVS POD POD POD POD POD POD POD POD POD POD POD POD

39 Container to Non-Container Communications In production environments certain services like high performance databases will be running as VMs or Bare Metal Servers This calls for the ability to easily provide communication between Kubernetes PODs and VMs/Bare Metal endpoints Simply deploy a contract between your EPGs, ACI will do the rest! This works for any VMM domain and Physical Domains, for example you can have a Container Domain using VXLAN speaking with a Microsoft SCVMM Domain using VLAN.

40 Kubernetes and external services without bottlenecks Fabric or security admin Distributed Kubernetes network policy complemented with ACI EPGs and contracts --- apiversion: v1 kind: Service metadata: name: oracle-db spec: ports: - name: sql protocol: TCP port: 3306 targetport: apiversion: v1 kind: Endpoints metadata: name: oracle-db subsets: - addresses: - ip: ports: - port: 3306 name: sql ACI Leaf K8s-node-01 OpFlex OVS POD POD POD POD K8s-node-02 OpFlex ACI Leaf POD POD POD POD Deployment EPG OVS WAN oracle-db PODs annotated (segmented) into dedicated EPG

41 Allow Nodes access to centralized storage leveraging contracts without bottlenecks [ ] spec: containers: - name: nginx image: nginx:1.7.9 volumemounts: - mountpath: /srv/www name: www-data readonly: true ports: - containerport: 80 volumes: - name: www-data nfs: server: path: "/var/nfs" ACI Leaf K8s-node-01 OpFlex OVS APIC implements distributed routing, switching and security on OVS and leaf ESXi-node ACI Leaf VDS WAN K8s Nodes mount the shared volume without bottlenecks. POD POD POD POD K8s-node-02 OpFlex OVS POD POD POD POD NFS-SERVER

42 Flexible Definition of EPG Boundaries Cluster Isolation Namespace Isolation Deployment Isolation EPG Network Policy Contract Contract Contrac t Contract Contrac t Single EPG for entire cluster, default behavior No need for any internal contracts Each namespace is mapped to its own EPG Contracts for inter-namespace traffic Each deployment mapped to an EPG Contracts tightly control service traffic 44

43 Container Demo

44 Container solutions

45 Cisco Solutions for Containers With Kubernetes CCP CCP + HX Cisco Container Platform with Cisco HX Complete Cisco Stack HyperFlex for easy infrastructure deployment and management Red Hat OpenShift Container Platform Cisco UCS Cisco UCS with RH OpenShift Container Platform Bare Metal deployments Software Defined Storage using Gluster Cisco HX IBM Cloud Private Docker Enterprise Edition VersaStack / FlexPod Cisco HX with RH OpenShift Container Platform IBM Cloud Private on Cisco UCS Portfolio Easy deployment and management with HyperFlex Virtualized deployment opportunities GTM with IBM on VersaStack GTM with IBM and NetApp on FlexPod Offering with HyperFlex Docker Enterprise Edition Multiple solution released FlexPod and Cisco-only options

46 Openshift

47 Integrated solution with Google Cloud Cluster Management Cisco Container Platform Kubernetes Management/Governance Cisco CloudCenter Google Kubernetes Engine Storage Cisco HyperFlex FlexDriver Security Cisco CSR1000v Stealthwatch Cloud Monitoring Cisco AppDynamics Networking Cisco ACI Contiv API/Mesh Management APIs Apigee Istio Open Service Broker Cisco HyperFlex Google Cloud Platform

48 HyperFlex 3.0 Kubernetes integration Technical Description Integration with Kubernetes FlexVolume Plugin framework Enables developers to leverage HyperFlex storage for stateful container storage HyperFlex Data Performance and Resiliency API K8s Node VM HX FlexVolume Driver Kubelet SW iscsi Initiator private host-only vswitch ESXi vmkernel interface iscsi LUN HX iscsi Proxy File vswitch-hx-storage-data HX Controller VM NFS Datastore HX ESXi Node

49 Questions

50 Kommende arrangementer

51

52