Wolfram Richter Red Hat OpenShift Container Netzwerk aus Sicht der Workload
Why this session? OpenShift is great for web applications, but we want to do X will this work? X { Analytics, Non-HTTP, High- Performance Computing, Big Data, Object storage, NAS, Replicated Databases, } Let s take a look from a networking PoV!
Agenda What is OpenShift? How plain Docker Networking works and what OpenShift does differently Container Networking across nodes Kubernetes Services Ingress: OpenShift Router Egress Pods and Network Policy
OpenShift Platform & Container as a Service Built for both traditional and cloud-native applications An integrated hybrid cloud application platform for application development and deployment Develop, build, and manage container based applications Easily turn source code into running applications with source-to-image capabilities
OpenShift High-Level Architecture
Container Networking Problem Statements As an X, I want my containerized applications to be able to connect to other services, so that they can perform meaningful work. As an X, I want my containerized applications to be accessible externally, so that a wide range of users can use them.
RFC1918 IP assigned by docker daemon
Host NIC Docker bridge Docker host
So we can use the ip command Container
Indicates which IF the veth device is connected to Container
Endpoint of the container s veth device Docker host
Source IP appears to be Node IP
Outbound traffic is masqueraded Inbound traffic is forwarded to container Docker host
Container Networking Problem Statement As an X, I want to use network attached storage from within my container, so I can provide stateful services (*). (*) and storage traffic shouldn t share application network bandwith
Container Networking Problem Statement As a X, I want name resolution to work inside the container like they would on a dedicated machine, so that I don t have to care about them.
/etc/hosts /etc/hostname /etc/resolv.conf
Files in the container fs are overwritten Container
OpenShift Networking Problem Statement As a X, I want container networking to work seamlessly across multiple nodes, so that I don t have to worry where which containers run (*). (*) while still maintaining compatibility with plain docker containers
Host NICs Docker bridge docker bridge <-> ovs bridge ovs bridge ovs bridge <-> host NICs OpenShift node
Look, there s no container connected OpenShift node
Container veth endpoints on ovs bridge OpenShift node
Pod IP address OpenShift node
Ping works from container on same host Container
Ping fails from container on different host Container
Each node has specific IP range
IP Range node 1 IP Range node 2 OpenShift node
IP packets destined for pod on other node is encapsulated via VXLAN
... and sent out via the node s IP stack (MTU impact!)
Port 1 is VXLAN OpenShift node
Flow rules that trigger VXLAN encapsulation Destination node IP address OpenShift node
Two pods in the same namespace on different nodes OpenShift node
can communicate with each other OpenShift pod
OpenShift Networking Problem Statement As a X, I want to ensure that a rogue pod cannot access pods in another project, so that I have a base level of security.
Project-specific VXLAN ID
Pod in a different namespace cannot be reached OpenShift pod
OpenShift Networking Problem Statement As a X, I want to be able to connect to other containerized services using a stable endpoint, so I don t have to reconfigure my application when other containers come and go.
Service IP Address OpenShift node
Namespace in search suffix list Name resolution via OpenShift dnsmasq on node OpenShift pod
Service name is resolved into IP adress OpenShift pod
Communication via service IP OpenShift pod
Kubernetes Service Modes User-space mode IPTables rules forward packages destined to the service IP address to the kube-proxy Kube-proxy will in turn initiate connections to the actual destination IP and proxy between the two endpoints Key advantage: can detect non-responding pods and retry connection to other pods IPTables mode kube-proxy continuously updates the node s IPTables rules forward packets directly to one of the target pod s IP Key advantage: increased throughput
OpenShift Networking Problem Statement As an X, I want my containerized applications to be accessible externally, so that a wide range of users can use them (*). (*) without having to care on which node a container/pod runs
Ingress router pod bound to host port
Host Port = port exposed on the node (containerized) haproxy
OpenShift Routing Layer 7 Routing: HTTP(S), TLS-SNI To properly route other protocols, deploy dedicated customized routers Alternatively instrument external load balancers such as F5, etc.
OpenShift Networking Problem Statement As an operator, I want to be able to fall back to the known working version of a service when deploying a new version so I have a safety net (blue/green deployments)
Router reconfiguration allows blue/green deployments oc patch route/api-gateway -p '{ "spec": { "to": { "name": "api-gateway-green" }}} oc patch route/api-gateway -p '{ "spec": { "to": { "name": "api-gateway-blue" }}}'
OpenShift Networking Problem Statement As an operator, I want my containerized applications to use specific source IP addresses to access external services, so I can restrict service access via (external) firewall rules.
Egress source IP Egress target IP (points to external service IPA ) Egress default GW
Points to egress-1 pod
Retrieving from egress-1 service works OpenShift pod
Egress source IP Egress target IP External service
if2: node s eth0 OpenShift egress pod
OpenShift Networking Problem Statement As an operator, I want to control which services my containerized applications can access, so I can limit access via internal means.
Egress Network Policy { }, } "kind": "EgressNetworkPolicy", "apiversion": "v1", "metadata": { "name": "default }, "spec": { } "egress": [ { "type": "Allow", "to": { "cidrselector": "1.2.3.0/24 } { "type": "Deny", "to": { "cidrselector": "0.0.0.0/32 } } ]
Summary If the question is OpenShift is great for web applications, but we want to do X will this work?, the answer is most likely yes (from a networking point of view) (*). (*) keep in mind that there is an MTU impact, multiple processing hops which impact latency, etc