Simplify Networking for Containers 叶磊曹水 华为中央软件院云网络实验室

Transcription

1 Simplify Networking for s 叶磊曹水 华为中央软件院云网络实验室

2 The Nature of Network 2

3 cloud native and containerised micro-services high density/ dynamic complex deployment scenarios online monitoring and control E2E Monitoring SLA (Application to Application) L2/L3 Overlay Tunnel VM s Public Cloud Private Cloud more applications and micro services are deployed in containers 3

4 deployment complexity simple flat container network model: CNI complex deployment scenarios public clouds: AWS/Azure/HEC private clouds: openstack/vmware/ baremetal NFV: SR- IOV/L2/L3 4

5 deployment complexity simple flat container network model: CNI existing solutions are suitable for limited cases with hard-coded plugins complex deployment scenarios require a flexible solution that always adapts the best technology based on specific situation public clouds: AWS/Azure/HEC private clouds: openstack/vmware/ baremetal NFV: SR- IOV/L2/L3 5

6 How we deal with so many scenarios for containers? Public Cloud Private Cloud OVERLAY OVERLAY OS OS Socket TCP/IP OVERLAY vnic DRIVER Socket TCP/IP OVERLAY vnic DRIVER Network (Iaas) OVERLAY OS Socket TCP/IP OVERLAY NIC DRIVER IRONIC Underlay vrouter Traditional OS Socket TCP/IP Bridge OVS L2 SND backend Underlay Kuryr Traditional OS Socket TCP/IP Open Backend Bare Mental Host OVERLAY L2 Hetero OS OS OS SuSE12 Socket TCP/IP OVERLAY NICs Driver Socket TCP/IP L2 NICs Driver Socket TCP/IP Bridge OVS L2 OVERLAY Native Driver OVERLAY OS Socket TCP/IP OVERLAY NICs Driver L2 OS Socket TCP/IP OVERLAY NICs Driver Pass through Contain eros API VF PMD VF PassThrough OVERLAY L2 OS API vnic PMD OVERLAY VF PMD Containe ros API vnic PMD L2 VF PMD XEN KVM HostGW Cloud Provider VPC vrouter Kuryr Neutron Bare mental Bare mental Bare mental Bare mental Cloud Provider 6

7 Why we need so many models Customized Socket Lib Customized Socket Lib Customized Socket Lib Kernel Network Customized Network PMD Function feature Rich, identical to Kernel Function feature Normal, according to Customized Function feature Poor, according to application Performance Normal Performance Good, about 3 times than Kenel Performance Very good, identical to wire speed Compatibility Very good Compatibility Normal, maybe miss some socket function Compatibility Poor, only ENV 7

8 Our solution: ican (intelligent Network) an extensible framework to program various container network data path and policies adapt to different orchestrators support end-to-end SLA between containerised applications 8

9 ican architecture monitoring report node node node kubernets agent kubernets agent kubernets agent ican agent ican agent ican agent node ican monitor master etcd aggregated report SNC configurations aggregated report kubernetes master ican SLA schd ext. aggregated report ican master Standard Netwok Component (SNC) models SLA-annotated policy 9

10 CNI Interface Extension { "cniversion": "0.2.0", "name": "IDM-M", "type": "bridge-veth", // type (plugin) specific "vlanid": 42, "ipam": { "type": "dhcp", "routes": [ { "dst": " /16", { "dst": " /16" ] // args may be ignored by plugins "args": { "labels" : { " phynet " : " Phy_Net1" Once with one ticket 1CNI ADD PaaS Once with multi ticket 1CNI ADD 2CNI Network Configuration 2CNI Network Configuration Node PodX PodY br-intx br-inty { "cniversion": "0.2.0", "name": "IDM-M", "type": "bridge-veth", // type (plugin) specific "vlanid": 42, "ipam": { "type": "dhcp", "routes": [ { "dst": " /16", { "dst": " /16" ] // args may be ignored by plugins "args": { "labels" : { " phynet " : " Phy_Net1" { "cniversion": "0.2.0", "name": "IDM-C", "type": "bridge-veth", // type (plugin) specific "vlanid": 43, "ipam": { "type": "dhcp", "routes": [ { "dst": " /16", { "dst": " /16" ] // args may be ignored by plugins "args": { "labels" : { " phynet " : " Phy_Net2" Eth0 Eth1 PhyNet1 1) Parameters on CNI Network Configuration, support Once or Multi entry; 2) Reuse the CNI s common agreement, all customized fields within args segment; PhyNet2 10

11 Standard Network Component (SNC) model abstract for network components in data-path interfaces, devices and templates l2 interface l3 interface l2 paired interface l2 paired interface L2 device L2 devices: bridge/macvlan/ovs/ L3 device L3 devices: router/ipvlan/ L2 dev:linux bridge L3 dev: IPS a template for Flannel data path l2 interface l3 interface 11

12 Unified Framework For Multi Models MNG MNG Canal Plugin Flannel Plugin Calico Plugin Flannel Type Calico Type SR-IOV type Linux BR Kernel Route Linux BR Linux BR Kernel Route GRE Tunnel Kernel VxLAN PGP Route Sync Kernel Route Kernel VxLAN PGP Route Sync IPIP Tunnel Kernel Route Kernel Route User SR-IVO Through Existing every Plugins only support its own model Though they employ common data module, the function is isolated After deconstruct different data path, we setup a DSL language to describe them,using abstracted standard component Unified Framework with Pluggable drivers for additional es, Linux BR, SR-IOV,... 12

13 Big Pic of Multi-modes && Multi-planes User Socket Lib User Socket Lib User Socket Lib User Socket Lib Kernel bonding Process APP User PMD Kernel bonding Process APP User PMD UIO APP PMD UIO UIO APP PMD UIO NIC NIC NIC NIC NIC NIC NIC NIC NIC NIC NIC NIC PHY-OM PHY-MNG PHY-DATA PHY-NET 13

14 Open stack Neutron Ml2 Solution Neutron Server Neutron Server ML2 Plugin ML2 Plugin API Network API Network Host A Host B Host C Host D Host A Host B Host C Host D Linuxbridge Agent Hyper-V Agent Open Agent Open Agent Modular Agent Modular Agent Modular Agent Modular Agent Existing ML2 Plugin works with existing agents Separate agents for Linuxbridge, Open, and Hyper-V Combine Open Source Agents, a single agent which can support Linuxbridge and Open Pluggable drivers for additional es, Infiniband, SR- IOV,... 14

15 ican Control Plane Integrated with Openstack Kubernetes Master Openstack Neutron controller Local Node Kuberlet Kuryr Agent CANAL Agent Neutron Server Distributed KV store (etcd) CANAL Master Monitoring controller SLA Manager Control Node C C C C C C IPAM 15

16 Monitoring based SNC Modeling Monitoring on local SNC components : Generate E2E monitoring data in master node: C1 C2 C3 E2E Monitoring Monitoring Master E2Ethrought:minimal throughput E2E Drop rate: deviations between RX and TX Throughput Analysis:data from local node vif vif vif vif vport vport vdev vport vif vif vport vdev vport Monitorin g Agent Monitorin g Agent Bandwidth Throughput Status QoS CPU utilization pif pif pif pport pdev pif pport pdev Point Monitor Item Latency: Source T1 T4 T2 T3 Dest Latency = ((T4 - T1) - (T3 - T2)) / 2 16

17 Simplify Network SLA modeling ican provides north bound interfaces for orchestration and applications to define their requirements through PG(Pod Group: a group of pods with the same functions), Linking (network requirement between PG), SLA Service types and Service LB Type. Given topology and link bandwidth, evaluate the offers when deploying pods. Essentially a evaluation for pod placement, and validate the deployment. 2-Tiers Network topology management Underlay Network(Stable and Predictable) and Overlay Network (Customizable and Dynamic) Support: bandwidth, latency and drop rate Bandwidth <5% Latency <10%, more non-deterministic, affected by many factors such as queuing in software switch and hardware, application response, server IO, etc 10Mbps (x3) Web 5Mbps (x6) DB User 1 Internet Web DB Web Latency: Low 10Mbps (x2) Web DB User 2 Internet Web DB Convert link requirement to node requirement Polices Deployment Scheduler validation 17

18 ican networking Powerful Monitoring Implement monitoring on-demand and E-to-E monitoring based on the topology Facilitate on-demand DSL based troubleshooting Cooperate with the SLA subsystem to assess the SLA quality Rich Network Support Powerful network component modeling : SNC and Modeling via Yang Rich network schemes, support L2, Overlay, NAT, VLAN, L3, BGP, VPC Accelerated Network Multi-dimension SLA& Security Performance Isolation with bandwidth, latency, drop rate(proactive Network SLA and Reactive Network SLA ) Security Isolation: VLAN/VXLAN, ACL 18

19 Thank You. Copyright 2016 Huawei Technologies Co., Ltd. All Rights Reserved. The information in this document may contain predictive statements including, without limitation, statements regarding the future financial and operating results, future product portfolio, new technology, etc. There are a number of factors that could cause actual results and developments to differ materially from those expressed or implied in the predictive statements. Therefore, such information is provided for reference purpose only and constitutes neither an offer nor an acceptance. Huawei may change the information at any time without notice.

20 SNC Template Execution Workflow Node Network Pool Configuration Name Linux- Bridge Linux- Bridge IP-Filter OVS Location Host Host Host Network Capability Strategy Cap- Name GRE GRE VxLAN VxLan L2- DHCP Reques t Linux- Bridge; GRE-Ko; IP-Filter OVS; IP-Filter OVS; IP-Filter EVS;IP- Filter Dnsmas q;linux- Bridge; Priority -Index ❶ Node Network Capability Configuration Cap- Name GRE Locati on Conta iner Linux- Bridge Priorit y- Index 1 GRE Host 1 L2- DHCP Network Configuration Template Resource Bridge GRE-Tunnel ❷ Host 0 Resource Bridge ❸ ❹ Create();Del ete();config Network Configuration Deploy Template Resource Deploy-with Parameter GRE-Tunnel Internal-Link IP-Filter veth Bridge OVS ❹ Resource Bridge Driverfunc GRE- Tunneel Bridge Driverfunc Create();Del ete();config ure().. Create();Del ete();config ure() ❹ IP- Filter Resource Driverfunc Internal- Link Create();Del ete();config ure().. ❶ Network-Agent Local initialized base on Node Network Pool Configuration and Network Capability Strategy, generate Node Network Capability Configuration(NNCC). ❷ Node received template deployment request, check NNCC. If node can t meet requirement, return failure, otherwise will return Network Configuration Deployment Template (NCDT) with information ❸; ❹After, send network deployment request to Network-Element as NCDT defined, Finally executed by related network driver; 20

21 Modeling for Standard Network Component Standard Network Component can help to : Decouple network control with implementation Replace and upgrade network components seperately Provide on-demanding network solution and SLA for application Paired-IF Paired-IF Paired-IF L3_IF L3_IF L3_IF L3_IF L3_IF L3_IF L3: IPVLAN L2_IF L2: MACVLAN L2_IF L2-dev L3: IPS IFD/IPA IPVLAN SNC: L3_IF L3_DEV L2_IF MACVLAN SNC:L3_IF L2_DEV L2_IF CALICO SNC: Paired-IF L2_DEV L3_DEV L3_IF L3_IF 21

22 Example: Support with Flannel(VxLAN backend mode)via SNC Modeling (kernel based Overlay) SNC interfaces: /* L2:SW device definition */ { /* members */ string port[]; /* methods */ CreateDevice(); // creat L2:SW device CreatePort(string port_name); /* Overlay:Flannel device definition */ { /* members */ string inf_l; string inf_r; /* methods */ CreateDevice(); Connect(string inf, string port); /* Link:VNIC-pair device definition */ { /* members */ string inf_l; string inf_r; /* methods */ CreateDevice(); Connect(string inf, string port) Flannel Template: Link-Device: vnic-pair Port_L L2-Device: Page 12 模板化实例 Port_R Overlay: flannel == Operating abstraction: - CreateSubnet() -- get subnet information via etcd API - L2:SW.CreateDevice() => "l2_sw_dev" - L2:SW.CreatePort(port_L) - L2:SW.CreatePort(port_R) - Overlay:Flannel.CreateDevice() => "flannel_dev" - Overlay:Flannel.Connect(flannel_dev.inf_L, l2_sw_dev.port_r) - Overlay:Flannel.Connect(flannel_dev.inf_R, eth0) - Link:vNIC-pair.CreateDevice() => "link_dev" - Link:vNIC-pair.Connect(link_dev.inf_R, l2_sw_dev.port_l) 22