Neutron: peeking behind the curtains

Transcription

1 Neutron: peeking behind the curtains (that is to say how Neutron builds virtual networks) Salvatore Orlando VMware IRC: salv-orlando sorlando(at)vmware.com

2 Before we start Make your voice heard! Audience interaction is very valuable. This will be a fairly technical deep-dive on Neutron internals. Please try not to fall asleep! We will focus exclusively on Neutron s built-in, purely FOSS components. Questions regarding other plugins are however welcome By the way, who is this chap talking to us? Professional procrastinator, SSC Napoli supporter Breaking Openstack since 2010 Founder and core team member of the OpenStack Networking project (Neutron)

3 Stuff we will talk about Neutron and its place in the OpenStack universe Neutron s architecture overview The ML2 plugin OVS agent: layer-2 network virtualization and VIF security L3 Agent: Routing, gateway, and floating IPs Configuration agents: DHCP and Metadata WIPs, limitations and alternatives

4 The OpenStack universe (and Neutron s place in it) 21 official programs 17 integrated, 4 in incubation

5 Neutron s logical resources External Gateway External Network External Gateway Router A Router B Internal Gateway Internal Gateway Internal Gateway Network A1 Network A2 Network B1 DHCP DHCP DHCP A11 A12 A21 B1 1 B1 2 Tenant A Tenant B

6 and their physical mappings Compute Node C1 Network Node A11 A21 Br-int Br-tun Br-tun Br-int DHCP L3 Br-ex Local VLAN tags converted into GRE keys (and vice versa) B11 Br-int Br-tun Br-tun Br-int B12 A12 Compute Node C2 Compute Node C3

7 Neutron architecture overview OVS agent L2-population GRE Neutron Server ML2 RPC REST L3 base plugin RPC - ML2 plugin - Open vswitch edges - GRE tunnels transport AMQP OVS Agent OVS Agent OVS Agent DHCP Agent Metadata Agent L3 Agent L3 Agent

8 The modular layer-2 plugin Driver-based vs. monolithic Type drivers for network transport type Mechanism drivers for: Interacting with backend (e.g.: Arista, Cisco N1kV) Providing additional features or optimize scale/performance (e.g: L2 population) More information: Modular, driver-based approach not (yet) available for layer-3 plugins Reference plugin implements all supported capabilities

9 The OVS Agent: responsibilities Builds GRE tunnel mesh Alternatively VxLAN or no mesh if VLANs are used as network transport Populates MAC forwarding table for quicker instance lookups (l2-population) Wires instance VIFs to the appropriate virtual network Secures virtual interfaces Basic anti-spoofing rules (ARP, MAC, IP) security group rules Runs on hypervisor Communicates with server via RPC over AMQP

10 The OVS Agent: architecture Conceptual representation of agent architecture Does not map to actual components/processes VIF To server via AMQP bus iptables Bridge OVS agent Agent main loop Security manager FDB manager OVS Monitor qvb qvo Open vswitch (br-int) Tunnels to other hosts

11 The GRE tunnel overlay mesh Full mesh between hosts n(n-1) total tunnels L2-in-L3 tunnels Abstracts from data center s network complexity Only needs IP connectivity to the destination host Nothing is free About 3% GRE headers overhead No hardware TSO. TSO done in software uses CPU and reduces overall throughput VxLAN (kind of) similar, but not discussed for time constraints

12 Handling BUMs: layer-2 population Broadcast, unknown unicasts, and multicasts can be a serious problem in large deployments However Neutron knows where target VM instances are Pre-populate forwarding tables Optimizes both GRE and VxLAN overlays L2-population implemented by: Server side driver On each host local ARP responder and forwarding table population For more info: h"ps://wiki.openstack.org/wiki/l2popula7on

13 Tunnel mesh with L2-population Traffic from VM A to VM G is encapsulated and sent to Host 4 according to the bridge forwarding table entry. VM A Host 1 VM B Proxy Arp The ARP request from VM A for VM G is intercepted and answered using a pre-populated neighbour entry VM H Host 5 Host 2 VM C Host 4 Host 3 VM G VM F VM E VM D Without L2-population ARP broadcasts would have filled the tunnel mesh

14 Wiring and securing interfaces Agent calls server for 3 interface(s) details To server via AMQP bus Port is wired assigning local VLAN tag 4 Security groups translated into iptables rules 5 OVS agent Agent main loop Security manager FDB manager OVS Monitor Agent loop collects new, updated, and removed interfaces 2 OVS monitor 1 detects new interfaces on integration bridge

15 Neutron s layer-3 services East/west routing Source NAT (instances external gateway) Destination NAT (floating IPs) Static routes External gateway Network VM External Network Router Floa7ng IP Network VM East/West rou7ng

16 Network namespaces Isolated copy of network stack Cloned from root namespace Scope limited to namespace veths can connect devices from child namespace to root namespace Ability to reuse IP addresses, routes, iptables rules Processes can spawn in namespace And access that namespace s network stack L3, DHCP and metadata agents rely on them Available in upstream Linux kernel since 2.6 series For more info:

17 Network namespaces example veth pair ip netns add A! ip link add tapa-root type veth peer name tapa-ns! ip link set tapa-ns netns A! ovs-vsctl add-port br-int tapa-root!! OVS internal interface HOST (root) lo eth0 NETNS A lo NETNS B lo taparoot tapans tapbint ip netns add B! br-int ip link add tapb-int! ip link set tapb-int netns B! ovs-vsctl add-port br-int tapb-int set Interface tapb-int type=internal!!!

18 Layer-3 agent: responsibilities Handle server RPC notifications for routers state changes E.g.: new interface, new floating IP Query server for current router state Ensures agent-side state is consistent with server Coalesce multiple changes in short timespan Apply configuration on host Add router interfaces into namespace Set default SNAT rule for external gateway Reconfigure DNAT/SNAT rules for floating IPs Apply extra static routes to network namespace

19 Layer-3 agent: how it works Distinct namespace for each logical router Internal interfaces for tenants networks Create interface, configure IP but does not wire port Gateway interfaces for uplink to external network Iptables rules in NAT table for default gateway and floating Ips Additional routes in namespaces routing table L3 Fwd NS-Rtr-C SNAT/ DNAT Br-int L3 Fwd NS-Rtr-B SNAT/ DNAT Br-ex L3 Fwd NS-Rtr-A SNAT/ DNAT

20 Configuration agents: DHCP Server RPC notifications for: Subnet state changes IP/MAC address pair changes Addresses distributed via dnsmasq Code allows for implementing more drivers (e.g.: ISC DHCP) Isolation ensured via network namespaces Overlapping IPs, anyone? Multiple instances for load distribution and HA

21 DHCP agent: how it works Root Network Namespace Reloaded with kill HUP on every change DHCP Network Namespace Spawn upon server notification if not yet configured br-int DHCP Interface dnsmasq Hosts Options Updated when active port set changes Set as startup, updated through DHCP options server extensions

22 Configuration agents: Metadata Proxies requests to nova metadata server Used by services such as cloudinit Namespace proxy captures requests for Metadata agent forwards requests to nova Bridge between tenant and management networks Routed networks: embedded in router namespace Instance reach via default route Non-routed networks: proxy running in DHCP namespace Static route injected into instance via DHCP option 121

23 Metadata agent: how it works Cloud-init (from instance ): curl Router (DHCP) Network Namespace Metadata proxy :80 Unix socket Add HTTP headers: X-Router-ID <uuid> X-Forwarded-For <instance_ip> X-Instance-ID: <instance_id> Tenant network Nova metadata server HTTP Metadata agent Management network

24 More stuff: high-level services Available with Service Plugin Every service plugin is driver based to allow for implementations different from the reference one Load balancing Reference implementation based on Haproxy VPN ipsec only at the moment Reference implementation uses openswan Requires l3 agent Edge firewall Reference implementation based on iptables Requires l3 agent Experimental

25 WIPs and limitations OVS agent scale: Better scalability must be achieved with high number of VIFs security group rules per host L3 agent: Traffic distribution (east/west, gateway) not yet supported HA/failover in progress IPAM logic baked in DB logic Security group rules as OVS flow entries Will avoid the need for bridge/ovs hybrid plugging IPv6 support General reliability and failure condition handling

26 Beyond the built-in solution Neutron s plugin mechanism allows for choosing among a wide range of solutions Open Source Open Daylight: ML2 driver leverages L3/DHCP/metadata agents Open Contrail Standalone plugin providing basic network services & NFV Commercial Available either as standalone plugins or ML2 drivers VMware (NSX/vCNS), Cisco (UCS/Nexus 1kv), NEC, Midonet, etc. etc.

27 Summary Overview of the reference Neutron network virtualization solution. OVS agent builds L2 networks and secures VIFs. L3 agent provides routing/nat. DHCP/Metadata agent for instance configuration. Current suite is not (yet) perfect. If it does not suit your needs Neutron supports a plethora of backends to choose from