OpenStack Havana On IPv6 Shixiong Shang Randy Tuttle Ciprian Popoviciu! Version 1.9.3
Agenda Introduction IPv6 and Cloud IPv6 Refreshment Proof of Concept Proposed Blueprint Next Steps 2
Introduction Nephos6! Service assurance company Founded in June, 2011 Twitter: @Nephos6 Web: http://www.nephos6.com Shixiong Shang Head of Engineering Twitter: @shshang Email: shshang@nephos6.com Ciprian Popoviciu! Founder, CEO IPv6 expert Twitter: @Nephos6 Email: chip@nephos6.com Randy Tuttle Network Consulting Engineer Twitter: @randyttl Email: rantuttl@cisco.com 3
IP Comparison Address IPv4 32-bit, Network Address Translation IPv6 128-bit, Multiple Scopes ICMP ICMP ICMPv6 Autoconfiguration DHCP SLAAC, DHCPv6, DHCP-PD Routing RIPv2, OSPFv2, ISIS, MP- BGP, EIGRP RIPng, OSPFv3, ISIS-ST/ MT, MP-BGP, EIGRPv6 IP Multicast IGMP/PIM/Multicast BGP MLD/PIM/Multicast BGP, Scope Identifier IPv6 Is an Evolution, Not a Revolution of the Internet Protocol 4
IPv6 and Cloud IPv6 Strength Business Value Sufficient address space Direct access to resources Simplified Address Assignment Native support of multicast and flow label New architectural models } } Easier management and lower operational cost Great opportunity for innovation The promise of Cloud cannot be fully met without IPv6 5
IPv6 Address Auto-Configuration Our focus today! SLAAC* Working in progress! DHCPv6 Address Assignment (non-link-local) By exchanging Router Solicitation and Router Advertisement messages with neighboring routers. From DHCPv6 server Additional Information None From DHCPv6 server Default Gateway The only way to announce default route is using Router Advertisement! Pros Plug and play IPv4-like approach, but better More control Cons Doesn t provide Hostname, DNS server, WINS, etc. Operational overhead (extra DHCP server, HA, etc.) * StateLess Address AutoConfiguration 6
SLAAC RFC 4861 - Neighbor Discovery for IP Version 6 (IPv6) and RFC 4862 - IPv6 Stateless Address Autoconfiguration Rely on ICMPv6 (IPv6 control plane!) Host Router Solicitation (RS) Router Advertisement (RA) subnet prefix lifetime autoconfig flag Router Router Solicitation (RS) ICMPv6 Type 133 Router Advertisement (RA) ICMPv6 Type 134 IPv6 Source A Link Local IPv6 Source A Link Local IPv6 Destination Link-local scope all-routers address (FF02::2) IPv6 Destination Link-local scope all-nodes address (FF02::1) VM sends Router Solicitation at boot time to solicit Router Advertisement Router sends RA to all-nodes address periodically Default route points to router s link-local address Router can also unicast RA back to VM upon receiving RS 7
SLAAC Address Calculation IPv6 SLAAC = network portion (i.e. /64 Prefix in RA) + interface id (i.e. EUI64) MAC FA 16 3E 73 83 D9 Insert 0xFFFE in the middle FA 16 3E FF FE 73 83 D9 Change 7th bit in OUI part 1111 1010 1111 1000 EUI- 64 F8 16 3E FF FE 73 83 D9 IPv6 address = 2001:7:10:180:F816:3EFF:FE73:83D9 8
OpenStack IPv6 Readiness OpenStack Havana Limited IPv6 support out of box OpenStack Icehouse Neutron will support IPv6 Neutron IPv6 roadmap is still in preliminary stage No clear IPv6 roadmap for other OpenStack projects Blueprint: IPv6 Feature Parity (working in progress ) Neutron-IPv6-Subteam (ongoing) Very limited documentation Biggest risk of all: IPv4 way of thinking 9
Proof Of Concept Success with both Grizzly and Havana! Mission Statement: To make these two inflection points, IPv6 and Cloud work together seamlessly! Motivation Goals We are believers What it is v.s. What it should be We are doers but we are not hackers, or developers :) All OpenStack infrastructure nodes should be able to communicate with each other by IPv6 OpenStack should be able to spin up dual-stack VMs in multi-tenant environment VMs should be able to gain connectivity to external IPv6 network beyond OpenStack s control 10
POC Architecture Controller Node nova-api nova-scheduler nova-consoleauth Network Node nova-novncproxy neutron-dhcp-agent Common Node nova-cert neutron-l3-agent horizon nova-conductor neutron-metadataagent Compute Node keystone cinder openvswitch nova-compute mysql db glance neutronopenvswitch-agent neutronopenvswitch-agent rabbitmq neutron-server dnsmasq openvswitch eth0 eth0 eth0 eth1 eth2 eth3 vlan 511 vlan 512 eth0 eth3 vlan 511 vlan 512 7.10.180.101 2001:7:10:180::101 7.10.180.102 2001:7:10:180::102 7.10.180.103 2001:7:10:180::103 7.10.180.104 2001:7:10:180::104 Management and API network 7.10.180.0/24 2001:7:10:180::/64 External Network Tenant Data Networks (Tenant 1: VLAN 511) (Tenant 2: VLAN 512) Management and API network Tenant 1 External Network 172.26.184.0/24 2001:172:26:184::/64 Router Tenant 2 External Network 172.26.185.0/24 2001:172:26:185::/64 Data Network 11
1. All OpenStack infrastructure nodes should be able to communicate with each other by IPv6 - IT IS ALL ABOUT CONFIGURATION 12
Enable IPv6 On Infrastructure Nodes Components Configuration Files Field Value Keystone /etc/keystone/keystone.conf bind_host 2001:7:10:180::101 Common MySQL DB /etc/mysql/my.cnf bind-address :: Apache /etc/apache2/ports.conf Listen 80 my_ip 2001:7:10:180::102 use_ipv6 true Nova /etc/nova/nova.conf osapi_compute_listen 2001:7:10:180::102 Controller metadata_listen novncproxy_host 7.10.180.102 2001:7:10:180::102 Glance /etc/glance/glance-api.conf /etc/glance/glanceregistry.conf bind_host 2001:7:10:180::102 registry_host net-glance.sandbox.com bind_host 2001:7:10:180::102 13
Enable IPv6 On Infrastructure Nodes Components Configuration Files Field Value my_ip 2001:7:10:180::102 Controller Cinder /etc/cinder/cinder.conf glance_host osapi_volume_listen 2001:7:10:180::102 2001:7:10:180::102 Neutron /etc/neutron/neutron.conf bind_host 2001:7:10:180::102 Network Neutron /etc/neutron/neutron.conf bind_host 2001:7:10:180::103 my_ip 2001:7:10:180::102 use_ipv6 true Compute Nova /etc/nova/nova.conf osapi_compute_listen metadata_listen 2001:7:10:180::102 7.10.180.102 novncproxy_host 2001:7:10:180::102 Neutron /etc/neutron/neutron.conf bind_host 2001:7:10:180::103 14
2. OpenStack should be able to spin up dual- stack VMs in multi- tenant environment - IT IS ALL ABOUT IPV6 ADDRESS ASSIGNMENT 15
Neutron Tenant Network Provisioning neutron router-create --tenant-id tenant2-id router2! IPv6 tenant subnet neutron net-create --tenant-id tenant2-id net2_192_168_2 -- provider:network_type vlan --provider:physical_network physnet3 --provider:segmentation_id 512! neutron subnet-create --tenant-id tenant2-id --ip-version 4 -- name sub2_192_168_2 net2_192_168_2 192.168.2.0/24 Specify IP version 6 neutron subnet-create tenant-id tenant2-id --ip-version 6 -- name sub2_2001_192_168_2 net2_192_168_2 2001:192:168:2::/64! neutron router-interface-add router2 sub2_192_168_2 neutron router-interface-add router2 sub2_2001_192_168_2 Port is associated with tenant subnet 16
Neutron Tenant Network dnsmasq binding interface (ipv4) qdhcp namespace ns- 74f270ff- 01 (192.168.2.2) 3. Need dnsmasq to send RA from default gateway interface 2. OpenStack needs to know this self- calculated IPv6 SLAAC address 1. Need ip6tables filter rules to enable ICMPv6 at inbound direction VM 192.168.2.3 (ipv6 address) tap74f270ff- 01 tap- intf RA br- eth2 qr- 2f573f07- d9 (192.168.2.1) br- int qr- 6dbfb73d- 89 (2001:192:168:2::1) qrouter namespace br- eth3 Network Node Compute Node br- int br- eth3 eth2 Default Gateway Interface (ipv4) Default Gateway Interface (ipv6) eth3 eth3 To External Network Tenant 2 Network 17
Enable RA Within Router Namespace Method spawn_process in neutron.agent.linux.dhcp.py on Network Node Derive router s namespace and gateway interface Add IP version check Enable dnsmasq with RA and SLAAC Bind to IPv6 qr- interface Specify IPv6 DHCP range. Taken from CLI Launch dnsmasq in router s namespace 18
3. VMs should be able to gain connectivity to external IPv6 network beyond OpenStack s control - Support dual- stack on a single external interface - Utilize existing VLAN/Segmentation ID! - Eliminate NAT and GARP for IPv6 subnets 19
Dual-Stack options Option #1: Use next-hop RA and SLAAC to allow external GW interface defined IPv6 address Option #2: Statically assign IPv6 address to external GW interface for the router neutron router-gateway-set router2 ext-net-185 20
Neutron External Network dnsmasq binding interface (ipv4) Namespace: qdhcp- bfc3d877-44b6-4879- a83e- d37455e77f71 ns- 74f270ff- 01 (192.168.2.2) dnsmasq binding interface (ipv6) Need ip6tables filter rules to enable ICMPv6 at inbound direction VM 192.168.2.3 (2001:192:168:2::1) tap74f270ff- 01 tap- intf br- eth2 qr- 2f573f07- d9 (192.168.2.1) br- int qr- 6dbfb73d- 89 (2001:192:168:2::1) qg- 3dac3be9-1b (172.26.185.70) (SLAAC or statically assigned) br- eth3 Network Node Compute Node br- int br- eth3 Namespace: qrouter- 94662c71- bf80-4c2f- 9841-09a2112e3f58 eth2 RA To External Network Disable NAT and GARP for IPV6 eth3 Tenant 2 Network eth3 21
Dual-stack options For Option #2, there exists a limitation on static IP address assignment for dual-stack implementation. The L3 (server and agent) only allows a single IP address per network (VLAN) within the Linux namespace representing the tenant's router. This limitation precluded the possibility of a dual-stack arrangement utilizing static assignments without code changes. 22
Dual-stack solution To accomplish a static dual- stack arrangement, ip_version, cidr, ip_address and gateway_ip, was essential for L3 agent to build dual- stack interface inside router s namespace. 23
Dual-stack configuration For the tenant router, learn the default route from the upstream router through RA. When adding an external gateway net.ipv6.conf.<gateway_interface>.accept_ra=2 net.ipv6.conf.<gateway_interface>.forwarding=1 net.ipv6.conf.<gateway_interface>.accept_ra_defrtr=1 Prevent learning a default route from RA from internal tenant network net.ipv6.conf.<internal_interface>.accept_ra_defrtr=0 When the subnet assigned is an IPv6, don t apply NAT configuration or perform GARP. 24
Summary Findings RA is not sent to IPv6 enabled internal tenant network by default DHCP process is bound to interface other than default gateway of tenant network IPv6 address chosen by OpenStack is not based on SLAAC standard Neighbor Discovery packet is dropped by ip6tables filter rules Fixes Enable RA on dnsmasq Launch dnsmasq process inside router namespace Calculate VM s IPv6 address based on unique MAC address Add ip6tables rules to allow ND related ICMPv6 packets NAT and GARP are turned on for IPv6 subnets. Not desirable! Whitepaper: Only perform NAT and GARP for IPv4 subnets http://www.nephos6.com/pdf/openstack-havana-on-ipv6.pdf 25
Proposed Blueprint From openstack-dev mailer: Short term, my goal is to get provider networks up and running, where instances can get RA's from an upstream router outside of OpenStack and configure themselves. Medium term, we want to make dnsmasq configuration more flexible. More long term, I'd like to make it so that if there is an upstream router doing RA's - Neutron should send a PD automatically on network creation, and populate a subnet from the response given by the upstream router. Service Provider focused; may not work entirely with L3 Agent without revisions Integrate this PoC work with Blueprint to address broader OpenStack community and address L3 Agent 26
Our Next Step Tactical DHCPv6 Migration Strategy SLAAC + DHCPv6 Support for dual-stack infrastructure Strategical IPv6 mindset IPv6 understanding / education Participation in IPv6 + Cloud efforts Icehouse release validation 27