Multicloud Networking Design & Deployment

Transcription

1

2 Multicloud Networking Design & Deployment Shannon McFarland CCIE#5245 Distinguished Consulting Engineer Cloud

3 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

4 Agenda Multicloud Networking Overview Deploying the On-Prem Resources Google Cloud Platform Amazon Web Services Microsoft Azure Conclusion

5 Disclaimer You won t learn security, routing, HA, performance best practices There are a gazillion ways to accomplish the same thing for ALL of this Running the Cisco CSR 1000v virtual router on Google Cloud Platform is not yet officially supported Be smart Know IPsec/IKE, BGP, OSPF, EIGRP and FHRP stuff Dead Peer Detection IPsec SA lifetimes IPsec SA replay window-size Perfect Forward Secrecy (PFS) BGP timers, Local Preference, MED, inbound soft reset (check if cloud provider supports dynamic inbound soft reset) BGP graceful restart - Note: Each cloud provider uses BGP graceful restart with default timers (120 sec) My configs do not show that due to slide space but know that it is enabled on each on-prem router router bgp bgp log-neighbor-changes bgp graceful-restart restart-time 120 bgp graceful-restart stalepath-time 360 bgp graceful-restart OSPF areas, interface types, timers EIGRP delay, queue, bandwidth, timers HSRP timers, tracking 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

6 Multicloud Networking Overview

7 Hybrid vs Multicloud Networking Hybrid Cloud Networking = Network transport from on-premises (on-prem) to a single public cloud provider Multicloud Networking = Network transport from on-prem to multiple public cloud providers and/or between multiple public cloud providers The technologies used can be identical for every connection or they can be per-provider, per-region, per-project, etc.. Common network transport ingredients for hybrid and multicloud: Encryption (IPsec/IKEv2/IKEv2, SSL, PKI) Routing (Static, BGP and with supported public cloud-hosted routers: OSPF, EIGRP) Tunneling (IPsec tunnel mode, GRE, mgre, MPLS, segment routing, etc..) Common network endpoint options: Native VPN (IPsec over Internet) using public cloud provider services that connect to on-prem router/firewall Commercial/Open Source VPN platform hosted on the public cloud provider connecting to an on-prem router/firewall Colocation/Direct Peering: Service from public cloud provider to on-prem via a 3 rd party colo facility Google Cloud Platform Dedicated Internconnect/Direct Peering/Carrier Peering: Amazon Web Services Direct Connect/PrivateLink: Microsoft Azure ExpressRoute: Cisco and/or its affiliates. All rights reserved. Cisco Public 7

8 Why Would You Use Multiple Cloud Providers? Cloud provider high availability Regional cloud provider access Feature disparity between providers, regions and/or services Per-project service requirements M&A may dictate public cloud provider preference (for a time) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8

9 Extending On-Prem Private Cloud to a Public Cloud

10 Cisco Cloud Services Router (CSR) 1000V Cisco IOS XE Software in a Virtual Appliance Form-Factor Software App OS App OS CSR 1000V Familiar IOS XE software with ASR1000 and ISR4000 Infrastructure Agnostic Runs on x86 platforms Supported Hypervisors: VMware ESXi, Linux KVM, Citrix Xen, Microsoft Hyper-V, Cisco NFVIS and CSP2100 Virtual Switch Hypervisor Server TBLSYkcoTUS6b4MFthdvhDrseo6MeN Supported Cloud Platforms: Amazon AWS, Microsoft Azure, and Google Cloud Platform (Summer 2018) Performance Elasticity Available licenses range from 10 Mbps to 10 Gbps CPU footprint ranges from 1vCPU to 8vCPU License Options Term based 1 year, 3 year or 5 year Smart License enabled Programmability NetConf/Yang, RESTConf, Guest Shell and SSH/Telnet 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

11 Public Cloud Provider Native VPN Services The Big Three Reference Google Cloud Platform (GCP): VPN: Dedicated Interconnect: Amazon Web Services (AWS): VPN: Direct Connect: Microsoft Azure: VPN: ExpressRoute: OpenStack public cloud goodness: Cisco and/or its affiliates. All rights reserved. Cisco Public 11

12 Starting Simple Public Cloud Provider Native IPsec VPN Service VPC Network /20 BGP AS65000 Google Cloud VPN Google Cloud Router IPsec/IKEv2 Tunnel Mode BGP AS65003 VMware, KVM, OpenStack-hosted CSR 1000v ebgp<>igp Redistribution BGP/OSPF/EIGRP Private Network /24 On-Prem Private Cloud Region: europe-west1 Cisco Cloud Services Router: Cisco and/or its affiliates. All rights reserved. Cisco Public 12

13 Add More On-Prem Stuff Public Cloud Provider Native IPsec VPN Service CSR1000v On-Prem Cloud 1 vsphere Hosted Cisco CSR Private Network /24 VPC Network /20 BGP AS65000 Routes this side should see: / /24 Google Cloud VPN Google Cloud Router BGP AS65002 BGP/OSPF/EIGRP CSR1000v BGP AS65003 BGP/OSPF/EIGRP Private Network /24 On-Prem Cloud 2 OpenStack Hosted Cisco CSR Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 13

14 On-Prem Physical/Virtual Public Cloud Provider Native IPsec VPN Service Physical Router ASR 1000 Private Network yyy.0/24 VPC Network /20 Google Cloud VPN ASA Firewall Private Network yyy.0/24 Google Cloud Router Physical Firewall 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14

15 Add More Public Cloud Providers to the Mix

16 Stepping into Multicloud Networking Multiple Native IPsec VPN Services VPC Network /20 VPC Network /16 Google Cloud VPN Google Cloud Router VPC Router VPN Gateway IPsec/IKEv2 Tunnel Mode Private Network /24 BGP/OSPF/EIGRP On-Prem Private Cloud 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16

17 Stepping into Multicloud Networking Multiple Native IPsec VPN Services VPC Network /20 Google Cloud VPN Google Cloud Router IPsec/IKEv2 Tunnel Mode BGP/OSPF/EIGRP Private Network /24 VPC Network /16 VPC Router VPN Gateway As the number of these connections increase and/or change frequently... You can see where this is going On-Prem Private Cloud 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17

18 Moving Away From Native VPN Services What Conditions Cause a Change in Design? If on-prem routers/firewalls are behind NAT - GCP and Azure VPN services do not support NAT-T You need different IPsec/IKE configurations than what the provider offers You need SSL-based VPNs You need to extend your on-prem IGP (OSPF/EIGRP) into the public cloud You need MPLS VPN QoS, specific network monitoring (IP SLA, NetFlow), Enterprise toolsets for configuration and monitoring Operational consistency 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18

19 DMVPN Enable Dynamic Multicloud Networking Cisco DMVPN VNet Network /16 Cisco CSR1000v Spoke Hub Cisco CSR1000v Private Network /24 BGP/OSPF/EIGRP VPC Network /16 Cisco CSR1000v Spoke DMVPN On-Prem Private Cloud Cisco DMVPN: Cisco and/or its affiliates. All rights reserved. Cisco Public 19

20 DMVPN Enable Dynamic Multicloud Networking Cisco DMVPN VNet Network /16 Cisco CSR1000v Spoke Hub Cisco CSR1000v FHRP Private Network /24 VPC Network /16 Cisco CSR1000v Spoke DMVPN On-Prem Private Cloud 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 20

21 DMVPN Enable Dynamic Multicloud Networking Cisco DMVPN VNet Network /16 Cisco CSR1000v Spoke Hub Cisco CSR1000v FHRP Private Network /24 VPC Network /16 Cisco CSR1000v Spoke DMVPN IGP Support: OSPF, EIGRP, ibgp QoS Policies IP SLA, NetFlow NAT-T (Transparency) MPLS etc... On-Prem Private Cloud 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 21

22 Cloud VPN vs Direct Connect/Express Route/Dedicated Interconnect

23 Options IPsec-over-the-Internet or Dedicated Connections IPsec VPN + Internet VPC Network /20 Google Cloud VPN IPsec/IKEv2 Tunnel Mode Private Network /24 BGP/OSPF/EIGRP On-Prem Private Cloud Google Cloud Router Colocation VPC Network /20 Google Cloud Router VPN Cloud Partner Interconnect Colocation Facility Private Network /24 BGP/OSPF/EIGRP On-Prem Private Cloud 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 23

24 VPN over the Internet vs Direct Connect/ExpressRoute/Dedicated Interconnect VPN over the Internet Direct/Express/Dedicated Throughput QoS Latency Inline Services Managed Services Winner Winner Winner Winner Winner Cost Time to Provision Flexibility Location Availability Winner Winner Winner Winner 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

25 The plan for this session: - GCP Native VPN Service - AWS with CSR and DMVPN - Azure with CSR and DMVPN

26 Google Cloud Platform Native VPN

27 Reference Google Cloud Platform VPN Gateway GCP Cloud VPN overview GCP Cloud VPN documentation GCP Advanced VPN documentation GCP does not support NAT-T 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 27

28 Topology for GCP to On-Prem CSR IPsec VPN BGP Routing BGP<>OSPF Redistribution Default Network /20 BGP AS Google Cloud VPN Google Cloud Router 35.xxx.xxx.x IPsec/IKEv2 Tunnel Mode xxx.xxx.x BGP AS65002 Cisco CSR1000v.1 Private Network /24 OSPF 10 Area 0 Hypervisor Routes this side should see: /24 Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 28

29 gcloud Create the VPN GW, External IP and Forwarding Rules Create a VPN gateway # gcloud compute target-vpn-gateways create csr-gcp-vm-gw --region us-west1 --network default Create an external IP to use for the VPN # gcloud compute addresses create gcp-to-csr --region us-west1 Capture the external IP address # gcloud compute addresses list --filter="gcp-to-csr NAME REGION ADDRESS STATUS gcp-to-csr us-west1 35.xxx.xxx.x RESERVED Create a forwarding rule for ESP, UDP500 and UDP4500 These are used by IKE/IPsec # gcloud compute forwarding-rules create csr-gcp-vm-rule-esp \ --region us-west1 \ --address 35.xxx.xxx.x \ --ip-protocol ESP \ --target-vpn-gateway csr-gcp-vm-gw # gcloud compute forwarding-rules create csr-gcp-vm-rule-udp500 \ --region us-west1 \ --address 35.xxx.xxx.x \ --ip-protocol UDP --ports 500 \ --target-vpn-gateway csr-gcp-vm-gw # gcloud compute forwarding-rules create csr-gcp-vm-rule-udp4500 \ --region us-west1 \ --address 35.xxx.xxx.x \ --ip-protocol UDP --ports 4500 \ --target-vpn-gateway csr-gcp-vm-gw 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 29

30 gcloud Create Cloud Router, VPN Tunnel and BGP session Create the Cloud router that is used for BGP (an existing router can be used) # gcloud compute routers create csr-gcp-vm-bgp-rtr \ --region us-west1 \ --asn=65000 \ --network default Create a VPN tunnel and link it to the router created in the previous step # gcloud compute vpn-tunnels create csr-gcp-vm-gw-tunnel-1 \ --region us-west1 \ --peer-address 192.xxx.xxx.x --shared-secret <pre-shared-password-goes-here> \ --ike-version 2 \ --target-vpn-gateway csr-gcp-vm-gw \ --router csr-gcp-vm-bgp-rtr Add a new interface to the router and set the BGP session IP address for the GCP side of the connection # gcloud compute routers add-interface csr-gcp-vm-bgp-rtr \ --interface-name if-csr-gcp-vm-bgp-rtr-01 \ --ip-address \ --mask-length 30 \ --vpn-tunnel csr-gcp-vm-gw-tunnel-1 \ --region us-west1 Create a new BGP peer This peer will be the Cisco CSR at the on-prem cloud # gcloud compute routers add-bgp-peer csr-gcp-vm-bgp-rtr \ --interface if-csr-gcp-vm-bgp-rtr-01 \ --peer-asn \ --peer-name csr-gcp-vm-bgp-peer \ --peer-ip-address \ --region us-west Cisco and/or its affiliates. All rights reserved. Cisco Public 30

31 ... Output summarized Cisco CSR Route Information Default Network /20 Google Cloud VPN BGP Google Cloud Router Area /24 csr-gcp-01# show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override, p - overrides from PfR S* /0 [1/0] via 192.xxx.xxx.x /20 is subnetted, 1 subnets B [20/100] via , 00:16: /16 is variably subnetted, 2 subnets, 2 masks C /30 is directly connected, Tunnel0 L /32 is directly connected, Tunnel0 192.xxx.xxx.x/24 is variably subnetted, 2 subnets, 2 masks C 192.xxx.xxx.x/26 is directly connected, GigabitEthernet1 L 192.xxx.xxx.x/32 is directly connected, GigabitEthernet /24 is variably subnetted, 2 subnets, 2 masks C /24 is directly connected, GigabitEthernet2 L /32 is directly connected, GigabitEthernet Cisco and/or its affiliates. All rights reserved. Cisco Public 31

32 Google VPN Dual/Redundant On- Prem Cisco CSRs

33 Reference Topology for Dual Cisco CSR Design yyy.yyy.y On-Prem Cloud 1 vsphere Hosted Cisco CSR OSPF 10 Area VM ESXi Host 1 Default Network /20 BGP AS65000 Compute Engine Google Cloud VPN Google Cloud Router 35.yyy.yyy.y 35.xxx.xxx.x BGP AS xxx.xxx.x BGP AS /24.3 Private Network HSRP VIP =.1 ESXi Host 2 Routes this side should see: /24 vsphere Distributed vswitch (DVS) with a Distributed PortGroup for the Private Network Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 33

34 Pre-Failure State (1) GCE Instance traceroutes via GCP BGP Path ~]$ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms On-Prem VM traceroutes via HSRP Active CSR ( ) [root@k8s-m-01 ~]# traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms HSRP Active CSR Route to GCP Default Network ( ) csr-gcp-01#show ip route... B /20 [20/100] via , 00:03:41 HSRP Standby CSR Route to GCP Default Network ( ) csr-gcp-02#show ip route... B /20 [20/100] via , 00:08:47 HSRP Active csr-gcp-01#show stand GigabitEthernet2 - Group 0 (version 2) State is Active HSRP Standby csr-gcp-02#show stand GigabitEthernet2 - Group 0 (version 2) State is Standby... Output summarized 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 34

35 Pre-Failure State (2)... Output summarized First Google Cloud Router BGP State # gcloud compute routers get-status csr-gcp-vm-bgp-rtr kind: compute#routerstatusresponse result: bestroutes: - creationtimestamp: ' T14:48: :00' destrange: /24 kind: compute#route nexthopip: priority: 0 - creationtimestamp: ' T14:48: :00' destrange: /24 kind: compute#route nexthopip: priority: 0 bestroutesforrouter: - creationtimestamp: ' T14:48: :00' destrange: /24 kind: compute#route nexthopip: priority: 0 bgppeerstatus: - advertisedroutes: - destrange: /20 kind: compute#route nexthopip: priority: 100 ipaddress: name: csr-gcp-vm-bgp-peer numlearnedroutes: 1 peeripaddress: state: Established status: UP uptime: 1 minutes, 48 seconds uptimeseconds: '108' network: Determining best path If Cloud Router receives multiple routes for the same destination, GCP uses route metrics and, in some cases, AS path length to determine the best path. To help you configure your on-premises routers, the following list describes the algorithm that GCP uses for egress traffic. If you have multiple BGP sessions on a single Cloud Router, GCP uses the route with the shortest AS path length. If routes have the same AS path length, GCP uses the route with the lower MED value. If routes have equal costs (same AS path length and metric), GCP uses ECMP to balance traffic across multiple paths. If you use multiple Cloud Routers, GCP uses only the MED value to determine the best path. The AS path length doesn't influence the path selection between multiple Cloud Routers. If a static and dynamic route have the same prefix and metric, GCP uses the static route Cisco and/or its affiliates. All rights reserved. Cisco Public 35

36 Pre-Failure State (3)... Output summarized Second Google Cloud Router BGP State # gcloud compute routers get-status csr-gcp-vm-bgp-rtr-02 kind: compute#routerstatusresponse result: bestroutes: - creationtimestamp: ' T14:48: :00' destrange: /24 kind: compute#route nexthopip: priority: 0 - creationtimestamp: ' T14:48: :00' destrange: /24 kind: compute#route nexthopip: priority: 0 bestroutesforrouter: - creationtimestamp: ' T14:43: :00' destrange: /24 kind: compute#route nexthopip: priority: 0 bgppeerstatus: - advertisedroutes: - destrange: /20 kind: compute#route nexthopip: priority: 100 ipaddress: name: csr-gcp-vm-bgp-peer-02 numlearnedroutes: 1 peeripaddress: state: Established status: UP uptime: 6 minutes, 50 seconds uptimeseconds: '410' network: Determining best path If Cloud Router receives multiple routes for the same destination, GCP uses route metrics and, in some cases, AS path length to determine the best path. To help you configure your on-premises routers, the following list describes the algorithm that GCP uses for egress traffic. If you have multiple BGP sessions on a single Cloud Router, GCP uses the route with the shortest AS path length. If routes have the same AS path length, GCP uses the route with the lower MED value. If routes have equal costs (same AS path length and metric), GCP uses ECMP to balance traffic across multiple paths. If you use multiple Cloud Routers, GCP uses only the MED value to determine the best path. The AS path length doesn't influence the path selection between multiple Cloud Routers. If a static and dynamic route have the same prefix and metric, GCP uses the static route Cisco and/or its affiliates. All rights reserved. Cisco Public 36

37 ... Output summarized Failure Scenario 1 HSRP Primary CSR VM Reload HSRP Debug on HSRP Standby csr-gcp-02# *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby: i/resign rcvd (110/ ) *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Active router is local, was *Sep 19 21:59:17.396: HSRP: Gi2 Nbr no longer active for group 0 (Standby) *Sep 19 21:59:17.396: HSRP: Gi2 Nbr Was active or standby - start passive holddown *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby router is unknown, was local *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Standby -> Active *Sep 19 21:59:17.396: %HSRP-5-STATECHANGE: GigabitEthernet2 Grp 0 state Standby -> Active *Sep 19 21:59:17.396: HSRP: Peer not present *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Redundancy "hsrp-gi2-0" state Standby -> Active *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Added to ARP (0000.0c9f.f000) *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Activating MAC c9f.f000 *Sep 19 21:59:17.396: HSRP: Gi2 Grp 0 Adding c9f.f000 to MAC address filter *Sep 19 21:59:17.396: HSRP: Gi2 IP Redundancy "hsrp-gi2-0" standby, local -> unknown *Sep 19 21:59:17.398: HSRP: Gi2 IP Redundancy "hsrp-gi2-0" update, Standby -> Active *Sep 19 21:59:20.379: HSRP: Gi2 IP Redundancy "hsrp-gi2-0" update, Active -> Active *Sep 19 21:59:57.361: %OSPF-5-ADJCHG: Process 10, Nbr on GigabitEthernet2 from FULL to DOWN, Neighbor Down: Dead timer expired On-Prem VM traceroutes via HSRP Newly Active CSR ( ) [root@k8s-m-01 ~]# traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 37

38 Failure Scenario 2 Shut HSRP Primary LAN Interface (BGP session is still active) Pre-Failure GCE Instance traceroutes via GCP BGP Path [shmcfarl@instance-3 ~]$ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms Post-Failure GCE Instance traceroutes via GCP BGP Path [shmcfarl@instance-3 ~]$ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 38

39 Failure Scenario 3 Shut IPsec Tunnel on HSRP Primary CSR With/Without HSRP Interface Tracking Pre-Failure GCE Instance traceroutes via GCP BGP Path [shmcfarl@instance-3 ~]$ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms Post-Failure GCE Instance traceroutes via GCP BGP Path BUT traffic is re-routed to the HSRP Primary ( ) before going to the end host [shmcfarl@instance-3 ~]$ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms ( ) ms * ms LAN Re-Route Issue Resolved Use Track track 10 interface Tunnel0 line-protocol interface GigabitEthernet2 description Private Network On-Prem ip address standby version 2 standby 0 ip standby 0 priority 110 standby 0 preempt standby 0 authentication md5 key-string F D720D standby 0 track 10 decrement 10 On-Prem LAN re-route to HSRP Active on router with failed IPsec Tunnel csr-gcp-01#show stand GigabitEthernet2 - Group 0 (version 2) State is Standby... Priority 100 (configured 110) Track object 10 state Down decrement 10 Tunnel failed and track changed HSRP state [shmcfarl@instance-3 ~]$ traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms ms ms 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 39

40 Reference Cisco CSR Config Primary crypto ikev2 proposal PHASE1-PROP encryption aes-cbc-256 integrity sha1 group 14 crypto ikev2 policy IKE-POL proposal PHASE1-PROP crypto ikev2 keyring KEY peer GCP-PEER address 35.yyy.yyy.y hostname csr-gcp-dmz-sjc pre-shared-key local <PSK_PASSWORD_GOES_HERE> pre-shared-key remote <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile IKEV2-SETUP match identity remote address authentication local pre-share authentication remote pre-share keyring local KEY lifetime crypto ikev2 dpd 10 2 periodic track 10 interface Tunnel0 line-protocol crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac mode tunnel crypto ipsec profile CSR-GCP set transform-set CSR-GCP-SET set pfs group14 set ikev2-profile IKEV2-SETUP... Output summarized interface Tunnel0 ip address ip mtu 1400 ip tcp adjust-mss 1360 tunnel source GigabitEthernet1 tunnel mode ipsec ipv4 tunnel destination 35.yyy.yyy.y tunnel protection ipsec profile CSR-GCP interface GigabitEthernet1 ip address 192.yyy.yyy.y interface GigabitEthernet2 description Private Network On-Prem ip address standby version 2 standby 0 ip standby 0 priority 110 standby 0 preempt standby 0 authentication md5 key-string 7 <HSRP_KEY> standby 0 track 10 decrement 10 router ospf 10 redistribute bgp subnets network area 0 router bgp bgp log-neighbor-changes neighbor remote-as neighbor timers address-family ipv4 redistribute ospf 10 neighbor activate neighbor soft-reconfiguration inbound ip route Cisco 192.yyy.yyy.y and/or its affiliates. All rights reserved. Cisco Public 40

41 Reference Cisco CSR Config Secondary crypto ikev2 proposal PHASE1-PROP encryption aes-cbc-256 integrity sha1 group 14 crypto ikev2 policy IKE-POL proposal PHASE1-PROP crypto ikev2 keyring KEY peer GCP-PEER address 35.xxx.xxx.x hostname csr-vpn-gw-02 pre-shared-key local <PSK_PASSWORD_GOES_HERE> pre-shared-key remote <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile IKEV2-SETUP match identity remote address authentication local pre-share authentication remote pre-share keyring local KEY lifetime crypto ikev2 dpd 10 2 periodic crypto ipsec transform-set CSR-GCP-SET esp-aes 256 esp-sha-hmac mode tunnel crypto ipsec profile CSR-GCP set transform-set CSR-GCP-SET set pfs group14 set ikev2-profile IKEV2-SETUP... Output summarized interface Tunnel0 ip address ip mtu 1400 ip tcp adjust-mss 1360 tunnel source GigabitEthernet1 tunnel mode ipsec ipv4 tunnel destination 35.xxx.xxx.x tunnel protection ipsec profile CSR-GCP interface GigabitEthernet1 ip address 192.xxx.xxx.x interface GigabitEthernet2 description Private Network On-Prem ip address standby version 2 standby 0 ip standby 0 priority 105 standby 0 preempt standby 0 authentication md5 key-string 7 <HSRP_KEY> router ospf 10 redistribute bgp subnets network area 0 router bgp bgp log-neighbor-changes neighbor remote-as neighbor timers address-family ipv4 redistribute ospf 10 neighbor activate neighbor soft-reconfiguration inbound ip route xxx.xxx.x 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 41

42 Amazon Web Services Native VPN

43 AWS VPN Gateway AWS VPN Overview AWS VPN Setup AWS does support NAT-T: Example templates for Cisco IOS: Cisco and/or its affiliates. All rights reserved. Cisco Public 43

44 Topology for AWS to On-Prem CSR IPsec VPN BGP Routing BGP <> OSPF Redistribution IPsec/IKEv2 Tunnel Mode Cisco CSR1000v VPC Network /16 VPC Router VPN Gateway BGP AS xxx.xxx.x xxx.xxx.x.1 Private Network /24 OSPF 10 Area 0 Hypervisor BGP AS65002 Routes this side should see: /24 Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 44

45 AWS CLI: Create VPC, VPN GW, Customer GW and VPN Connection Create a new AWS VPC (or use an existing one) # aws ec2 create-vpc --cidr-block /16 Create VPN Gateway and set the AWS BGP ASN # aws ec2 create-vpn-gateway --type ipsec.1 --amazon-side-asn Attach VPN Gateway to the VPC # aws ec2 attach-vpn-gateway --vpc-id vpc-ce2124aa --vpn-gateway-id vgw-64277e21 Create a new customer gateway with the on-prem BGP ASN and the on-prem router IP address (do this for each connection) # aws ec2 create-customer-gateway --bgp-asn public-ip 192.xxx.xxx.x --type ipsec.1 Create a new VPN connection # aws ec2 create-vpn-connection --customer-gateway-id cgw-d6055d93 --type ipsec.1 --vpn-gateway-id vgw-64277e21 Note: Lots of output will come from the above VPN creation command. This information can be used to build the on-prem CSR config. The best method for getting the configuration is shown on the next slide. Enable route propagation for the VPC # aws ec2 enable-vgw-route-propagation --gateway-id vgw-64277e21 --route-table-id rtb-515e8e36 Permit SSH and ICMP # aws ec2 authorize-security-group-ingress --group-name default --protocol tcp --port 22 --cidr /0 # aws ec2 authorize-security-group-ingress --group-name default --protocol icmp --port -1 --cidr / Cisco and/or its affiliates. All rights reserved. Cisco Public 45

46 Optional: Download Router Configuration VPC Dashboard > VPN Connections 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 46

47 Reference Cisco CSR Config - Primary crypto isakmp policy 200 encryption aes 128 authentication pre-share group 2 lifetime hash sha crypto keyring keyring-vpn-cec local-address 192.xxx.xxx.x pre-shared-key address 52.xxx.xxx.x key <PSK_PASSWORD_GOES_HERE> crypto isakmp profile isakmp-vpn-cec local-address 192.xxx.xxx.x match identity address 52.xxx.xxx.x keyring keyring-vpn-cec crypto ipsec transform-set ipsec-prop-vpn-cec esp-aes 128 esp-sha-hmac mode tunnel crypto ipsec profile ipsec-vpn-cec set pfs group2 set security-association lifetime seconds 3600 set transform-set ipsec-prop-vpn-cec crypto ipsec df-bit clear crypto isakmp keepalive on-demand crypto ipsec fragmentation before-encryption... Output summarized interface Tunnel1 ip address ip virtual-reassembly ip mtu 1400 tunnel source 192.xxx.xxx.x tunnel destination 52.xxx.xxx.x tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-cec ip tcp adjust-mss 1379 router ospf 10 redistribute bgp subnets network area 0 router bgp neighbor remote-as neighbor activate neighbor timers address-family ipv4 redistribute ospf 10 neighbor remote-as neighbor activate neighbor soft-reconfiguration inbound 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 47

48 ... Output summarized Verify Routing and Reachability On the on-prem CSR check the route for the the AWS VPC network /16 csr-mc-01#show ip route i B /16 [20/100] via , 00:13:35 On AWS check for the route for the on-prem network ( /24) # aws ec2 describe-route-tables grep ROUTES /24 vgw-64277e21 EnableVgwRoutePropagation active Connect to an AWS instances and ping to the on-prem private network ubuntu@ip :~$ ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=63 time=4.95 ms 64 bytes from : icmp_seq=2 ttl=63 time=4.47 ms VPC Network /16.1 BGP Cisco CSR1000v.1.30 VM Private Network /24 Hypervisor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48

49 Topology for Dual Cisco CSR on AWS BGP AS On-Prem Cloud 1 vsphere Hosted Cisco CSR OSPF 10 Area 0.2 ESXi Host 1 VPC Network /16 VPC Router VPN Gateway BGP AS /24.3 Private Network HSRP VIP =.1 ESXi Host BGP AS65002 Routes this side should see: /24 vsphere Distributed vswitch (DVS) with a Distributed PortGroup for the Private Network Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 49

50 Microsoft Azure Native VPN

51 Microsoft Azure VPN Gateway Azure VPN Overview In order to use BGP you must use Route-Based VPN and SKUs VpnGw1, VpnGw2, VpnGw3, Standard or HighPerformance SKUs : Azure does not support NAT-T 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 51

52 Azure to On-Prem CSR IPsec VPN BGP Routing BGP <> OSPF Redistribution IPsec/IKEv2 Tunnel Mode Cisco CSR1000v Vnet Subnet /16 VPN Gateway 40.xxx.xxx.x xxx.xxx.x.1 Private Network /24 OSPF 10 Area 0 BGP AS64512 Hypervisor BGP AS Cisco and/or its affiliates. All rights reserved. Cisco Public 52

53 Azure CLI: Create Resource Group, Networks, Subnets Create a new Azure Resource Group (rg) # az group create --name azure-vpn-rg --location westus # az configure --defaults location=westus # az configure --defaults group=azure-vpn-rg Create a new virtual network (vnet) and a new outside subnet # az network vnet create \ --name vnet1 \ --address-prefix /16 \ --subnet-name outside \ --subnet-prefix /24 Create a inside subnet # az network vnet subnet create \ --vnet-name vnet1 \ --name inside \ --address-prefix /24 Create a new subnet that is used for the IPsec/BGP interface on the Azure side # az network vnet subnet create \ --vnet-name vnet1 \ --name gatewaysubnet \ --address-prefix / Cisco and/or its affiliates. All rights reserved. Cisco Public 53

54 Azure CLI: Create a Public IP, VPN/Vnet Gateway & Local Gateway Create a new public IP address (Using Azure VPN service, the allocation must be dynamic ) # az network public-ip create \ --name azure-vpn-gw-eip \ --allocation-method dynamic Create Vnet gateway using RouteBased (BGP) and a supported sku (see earlier links for requirements). THIS TAKES AWHILE # az network vnet-gateway create \ --name vpn-gw \ --public-ip-address azure-vpn-gw-eip \ --vnet vnet1 \ --gateway-type Vpn \ --sku VpnGw1 \ --vpn-type RouteBased \ --asn Once the Vnet gateway is up, get the Azure-side BGP Peering address (Needed for on-prem configuration) # az network vnet-gateway list grep bgppeeringaddress "bgppeeringaddress": " ", Create the local gateway (on-prem target). Local prefix/bgp peer should be the on-prem CSR tunnel info. Can t be in Azure vnet range # az network local-gateway create \ --gateway-ip-address 192.xxx.xxx.x \ --name azure-lng \ --local-address-prefixes /32 \ --asn \ --bgp-peering-address Cisco and/or its affiliates. All rights reserved. Cisco Public 54

55 Azure CLI: Vnet GW, Local GW, VPN Connection Copy the full path from the id line (under the gatewaytype: Vpn line) that is shown in the vnet-gateway output # az network vnet-gateway show --name vpn-gw "gatewaytype": "Vpn", "id": "/subscriptions/<your_id>/resourcegroups/azure-vpn-rg/providers/microsoft.network/virtualnetworkgateways/vpn-gw", Copy the full path from the id line that is shown in the vnet-gateway output # az network local-gateway show --name azure-ln "id": "/subscriptions/<your_id>/resourcegroups/azure-vpn-rg/providers/microsoft.network/localnetworkgateways/azure-lng" Create the VPN connection using information from above # az network vpn-connection create \ --name azure-to-csr \ --vnet-gateway1 /subscriptions/<your_id>/resourcegroups/azure-vpn-rg/providers/microsoft.network/virtualnetworkgateways/vpn-gw \ --enable-bgp \ --shared-key <YOUR_PRE_SHARED_KEY>" \ --local-gateway2 /subscriptions/<your_id>/resourcegroups/azure-vpn-rg/providers/microsoft.network/localnetworkgateways/azure-lng Optional: Create a new test VM on Azure and associate it with the inside subnet # az vm create \ --name AzTestVm \ --authentication-type ssh \ --ssh-key-value "$(< ~/.ssh/id_rsa.pub)" \ --image Canonical:UbuntuServer:16.04-LTS:latest \ --size Standard_DS1_v2 \ --vnet-name vnet1 \ --subnet inside \ --public-ip-address-allocation dynamic 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 55

56 Connect to the Azure CSR Enable Interfaces Connect to the new Azure-hosted CSR and enable the GigabitEthernet 2 interface for DHCP # ssh csr-azure@40.xxx.xxx.x csr-azure-01#configure terminal Enter configuration commands, one per line. End with CNTL/Z. csr-azure-01(config)#interface gigabitethernet 2 csr-azure-01(config-if)#ip address dhcp csr-azure-01(config-if)#no shutdown Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct IP addresses: csr-azure-01#show ip interface brief Interface IP-Address OK? Method Status Protocol GigabitEthernet YES DHCP up up GigabitEthernet YES DHCP up up VirtualPortGroup YES TFTP up up Note: This can all be automated (along with the DMVPN configs) by creating Azure Automation/Resource Manager 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 56

57 On-Prem Cisco CSR IPsec/Routing Config crypto ikev2 proposal PHASE1-PROP encryption aes-cbc-256 integrity sha1 group 2 crypto ikev2 policy IKE-POL proposal PHASE1-PROP crypto ikev2 keyring KEY peer AZURE-PEER address 40.xxx.xxx.x pre-shared-key local <PSK_PASSWORD_GOES_HERE> pre-shared-key remote <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile IKEV2-SETUP match identity remote address authentication local pre-share authentication remote pre-share keyring local KEY lifetime crypto ikev2 dpd 10 2 periodic crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set CSR-AZURE-SET esp-aes 256 esp-sha-hmac mode tunnel crypto ipsec profile CSR-AZURE set transform-set CSR-AZURE-SET set pfs group14 set ikev2-profile IKEV2-SETUP... Output summarized interface Tunnel2 ip address ip mtu 1400 ip tcp adjust-mss 1360 tunnel source GigabitEthernet1 tunnel mode ipsec ipv4 tunnel destination 40.xxx.xxx.x tunnel protection ipsec profile CSR-AZURE interface GigabitEthernet1 description Internet ip address 192.xxx.xxx.x no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id redistribute bgp subnets network area 0 router bgp bgp log-neighbor-changes neighbor remote-as neighbor ebgp-multihop 255 address-family ipv4 redistribute ospf 10 neighbor activate neighbor soft-reconfiguration inbound ip route xxx.xxx.x ip route Tunnel Cisco and/or its affiliates. All rights reserved. Cisco Public 57

58 ... Output summarized Verify Routing and Reachability On the on-prem CSR check the route for the AWS VPC network /24 csr-mc-01#show ip route i B /16 [20/0] via , 00:51:26 On AWS check for the route for the on-prem network ( /24) PS Azure:\> Get-AzureRmEffectiveRouteTable -NetworkInterfaceName AzTestVmVMNic -ResourceGroupName azure-vpn-rg Format-Table Name State Source AddressPrefix NextHopType NextHopIpAddress Active VirtualNetworkGateway { /24} VirtualNetworkGateway {40.xxx.xxx.x} Connect to an AWS instances and ping to the on-prem private network shmcfarl@aztestvm:~$ ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=254 time=4.48 ms 64 bytes from : icmp_seq=2 ttl=254 time=4.38 ms Inside Subnet /24 VPN Gateway 40.xxx.xxx.x xxx.xxx.x Cisco CSR1000v.1.30 Private Network /24 Hypervisor VM 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 58

59 DMVPN

60 DMVPN (Dynamic Multipoint VPN) Cisco DMVPN Cisco Live DMVPN Cisco IWAN CVD DMVPN is a Cisco innovation for building GRE/mGRE + IPsec VPN connections in a dynamic and scalable manner 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 60

61 Terminology and Features Core Network /17 Overlay Addresses / /24 Tunnel Address Hub1 Hub 2 Tunnel: Physical: Tunnel: Physical: NBMA Address Tunnel: Physical: Tunnel: Physical: Transport Network Spoke 1 GRE/IPsec Tunnels Spoke 2 Overlay Network / /24 On Demand Spoke Tunnels 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 61

62 DMVPN Components Next Hop Resolution Protocol (NHRP) Creates a distributed (NHRP) mapping database of all the spoke s tunnel to real (public interface) addresses Multipoint GRE Tunnel Interface (mgre) Single GRE interface to support multiple GRE/IPsec tunnels Simplifies size and complexity of configuration IPsec tunnel protection Dynamically creates and applies encryption policies Routing Dynamic advertisement of branch networks; almost all routing protocols (EIGRP, RIP, OSPF, BGP, ODR) are supported 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 62

63 . DMVPN Implementation Spoke-to-hub tunnels Spoke-to-spoke tunnels 2547oDMVPN tunnels Hub and spoke (Phase 1) Spoke-to-spoke (Phase 2) VRF-lite Server Load Balancing Hierarchical (Phase 3) 2547oDMVPN 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 63

64 Google Cloud Platform Cisco CSR & DMVPN

65 DISCLAIMER As of this moment, Cisco CSR 1000v on Google Cloud Platform is in a proof-ofconcept phase and it is not a shipping solution While the functional operation of the Cisco CSR 1000v with DMVPN on GCP in the following slides match those of AWS and Azure, the actual provisioning of the CSR as an instance in GCE, is specific to Google s environment and subject to change Stay tuned for additional information on the release of Cisco CSR 1000v on GCP 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

66 GCP to On-Prem CSR IPsec VPN Example 1 BGP <> OSPF Redistribution Default Network /20 BGP AS65000 Compute Engine Google Cloud VPN Google Cloud Router 35.xxx.xxx.x IPsec/IKEv2 Tunnel Mode xxx.xxx.x BGP AS65002 Cisco CSR1000v.1.30 VM Private Network /24 OSPF 10 Area 0 Hypervisor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 66

67 GCP CSR to On-Prem CSR IPsec VPN Example 2 NOT OFFICIALLY RELEASED Compute Engine inside-network / Cisco CSR1000v Default Network / xxx.xxx.x 192.xxx.xxx.x Cisco CSR1000v.1.30 VM Private Network /24 OSPF 10 Area 0 Hypervisor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67

68 GCP CSR to On-Prem CSR IPsec VPN Example 2 NOT OFFICIALLY RELEASED Compute Engine inside-network / Cisco CSR1000v Default Network / xxx.xxx.x IPsec/IKEv2 Tunnel Mode OSPF 192.xxx.xxx.x Cisco CSR1000v.1.30 VM Private Network /24 OSPF 10 Area 0 Hypervisor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 68

69 NOT OFFICIALLY RELEASED GCP CSR to On-Prem CSR DMVPN Compute Engine inside-network / Cisco CSR1000v Default Network / xxx.xxx.x Spoke CSR Tunnel: DMVPN OSPF 192.xxx.xxx.x Hub CSR Tunnel: Cisco CSR1000v.1.30 VM Private Network /24 OSPF 10 Area 0 Hypervisor Routes this side should see: /24 Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 69

70 gcloud Create the GCP External IP, Inside VPC Network & Route Create a new external IP reservation that will be used for the GCP CSR NATed connection (or use an existing one) # gcloud compute addresses create csr-to-csr-ext-ip --region us-west1 Capture the external IP address # gcloud compute addresses list --filter="csr-to-csr-ext-ip" NAME REGION ADDRESS STATUS csr-to-csr-ext-ip us-west1 35.xxx.xxx.x RESERVED Create a new GCP inside network that will be attached to the inside interface of the CSR # gcloud compute networks create inside-network --subnet-mode=custom Create a new GCP inside subnet - Associate it with the inside network # gcloud compute networks subnets create inside-subnet \ --network=inside-network \ --range= /24 Create a new GCP route from the CSR inside network to the on-prem private network which routes through the IPsec VPN # gcloud compute routes create inside-to-csr-private \ --network=inside-network \ --destination-range= /24 \ --next-hop-address= NOT OFFICIALLY RELEASED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 70

71 gcloud Create GCP Firewall Rules Create a new GCP firewall rule to allow traffic into the inside CSR network from the default network # gcloud compute firewall-rules create allow-default-to-csr-inside \ --direction=ingress \ --network=inside-network \ --action=allow \ --rules=all \ --source-ranges= /0 Create a new GCP firewall rule to allow traffic between the default network and the on-prem CSR public IP for IKE, IPsec # gcloud compute firewall-rules create csr-csr-vpn \ --direction=ingress \ --network=default \ --action=allow \ --rules=udp:500,udp:4500,esp \ --source-ranges=192.xxx.xxx.x NOT OFFICIALLY RELEASED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71

72 gcloud Create CSR and Test Instances Create a new GCE CSR instance and set fixed IPv4 addresses to each of the two interfaces # gcloud compute instances create "csr-gcp-01" \ --zone "us-west1-a" \ --machine-type "n1-standard-4" \ --network-interface subnet="default",private-network-ip=" ",address="35.xxx.xxx.x" \ --can-ip-forward \ --network-interface subnet="inside-subnet",private-network-ip=" ",no-address \ --image name_of_csr_image" \ --boot-disk-size "10" \ --boot-disk-type "pd-standard" \ --boot-disk-device-name "csr-gcp-01" Create a new GCE test instance that will be used to validate the VPN and routing # gcloud compute instances create "csr-inside-vm" \ --zone "us-west1-a" \ --machine-type "g1-small" \ --subnet "inside-subnet" \ --private-network-ip " " \ --image "debian-9-stretch-v " \ --image-project "debian-cloud" \ --boot-disk-size "10" \ --boot-disk-type "pd-standard" \ --boot-disk-device-name "csr-inside-vm" NOT OFFICIALLY RELEASED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 72

73 Connect to the GCP CSR Enable Interfaces Connect to the new GCP-hosted CSR and enable the GigabitEthernet 2 interface for DHCP... Output summarized # gcloud compute ssh cisco-user@csr-gcp-01 csr1kv-gcp#configure terminal Enter configuration commands, one per line. End with CNTL/Z. csr1kv-gcp(config)#interface gigabitethernet 2 csr1kv-gcp(config-if)#ip address dhcp csr1kv-gcp(config-if)#no shutdown Wait a few seconds and check to make sure that both interfaces on the CSR are up and with the correct IP addresses: csr1kv-gcp#show ip interface brief Interface IP-Address OK? Method Status Protocol GigabitEthernet YES TFTP up up GigabitEthernet YES DHCP up up NOT OFFICIALLY RELEASED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 73

74 GCP Cisco CSR DMVPN Config Spoke crypto ikev2 proposal AES/GCM/256 encryption aes-gcm-256 prf sha512 group 19 crypto ikev2 policy AES/GCM/256 match fvrf any proposal AES/GCM/256 crypto ikev2 keyring DMVPN-KEYRING peer ANY address pre-shared-key <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile DMVPN-IKEv2-PROFILE description PSK Profile match identity remote address identity local address 35.xxx.xxx.x authentication remote pre-share authentication local pre-share keyring local DMVPN-KEYRING dpd 40 5 on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 mode transport crypto ipsec profile DMVPN-IPSEC-PROFILE set transform-set AES256/GCM/TRANSFORM set ikev2-profile DMVPN-IKEv2-PROFILE... Output summarized interface Tunnel0 description DMVPN ip address no ip redirects ip mtu 1400 ip nhrp authentication cisco123 ip nhrp network-id 100 ip nhrp nhs nbma 192.xxx.xxx.x multicast ip tcp adjust-mss 1360 ip ospf authentication-key 7 110A D5A5E57 ip ospf network point-to-multipoint ip ospf hello-interval 10 tunnel source GigabitEthernet1 tunnel mode gre multipoint tunnel key 100 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE interface GigabitEthernet1 description Internet ip address no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id network area 1 network area 0 ip route NOT OFFICIALLY RELEASED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 74

75 On-Prem Cisco CSR DMVPN Config Hub crypto ikev2 proposal AES/GCM/256 encryption aes-gcm-256 prf sha512 group 19 crypto ikev2 policy AES/GCM/256 match fvrf any proposal AES/GCM/256 crypto ikev2 keyring DMVPN-KEYRING peer ANY address pre-shared-key <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile DMVPN-IKEv2-PROFILE description PSK Profile match identity remote address identity local address 192.xxx.xxx.x authentication remote pre-share authentication local pre-share keyring local DMVPN-KEYRING dpd 40 5 on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 mode transport crypto ipsec profile DMVPN-IPSEC-PROFILE set transform-set AES256/GCM/TRANSFORM set ikev2-profile DMVPN-IKEv2-PROFILE... Output summarized interface Tunnel0 description DMVPN ip address no ip redirects ip mtu 1400 ip nhrp authentication cisco123 ip nhrp map multicast dynamic ip nhrp network-id 100 ip nhrp redirect ip tcp adjust-mss 1360 ip ospf authentication-key 7 121A0C D5679 ip ospf network point-to-multipoint ip ospf hello-interval 10 tunnel source GigabitEthernet1 tunnel mode gre multipoint tunnel key 100 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE interface GigabitEthernet1 description Internet ip address 192.xxx.xxx.x no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id network area 0 network area 0 ip route xxx.xxx.x 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 75

76 Verify Routing and Reachability... Output summarized On the GCP CSR, check for the private network route from the on-prem side( /24) csr1kv-gcp#show ip route i O /24 [110/1001] via , 00:09:51, Tunnel0 On the on-prem CSR, check for the VPC inside network route (10.1.0/24) csr-mc-01#show ip route i O IA /24 [110/1001] via , 00:40:08, Tunnel0 Check the DMVPN Next-Hop Resolution Protocol (NHRP) Status csr1kv-gcp#show ip nhrp /32 via Tunnel0 created 5d14h, never expire Type: static, Flags: NBMA address: 192.xxx.xxx.x csr-mc-01#show ip nhrp /32 via Tunnel0 created 00:40:25, expire 00:08:20 Type: dynamic, Flags: registered used nhop NBMA address: 35.xxx.xxx.x (Claimed NBMA address: ) Connect to the GCP test instance that was created earlier and ping to the on-prem private network # gcloud compute ssh "csr-inside-vm shmcfarl@csr-inside-vm:~$ ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=62 time=22.1 ms 64 bytes from : icmp_seq=2 ttl=62 time=23.3 ms 64 bytes from : icmp_seq=3 ttl=62 time=23.6 ms NOT OFFICIALLY RELEASED 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76

77 Amazon Web Services Cisco CSR & DMVPN

78 AWS with Cisco CSR 1000v Support Amazon Web Services Marketplace + Cisco CSR: _search_box Cisco CSR for AWS Deployment DMVPN AWS_3.html Deployment Cisco Live Session for AWS with Cisco CSR: #/session/ AhER Transit VPC with CSR: Cisco and/or its affiliates. All rights reserved. Cisco Public 78

79 AWS to On-Prem CSR IPsec VPN Example 1 BGP <> OSPF Redistribution IPsec/IKEv2 Tunnel Mode Cisco CSR1000v VPC Network /16 VPC Router VPN Gateway BGP AS xxx.xxx.x xxx.xxx.x.1 Private Network /24 OSPF 10 Area 0 Hypervisor BGP AS Cisco and/or its affiliates. All rights reserved. Cisco Public 79

80 AWS CSR to On-Prem CSR IPsec VPN Example 2 VPC Network /24 Public-side Network /24 Cisco CSR1000v VPC Router 52.xxx.xxx.x IPsec/IKEv2 Tunnel Mode 192.xxx.xxx.x Cisco CSR1000v.1 Private Network /24 OSPF 10 Area 0 Hypervisor OSPF 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 80

81 AWS CSR to On-Prem CSR DMVPN Public-side Network /24 VPC Network /24 Cisco CSR1000v VPC Router 52.xxx.xxx.x Spoke CSR Tunnel: xxx.xxx.x Hub CSR Tunnel: Cisco CSR1000v.1 Private Network /24 OSPF 10 Area 0 Hypervisor DMVPN Routes this side should see: /24 OSPF Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 81

82 AWS CLI: Create VPC, Subnets and Internet GW Create a new AWS VPC (vpc) # aws ec2 create-vpc --cidr-block /16 Create a new subnet in the VPC (this one will be used for the CSR s outside interface) # aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block /24 Create another new subnet in the VPC (this one will be used for the CSR s inside interface) # aws ec2 create-subnet --vpc-id vpc-66a0a102 --cidr-block /24 Create a new AWS Internet Gateway (igw) # aws ec2 create-internet-gateway Attach the Internet gateway to the VPC # aws ec2 attach-internet-gateway --vpc-id vpc-66a0a102 --internet-gateway-id igw-591fba3d 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 82

83 AWS CLI: Create Route Tables Create a new route table in the VPC (rtb) that will be used by the CSR s outside subnet # aws ec2 create-route-table --vpc-id vpc-66a0a102 Create a new default route in the route table and point it to the Internet gateway # aws ec2 create-route --route-table-id rtb-aaa37dcd --destination-cidr-block /0 --gateway-id igw-591fba3d Associate the new routable with the outside VPC subnet # aws ec2 associate-route-table --subnet-id subnet-0c15b86b --route-table-id rtb-aaa37dcd Create a new route table in the VPC (rtb) that will be used by the CSR s inside subnet # aws ec2 create-route-table --vpc-id vpc-66a0a102 Create a new default route in the route table for the inside subnet and point it to the Internet gateway # aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block /0 --gateway-id igw-591fba3d Create a new default route in the route table and point it to the Internet gateway # aws ec2 create-route --route-table-id rtb-3741e750 --destination-cidr-block /24 --network-interface-id eni-af67db80 Associate the new route table with the inside VPC subnet # aws ec2 associate-route-table --subnet-id subnet-c617baa1 --route-table-id rtb-3741e Cisco and/or its affiliates. All rights reserved. Cisco Public 83

84 AWS CLI: Create a Security Group/Rules Reference Create a new security group for the outside facing interface (Optional: You can just use an existing group) # aws ec2 create-security-group --group-name csr --description csr-rules --vpc-id vpc-66a0a102 Create a new security group rule for SSH to the CSR # aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 --protocol tcp --port 22 --cidr /0 Create a new security group rule for ICMP from the other CSRs (On-Prem and GCP CSR [optional: Just showing the format for your use]) # aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \ --ip-permissions '[{"IpProtocol": "icmp", "FromPort": -1, "ToPort": -1, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}] Create a new security group rule for ESP (IP 50) from the other CSRs # aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \ --ip-permissions '[{"IpProtocol": "50", "IpRanges": [{"CidrIp": 192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]' Create a new security group rule for IKE from the other CSRs # aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \ --ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}]' Create a new security group rule for IKE/NAT-T from the other CSRs # aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \ --ip-permissions '[{"IpProtocol": "17", "FromPort": 4500, "ToPort": 4500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}]}] Optional: You may want to create a security group just for the inside subnet that has different rules than the one for the outside subnet Create a new security group for the outside facing interface (Optional: You can just use an existing group) # aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr /24 Create a new security group for the outside facing interface (Optional: You can just use an existing group) # aws ec2 authorize-security-group-ingress --group-id sg-84aef7e2 --protocol all --cidr / Cisco and/or its affiliates. All rights reserved. Cisco Public 84

85 { } AWS CLI: Run a new CSR Instance Using Previous Parameters csr-create.json "ImageId": "ami-99e5d0f9", "InstanceType": "t2.medium", "KeyName": "mc-aws-key", "NetworkInterfaces": [ { "DeviceIndex": 0, "Description": "Primary network interface", "Groups": [ "sg-65c39b03" ], "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": " " } ], "SubnetId": "subnet-0c15b86b" }, { "DeviceIndex": 1, "PrivateIpAddresses": [ { "Primary": true, "PrivateIpAddress": " " } ], "SubnetId": "subnet-c617baa1" } ] Create a CSR instance using the JSON file shown to the left # aws ec2 run-instances --cli-input-json file://csr-create.json Create a tag/name and associate it with the CSR (Optional) # aws ec2 create-tags --resources i-0f2a0ee857e9c2540 \ --tags Key=Name,Value=csr-aws-01 Create a new External IP (EIP) allocation (or use an existing one) # aws ec2 allocate-address eipalloc-ab35cb96 vpc 52.xxx.xxx.x Associate the EIP with the outside interface of the CSR (GigabitEthernet 1) # aws ec2 associate-address --allocation-id eipalloc-ab35cb96 \ --network-interface-id eni-dd5bd6f2 Modify the inside subnet to disable source/destination checking # aws ec2 modify-network-interface-attribute \ --network-interface-id eni-af67db80 \ --source-dest-check "{\"Value\": false}" A note about NAT: If you plan to use the CSR for NAT operation, you must disable source/destination checking on the outside CSR interface/subnet de/vpc_nat_instance.html#eip_disable_srcdestcheck 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 85

86 Connect to the AWS CSR Enable Interfaces Connect to the new AWS-hosted CSR and enable the GigabitEthernet 2 interface for DHCP # ssh -i "mc-aws-key.pem" ec2-user@52.xxx.xxx.x csr-aws-01#configure terminal Enter configuration commands, one per line. End with CNTL/Z. csr-aws-01(config)#interface gigabitethernet 2 csr-aws-01(config-if)#ip address dhcp csr-aws-01(config-if)#no shutdown Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct IP addresses: csr-aws-01#show ip interface brief Interface IP-Address OK? Method Status Protocol GigabitEthernet YES DHCP up up GigabitEthernet YES DHCP up up VirtualPortGroup YES TFTP up up Note: This can all be automated (along with the DMVPN configs) by creating AWS CloudFormation templates 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86

87 AWS Cisco CSR DMVPN Config Spoke crypto ikev2 proposal AES/GCM/256 encryption aes-gcm-256 prf sha512 group 19 crypto ikev2 policy AES/GCM/256 match fvrf any proposal AES/GCM/256 crypto ikev2 keyring DMVPN-KEYRING peer ANY address pre-shared-key <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile DMVPN-IKEv2-PROFILE description PSK Profile match identity remote address identity local address 52.xxx.xxx.x authentication remote pre-share authentication local pre-share keyring local DMVPN-KEYRING dpd 40 5 on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 mode transport crypto ipsec profile DMVPN-IPSEC-PROFILE set transform-set AES256/GCM/TRANSFORM set ikev2-profile DMVPN-IKEv2-PROFILE... Output summarized interface Tunnel0 description DMVPN ip address no ip redirects ip mtu 1400 ip nhrp authentication cisco123 ip nhrp network-id 100 ip nhrp nhs nbma 192.xxx.xxx.x multicast ip tcp adjust-mss 1360 ip ospf authentication-key 7 110A D5A5E57 ip ospf network point-to-multipoint ip ospf hello-interval 10 tunnel source GigabitEthernet1 tunnel mode gre multipoint tunnel key 100 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE interface GigabitEthernet1 description Internet ip address dhcp no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id network area 2 network area 0 ip route Cisco and/or its affiliates. All rights reserved. Cisco Public 87

88 On-Prem Cisco CSR DMVPN Config Hub Nothing ever changes on the hub for each example crypto ikev2 proposal AES/GCM/256 encryption aes-gcm-256 prf sha512 group 19 crypto ikev2 policy AES/GCM/256 match fvrf any proposal AES/GCM/256 crypto ikev2 keyring DMVPN-KEYRING peer ANY address pre-shared-key <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile DMVPN-IKEv2-PROFILE description PSK Profile match identity remote address identity local address 192.xxx.xxx.x authentication remote pre-share authentication local pre-share keyring local DMVPN-KEYRING dpd 40 5 on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 mode transport crypto ipsec profile DMVPN-IPSEC-PROFILE set transform-set AES256/GCM/TRANSFORM set ikev2-profile DMVPN-IKEv2-PROFILE... Output summarized interface Tunnel0 description DMVPN ip address no ip redirects ip mtu 1400 ip nhrp authentication cisco123 ip nhrp map multicast dynamic ip nhrp network-id 100 ip nhrp redirect ip tcp adjust-mss 1360 ip ospf authentication-key 7 121A0C D5679 ip ospf network point-to-multipoint ip ospf hello-interval 10 tunnel source GigabitEthernet1 tunnel mode gre multipoint tunnel key 100 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE interface GigabitEthernet1 description Internet ip address 192.xxx.xxx.x no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id network area 0 network area 0 ip route xxx.xxx.x 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 88

89 ... Output summarized Verify Routing and Reachability On the on-prem CSR check the route for the AWS VPC network /24 csr-mc-01#show ip route i O IA [110/1001] via , 00:11:41, Tunnel0 On AWS check for the route for the on-prem network ( /24) csr-aws-01#show ip route i O /24 [110/1001] via , 6d17h, Tunnel0 Connect to an AWS instances and ping to the on-prem private network [ec2-user@ip ~]$ ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=62 time=2.75 ms 64 bytes from : icmp_seq=2 ttl=62 time=2.93 ms 64 bytes from : icmp_seq=3 ttl=62 time=2.75 ms VPC Network /24 Cisco CSR1000v.10 Spoke CSR Tunnel: OSPF Hub CSR Tunnel: Cisco CSR1000v.1.30 Private Network /24 Hypervisor VM 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 89

90 Amazon Web Services Marketplacebased Launch Walk-thru For Reference

91 AWS Marketplace CSR Launch Console (1) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 91

92 AWS Launch CSR as an Instance Console (1) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 92

93 AWS Launch CSR as an Instance Console (2) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93

94 AWS Launch CSR as an Instance Console (3) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 94

95 AWS Launch CSR as an Instance Console (4) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 95

96 AWS Launch CSR as an Instance Console (5) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 96

97 AWS Launch CSR as an Instance Console (6) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 97

98 AWS Launch CSR as an Instance Console (7) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 98

99 AWS Launch CSR as an Instance Console (8) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 99

100 AWS Launch CSR as an Instance Console (9) 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 100

101 AWS Launch CSR as an Instance Console (10) Cisco and/or its affiliates. All rights reserved. Cisco Public

102 Microsoft Azure Cisco CSR and DMVPN

103 Azure to On-Prem CSR IPsec VPN Example 1 BGP <> OSPF Redistribution IPsec/IKEv2 Tunnel Mode Cisco CSR1000v Vnet Subnet /16 VPN Gateway 40.xxx.xxx.x xxx.xxx.x.1 Private Network /24 OSPF 10 Area 0 BGP AS64512 Hypervisor BGP AS Cisco and/or its affiliates. All rights reserved. Cisco Public 103

104 Azure CSR to On-Prem CSR IPsec VPN Example 2 Inside Subnet /24 Outside Subnet /24 Cisco CSR1000v 40.xxx.xxx.x IPsec/IKEv2 Tunnel Mode 192.xxx.xxx.x Cisco CSR1000v.1 Private Network /24 OSPF 10 Area 0 Hypervisor OSPF 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 104

105 Azure CSR to On-Prem CSR DMVPN Outside Subnet /24 Inside Subnet Cisco CSR1000v 40.xxx.xxx.x 192.xxx.xxx.x Cisco CSR1000v.1 Private Network /24 Spoke CSR Tunnel: Hub CSR Tunnel: /24 OSPF 10 Area 0 Hypervisor DMVPN Routes this side should see: /24 OSPF Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 105

106 Microsoft Azure with Cisco CSR 1000v Microsoft Azure Marketplace Cisco CSR 1000v with Azure Deployment Cisco and/or its affiliates. All rights reserved. Cisco Public 106

107 Azure CLI: Create Resource Group, Networks, Subnets Create a new Azure Resource Group (rg) # az group create --name multicloud-rg --location westus Create a new public (external IP) IPv4 address to be used for the CSR s outside interface # az network public-ip create --resource-group multicloud-rg --name csr-azure-01-eip --allocation-method static Create a new virtual network (vnet) and a subnet to be used for the CSR s outside interface # az network vnet create \ --resource-group multicloud-rg \ --name mc-csr-vnet \ --address-prefix /16 \ --subnet-name csr-outside \ --subnet-prefix /24 Create a new subnet for the CSR s inside interface and associate it with the vnet created above # az network vnet subnet create \ --resource-group multicloud-rg \ --vnet-name mc-csr-vnet \ --name csr-inside \ --address-prefix / Cisco and/or its affiliates. All rights reserved. Cisco Public 107

108 Azure CLI: Create Route Tables Create a new route table (rt) that will be used for the CSR s outside subnet # az network route-table create \ --resource-group multicloud-rg \ --name csr-outside-rt Create a new route table that will used for the CSR s inside subnet # az network route-table create \ --resource-group multicloud-rg \ --name csr-inside-rt Create a new route table entry for the inside subnet to reach the on-prem network ( ) via the CSR s IP ( ) # az network route-table route create \ --resource-group multicloud-rg \ --name csr-to-on-prem-route \ --route-table-name csr-inside-rt \ --address-prefix /24 \ --next-hop-type VirtualAppliance \ --next-hop-ip-address Associate the outside route table with the outside subnet # az network vnet subnet update \ --resource-group multicloud-rg \ --vnet-name mc-csr-vnet \ --name csr-outside \ --route-table csr-outside-rt Associate the inside route table with the inside subnet # az network vnet subnet update \ --resource-group multicloud-rg \ --vnet-name mc-csr-vnet \ --name csr-inside \ --route-table csr-inside-rt 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 108

109 Azure CLI: Create Network Security Group (NSG) Create a new Network Security Group (NSG) to be used for the outside CSR interface # az network nsg create \ --resource-group multicloud-rg \ --name csr-nsg-outside Create a new NSG rule to allow inbound SSH access to the CSR (can make it specific to an IP/Prefix) # az network nsg rule create \ --resource-group multicloud-rg \ --nsg-name csr-nsg-outside \ --name SSHRule \ --priority 100 \ --source-address-prefixes 'Internet' \ --source-port-ranges '*' \ --destination-address-prefixes '*' \ --destination-port-ranges 22 \ --access Allow \ --protocol Tcp \ --direction inbound Create a new NSG rule to allow inbound UDP 500 (IKE) traffic to the CSR (can make it specific to an IP/Prefix) # az network nsg rule create \ --resource-group multicloud-rg \ --nsg-name csr-nsg-outside \ --name UDP-500 \ --priority 101 \ --source-address-prefixes 'Internet' \ --source-port-ranges '*' \ --destination-address-prefixes '*' \ --destination-port-ranges 500 \ --access Allow \ --protocol Udp \ --direction inbound 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 109

110 Azure CLI: Create NSG Rule & NICs Reference Create a new Network Security Group (NSG) to be used for the outside CSR interface # az network nsg rule create \ --resource-group multicloud-rg \ --nsg-name csr-nsg-outside \ --name UDP-4500 \ --priority 102 \ --source-address-prefixes 'Internet' \ --source-port-ranges '*' \ --destination-address-prefixes '*' \ --destination-port-ranges 4500 \ --access Allow \ --protocol Udp \ --direction inbound Create a new NIC to be used by the CSR s outside interface. Associate the NIC with the NSG, Subnet, Public IP & enable forwarding # az network nic create \ --resource-group multicloud-rg \ --name csr-nic-g1 \ --vnet-name mc-csr-vnet \ --subnet csr-outside \ --network-security-group csr-nsg-outside \ --ip-forwarding true \ --public-ip-address csr-azure-01-eip Create a new NIC to be used by the CSR s inside interface. Associate the NIC with the NSG, Subnet and enable forwarding # az network nic create \ --resource-group multicloud-rg \ --name csr-nic-g2 \ --vnet-name mc-csr-vnet \ --subnet csr-inside \ --ip-forwarding true 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 110

111 Azure CLI: Run a new CSR Instance Using Previous Parameters Create a CSR VM using a Azure Marketplace image. Associate the VM with two the NICs created earlier. # Note: The VM can be created with a large number of options to include SSH keys, image (BYOL, # of NICs), and size # az vm create \ --resource-group multicloud-rg \ --name csr-azure-01 \ --admin-username csr-azure \ --admin-password <PASSWORD> \ --authentication-type password \ --image cisco:cisco-csr-1000v:16_6: \ --nics csr-nic-g1 csr-nic-g2 \ --size Standard_D2_v Cisco and/or its affiliates. All rights reserved. Cisco Public 111

112 Connect to the Azure CSR Enable Interfaces Connect to the new Azure-hosted CSR and enable the GigabitEthernet 2 interface for DHCP # ssh csr-azure@40.xxx.xxx.x csr-azure-01#configure terminal Enter configuration commands, one per line. End with CNTL/Z. csr-azure-01(config)#interface gigabitethernet 2 csr-azure-01(config-if)#ip address dhcp csr-azure-01(config-if)#no shutdown Wait a few seconds and check to make sure that both interfaces on the CSR are up and have the correct IP addresses: csr-azure-01#show ip interface brief Interface IP-Address OK? Method Status Protocol GigabitEthernet YES DHCP up up GigabitEthernet YES DHCP up up VirtualPortGroup YES TFTP up up Note: This can all be automated (along with the DMVPN configs) by creating Azure Automation/Resource Manager 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 112

113 Azure Cisco CSR DMVPN Config Spoke crypto ikev2 proposal AES/GCM/256 encryption aes-gcm-256 prf sha512 group 19 crypto ikev2 policy AES/GCM/256 match fvrf any proposal AES/GCM/256 crypto ikev2 keyring DMVPN-KEYRING peer ANY address pre-shared-key <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile DMVPN-IKEv2-PROFILE description PSK Profile match identity remote address identity local address 40.xxx.xxx.x authentication remote pre-share authentication local pre-share keyring local DMVPN-KEYRING dpd 40 5 on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 mode transport crypto ipsec profile DMVPN-IPSEC-PROFILE set transform-set AES256/GCM/TRANSFORM set ikev2-profile DMVPN-IKEv2-PROFILE... Output summarized interface Tunnel0 description DMVPN ip address no ip redirects ip mtu 1400 ip nhrp authentication cisco123 ip nhrp network-id 100 ip nhrp nhs nbma 192.xxx.xxx.x multicast ip tcp adjust-mss 1360 ip ospf authentication-key 7 110A D5A5E57 ip ospf network point-to-multipoint ip ospf hello-interval 10 tunnel source GigabitEthernet1 tunnel mode gre multipoint tunnel key 100 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE interface GigabitEthernet1 description Internet ip address dhcp no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id network area 0 network area 3 ip route Cisco and/or its affiliates. All rights reserved. Cisco Public

114 On-Prem Cisco CSR DMVPN Config Hub - Nothing ever changes on the hub for each example crypto ikev2 proposal AES/GCM/256 encryption aes-gcm-256 prf sha512 group 19 crypto ikev2 policy AES/GCM/256 match fvrf any proposal AES/GCM/256 crypto ikev2 keyring DMVPN-KEYRING peer ANY address pre-shared-key <PSK_PASSWORD_GOES_HERE> crypto ikev2 profile DMVPN-IKEv2-PROFILE description PSK Profile match identity remote address identity local address 192.xxx.xxx.x authentication remote pre-share authentication local pre-share keyring local DMVPN-KEYRING dpd 40 5 on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec transform-set AES256/GCM/TRANSFORM esp-gcm 256 mode transport crypto ipsec profile DMVPN-IPSEC-PROFILE set transform-set AES256/GCM/TRANSFORM set ikev2-profile DMVPN-IKEv2-PROFILE... Output summarized interface Tunnel0 description DMVPN ip address no ip redirects ip mtu 1400 ip nhrp authentication cisco123 ip nhrp map multicast dynamic ip nhrp network-id 100 ip nhrp redirect ip tcp adjust-mss 1360 ip ospf authentication-key 7 121A0C D5679 ip ospf network point-to-multipoint ip ospf hello-interval 10 tunnel source GigabitEthernet1 tunnel mode gre multipoint tunnel key 100 tunnel protection ipsec profile DMVPN-IPSEC-PROFILE interface GigabitEthernet1 description Internet ip address 192.xxx.xxx.x no ip redirects no ip unreachables no ip proxy-arp negotiation auto router ospf 10 router-id network area 0 network area 0 ip route xxx.xxx.x 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 114

115 ... Output summarized Verify Routing and Reachability On the on-prem CSR check the route for the AWS VPC network /24 csr-mc-01#show ip route i O IA /24 [110/1001] via , 00:19:15, Tunnel0 On AWS check for the route for the on-prem network ( /24) csr-azure-01#show ip route i O /24 [110/1001] via , 00:17:57, Tunnel0 Connect to an AWS instances and ping to the on-prem private network shmcfarl@aztestvm:~$ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=2 ttl=62 time=3.99 ms 64 bytes from : icmp_seq=3 ttl=62 time=6.44 ms Inside Subnet /24 Cisco CSR1000v.4 Spoke CSR Tunnel: OSPF Hub CSR Tunnel: Cisco CSR1000v.1.30 Private Network /24 Hypervisor VM 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 115

116 Azure Marketplace-based Launch Walk-thru For Reference

117 Azure Marketplace/Resource Search 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 117

118 Azure Marketplace There are multiple CSR types to pick from 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 118

119 Azure Marketplace 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 119

120 Deployment Flow Cisco and/or its affiliates. All rights reserved. Cisco Public

121 Linking DMVPN Sites

122 DMVPN Enable Dynamic Multicloud Networking Cisco DMVPN VNet Network Spoke /24 Cisco CSR1000v Cisco CSR1000v Private Network VPC Network Spoke Hub / /24 Cisco CSR1000v BGP/OSPF On-Prem Private Cloud VPC Network DMVPN /24 Cisco CSR1000v Spoke 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 122

123 General Guidelines for DMVPN Between Clouds Reference Set the VPC routes for each site gcloud compute routes create inside-to-aws \ --network=inside-network \ --destination-range= /24 \ --next-hop-address= gcloud compute routes create inside-to-azure \ --network=inside-network \ --destination-range= /24 \ --next-hop-address= Set the firewall/security groups/network security groups for each site/protocol Create a very specific per-site route (AWS example allowing UDP500 from each cloud provider public IP) aws ec2 authorize-security-group-ingress --group-id sg-65c39b03 \ --ip-permissions '[{"IpProtocol": "17", "FromPort": 500, "ToPort": 500, "IpRanges": [{"CidrIp": "192.x.x.x/32"}, {"CidrIp": "35.x.x.x/32"}, {"CidrIp": "40.x.x.x/32"}]}] Alternatively, you can open it up (Azure example) az network nsg rule create \ --resource-group multicloud-rg \ --nsg-name csr-nsg-outside \ --name UDP-4500 \ --priority 102 \ --source-address-prefixes 'Internet' \ --source-port-ranges '*' \ --destination-address-prefixes '*' \ --destination-port-ranges 4500 \ --access Allow \ --protocol Udp \ --direction inbound 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

124 Routing Example All Sites... Output summarized For spoke-to-spoke direct routing with DMVPN/NHRP: ip nhrp redirect on the hubs ip nhrp shortcut on the spokes Hub On-Prem CSR csr-mc-01#show ip route ospf /8 is variably subnetted, 7 subnets, 2 masks O IA /24 [110/1001] via , 02:40:45, Tunnel0 O /32 [110/1000] via , 02:40:45, Tunnel0 O /32 [110/1000] via , 01:18:49, Tunnel0 O /32 [110/1000] via , 00:56:19, Tunnel0 O IA /24 [110/1001] via , 00:55:34, Tunnel /24 is subnetted, 1 subnets O IA [110/1001] via , 01:18:49, Tunnel0 Spoke Google Cloud Platform CSR csr1kv-gcp#show ip route ospf /8 is variably subnetted, 10 subnets, 2 masks O /32 [110/1000] via , 02:43:14, Tunnel0 O % /32 [110/2000] via , 01:21:14, Tunnel0 O % /32 [110/2000] via , 00:58:47, Tunnel0 O IA% /24 [110/2001] via , 00:58:00, Tunnel /24 is subnetted, 1 subnets O IA% [110/2001] via , 01:21:14, Tunnel0 O /24 [110/1001] via , 02:43:14, Tunnel0 Spoke Amazon Web Services CSR csr-aws-01#show ip route ospf /8 is variably subnetted, 7 subnets, 2 masks O IA% /24 [110/2001] via , 01:21:32, Tunnel0 O % /32 [110/2000] via , 01:21:32, Tunnel0 O /32 [110/1000] via , 01:21:32, Tunnel0 O % /32 [110/2000] via , 00:59:01, Tunnel0 O IA% /24 [110/2001] via , 00:58:14, Tunnel0 O /24 [110/1001] via , 01:21:32, Tunnel0 Spoke Azure CSR IA - OSPF inter area % - next hop override csr-azure-01#show ip route ospf /8 is variably subnetted, 10 subnets, 2 masks O IA% /24 [110/2001] via , 00:58:44, Tunnel0 O % /32 [110/2000] via , 00:58:44, Tunnel0 O /32 [110/1000] via , 00:58:44, Tunnel0 O % /32 [110/2000] via , 00:58:44, Tunnel /24 is subnetted, 1 subnets O IA% [110/2001] via , 00:58:44, Tunnel0 O /24 [110/1001] via , 00:58:44, Tunnel Cisco and/or its affiliates. All rights reserved. Cisco Public 124

125 NHRP Example Hub/Spoke Hub On-Prem CSR csr-mc-01#show ip nhrp /32 via Tunnel0 created 02:02:42, expire 00:08:17 Type: dynamic, Flags: registered used nhop NBMA address: 35.xxx.xxx.x (Claimed NBMA address: ) /32 via Tunnel0 created 00:42:52, expire 00:09:17 Type: dynamic, Flags: registered used nhop NBMA address: 52.xxx.xxx.x (Claimed NBMA address: ) /32 via Tunnel0 created 00:18:12, expire 00:08:26 Type: dynamic, Flags: registered used nhop NBMA address: 40.xxx.xxx.x (Claimed NBMA address: ) csr-mc-01#show ip nhrp multicast I/F NBMA address Tunnel0 35.xxx.xxx.x Flags: dynamic (Enabled) Tunnel0 52.xxx.xxx.x Flags: dynamic (Enabled) Tunnel0 40.xxx.xxx.x Flags: dynamic (Enabled) Spoke Azure VM traceroute traceroute to ( ), 30 hops max, 60 byte packets ( ) ms ms ms ( ) ms * ms Spoke Azure CSR csr-azure-01#show ip nhrp /24 via Tunnel0 created 00:06:26, expire 00:03:32 Type: dynamic, Flags: router rib nho NBMA address: 35.xxx.xxx.x (Claimed NBMA address: ) /32 via Tunnel0 created 00:06:26, expire 00:03:32 Type: dynamic, Flags: router nhop rib nho NBMA address: 35.xxx.xxx.x (Claimed NBMA address: ) /32 via Tunnel0 created 00:21:28, never expire Type: static, Flags: NBMA address: 192.xxx.xxx.x /32 via Tunnel0 created 00:12:29, expire 00:02:40 Type: dynamic, Flags: router nhop rib nho NBMA address: 52.xxx.xxx.x (Claimed NBMA address: ) /24 via Tunnel0 created 00:08:30, expire 00:03:33 Type: dynamic, Flags: router unique local NBMA address: (no-socket) /24 via Tunnel0 created 00:07:19, expire 00:02:40 Type: dynamic, Flags: router rib nho NBMA address: 52.xxx.xxx.x (Claimed NBMA address: ) csr-azure-01#show ip nhrp multicast I/F NBMA address Tunnel0 192.xxx.xxx.x 2018 Cisco Flags: and/or its affiliates. nhs All rights reserved. Cisco (Enabled) Public

126 Split-Tunneling/Routing Options

127 Split-Tunnel/Routing Options All three public cloud providers allow for either split-tunneling or forced/direct routing Split-tunneling: Public cloud resources (instances/vms, container clusters) will use the default VPC gateway for nonon-prem routes Public cloud resources will use the on-prem-specific routes advertised by the CSR Forced/Direct routing All public cloud resources will use the VPN connection as their default route for ALL traffic (forces traffic through the on-prem site) External/NAT Compute Engine VPC Subnetwork GW Routing Google Cloud VPN BGP 35.xxx.xxx.x 192.xxx.xxx.x Cisco CSR1000v Google Cloud Router 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 127

128 CSR High Availability

129 Public Cloud Provider CSR High-Availability Common challenge with all public cloud provider is that there is not true layer 2 support on a VPC subnet this prevents FHRPs from working properly Must setup a monitoring/tracking feature to watch for CSR interface/instance failure and adjust the VPC route table to point to 2 nd CSR inside interface AWS CSR High-Availability: s_chapter_0100.pdf Azure CSR High-Availability: Cisco and/or its affiliates. All rights reserved. Cisco Public 129

130 Automation Challenges

131 Automating the Multicloud Network Challenges: Different toolsets for different jobs (Ansible, Python, Bash scripts, Terraform, etc..) Different toolsets for different clouds (Heat for OpenStack, CloudFormation for AWS, Deployment Manager for GCP, Azure Automation) Different toolsets for different vendor products (Cisco NSO, CloudCenter, Prime, YANG development kit, etc..) There is no silver bullet - Start simple: Use what your team knows Perform a gap analysis on what you have against what you need Initially, automate the things that hurt a lot to do by hand and that change frequently I use free tools but that doesn t mean the process is free I use public cloud clients (gcloud, aws cli, azure cli) for services that don t change frequently or that need very unique/nonrepeatable configurations I use public cloud provider automation tools (GCP Deployment Manager) for in-project work (new instances with new networks for a GCP-only project) I use REST for things that change a lot When you want to stop pulling your hair out, move to something that can front-end each API that you need to talk to and treat the environment as a whole Cisco CloudCenter: Cisco and/or its affiliates. All rights reserved. Cisco Public 131

132 Amazon CloudFormation Template-based (JSON/YAML) Build a stack(s) from a template file Sometimes you need to run more than one stack (in order) to get what you need Race conditions: The wait condition does not universally work for all resources Common issues: External/Public IP assignment when using 2 or more interfaces - Can t assign the External IP until the instance is running and the interfaces are ready Assigning private routes to the inside interface of the CSR Routes won t install until CSR interfaces are available Use Outputs to export values that the next stack will need to build the next set of resources Example templates: Make your own changes to the files Deploy CsrStack-pw-cleaned first Deploy CsrEipRoute next Point to the name of the first stack 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 132

133 Google Cloud Platform Deployment Manager Configuration files (YAML), Templates (Python/Jinja2), Schema files (JSON) Sometimes you need to run more than one stack (in order) to get what you need Race conditions Use Outputs to export values that the next stack will need to build the next set of resources Example templates: Make your own changes to the files: <ZONE>, <PROJECT>, <IMAGE>, etc. Deploy the main stack: gcloud deployment-manager deployments create gcp-stack \ --config gcp_main_stack.yaml \ --automatic-rollback-on-error Deploy any custom routes that may be needed for other sites gcloud deployment-manager deployments create gcp-stack-route \ --config inside-private-routes.yaml \ --automatic-rollback-on-error 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 133

134 Microsoft Azure Automation/Resource Manager Runbooks (create graphically, PowerShell, Python) Read and select these carefully: Resource Manager: Example template: Cisco and/or its affiliates. All rights reserved. Cisco Public 134

135 Call APIs Directly Google Cloud Platform: Amazon Web Services: Microsoft Azure: Cisco and/or its affiliates. All rights reserved. Cisco Public 135

136 Google VPN Creating Google VPN, Router, IPsec, BGP via REST APIs

137 Google Cloud API Creating GCP Cloud VPN/Routers Assumptions/environment: Understand how to authenticate to GCP APIs: In this example, the Paw application was used to craft GET, POST and PATCH calls Some configurations have been sanitized for security purposes Have on-prem Cloud infrastructure deployed and a CSR/ASR configured (can be done after GCP side is deployed) In this example, the configuration will be deployed against the OpenStack use case discussed in the earlier slides In this example, the default network created by GCP will be used Note: gcloud has VERY long delays in commands if you have IPv6 enabled on your local machine set to link-local mode on your Mac 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 137

138 Reference Topology for GCP API Example OSPF<>BGP Redistribution Default Network /20 BGP AS Google Cloud VPN Google Cloud Router 35.yyy.yyy.y IPsec/IKEv2 Tunnel Mode yyy.yyy.y BGP AS Private Network /24 OSPF 10 Area 0 Cisco DMZ OpenStack Hosted Cisco CSR Routes this side should see: /24 Routes this side should see: / Cisco and/or its affiliates. All rights reserved. Cisco Public 138

139 GCP API (1) Create VPN GW and External IP... Output summarized POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/targetvpngateways HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 138 POST: Create VPN Gateway { "name": "csr-gcp-os-aio-gw", "network": "projects/<gcp_project_number>/global/networks/default", "region": "projects/<gcp_project_number>/regions/us-west1" } POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 29 POST: Create External IP Address { } "name": "gcp-to-os-dmz" GET /compute/v1/projects/<gcp_project_number>/regions/us-west1/addresses/gcp-to-os-dmz HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close GET: Get the External IP Address RESPONSE - SUMMARIZED: "name": "gcp-to-os-dmz", "description": "", "address": 35.yyy.yyy.y", "status": "RESERVED", 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

140 GCP API (2) Create Forwarding Rules POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingrules HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 257 { "name": "csr-gcp-os-aio-rule-esp", "IPProtocol": "ESP", "IPAddress": "35.yyy.yyy.y", "region": "projects/<gcp_project_number>/regions/us-west1", "target": " } POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingrules HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 278 { "name": "csr-gcp-os-aio-rule-udp500", "IPProtocol": "UDP", "IPAddress": "35.yyy.yyy.y", "region": "projects/<gcp_project_number>/regions/us-west1", "target": " "portrange": "500" } POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/forwardingrules HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 280 { "name": "csr-gcp-os-aio-rule-udp4500", "IPProtocol": "UDP", "IPAddress": "35.yyy.yyy.y", "region": "projects/<gcp_project_number>/regions/us-west1", "target": " "portrange": "4500" }... Output summarized POST: Create Forwarding rule for ESP POST: Create Forwarding rule for UDP 500 POST: Create Forwarding rule for UDP Cisco and/or its affiliates. All rights reserved. Cisco Public 140

141 GCP API (3) Create Cloud Router & BGP Session... Output summarized POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/routers HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 574 { "name": "csr-gcp-os-bgp-rtr", "bgp": { "asn": "65000" }, "interfaces": [ { "name": "if-csr-gcp-os-bgp-rtr-02", "linkedvpntunnel": " "iprange": " /30" } ], "bgppeers": [ { "name": "csr-gcp-os-bgp-peer", "interfacename": "if-csr-gcp-os-bgp-rtr-02", "ipaddress": " ", "peeripaddress": " ", "peerasn": "65003" } ], "region": "projects/<gcp_project_number>/regions/us-west1", "network": " } POST: Create Cloud Router, BGP session and link to the Cloud VPN tunnel 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 141

142 ... Output summarized GCP API (5) Create Cloud VPN Tunnel POST /compute/v1/projects/<gcp_project_number>/regions/us-west1/vpntunnels HTTP/1.1 Authorization: Bearer XXXX Content-Type: application/json; charset=utf-8 Host: Connection: close Content-Length: 417 { "name": "csr-gcp-os-aio-gw-tunnel-1", "sharedsecret": " <pre-shared-password-goes-here> ", "router": " "peerip": "192.yyy.yyy.y", "region": "projects/<gcp_project_number>/regions/us-west1", "ikeversion": "2", "targetvpngateway": " } POST: Create a Cloud VPN tunnel and associated it with the Cloud router 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 142

143 Demo

144 Summary Public cloud native IPsec VPN support is good, but it is always point-to-point, does not have consistent support for NAT and lacks network-rich features DMVPN with Cisco CSR, ASR, ISR can greatly improve the deployment, HA, scalability and operations of the VPN connections Multicloud between multiple public cloud providers and on-prem look like distinctly separate hybrid cloud deployments but.. You have to take into consideration: Team knowledge of public cloud operations, tools, automation Cross cloud tools and automation Diversity of network designs, protocols, security Multi-region designs Availability zones within and across providers 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 144

145 Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot/# 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

146 Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at Cisco and/or its affiliates. All rights reserved. Cisco Public

147 Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 147

148 Thank you

149

150 Reference

151 Application Deployment

152 GKE, Cloud VPN, Cloud Router and an On-Prem CSR Deployment with Dynamic Routing (IP Alias)

153 Google Container Engine (GKE) Dynamic Routing Prior to the IP alias feature, GKE clusters did not advertise their IP ranges via the GCP Cloud Router (BGP) service: IP alias and self-directed alias ranges, cluster IP ranges and service IP ranges can all be enabled via REST, gcloud and the GKE console # gcloud beta container clusters create gke-cls-istio \ > --enable-ip-alias \ > --create-subnetwork name=gke-istio-subnetwork 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 153

154 GKE Dynamic Routing with On-Prem CSR Google Container Cluster (GKE) External/ NAT Pods cbr0 eth / Pods cbr0 eth / Pods cbr0 eth / VPC Subnetwork GW Routing Google Cloud VPN BGP 35.xxx.xxx.x 192.xxx.xxx.x Cisco CSR1000v.1 Private Network / Default Network: - Subnetwork: - Nodes: /22 - Container Range: /14 - Services Range: /20 Google Cloud Router Hypervisor 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 154

155 Google Container Engine - Setup Google Container Cluster (GKE) Create a basic GKE cluster with IP alias enabled # gcloud beta container clusters create gke-cls-istio \ > --enable-ip-alias \ > --create-subnetwork name=gke-istio-subnetwork Get a list of the nodes # kubectl get nodes NAME STATUS AGE VERSION gke-gke-cls-istio-default-pool-6724d65b-6lsc Ready 2m v1.7.6 gke-gke-cls-istio-default-pool-6724d65b-x04p Ready 2m v1.7.6 gke-gke-cls-istio-default-pool-6724d65b-zgdq Ready 2m v1.7.6 Check the IP ranges of the new subnetwork gke-istio-subnetwork # gcloud compute networks subnets describe gke-istio-subnetwork grep ipcidrrange ipcidrrange: /22 - ipcidrrange: /14 - ipcidrrange: / Pods cbr0 eth / Pods cbr0 eth / Pods cbr0 eth /24 Default Network: - Subnetwork: - Nodes: /22 - Container Range: /14 - Services Range: / Cisco and/or its affiliates. All rights reserved. Cisco Public 155

156 Google Container Engine Node/Pod IP Verification NAME STATUS AGE VERSION gke-gke-cls-istio-default-pool-6724d65b-6lsc Ready 2m v1.7.6 gke-gke-cls-istio-default-pool-6724d65b-x04p Ready 2m v1.7.6 gke-gke-cls-istio-default-pool-6724d65b-zgdq Ready 2m v1.7.6 Using the node list from above, check the IPs assignments of each node # kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-zgdq grep 'InternalIP\ PodCIDR' InternalIP: PodCIDR: /24 # kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-6lsc grep 'InternalIP\ PodCIDR' InternalIP: PodCIDR: /24 # kubectl describe nodes gke-gke-cls-istio-default-pool-6724d65b-x04p grep 'InternalIP\ PodCIDR' InternalIP: PodCIDR: /24 Google Container Cluster (GKE) Pods cbr0 eth / Pods cbr0 eth / Pods cbr0 eth / Cisco and/or its affiliates. All rights reserved. Cisco Public 156

157 ... Output summarized GKE/GCP and On-Prem CSR Dynamic Routing Get the advertised route list from the GCP Cloud Router # gcloud compute routers get-status csr-gcp-vm-bgp-rtr... result:... bgppeerstatus: - advertisedroutes: - destrange: /20 kind: compute#route nexthopip: priority: destrange: /14 kind: compute#route nexthopip: priority: destrange: /22 kind: compute#route nexthopip: priority: 100 Check the BGP routes on the on-prem CSR csr-gcp-01#show ip route bgp... B /22 [20/100] via , 00:00:04 B /20 [20/100] via , 00:00:04 B /14 [20/100] via , 00:00: BGP Google Cloud Router Google Cloud VPN Cisco CSR 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 157

158 GKE and CSR Routing/Access Verification From a VM at the on-prem network ( /24), ping a GKE nodes IP and the cbr0 interface on that node [root@k8s-m-01 ~]# ip a... 2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:50:56:bc:4b:91 brd ff:ff:ff:ff:ff:ff inet /24 brd scope global ens192 valid_lft forever preferred_lft forever inet6 fe80::50de:b58f:8dc8:2fd5/64 scope link valid_lft forever preferred_lft forever Google Container Cluster (GKE) [root@k8s-m-01 ~]# ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=63 time=25.4 ms 64 bytes from : icmp_seq=2 ttl=63 time=24.3 ms [root@k8s-m-01 ~]# ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=63 time=25.2 ms 64 bytes from : icmp_seq=2 ttl=63 time=24.1 ms Pods cbr0 eth / Pods cbr0 eth / Pods cbr0 eth / Cisco and/or its affiliates. All rights reserved. Cisco Public 158

159 GKE Pod Routing/Access Verification Deploy an nginx pod # kubectl run my-nginx --image=nginx --port=80 deployment "my-nginx" created # kubectl get pods NAME READY STATUS RESTARTS AGE my-nginx jbjl 1/1 Running 0 14s Find the IP addres of the pod # kubectl describe pods my-nginx jbjl grep IP: IP: Ping the IP address of the pod from the on-prem VM [root@k8s-m-01 ~]# ping PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=62 time=24.9 ms 64 bytes from : icmp_seq=2 ttl=62 time=24.4 ms curl the nginx pod [root@k8s-m-01 ~]# curl -o /dev/null -s -w "%{http_code}\n" Cisco and/or its affiliates. All rights reserved. Cisco Public 159

160 Google Container Engine Deploy Pods Deploy NGINX as a test # kubectl run my-nginx --image=nginx --replicas=3 --port=80 deployment "my-nginx" created Check to make sure the pods are running # kubectl get pods NAME READY STATUS RESTARTS AGE my-nginx x8mp 1/1 Running 0 6s my-nginx rt9sp 1/1 Running 0 6s my-nginx vhq6f 1/1 Running 0 6s Get the IPv4 address for each pod # kubectl describe pods my-nginx x8mp grep IP: IP: # kubectl describe pods my-nginx rt9sp grep IP: IP: # kubectl describe pods my-nginx vhq6f grep IP: IP: Cisco and/or its affiliates. All rights reserved. Cisco Public 160

161 Google VPN <> OpenStack Hosted Cisco CSR BGP Routing