Cisco Multicloud Portfolio: Cloud Connect

Transcription

1 Design and Deployment Guide Cisco Multicloud Portfolio: Cloud Connect Design and Deployment Guide for Private Data Center to AWS VPC October Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 105

2 Contents Executive summary... 3 Cisco Multicloud Portfolio: overview... 3 Cloud Connect: overview... 4 Cloud Connect: Use cases... 4 Cloud Connect: Benefits... 5 Technology overview... 5 Solution design... 6 Design principles... 6 Solution deployment... 8 Configure the native IPsec VPN connections in the AWS VPC... 9 Procedure 1: Create two AWS customer gateways... 9 Procedure 2: Create AWS virtual private gateways (VGWs) Procedure 3: Create the AWS VPN connections Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS VPC Procedure 1: Configure a BFD template Procedure 2: Configure IKEv Procedure: 3 Configure IPsec Procedure 4: Configure tunnel interfaces Procedure 5: Configure routing Procedure 5: Verify that the VPN connections are working properly Configure the IPsec VPN connections on the Cisco CSR 1000V Routers in the AWS Transit VPC Procedure 1: Configure a VRF for the tunnel interface to the private network Procedure 2: Configure a BFD template Procedure 3: Configure IKEv Procedure 4: Configure IPsec Procedure 5: Configure tunnel interfaces Procedure 6: Configure dynamic routing Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS Transit VPC Procedure 1: Configure IKEv Procedure 2: Configure IPsec Procedure 3: Configure tunnel interfaces Procedure 4: Configure routing Procedure 5: Verify that the VPN connections are working properly Configure visibility into application flows Procedure 1: Configure AVC Procedure 2: View AVC data via the CLI Optional Procedure 3: View AVC data through the web interface Procedure 3: Configure flow monitoring and data export Procedure 4. View flow-monitoring information via the LiveNX client Procedure 5. View flow-monitoring information via the LiveAction web interface Appendix A: Design considerations Positioning of the VPN gateway in the private network Network Address Translation (NAT) High availability Symmetric routing for AVC Access control and path isolation Appendix B: Cisco ASR 1000 Series Router configuration Appendix C: AWS VPN configuration example Appendix D: Flow configuration example Additional resources Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 105

3 Executive summary This guide focuses on how to deploy an Amazon Web Services (AWS) virtual private cloud (VPC) with VPN connectivity back to a private network using a redundant pair of Cisco ASR 1000 Series Aggregation Services Routers at the corporate site. The design uses an AWS managed VPN connection through a virtual private gateway (VGW) attached to the VPC. Advanced features such as Application Visibility & Control (AVC) / Next- Generation Network-Based Application Recognition (NBAR2) and Flexible NetFlow data export are discussed for traffic and application-level visibility at the ASR 1000 Series routers within the private network. The audience for this document includes network design engineers, network operations personnel, and security operations personnel who wish to implement secure VPN connectivity from their private data centers to an AWS VPC. This guide also discusses the connection of an AWS Transit VPC to a private network using IPSec VPN connectivity. The AWS Transit VPC design is documented in the following link: Cisco Multicloud Portfolio: overview In a multicloud world, growing complexity is driving a cloud gap between what your customers require and what your people, processes, and tools can support. With the Cisco Multicloud Portfolio, we make it simple: simple to connect, simple to protect, and simple to consume. The Cisco Multicloud Portfolio is a set of essential products, software, and services supported with simplified ordering and design deployment guides to help you when it comes to multicloud adoption. The Cisco Multicloud Portfolio consists of four component portfolios (Figure 1): Cloud Advisory: Helps you design, plan, accelerate, and remove risk from your multicloud migration. Cloud Connect: Securely extends your private networks into public clouds and helps ensure the appropriate application experience. Cloud Protect: Protects your multicloud identities, direct-to-cloud connectivity, data, and applications, including software as a service (SaaS), and detects infrastructure and application threats on premises and in public clouds. Cloud Consume: Helps you deploy, monitor, and optimize applications in multicloud and container environments Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 105

4 Figure 1. Multicloud Portfolio: Cloud Advisory, Cloud Connect, Cloud Protect, and Cloud Consume Cloud Connect: overview Cloud Connect consists of essential products that help securely extend your private networks including the data center, branches, and campuses to public clouds and to help ensure that the application experience is optimal: Cisco Cloud Services Router (CSR) 1000V Series Cisco vedge with Cisco Umbrella For detailed use cases, see the section about Cloud Connect on the portfolio s solution page at Cloud Connect: Use cases Cloud Connect delivers value in the following use cases: Securely extending a private network to single or multiple public cloud environments. Includes multiple clouds (for example, multiple AWS and Azure), multiple regions in a cloud, or multiple VPCs in a cloud; VPN; multicloud and multi-vpc connectivity; scaling; and performance optimization- transit VPC. Also supports extending data centers into the cloud and enabling direct branch-to-cloud connectivity (when a branch has a Cisco 4000 Series Integrated Services Router [ISR] and wants to connect the clouds or a branch has vedge and requires a software-defined WAN [SD-WAN] extension to the cloud). Optimizing data center and branch connectivity performance to cloud infrastructure as a Service (IaaS) and SaaS. Includes best path to destination (SD-WAN), cloud segmentation, monitoring to assure best performance, visibility into traffic going to applications, and traffic shaping/quality of Service (QoS). Also supports extending data centers into the cloud and enabling direct branch-to-cloud connectivity (when a branch has a 4000 Series ISR and wants to connect the clouds or a branch has vedge and requires an SD- WAN extension to the cloud) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 105

5 Securing access to the Internet and SaaS from the branch. Includes connecting and protecting branch office users directly to the multicloud environment using Direct Internet Access (DIA), SD-WAN (vedge), and secure Internet gateways (Cisco Umbrella). Cloud Connect: Benefits Cloud Connect benefits include the ability to: Extend a private network to a multicloud environment while leveraging existing investments Apply consistent security policies across a private and public cloud footprint Enhance and secure the app experience on a cloud network by enabling visibility and path selection and optimization Centralize management in a manner that is intuitive, fast, and easy to design, provision, and apply policies across the entire network Achieve faster and more simple adoption of cloud Improve TCO Access a richer networking security feature set and higher performance Improve ease of use through consistency of management tools for both on-premises and cloud Simplify implementation through increased visibility into the public cloud network Technology overview In a multicloud world, IT managers are quickly realizing the benefits of cloud computing services such as infrastructure as a service (IaaS). IaaS providers, such as AWS, allow organizations to more rapidly and costeffectively prototype new applications. Instead of procuring, installing, and managing hardware which could takes months to accomplish you can easily use the on-demand and scalable compute services in AWS. This allows you to focus your resources on applications rather than on managing the data center and physical infrastructure. With the use of IaaS, expenses shift from fixed costs for hardware, software, and data center infrastructure to variable costs based on the usage of compute resources and the amount of data transferred between the private data center and the IaaS provider. Thus, you must also be able to monitor the usage of such resources for cost tracking and/or internal billing purposes. To realize the full benefits that IaaS offers, secure, highly available connectivity to the AWS VPC must be established. The use of VPN connectivity to an AWS VPC provides a cost-effective solution for providing that connectivity. IP Security (IPsec) encryption of traffic, using strong cryptographic cipher suites, provides secure connectivity between your private data center and the AWS VPC. High availability is accomplished through the deployment of redundant VPN gateways. This deployment guide uses a pair of Cisco ASR 1002-X Routers running Cisco IOS XE Software Release code licensed with the Advanced Enterprise feature set to provide IPsec VPN and routing services to connect to AWS Virtual Private Gateways (VGWs). Configuration of the ASR 1002-X routers is through the Command-Line Interface (CLI) for this version of this guide. Cisco ASR 1002-X Routers support AVC/NBAR2 and Flexible NetFlow. The application-flow information provided by the Cisco ASR 1002-X Routers can be leveraged by a flow collector provided by a Cisco partner, LiveAction ( through the LiveNX network performance and analytics platform. LiveNX Enterprise Edition, Version 7.0.1, was used for this guide. LiveNX provides a centralized console for visualizing application flows through the ASR 1002-X routers Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 105

6 Finally, the AWS web-based console ( was used for configuring the AWS VPN connections. Solution design Design principles The procedures in this section provide examples for most settings. The actual settings and values that you use are determined by your current network configuration. Figure 2 shows the overall high-level design for this deployment guide. A redundant set of Cisco ASR 1000 Series Routers serve as dedicated VPN gateways for the private network. Figure 2. Private network to AWS VPC design The top part of the figure shows the connectivity from the private network to an AWS Transit VPC, using a redundant pair of Cisco CSR 1000V Routers deployed in the Transit VPC. Each Cisco ASR 1000 Series Router in the private network establishes a single IPsec VPN tunnel to one of the Cisco CSR 1000V routers deployed in the Transit VPC Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 105

7 The bottom part of the figure shows connectivity from the private network to a VPC using the native IPsec VPN gateway of AWS. Each Cisco ASR 1000 Series Router in the private network establishes two IPsec VPN tunnels to the public IP addresses of the AWS VGWs in the VPC, for a total of four IPsec VPN tunnels in this design. (For more information, see Tables 1 and 2.) Each of the Cisco CSR 1000V Routers in the Transit VPC also establishes two IPsec VPN tunnels to the public IP addresses of the AWS VGWs in the spoke VPCs as shown in the top right section of the figure above. For more information, refer to the AWS Transit VPC with Cisco Cloud Services Router 1000V deployment guide at the following URL: Note: You may not necessarily deploy both native IPsec VPN connectivity directly to an AWS VPC and IPsec VPN connectivity through an AWS Transit VPC design simultaneously. However, this design demonstrates full routing from the spoke VPCs connected through the Transit VPC, all the way out to the VPC connected through the native IPsec VPN connection directly from the private network. This scenario may also be temporarily deployed during a transition to a Transit VPC design, because the AWS deployment scales beyond one or two VPCs connected directly through AWS-native IPsec VPN connections. Table 1. AWS VPC native IPsec VPN information Description Name IP address First AWS customer gateway Multi-Cloud_ASR1K Second AWS customer gateway Multi-Cloud_ASR1K AWS virtual private gateway (VGW) Multi-Cloud_VGW AWS first VPN connection outside IP addresses Multi-Cloud_VPN and AWS first VPN connection tunnel addresses Multi-Cloud_VPN /30 and /30 AWS second VPN connection outside IP address Multi-Cloud_VPN and AWS second VPN connection tunnel addresses Multi-Cloud_VPN /30 and /30 AWS BGP ASN AWS VPC Multi-Cloud_VPC /16 AWS VPC VPN-only subnet Multi-Cloud_Subnet /24 Table 2. AWS Transit VPC VPN information Description Name IP address Transit VPC CSR1 public IP address TVPC-CSR Transit VPC CSR2 public IP address TVPC-CSR TVPC-CSR1 to ASR1K-1 IPsec VPN connection tunnel addresses TVPC-CSR2 to ASR1K-2 IPsec VPN connection tunnel addresses TVPC-CSR1 Tunnel /30 TVPC-CSR2 Tunnel /30 Transit VPC BGP ASN Table 3. Private network VPN information Description Name IP address Public IP address of first VPN router Public IP address of second VPN router Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 105

8 Description Name IP address Front-side interface of first VPN router (1:1 NAT to ) Back-side interface of first VPN router VPC tunnel addresses of first VPN router ASR1K-1 GigabitEthernet0/0/0 ASR1K-1 GigabitEthernet0/0/2 ASR1K-1 Tunnel1 and Tunnel / /29 Transit VPC tunnel address of the first VPN router ASR 1K-1 Tunnel /30 Front-side interface of second VPN router (1:1 NAT to ) Back-side interface of second VPN router VPC tunnel addresses of second VPN router ASR1K-2 GigabitEthernet0/0/0 ASR1K-2 GigabitEthernet0/0/2 ASR1K-2 Tunnel1 and Tunnel /30 Tunnel1 and /30 Tunnel / /29 Transit VPC tunnel address of the second VPC router ASR1K-2 Tunnel /30 Outside interface of ASAv data center firewall active/standby pair Inside interface of ASAv data center firewall active/standby pair Nexus 5000 Series distribution-layer switch #1 SVI for VLAN 20 Nexus 5000 Series distribution-layer switch #1 SVI for VLAN 70 Nexus 5000 Series distribution-layer switch #2 SVI for VLAN 20 Nexus 5000 Series distribution-layer switch #2 SVI for VLAN 70 Nexus 5000 Series distribution-layer switch shared HSRP IP address ASAv-1 GigabitEthernet0/ /29 ASAv-1 GigabitEthernet0/ /29 N5K-1 VLAN /29 N5K-1 VLAN /29 N5K-2 VLAN /29 N5K-2 VLAN /29 HSRP Group /29 Private network BGP ASN N/A Private network EIGRP ASN 100 NA /30 Tunnel1 and /30 Tunnel2 Solution deployment The overall processes for this deployment guide are split into three sections. The first set of processes configures IPsec VPN connectivity between the private network and a VPC, using the native IPsec VPN connectivity of AWS. The processes are as follows: Configure the native IPsec VPN connections in the AWS VPC Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS VPC The second set of processes configures the IPsec VPN connectivity between the private network and an AWS Transit VPC, using the Cisco CSR 1000V Routers already deployed in the Transit VPC. Only deploy these processes if you are connecting a Transit VPC to the private network. The processes are as follows: Configure the IPsec VPN connections on the Cisco CSR 1000V Routers in the AWS Transit VPC Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS Transit VPC The final process is optional. The below process provides visibility into application flows to and from the private network and the AWS VPCs. Configure visibility into application flows The below figure provides guidelines on how to read the commands given in this guide Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 105

9 Figure 3. How to read the commands in this guide Configure the native IPsec VPN connections in the AWS VPC Use this process to create the following: The AWS customer gateways to which the AWS virtual private gateway will establish IPsec VPN connections The AWS virtual private gateway (VGW) The VPN connections that connect the AWS virtual private gateway with the AWS customer gateways This process assumes that the AWS VPC and a subnet (VPN-only subnet) within that VPC have already been created. Procedure 1: Create two AWS customer gateways In order to support a redundant pair of Cisco ASR 1000 Series Routers in your private data center, you need to create two AWS customer gateways. Each AWS customer gateway must have a unique public IP address (an IP address routable over the Internet). Each AWS customer gateway is paired with a single AWS VPN connection. From the AWS VPC Dashboard, select Customer Gateways from the panel on the left side of the screen. Click the Create Customer Gateway button at the top of the screen to bring up the Create Customer Gateway screen (see Figure 4). Figure 4. Create Customer Gateway screen 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 105

10 Step 1. Name the two AWS customer gateways. An AWS customer gateway is the logical construct configured within AWS that represents the VPN router at your private data center to which the AWS VGW will establish an IPsec VPN connection. Provide a name for each AWS customer gateway you are creating. In this example, the first AWS customer gateway is named Multi- Cloud_ASR1K-1 and the second AWS customer gateway is named Multi-Cloud_ASR1K-2. Step 2. Select the type of routing. Routing between the AWS VPC and your private network can be done via either static routes or dynamic routing using Border Gateway Protocol (BGP). For this deployment guide, dynamic routing is used. Select the Dynamic radio button next to Routing. Step 3. Configure the BGP ASN of the private network. This guide uses dynamic routing, so BGP peering between two autonomous systems (AWS and the private network) must be established. You must configure a BGP autonomous system number (ASN) for your private network. AWS will provide a default ASN of if you do not configure your ASN. This deployment guide uses the default ASN of provided by AWS for the private network. Step 4. Configure the IP addresses of the AWS customer gateways. For this guide, the IP addresses of the Cisco ASR 1000 Series Routers that function as the AWS customer gateways are both be-hind a firewall that is performing static 1:1 NAT. Therefore the external NAT-translated IP addresses (public IP addresses) of the front-side interfaces (GigabitEthernet0/0/0) of the routers are used when creating the AWS customer gateways. These addresses are provided in Table 2. Step 5. Create an AWS customer gateway. Click the Create Customer Gateway button when you have provided the information for each of the four steps above. This will bring up a screen indicating that a customer gateway was successfully created. Close this screen and you will be taken back to the main Customer Gateways screen. It should show the customer gateway you just created, and its state should be available (see Figure 5). Figure 5. Customer Gateways screen with AWS customer gateway 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 105

11 Step 6. Create a second AWS customer gateway. Repeat steps 1 through 5 to create a second AWS customer gateway. When completed, the main screen should show the availability of both gateways (see Figure 5). Procedure 2: Create AWS virtual private gateways (VGWs) Each AWS VPC supports a single AWS VGW. From the AWS VPC dashboard, select Virtual Private Gateways from the left panel. Click the Create Virtual Private Gateway button at the top of the screen to bring up the Create Virtual Private Gateway screen (see Figure 6). Figure 6. Creating an AWS virtual private gateway Step 1. Name the virtual private gateway The virtual private gateway is a logical construct that represents the VPN router at AWS. It will establish IPsec VPN connections to your private network. Provide a name for the virtual private gateway. In this guide, the virtual private gateway is named Multi-Cloud_VGW. Step 2. Configure the BGP ASN of the AWS side of the VPN connection. Since this deployment guide uses dynamic routing, BGP peering between two autonomous systems (AWS and the private network) must be established. You must configure the BGP ASN for the AWS side of the VPN connection. AWS will provide an ASN of 7224 if you select the Amazon default ASN. Alternatively, you can select Custom ASN and provide an ASN. This deployment guide uses a BGP ASN of for the VPC. Step 3. Create the AWS virtual private gateway. Click the Create Virtual Private Gateway button when you have provided the information for each of the two steps above. This will bring up a screen indicating that the AWS virtual private gateway was successfully created. Close this screen and you will be taken back to the main Virtual Private Gateways screen. It should show the AWS virtual private gateway you just created (see Figure 7). Note that, unlike Figure 7, the state of your AWS virtual private gateway will be "detached" because it is not attached to any VPC when it is first created Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 105

12 Figure 7. Virtual Private Gateways screen Step 4. Attach the AWS virtual private gateway to a VPC. From the main Virtual Private Gateways screen, click Actions and select Attach to VPC. This will bring up another screen with a drop-down menu of available VPCs. Select the VPC to which you wish to attach the virtual private gateway, and click the Yes, Attach button at the bottom right of the screen (see Figure 8). Figure 8. Attaching the virtual private gateway to a VPC This will take you back to the main Virtual Private Gateways screen. After a few moments, the state of the AWS VGW you created will be attached and will show the name of the VPC (Multi-Cloud_VPC for this deployment guide) to which it is attached (see Figure 7). Procedure 3: Create the AWS VPN connections To support a redundant pair of Cisco ASR 1000 Series Routers in your private network, two AWS VPN connections need to be created. Each AWS VPN connection is paired to a single AWS customer gateway. An AWS VPN connection is a logical construct. In each AWS VPN connection, a redundant pair of IPsec connections is established to the ASR 1000 Series router in your private network. For this guide, two AWS VPN connections are created. Therefore, a total of four IPsec connections are created two to each ASR 1000 Series router in your private network Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 105

13 From the AWS VPC Dashboard, select VPN Connections from the panel on the left side of the screen. Click the Create VPN Connections button at the top of the screen to bring up the Create VPN Connections screen (see Figure 9). Figure 9. Creating a VPN connection Step 1. Name the VPN connections. Provide a name for each VPN connection. In this deployment guide, the first VPN connection is named Multi- Cloud_VPN1 and the second VPN connection is named Multi-Cloud_VPN2. Step 2. Select the AWS virtual private gateway for the VPN connections. From the drop-down menu, select the virtual private gateway you just created to associate with the VPN connections. Since there is only one virtual private gateway (VGW), both AWS VPN connections will be associated with the same VGW Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 105

14 Step 3. Select the customer gateway for the VPN connections. An AWS VPN connection is actually a redundant pair of IPsec VPN connections between the AWS virtual private gateway and a Cisco ASR 1000 Series Router in the private network (represented logically by an AWS customer gateway). Since the customer gateways have already been created within AWS, select the Existing radio button. Select the first customer gateway created in the procedure above from the Customer Gateway ID drop-down menu for the first VPN connection. For the second VPN connection, select the second customer gateway created in Procedure 1, above. Step 4. Select the routing method. The design followed in this deployment guide uses BGP routing between the AWS VPC and the private network. Select the Dynamic (which requires BGP) radio button adjacent to Routing Options. Step 5. Select tunnel options. The VPN connection between the AWS virtual private gateway and the Cisco ASR 1000 Series Routers uses IPsec virtual tunnel interfaces (VTIs). Tunnel Options allow you to select the IP addresses used for the tunnel interfaces. Since two IPsec tunnels are created with each VPN connection, the IP address ranges for Tunnels 1 and 2 can be specified. If an IPv4 Classless Inter-Domain Routing (CIDR) subnet is not specified, AWS will automatically assign subnets for each tunnel. The IP address ranges for Tunnels 1 and 2 are generated by AWS for both VPN connections in this deployment guide. VPN connections between AWS virtual private gateways and Cisco ASR 1000 Series Routers use preshared keys for Internet Security Association and Key Management Protocol (ISAKMP) authentication of the VPN peers. Tunnel Options allow you to select the preshared key for each of the tunnel interfaces, each of which corresponds to an ISAKMP peer configured on the ASR 1000 Series routers. If preshared keys are not specified, AWS will automatically assign them for each tunnel. The preshared keys are generated by AWS for both VPN connections in this deployment guide. Step 6. Create the VPN connection. Click the Create VPN Connection button when you have provided the information for each of the five steps above. This will bring up a screen indicating that the AWS VPN connection was successfully created. Close this screen and you will be taken back to the main VPN Connections screen. It should show the AWS VPN Connection you just created (see Figure 10). Note that the initial state of the AWS VPN connection just created may be pending. After a few moments, the state should be available (see Figure 10) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 105

15 Figure 10. VPN Connections screen If the Cisco ASR 1000 Series Routers at the private network have not been configured yet, the Tunnel Details tab may show the status of the two IPsec connections as being DOWN. However, as soon as the ASR 1000 Series routers are configured, both VPN tunnels should show a status of UP (see Figure 10). This provides a quick means of troubleshooting the status of the VPN tunnels from AWS. Step 7. Download the configuration information for the customer gateway. AWS provides an example of how the customer gateway should be configured in order to establish the two IPsec connections between the AWS virtual private gateway and the device used as the VPN gateway at your private network. From the main VPN Connections screen, click the Download Configuration button. Figure 11. Selecting the device for configuration download 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 105

16 The design followed in this deployment guide uses Cisco ASR 1000 Series Routers at the private network. Select Cisco Systems, Inc. as the Vendor, Cisco ASR 1000 as the Platform, and IOS as the Software (see Figure 11). Click the Download button to download the configuration information to your PC. You can close the pop-up window once the download is complete. You will use this information when configuring the ASR 1000 Series routers. An example of the configuration information downloaded from AWS is shown in Appendix B. Step 8. Create the second VPN connection. Repeat steps 1 through 7 above to create a second AWS VPN connection. Use the second AWS customer gateway created in Step 6 of Procedure 1 above for this AWS VPN connection. Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS VPC Use this process to configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers in your private network. These connect to the native IPsec VPN connections in the AWS VPC that you created in the previous process. The following procedures use the configurations downloaded for each of the two AWS VPN connections you have already created. This process configures the following: BFD for more rapid convergence IKEv1 IPsec Tunnel interfaces for the VPN connections Dynamic routing The final procedure verifies that the IPsec VPN connections are up and dynamic routing has been established. Procedure 1: Configure a BFD template BFD can be used to more rapidly detect the loss of connectivity between devices and have routing protocols converge faster. Step 1. Configure a BFD template on both of the Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2) in the private network: bfd-template single-hop bfdtemplate1 interval min-tx 1000 min-rx 1000 multiplier 3 The minimum transmit and receive interval is set for 1000 milliseconds (1 second) in the configuration above. A loss of three consecutive messages (3 seconds) causes the BFD peer to be declared down. The BFD transmitand-receive intervals are not aggressive in this deployment guide. This is done intentionally to minimize the possibility of route flapping. Tune the intervals and multiplier based on the requirements for convergence within your network and the capabilities of your network devices Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 105

17 Step 2. Apply the BFD template to the required interface on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2): interface GigabitEthernet0/0/2 bfd template bfdtemplate1 BFD is not supported over AWS-native IPsec VPN connections; therefore, BFD is not implemented across the tunnel interfaces. For this deployment guide, BFD is implemented for the ibgp peering that connects both ASR 1000 Series routers (ASR1K-1 and ASR1K-2). The ibgp peering will be established across the GigabitEthernet0/0/2 interface. You will still need to configure BGP to use BFD to determine when the peer is down. This will be done in an upcoming procedure. Note: BFD can also be implemented between Enhanced Interior Gateway Routing Protocol (EIGRP) peers within the private network. Since the primary focus of this deployment guide is the connectivity out to AWS, this configuration is outside the guide s scope. Procedure 2: Configure IKEv1 An IKEv1 policy defines a combination of security parameters to be used during Internet Key Exchange (IKE) negotiation. Policy numbers must be unique within the router configuration. The ISAKMP security association (SA) is configured to propose a timeout of 28,800 seconds (8 hours). We recommend that, for increased security, you configure the following in the IKEv1 policies: Use Advanced Encryption Standard (AES) encryption with a 256-bit key (AES-256) rather than the 128-bit key (AES-128) shown in the configuration example downloaded from AWS in the previous process. Use the Secure Hash Algorithm 2 (SHA-2) hashing algorithm with a 256-bit digest, rather than the SHA-1 algorithm show in the configuration example downloaded from AWS in the previous process. This is also known as SHA-256. Use Diffie-Hellman group 14 and above, rather than group 2, shown in the configuration example downloaded from AWS in the previous process. This design document configures Diffie-Hellman group 24. All of the above security parameters are currently supported by the AWS virtual private gateway. The AWS VGW supports only IKEv1 and preshared keys for authentication of peers. If IKEv2 and/or a different authentication method such as RSA signatures or RSA-encrypted nonces is required, you will need to deploy a VPN appliance such as the Cisco CSR 1000V Router. This guide covers only the deployment of the AWS VGW at the VPC, configured in the previous process. Step 1. Configure the IKEv1 policy for the IPsec VPN tunnels on both Cisco ASR 1000 Series routers (ASR1K-1 and ASR1K-2): crypto isakmp policy 200 encr aes 256 hash sha256 authentication pre-share group 24 lifetime Note: IKEv1 policies are configured as ISAKMP policies within Cisco IOS and IOS-XE devices Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 105

18 Step 2. Configure the IKE preshared keys on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2). IKE preshared keys are used to authenticate the ASR 1000 Series router with the AWS VGW during the establishment of the ISAKMP SA. The configuration uses keyrings to store the preshared keys for each of the IPsec VPN tunnels. The following is the configuration for the first ASR 1000 Series router (ASR1K-1): crypto keyring keyring-vpn-0e69ba34dc79a6a1c-1 local-address GigabitEthernet0/0/0 pre-shared-key address key <secret-key> crypto keyring keyring-vpn-0e69ba34dc79a6a1c-0 local-address GigabitEthernet0/0/0 pre-shared-key address key <secret key> The following is the configuration for the second ASR 1000 Series router (ASR1K-2): crypto keyring keyring-vpn-09d08c4b c-1 local-address GigabitEthernet0/0/0 pre-shared-key address key <secret key> crypto keyring keyring-vpn-09d08c4b c-0 local-address GigabitEthernet0/0/0 pre-shared-key address key <secret key> A separate preshared key is configured for each of the two IPsec VPN tunnels from each ASR 1000 Series router to each AWS VPN connection. Step 3. Associate the keyrings with the AWS VPN connection peer. The following is the configuration for the first ASR 1000 Series router (ASR1K-1): crypto isakmp profile isakmp-vpn-0e69ba34dc79a6a1c-0 keyring keyring-vpn-0e69ba34dc79a6a1c-0 match identity address local-address GigabitEthernet0/0/0 crypto isakmp profile isakmp-vpn-0e69ba34dc79a6a1c-1 keyring keyring-vpn-0e69ba34dc79a6a1c-1 match identity address local-address GigabitEthernet0/0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 105

19 The following is the configuration for the second ASR 1000 Series router (ASR1K-2): crypto isakmp profile isakmp-vpn-0e69ba34dc79a6a1c-0 keyring keyring-vpn-0e69ba34dc79a6a1c-0 match identity address local-address GigabitEthernet0/0/0 crypto isakmp profile isakmp-vpn-0e69ba34dc79a6a1c-1 keyring keyring-vpn-0e69ba34dc79a6a1c-1 match identity address local-address GigabitEthernet0/0/0 Procedure: 3 Configure IPsec An IPsec transform set defines a combination of security parameters to be used for IPsec SAs. The transform set names must be unique within the Cisco ASR 1000 Series Router configuration. We recommended that you configure the following within the IPsec transform sets for increased security: Use AES-256 rather than AES-128 shown in the configuration example downloaded from AWS in the previous process. Use the SHA-2 hashing algorithm with a 256-bit digest, rather than the SHA-1 algorithm show in the configuration example downloaded from AWS in the previous process. This is also known as SHA-256. Use Diffie-Hellman group 14 or higher, rather than group 2, shown in the configuration example downloaded from AWS in the previous process, for perfect forward secrecy (PFS). This deployment guide configures Diffie-Hellman group 24. The above security parameters are currently supported by the AWS virtual private gateway. Step 1. Configure the IPsec transform sets. The following is the configuration for the first Cisco ASR 1000 Series Router (ASR1K-1): crypto ipsec transform-set ipsec-prop-vpn-0e69ba34dc79a6a1c-0 esp-aes 256 espsha256-hmac mode tunnel crypto ipsec transform-set ipsec-prop-vpn-0e69ba34dc79a6a1c-1 esp-aes 256 espsha256-hmac mode tunnel The following is the configuration for the second ASR 1000 Series router (ASR1K-2): crypto ipsec transform-set ipsec-prop-vpn-09d08c4b c-0 esp-aes 256 espsha256-hmac mode tunnel crypto ipsec transform-set ipsec-prop-vpn-09d08c4b c-1 esp-aes 256 espsha256-hmac mode tunnel 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 105

20 Step 2. Configure the IPsec profiles. The following is the configuration for the first Cisco ASR 1000 Series Router (ASR1K-1): crypto ipsec profile ipsec-vpn-0e69ba34dc79a6a1c-0 set transform-set ipsec-prop-vpn-0e69ba34dc79a6a1c-0 set pfs group24 crypto ipsec profile ipsec-vpn-0e69ba34dc79a6a1c-1 set transform-set ipsec-prop-vpn-0e69ba34dc79a6a1c-1 set pfs group24 The following is the configuration for the second ASR 1000 Series router (ASR1K-2): crypto ipsec profile ipsec-vpn-09d08c4b c-0 set transform-set ipsec-prop-vpn-09d08c4b c-0 set pfs group24 crypto ipsec profile ipsec-vpn-09d08c4b c-1 set transform-set ipsec-prop-vpn-09d08c4b c-1 set pfs group24 We also recommend that you configure the following: Increase the IPsec anti-replay window to the full 1024 window size, in order to eliminate any potential antireplay problems. Configure IKE dead-peer detection (DPD) between the peers on an on-demand basis with a keepalive interval of 10 seconds, and a retry interval of 10 seconds. An on-demand basis means that IKE DPD packets are sent only if no traffic has been received from the peer for 10 seconds. IKE DPD is used to ensure the IPsec tunnels are not torn down during idle periods. Step 3. Configure additional IKE and IPsec parameters specified in the AWS configuration on both ASR 1000 routers (ASR1K-1 and ASR1K-2): crypto ipsec df-bit clear crypto isakmp keepalive on-demand crypto ipsec security-association replay window-size 1024 crypto ipsec fragmentation before-encryption Procedure 4: Configure tunnel interfaces The IPsec connections on the Cisco ASR 1000 Series Routers in your private network use IPsec virtual tunnel interface (VTI) configurations. Two tunnel interfaces are created for each AWS VPN connection on each ASR 1000 Series router, since each AWS VPN connection establishes a redundant pair of IPsec connections to the router. Step 1. Configure the tunnel interfaces Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 105

21 The following is the configuration for the first Cisco ASR 1000 Series Router (ASR1K-1): interface Tunnel1 description IPsec VPN Tunnel for AWS Connection ID vpn-0e69ba34dc79a6a1c-0 ip address ip tcp adjust-mss 1387 tunnel source GigabitEthernet0/0/0 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-0e69ba34dc79a6a1c-0 ip virtual-reassembly no shutdown interface Tunnel2 description IPsec VPN Tunnel for AWS Connection ID vpn-0e69ba34dc79a6a1c-1 ip address ip tcp adjust-mss 1387 tunnel source GigabitEthernet0/0/0 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-0e69ba34dc79a6a1c-1 ip virtual-reassembly no shutdown The following is the configuration for the second ASR 1000 router (ASR1K-2): interface Tunnel1 description IPsec VPN Tunnel for AWS Connection ID vpn-09d08c4b c-0 ip address ip tcp adjust-mss 1387 tunnel source GigabitEthernet0/0/0 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-09d08c4b c-0 ip virtual-reassembly no shutdown interface Tunnel2 description IPsec VPN Tunnel for AWS Connection ID vpn-09d08c4b c-1 ip address ip tcp adjust-mss 1387 tunnel source GigabitEthernet0/0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 105

22 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-09d08c4b c-1 ip virtual-reassembly no shutdown Procedure 5: Configure routing This guide uses BGP routing between the AWS VPC and your private network. Within your private network, Enhanced Interior Gateway Routing Protocol (EIGRP) and static routing are implemented. EIGRP is configured with an MD5 hashed key for additional security. The routes learned via BGP are redistributed into EIGRP. Step 1. Configure the EIGRP key chain and key on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2): key chain EIGRP-KEY key 1 key-string <secret key> Note: Use the global configuration command service password-encryption as an additional security setting within Cisco routers to hide passwords when viewing the configuration. Step 2. Configure EIGRP routing on both Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2). Note that EIGRP autono-mous-system 100 is configured within the private network: router eigrp Multicloud address-family ipv4 unicast autonomous-system 100 af-interface default passive-interface exit-af-interface af-interface GigabitEthernet0/0/2 authentication mode md5 authentication key-chain EIGRP-KEY no passive-interface exit-af-interface network exit-address-family This guide uses two interfaces (GigabitEthernet0/0/0 and GigabitEthernet0/0/2) on each of the Cisco ASR 1000 Series Routers to separate encrypted traffic from unencrypted traffic. They also keep the ASR 1000 Series routers from being "one-armed routers" for VPN connections to the AWS VPC Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 22 of 105

23 Interface GigabitEthernet0/0/0 is the source interface for the IPsec connections on both routers. This has been configured under interfaces Tunnel1 and Tunnel2, above. Interface GigabitEthernet0/0/0 is intended only for encrypted traffic to the AWS VPC. However, interface GigabitEthernet0/0/2 is intended to be the interface through which unencrypted traffic destined for the AWS VPC passes. EIGRP routing is only active on this interface. Step 3. Configure static routes. The following is the configuration for the first ASR 1000 Series router (ASR1K-1): ip route ip route The following is the configuration for the second ASR 1000 Series router (ASR1K-2): ip route ip route To prevent routing of traffic not destined for the AWS VPC through the Cisco ASR 1000 Series Routers, EIGRP routing is not configured to be active on interface GigabitEthernet0/0/0. Instead, configure static routes to the public IP addresses of the AWS VPN connections pointing to a shared IP address of a Hot Standby Router Protocol (HSRP) group on the pair of redundant upstream data center distribution-layer switches. This ensures that normal traffic in the private network is not routed through the ASR 1000 Series routers. The upstream HSRP group ensures that, if one of the two upstream distribution-layer switches fails, the second switch will take over to minimize the disruption. Step 4. Configure BGP routing. This guide uses BGP routing between the AWS VPC and your private network. The BGP ASN specified for your private network in the AWS customer gateway configurations was The BGP ASN of the AWS VPC was specified as The following is the configuration for the first Cisco ASR 1000 Series Router (ASR1K-1): route-map ipsec-vpn-0e69ba34dc79a6a1c permit 10 set as-path prepend router bgp bgp router-id interface Loopback0 bgp log-neighbor-changes neighbor remote-as neighbor timers neighbor password <secret key> neighbor update-source GigabitEthernet0/0/2 neighbor remote-as neighbor update-source Tunnel2 neighbor timers neighbor remote-as neighbor update-source Tunnel Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 23 of 105

24 neighbor timers address-family ipv4 neighbor activate neighbor soft-reconfiguration inbound neighbor activate neighbor default-originate neighbor soft-reconfiguration inbound neighbor route-map ipsec-vpn-0e69ba34dc79a6a1c out neighbor activate neighbor default-originate neighbor soft-reconfiguration inbound neighbor route-map ipsec-vpn-0e69ba34dc79a6a1c out exit-address-family The following is the configuration for the second ASR 1000 Series router (ASR1K-2): route-map ipsec-vpn-09d08c4b c permit 10 set as-path prepend router bgp bgp router-id interface Loopback0 bgp log-neighbor-changes neighbor remote-as neighbor timers neighbor password <secret key> neighbor update-source GigabitEthernet0/0/2 neighbor fall-over bfd neighbor remote-as neighbor timers neighbor remote-as neighbor timers address-family ipv4 neighbor activate neighbor soft-reconfiguration inbound neighbor activate neighbor default-originate neighbor soft-reconfiguration inbound neighbor route-map ipsec-vpn-09d08c4b c out neighbor activate 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 24 of 105

25 neighbor default-originate neighbor soft-reconfiguration inbound neighbor route-map ipsec-vpn-09d08c4b c out exit-address-family In the configuration above, the IP addresses that are the remote side of interfaces Tunnel1 and Tunnel2 are configured as BGP peers. Since AWS virtual private gateways support only IPv4, only "address-family ipv4" is configured. Because this deployment guide utilizes Application Visibility and Control (AVC) deployed on the Cisco ASR 1000 Series Routers in the private network, asymmetric routing to and from the VPC is required. ASR1K-1 is used as the primary path to and from the VPC. In order to ensure that the AWS VGW sends traffic destined for endpoints in the private network to ASR1K-1, autonomous system (AS) prepending is implemented and configured differently for ASR1K-1 and ASR1K-2. For ASR1K-1, a route map in the outbound direction is applied to each external BGP (ebgp) neighbor. The route map prepends the AS number once to the AS path, as follows: route-map ipsec-vpn-0e69ba34dc79a6a1c permit 10 set as-path prepend For ASR1K-2, a route map in the outbound direction is also applied to each ebgp neighbor. However, the route map prepends the AS number twice to the AS path, as follows: route-map ipsec-vpn-09d08c4b c permit 10 set as-path prepend Since BGP routing uses the AS-path length in determining which is the preferred path, the AWS VGW sees the path to routes available at the private network as being preferred through ASR1K-1 over ASR1k-2. If ASR1K-1 fails, or the IPsec tunnels to ASR1K-1 go down, the route through ASR1K-2 will automatically become the preferred route. This deployment guide assumes the VPC subnet has no Internet access through AWS; therefore, default routes are originated by the Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2) for each of the AWS BGP peers. All routing to the Internet is backhauled from the VPC to the private network before exiting the private network at the Internet edge firewall (see Figure 2, above). An internal BGP (ibgp) peer is established between both ASR 1000 Series routers. This is created by adding the IP address of the inside (GigabitEthernet0/0/2) interface of the second ASR Series 1000 router as another BGP peer using an ASN of BFD is implemented on the ibgp peer. An authentication password is implemented between the ibgp peers as an added security measure. For this deployment guide, routes learned via BGP are redistributed into EIGRP. A route map named bgp-toeigrp is used to control which BGP-learned routes are redistributed into EIGRP Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 25 of 105

26 Step 5. Configure the route-map and prefix-list commands for redistributing BGP routes into EIGRP on both Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2): ip prefix-list bgp-to-eigrp seq 100 permit /16 ip prefix-list bgp-to-eigrp seq 110 permit /20 ip prefix-list bgp-to-eigrp seq 120 permit /20 ip prefix-list bgp-to-eigrp seq 130 permit /27 route-map bgp-to-eigrp permit 10 description Route-map for BGP to EIGRP Redistribution match ip address prefix-list bgp-to-eigrp Step 6. Redistribute routes learned from BGP into EIGRP. The following is the configuration for the first ASR 1000 Series router (ASR1K-1): router eigrp Multicloud address-family ipv4 unicast autonomous-system 100 topology base default-metric redistribute bgp route-map bgp-to-eigrp exit-af-topology exit-address-family The following is the configuration for the second ASR 1000 Series router (ASR1K-2): router eigrp Multicloud address-family ipv4 unicast autonomous-system 100 topology base default-metric redistribute bgp route-map bgp-to-eigrp exit-af-topology exit-address-family The VPC network address range of /16, and the Transit VPC address ranges of /20, /20, and /27 redistributed from BGP to EIGRP. Because this deployment guide utilizes AVC deployed on the ASR 1000 Series routers in the private network, asymmetric routing to and from the VPC is required. ASR1K-1 is used as the primary path to and from the VPC. In order to ensure that all Layer 3 switches and routers in the private network send traffic destined for endpoints in the VPC to ASR1K-1, the default metrics for redistribution of BGP routes into EIGRP are different for ASR1K-1 and ASR1K Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 26 of 105

27 The default-metric command for EIGRP includes the following five parameters (listed in their order of appearance in the commands shown above): Bandwidth: Bandwidth of the route in kilobytes per second (KB/sec) Delay: Route delay in tens of microseconds from 1 to any multiple of 39.1 microseconds Reliability: Reliability of the route from 255 (100 percent reliable) to 0 (no reliability) Loading: Effective bandwidth of the router from 255 (100 percent loaded) to 1 (no loading) MTU: The smallest value allowed for maximum transmission unit (MTU), from 1 byte to bytes For ASR1K-1, set the default-metric command as follows for this deployment guide: default-metric For ASR1K-2, set the default-metric command as follows for this deployment guide: default-metric This will configure the bandwidth parameter such that ASR1K-1 advertises that it has double the bandwidth to the VPC than ASR1K-2. This causes the path to the VPC network ( /16) through ASR1K-1 to be the preferred route. Should ASR1K-1 fail, or the IPsec tunnels to ASR1K-1 go down, the route through ASR1K-2 will automatically become the preferred route. Step 7. Configure the route-map and prefix-list commands for redistributing EIGRP routes into BGP on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2): ip prefix-list eigrp-to-bgp seq 10 permit /30 ip prefix-list eigrp-to-bgp seq 20 permit /30 ip prefix-list eigrp-to-bgp seq 30 permit /30 ip prefix-list eigrp-to-bgp seq 40 permit /32 ip prefix-list eigrp-to-bgp seq 50 permit /32 ip prefix-list eigrp-to-bgp seq 60 permit /32 ip prefix-list eigrp-to-bgp seq 70 permit /32 ip prefix-list eigrp-to-bgp seq 80 permit /32 ip prefix-list eigrp-to-bgp seq 90 permit /30 ip prefix-list eigrp-to-bgp seq 100 permit /30 ip prefix-list eigrp-to-bgp seq 110 permit /28 ip prefix-list eigrp-to-bgp seq 120 permit /29 ip prefix-list eigrp-to-bgp seq 130 permit /29 ip prefix-list eigrp-to-bgp seq 140 permit /29 ip prefix-list eigrp-to-bgp seq 150 permit /29 ip prefix-list eigrp-to-bgp seq 160 permit /24 route-map eigrp-to-bgp permit 10 description Route-map for EIGRP to BGP Redistribution match ip address prefix-list eigrp-to-bgp 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 27 of 105

28 A route map named eigrp-to-bgp is used to control which EIGRP-learned routes are redistributed into BGP. For this deployment guide, all of the private network subnets are advertised. Step 8. Redistribute routes learned via EIGRP into BGP on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2): router bgp address-family ipv4 redistribute eigrp 100 route-map eigrp-to-bgp exit-address-family For this deployment guide, routes learned via EIGRP are redistributed into BGP. This can aid in troubleshooting, since the private network routes will appear within the VPC subnet route table if routes are propagated in the route table. Note that some caution needs to be taken when implementing this method. AWS only supports up to 100 propagated routes per route table. If more than 100 prefixes are required, you may need to deploy a VPN router such as the Cisco CSR 1000V Router instead of the AWS VGW. Alternatives such as just having the ebgp peer originate a default route, or simply configuring a network under the IPv4 address-family configuration, could also be implemented, depending on your requirements. Procedure 5: Verify that the VPN connections are working properly The IPsec VPN connections should come up automatically once the configurations on the Cisco ASR 1000 Series Routers and the AWS VPC are completed. The following are the steps to verify that these VPN connections are operating properly: Step 1. Run a "show interface" command on both tunnel interfaces. The tunnel interfaces should appear with no errors, as shown in the example below: ASR1K-1#show interface Tunnel1 Tunnel1 is up, line protocol is up Hardware is Tunnel Description: IPsec VPN Tunnel for AWS Connection ID vpn-0e69ba34dc79a6a1c-0 Internet address is /30 MTU 9922 bytes, BW 100 Kbit/sec, DLY usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation TUNNEL, loopback not set Keepalive not set Tunnel linestate evaluation up Tunnel source (GigabitEthernet0/0/0), destination Tunnel Subblocks: src-track: Tunnel1 source tracking subblock associated with GigabitEthernet0/0/0 Set of tunnels with source GigabitEthernet0/0/0, 3 members (includes iterators), on interface <OK> Tunnel protocol/transport IPSEC/IP Tunnel TTL 255 Tunnel transport MTU 1422 bytes 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 28 of 105

29 Tunnel transmit bandwidth 8000 (kbps) Tunnel receive bandwidth 8000 (kbps) Tunnel protection via IPSec (profile "ipsec-vpn-0e69ba34dc79a6a1c-0") Last input never, output never, output hang never Last clearing of "show interface" counters 1w0d Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 5 Queueing strategy: fifo Output queue: 0/0 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec packets input, bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort packets output, bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out Step 2. Run a "show ip route" command to verify that the BGP route to the AWS VPC address space appears within the ASR 1000 Series routers: ASR1K-1# show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override, p - overrides from PfR Gateway of last resort is to network D*EX /0 [170/537600] via , 1w0d, GigabitEthernet0/0/ /8 is variably subnetted, 34 subnets, 7 masks B /16 [20/100] via , 02:51:09 D /30 [90/20480] via , 1w0d, GigabitEthernet0/0/2 D /30 [90/20480] via , 1w0d, GigabitEthernet0/0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 29 of 105

30 The route to network /16 in the route table above known through BGP via indicates that BGP routes are being distributed between the AWS VPC and the private network. The next two processes configure the IPsec VPN connectivity between the private network and an AWS Transit VPC, using the Cisco CSR 1000V Routers already deployed within the Transit VPC. Only deploy these processes if you are connecting a Transit VPC to the private network. Configure the IPsec VPN connections on the Cisco CSR 1000V Routers in the AWS Transit VPC AWS automatically generates the VPN configurations that are used for connecting spoke VPCs to the transit VPC on the Cisco CSR 1000V routers in the Transit VPC. For more information, refer the following deployment guide: AWS will not automatically generate the IPsec VPN connections required to connect a private network to the Transit VPC. Use this process to configure one IPsec VPN connection on each of the CSR 1000V routers within the Transit VPC. Each of these IPsec VPN connections will connect to one of the Cisco ASR 1000 Series Routers in the private network. This process assumes you have Security Shell Protocol (SSH) access to the CSR 1000V routers in the Transit VPC, and that you have downloaded the AWS key pair used to access the instances when you created the CSR 1000V routers. This process configures the following: A new VRF for the tunnel interface to the private network BFD for more rapid convergence IKEv1 IPsec Tunnel interfaces for the VPN connection Dynamic routing Procedure 1: Configure a VRF for the tunnel interface to the private network The AWS Transit VPC model uses a separate VRF for each spoke VPC. Multi-Protocol BGP (MP-BGP) is then used to leak routes between the spokes. The following configuration shows an example of how this is done with two spoke VPCs: ip vrf vpn-43bca022 rd 64512:101 route-target export 64512:0 route-target import 64512:0 ip vrf vpn-49bca028 rd 64512:103 route-target export 64512:0 route-target import 64512: Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 30 of 105

31 ip vrf vpn0 rd 64512:0 In this example, the two spoke VRFs vpn-43bca022 and vpn-49bca028 are automatically defined by AWS with route distinguishers that reflect the BGP ASN (64512) and one of the tunnel interfaces of each spoke (101 and 102). Spokes use the AWS-native IPsec VPN service, which automatically creates two IPsec VPN connections, and therefore two tunnel interfaces to each spoke. A third VRF, vpn0, is also automatically defined with a route distinguisher (RD) of 64512:0, as in the example above. Each of the spoke VPCs exports routes to, and imports routes from, VRF vpn0 through the route-target statements defined under each spoke VRF. This allows for full BGP reachability between the VRFs. Note that the IP address ranges between the VPCs do not overlap. The design followed in this deployment guide connects the private network to the Transit VPC as if it were another spoke. Step 1. Configure a VRF for the private network connection on both of the Cisco CSR 1000V Routers (TVPC- CSR1 and TVPC-CSR2) in the Transit VPC: ip vrf vpn-private-dc rd 64512:100 route-target export 64512:0 route-target import 64512:0 The private network VPC vpn-private-dc is defined with a route distinguisher that reflects the BGP ASN (64512) and the tunnel interface of the connection (100) that will be configured in an upcoming procedure. Keeping with the conventions used by AWS minimizes any chances of issues when any new spoke VPCs are added in the future. The private network VPC exports routes to, and imports routes from, VRF vpn0 through the route-target statements defined under the VRF. This allows for full BGP reachability between the spoke VRFs and the private network VRF. Procedure 2: Configure a BFD template BFD can be used to rapidly detect the loss of connectivity between devices and have routing protocols converge faster. Although BFD is not supported on AWS-native IPsec VPN connections, it can be added on the IPsec VPN connections between the Cisco CSR 1000V Routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC and the Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2) in the private network. Note: Adding BFD will incrementally increase the traffic across the VPN connections from the private network to the Transit VPC. You should note that part of the overall AWS costs include data transfer costs, which are based on the amount of data transferred to or from EC2 instances in the VPC. This includes CSR 1000V routers that run on EC2 instances. Step 1. Configure a BFD template on both Cisco CSR 1000V Routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC: bfd-template single-hop bfdtemplate1 interval min-tx 1000 min-rx 1000 multiplier Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 31 of 105

32 The minimum transmit and receive interval is set to 1000 milliseconds (1 second) in the configuration above. A loss of three consecutive messages (3 seconds) causes the BFD peer to be declared down. The BFD transmit-andreceive intervals are not aggressive in this deployment guide. This is done intentionally to minimize the possibility of route flapping. Tune the intervals and multiplier based on the requirements for convergence within your network and the capabilities of your network devices. The BFD template will be applied when the tunnel interfaces are created in an upcoming procedure. You will still need to configure BGP to use BFD to determine when the peer is down. Further information is provided in the later procedures in this document. Procedure 3: Configure IKEv1 This deployment guide uses IKEv1 policies to be consistent with AWS-native VPN service, which only supports IKEv1 policies. The AWS native VPN service is already used to connect spoke VPCs with the Transit VPC. Cisco CSR 1000V Routers also support IKEv2, if your security policy requires IKEv2. An IKEv1 policy defines a combination of security parameters to be used during IKE negotiation. Policy numbers must be unique within the router configuration. The ISAKMP security association (SA) is configured to propose a timeout of 28,800 seconds (8 hours). We recommend that, for increased security, you configure the following in the IKEv1 policies: Use Advanced Encryption Standard (AES) encryption with a 256-bit key (AES-256). Use the Secure Hash Algorithm 2 (SHA-2) hashing algorithm with a 256-bit digest. This is also known as SHA-256. Use Diffie-Hellman group 24. Step 1. Configure the IKE policy for the IPsec VPN tunnel on both of the Cisco CSR 1000V Routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC: crypto isakmp policy 500 encr aes 256 hash sha256 authentication pre-share group 24 lifetime Note: IKEv1 policies are configured as ISAKMP policies within Cisco IOS and IOS-XE devices. Step 2. Configure the IKE preshared key on both of the CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC. IKE preshared keys are used to authenticate CSR 1000V routers during the establishment of the ISAKMP SA. The following configuration uses a keyring to store a preshared key for the IPsec VPN tunnel Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 32 of 105

33 The following is the configuration for the first CSR 1000V router (TVPC-CSR1) in the Transit VPC: crypto keyring keyring-vpn-private-dc local-address GigabitEthernet1 pre-shared-key address key <secret key> The preshared key configured for the first CSR 1000V router (TVPC-CSR1) in the Transit VPC points to the public IP address of the first Cisco ASR 1000 Series Router (ASR1K-1) in the private network. The following is the configuration for the second CSR 1000V router (TVPC-CSR2) in the Transit VPC: crypto keyring keyring-vpn-private-dc local-address GigabitEthernet1 pre-shared-key address key <secret key> The preshared key configured for the second CSR 1000V router (TVPC-CSR2) in the Transit VPC points to the public IP address of the second ASR 1000 Series router (ASR1K-2) in the private network. Step 3. Associate the keyring with the IPSec connection peer on both of the CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC. The following is the configuration for the first CSR 1000V router (TVPC-CSR1) in the Transit VPC: crypto isakmp profile isakmp-vpn-private-dc keyring keyring-vpn-private-dc match identity address local-address GigabitEthernet1 The identity address configured for the first CSR 1000V router (TVPC-CSR1) in the Transit VPC points to the public IP address of the first ASR 1000 Series router (ASR1K-1) in the private network. The following is the configuration for the second CSR 1000V router (TVPC-CSR2) in the Transit VPC: crypto isakmp profile isakmp-vpn-private-dc keyring keyring-vpn-private-dc match identity address local-address GigabitEthernet1 The identity address configured for the second CSR 1000V router (TVPC-CSR2) in the Transit VPC points to the public IP address of the second Cisco ASR 1000 Series router (ASR1K-2) in the private network. Procedure 4: Configure IPsec An IPsec transform set defines a combination of security parameters to be used for IPsec SAs. The transform set name must be unique within the Cisco CSR 1000V Router configuration. We recommended that, for increased security, you configure the following within the IPsec transform set: Use AES-256. Use the SHA-2 hashing algorithm with a 256-bit digest. This is also known as SHA-256. Use Diffie-Hellman group Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 33 of 105

34 Step 1. Configure the IPsec transform set on both Cisco CSR 1000V Routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC: crypto ipsec transform-set ipsec-prop-private-dc esp-aes 256 esp-sha256-hmac mode tunnel Step 2. Configure the IPsec profile on both CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC: crypto ipsec profile ipsec-vpn-private-dc set transform-set ipsec-prop-private-dc set pfs group24 Additional IKE and IPsec parameters such as the IPsec anti-replay window size and IKE dead-peer detection (DPD) should already be set automatically on the CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC when AWS configures IPsec for the spoke VPCs. Procedure 5: Configure tunnel interfaces The IPsec connections on the Cisco CSR 1000V Routers in the Transit VPC use IPsec virtual tunnel interface (VTI) configurations. A single tunnel interface is created on each CSR 1000V, connecting to one of the Cisco ASR 1000 Series Routers in the private network. Step 1. Configure the tunnel interfaces. The following is the configuration for the first Cisco CSR 1000V Router (TVPC-CSR1): interface Tunnel100 description VPN from private dc ASR1K-1 to transit vpc TVPC-CSR1 ip vrf forwarding vpn-private-dc ip address ip tcp adjust-mss 1387 bfd template bfdtemplate1 tunnel source GigabitEthernet1 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-private-dc ip virtual-reassembly no shutdown The following is the configuration for the second CSR 1000V router (TVPC-CSR2): interface Tunnel100 description VPN from private dc ASR1K-2 to transit vpc TVPC-CSR2 ip vrf forwarding vpn-private-dc ip address ip tcp adjust-mss 1387 bfd template bfdtemplate1 tunnel source GigabitEthernet1 tunnel destination Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 34 of 105

35 tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-private-dc ip virtual-reassembly no shutdown Note that the tunnel interfaces between the CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC and the Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2) in the private network use the private network address space ( ) in this deployment guide. This choice was arbitrary and designed to make sure these tunnel addresses never conflict with any future tunnel interfaces that may be dynamically generated by AWS when adding future spoke VPCs. You can use a completely different address space, separate from the Transit VPC, spoke VPCs, and the private network, according to your requirements. Procedure 6: Configure dynamic routing This guide uses BGP routing between the Transit VPC and your private network. The BGP ASN specified for your private network in the AWS customer gateway configurations was The BGP ASN of the AWS Transit VPC was specified as The AWS Transit VPC model includes the ability to add a tag to the virtual private gateway (VGW) of each spoke VPC, indicating the preferred path from the spoke VPC to the Transit VPC. An example of the preference tag is shown below for a spoke VPC. Figure 12. Spoke VPC preference tag 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 35 of 105

36 The format of the tag is as follows: Key: transitvpc:preferred-path Value: CSR1 or CSR2 A value of CSR1 corresponds to the first Cisco CSR 1000V Router (TVPC-CSR1) in the Transit VPC. A value of CSR2 corresponds to the second CSR 1000V router (TVPC-CSR2) in the Transit VPC. The effect of configuring a preferred path is that AWS will automatically configure two route maps on both of the CSR 1000V routers in the Transit VPC. The following is an example of the route maps configured on the first CSR 1000V router (TVPC-CSR1) in the Transit VPC. route-map rm-vpn-42bca022 permit 10 set as-path prepend route-map rm-vpn-42bca028 permit 10 set as-path prepend The following is an example of the route maps configured on the second CSR 1000V router (TVPC-CSR2) in the Transit VPC: route-map rm-vpn-42bca023 permit 10 set as-path prepend route-map rm-vpn-42bca029 permit 10 set as-path prepend These same route maps will be leveraged to configure a preference from the private network to the Transit VPC. Step 1. Configure BGP routing on both CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC. BGP routing should already be automatically configured on both CSR 1000V routers in the Transit VPC. This is a result of adding spoke VPCs to the Transit VPC. The following shows the additional BGP configuration needed to establish a BGP routing peer from each CSR 1000V router to one of the Cisco ASR 1000 Series Routers (ASR1K- 1 or ASR1K-2) in the private network. The following is the configuration for the first CSR 1000V router (TVPC-CSR1), which peers with ASR1K-1 in the private network: router bgp address-family ipv4 vrf vpn-private-dc neighbor remote-as neighbor password <secret key> neighbor timers neighbor fall-over bfd neighbor activate neighbor next-hop-self neighbor as-override 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 36 of 105

37 neighbor soft-reconfiguration inbound neighbor route-map rm-vpn-49bca028 out exit-address-family The following is the configuration for the second CSR 1000V router (TVPC-CSR2), which peers with ASR1K-2 in the private network: router bgp address-family ipv4 vrf vpn-private-dc neighbor remote-as neighbor password <secret key> neighbor timers neighbor fall-over bfd neighbor activate neighbor next-hop-self neighbor as-override neighbor soft-reconfiguration inbound exit-address-family In the configuration above, the IP addresses that are the remote side of Tunnel100 interfaces are configured as BGP peers. Because this deployment guide utilizes AVC deployed on the Cisco ASR 1000 Series Routers in the private network, asymmetric routing to and from the Transit VPC is required. The path between TVPC-CSR1 and ASR1K- 1 is used as the primary path to and from the Transit VPC. In order to ensure the private network sends traffic destined for endpoints in the spoke VPCs to TVPC-CSR1, AS prepending is implemented and configured differently for TVPC-CSR1 and TVPC-CSR2. For TVPC-CSR1, the route map used in the BGP configuration prepends BGP AS once to the routes advertised to ASR1K-1. For TVPC-CSR2, the route map used in the BGP configuration prepends BGP AS twice to the routes advertised to ASR1K-2. Since BGP routing uses the AS path length in determining which is the preferred path, the private network sees the path to routes available in the spoke VPCs as being preferred through ASR1K-1 over ASR1K-2. Should ASR1K-1 fail, or the IPsec tunnel to ASR1K-1 go down, the route through ASR1K-2 will automatically become the preferred route. BFD is enabled on the BGP peers between the CSR 1000V routers in the Transit VPC and the ASR 1000 Series routers in the private network for more rapid convergence Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 37 of 105

38 Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS Transit VPC This process configures the following: BFD for more rapid convergence IKEv1 IPsec Tunnel interfaces for the VPN connection Routing Procedure 1: Configure IKEv1 This deployment guide uses IKEv1 policies to be consistent with AWS-native VPN service, which only supports IKEv1 policies. Cisco CSR 1000V Routers also support IKEv2, if the latter is required by your security policy. An IKEv1 policy was previously configured in Configuring the Cisco ASR 1000 Series Router VPN connections, above. This same policy will be used for the IPsec VPN connections between the Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2) and the Cisco CSR 1000V Routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC. Step 1. Configure the IKE preshared key on both of the Cisco ASR 1000 Series Routers (ASR1K-1 and ASR1K-2) in the private network. IKE preshared keys are used to authenticate the ASR 1000 Series routers during the establishment of the ISAKMP SA. The following configuration uses a keyring to store a preshared key for the IPsec VPN tunnel. The following is the configuration for the first ASR 1000 Series router (ASR1K-1) in the private network: crypto keyring keyring-vpn-private-dc local-address GigabitEthernet0/0/0 pre-shared-key address key <secret key> The preshared key configured for the first ASR 1000 Series router (ASR1K-1) in the private network points to the public IP address of the first CSR 1000V router (TVPC-CSR1) in the Transit VPC. The following is the configuration for the second ASR 1000 Series router (ASR1K-2) in the private network: crypto keyring keyring-vpn-private-dc local-address GigabitEthernet0/0/0 pre-shared-key address key <secret key> The preshared key configured for the second ASR 1000 Series router (ASR1K-2) in the private network points to the public IP address of the second CSR 1000V router (TVPC-CSR2) in the Transit VPC. Step 2. Associate the keyring with the IPsec connection peer on both of the ASR 1000 Series routers (ASR1K-1 and ASR1k-2) in the private network. The following is the configuration for the first ASR 1000 Series router (ASR1K-1) in the private network: crypto isakmp profile isakmp-vpn-private-dc keyring keyring-vpn-private-dc match identity address local-address GigabitEthernet0/0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 38 of 105

39 The identity address configured for the first ASR 1000 Series router (ASR1K-1) in the private network points to the public IP address of the first CSR 1000V router (TVPC-CSR1) in the Transit VPC. The following is the configuration for the second ASR 1000 Series router (ASR1K-2) in the private network: crypto isakmp profile isakmp-vpn-private-dc keyring keyring-vpn-private-dc match identity address local-address GigabitEthernet0/0/0 The identity address configured for the second ASR 1000 Series router (ASR1K-2) in the private network points to the public IP address of the second CSR 1000V router (TVPC-CSR2) in the Transit VPC. Procedure 2: Configure IPsec An IPsec transform set defines a combination of security parameters to be used for IPsec SAs. The transform set name must be unique in the Cisco ASR 1000 Series Router configuration. We recommended that you configure the following within the IPsec transform set for increased security: Use AES-256. Use the SHA-2 hashing algorithm with a 256-bit digest. This is also known as SHA-256. Use Diffie-Hellman group 24. Step 1. Configure the IPsec transform set on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2) in the private network. crypto ipsec transform-set ipsec-prop-private-dc esp-aes 256 esp-sha256-hmac mode tunnel Step 2. Configure the IPsec profile on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2) in the private network. crypto ipsec profile ipsec-vpn-private-dc set transform-set ipsec-prop-private-dc set pfs group24 Additional IKE and IPsec parameters such as the IPsec anti-replay window size and IKE dead-peer detection (DPD) were already configured in Configuring the Cisco ASR 1000 Series Router VPN connections, above. Procedure 3: Configure tunnel interfaces The IPsec connections on the Cisco ASR 1000 Series Routers in the private network use IPsec virtual tunnel interface (VTI) configurations. A single tunnel interface is created for each ASR 1000 Series router, connecting the router to one of the Cisco CSR 1000V Routers in the Transit VPC. Step 1. Configure the tunnel interfaces. The following is the configuration for the first ASR 1000 Series router (ASR1K-1): interface Tunnel100 description VPN from private dc ASR1K-1 to transit vpc TVPC-CSR1 ip address ip tcp adjust-mss 1387 bfd template bfdtemplate Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 39 of 105

40 tunnel source GigabitEthernet0/0/0 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-private-dc ip virtual-reassembly no shutdown The following is the configuration for the second ASR 1000 Series router (ASR1K-2): interface Tunnel100 description VPN from private dc ASR1K-2 to transit vpc TVPC-CSR2 ip vrf forwarding vpn-private-dc ip address ip tcp adjust-mss 1387 bfd template bfdtemplate1 tunnel source GigabitEthernet0/0/0 tunnel destination tunnel mode ipsec ipv4 tunnel protection ipsec profile ipsec-vpn-private-dc ip virtual-reassembly no shutdown Note that the tunnel interfaces between the ASR 1000 Series routers (ASR1K-1 and ASR1K-2) in the private network and the CSR 1000V routers (TVPC-CSR1 and TVPC-CSR2) in the Transit VPC use the private network address space ( ) in this deployment guide. This choice was arbitrary and designed to make sure these tunnel addresses never conflict with any future tunnel interfaces that may be dynamically generated by AWS when adding future spoke VPCs. You can use a completely different address space, separate from the Transit VPC, spoke VPCs, and the private network, according to your requirements. Procedure 4: Configure routing This guide uses two interfaces (GigabitEthernet0/0/0 and GigabitEthernet0/0/2) on each of the Cisco ASR 1000 Series Routers to separate encrypted traffic from unencrypted traffic. The two interfaces also keep the ASR 1000 Series routers from being "one-armed routers" for VPN connections to the AWS Transit VPC. Interface GigabitEthernet0/0/0 is the source interface for the IPsec connections on both routers. This has been configured under interface Tunnel100 above. In other words, interface GigabitEthernet0/0/0 is intended only for encrypted traffic to the AWS Transit VPC. Interface GigabitEthernet0/0/2 is intended to be the interface through which unencrypted traffic destined for the AWS Transit VPC passes. Step 1. Configure static routes. The following is the configuration for the first ASR 1000 Series router (ASR1K-1): ip route The following is the configuration for the second ASR 1000 Series router (ASR1K-2): ip route Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 40 of 105

41 To prevent routing of traffic not destined for the AWS VPC through the ASR 1000 Series routers, EIGRP routing is not configured to be active on interface GigabitEthernet0/0/0. Instead, configure static routes to the public IP addresses of the AWS VPN connections pointing to a shared IP address of a Hot Standby Router Protocol (HSRP) group on the pair of redundant upstream data center distribution-layer switches. This ensures that normal traffic in the private network is not routed through the ASR 1000 Series routers. The upstream HSRP group ensures that if one of the two upstream distribution-layer switches fails, the second switch will take over to minimize the disruption. This guide uses BGP routing between the Transit VPC and the private network. The BGP ASN specified for your private network in the AWS customer gateway configurations was The BGP ASN of the AWS Transit VPC was specified as The route maps configured in Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS Transit VPC, above, will again be leveraged to configure a preference from the Transit VPC to the private network. Step 2. Configure BGP routing on both ASR 1000 Series routers (ASR1K-1 and ASR1K-2) in the private network. BGP routing should already be configured on both ASR 1000 Series routers in the private network, as part of the Configure the IPsec VPN connections on the Cisco ASR 1000 Series Routers to the AWS Transit VPC process described previously. This is a result of connecting individual VPCs to the private network using the AWS-native IPsec VPN service. The following shows an additional BGP configuration needed to establish a BGP routing peer from each of the ASR 1000 Series routers (ASR1K-1 or ASR1K-2) in the private network to one of the Cisco CSR 1000V Routers in the Transit VPC. The following is the configuration for the first ASR 1000 Series router (ASR1K-1), which peers with TVPC-CSR1 in the Transit VPC: router bgp neighbor remote-as neighbor password <secret key> neighbor timers neighbor fall-over bfd address-family ipv4 neighbor activate neighbor soft-reconfiguration inbound neighbor route-map ipsec-vpn-0e69ba34dc79a6a1c out exit-address-family The following is the configuration for the second ASR 1000 Series router (ASR1K-2), which peers with TVPC- CSR2 in the Transit VPC: router bgp neighbor remote-as neighbor password <secret key> neighbor timers neighbor fall-over bfd 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 41 of 105

42 address-family ipv4 neighbor activate neighbor soft-reconfiguration inbound neighbor route-map ipsec-vpn-09d08c4b c out exit-address-family In the configuration above, the IP addresses that are the remote side of Tunnel100 interfaces are configured as BGP peers. Because this deployment guide utilizes AVC deployed on the ASR 1000 Series routers in the private network, asymmetric routing to and from the Transit VPC is required. The path between TVPC-CSR1 and ASR1K-1 is used as the primary path to and from the Transit VPC. In order to ensure the Transit VPC sends traffic destined for endpoints in the private network to ASR1K-1, AS prepending is implemented and configured differently for ASR1K-1 and ASR1K-2. For ASR1K-1, the route map used in the BGP configuration prepends BGP AS once to the routes advertised to TVPC-CSR1. For ASR1K-2, the route map used in the BGP configuration prepends BGP AS 6500 twice to the routes advertised to TVPC-CSR2. Since BGP routing uses the AS path length in determining which is the preferred path, the spoke VPCs see the path to routes available in the private network as being preferred through TVPC-CSR1 over TVPC- CSR2. Should TVPC-CSR1 fail, or the IPsec tunnel to TVPC-CSR1 go down, the route through TVPC-CSR2 will automatically become the preferred route. BFD is enabled on the BGP peers between the CSR 1000V routers in the Transit VPC and the ASR 1000 Series routers in the private network for more rapid convergence. Procedure 5: Verify that the VPN connections are working properly The IPsec VPN connections should come up automatically once the configurations on the Cisco ASR 1000 Series Routers and the Cisco CSR 1000V Routers in the AWS Transit VPC are completed. The following are the steps to verify that these VPN connections are operating properly: Step 1. Run a "show interface" command interface Tunnel100 on both ASR 1000 Series routers in the private network and both CSR 1000V routers in the AWS Transit VPC. The tunnel interfaces should appear with no errors, as shown in the example below: ASR1K-1#show interface Tunnel100 Tunnel100 is up, line protocol is up Hardware is Tunnel Description: vpn from private dc ASR1K-1 to transit vpc TVPC-CSR1 Internet address is /30 MTU 9922 bytes, BW 100 Kbit/sec, DLY usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation TUNNEL, loopback not set Keepalive not set Tunnel linestate evaluation up Tunnel source (GigabitEthernet0/0/0), destination Tunnel Subblocks: src-track: 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 42 of 105

43 Tunnel100 source tracking subblock associated with GigabitEthernet0/0/0 Set of tunnels with source GigabitEthernet0/0/0, 3 members (includes iterators), on interface <OK> Tunnel protocol/transport IPSEC/IP Tunnel TTL 255 Tunnel transport MTU 1422 bytes Tunnel transmit bandwidth 8000 (kbps) Tunnel receive bandwidth 8000 (kbps) Tunnel protection via IPSec (profile "ipsec-vpn-private-dc") Last input never, output never, output hang never Last clearing of "show interface" counters 1w1d Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 15 Queueing strategy: fifo Output queue: 0/0 (size/max) 30 second input rate 0 bits/sec, 0 packets/sec 30 second output rate 0 bits/sec, 0 packets/sec packets input, bytes, 0 no buffer Received 0 broadcasts (0 IP multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort packets output, bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out Step 2. Run a "show ip route" command to verify that the BGP routes to the AWS spoke VPC address spaces appear in the ASR 1000 Series routers: ASR1K-1# show ip route Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP a - application route + - replicated route, % - next hop override, p - overrides from PfR Gateway of last resort is to network D*EX /0 [170/537600] via , 1w1d, GigabitEthernet0/0/ Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 43 of 105

44 /8 is variably subnetted, 32 subnets, 7 masks B /16 [20/100] via , 01:04:14 D /30 [90/20480] via , 1w1d, GigabitEthernet0/0/2 D /30 [90/20480] via , 1w1d, GigabitEthernet0/0/2 D /30 [90/25600] via , 1w1d, GigabitEthernet0/0/2 C /32 is directly connected, Loopback0 D /32 [90/10880] via , 1w1d, GigabitEthernet0/0/2 D /32 [90/21120] via , 1w1d, GigabitEthernet0/0/2 D /32 [90/21120] via , 1w1d, GigabitEthernet0/0/2 C /30 is directly connected, GigabitEthernet0/0/1 L /32 is directly connected, GigabitEthernet0/0/1 D /30 [90/15360] via , 1w1d, GigabitEthernet0/0/2 D /29 [90/20480] via , 1w1d, GigabitEthernet0/0/2 D /29 [90/20480] via , 1w1d, GigabitEthernet0/0/2 D /28 [90/20480] via , 1w1d, GigabitEthernet0/0/2 C /29 is directly connected, GigabitEthernet0/0/0 L /32 is directly connected, GigabitEthernet0/0/0 D /29 [90/20480] via , 1w1d, GigabitEthernet0/0/2 C /30 is directly connected, Tunnel100 L /32 is directly connected, Tunnel100 C /29 is directly connected, GigabitEthernet0/0/2 L /32 is directly connected, GigabitEthernet0/0/2 D /29 [90/15360] via , 1w1d, GigabitEthernet0/0/2 D EX /24 [170/ ] via , 2d09h, GigabitEthernet0/0/2 D EX /24 [170/ ] via , 2d09h, GigabitEthernet0/0/2 D EX /24 [170/ ] via , 1d03h, GigabitEthernet0/0/2 B /32 [20/0] via , 2d09h B /32 [20/0] via , 2d09h B /32 [20/0] via , 2d09h B /24 [20/0] via , 2d09h B /24 [20/0] via , 2d09h B /20 [20/0] via , 01:03:37 B /20 [20/0] via , 01:03: /32 is subnetted, 1 subnets S [1/0] via /32 is subnetted, 1 subnets S [1/0] via Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 44 of 105

45 /32 is subnetted, 1 subnets S [1/0] via /16 is variably subnetted, 4 subnets, 2 masks C /30 is directly connected, Tunnel2 L /32 is directly connected, Tunnel2 C /30 is directly connected, Tunnel1 L /32 is directly connected, Tunnel1 D /24 [90/20480] via , 1w1d, GigabitEthernet0/0/2 The routes to networks /20 and /20 in the route table above known through BGP via indicate that BGP routes are being distributed between the AWS Transit VPC and the private network. The final process is optional. This process provides visibility into application flows to and from the private network and the AWS VPCs. Configure visibility into application flows Applications running on AWS Elastic Compute Cloud (EC2) instances within an AWS VPC are not usually under the administrative control of network operations personnel. You can configure visibility into which application flows are using the VPN connections between the private network and the AWS VPC (including the Transit VPC), as well as how much data is being sent. This can prove valuable in managing the costs of the overall AWS service. It can also potentially simplify internal billing of the cost of the service back to the various internal organizations. The following procedures show how to provide visibility into application-level flows between AWS VPCs and your private network. Procedure 1: Configure AVC Cisco Application Visibility and Control (AVC) can be enabled on Cisco ASR 1000 Series Routers to provide application-level visibility into traffic flows between AWS VPCs and your private network. This is accomplished by enabling Network-Based Application Recognition (NBAR2) protocol discovery on one or more interfaces on the ASR 1000 Series routers. Step 1. Enable NBAR2 protocol discovery. interface GigabitEthernet0/0/2 ip nbar protocol-discovery This guide separates IPsec-encrypted traffic (GigabitEthernet0/0/0) from unencrypted traffic (GigabitEthernet0/0/2). Since all traffic to and from the AWS VPC will pass through GigabitEthernet0/0/2 unencrypted, NBAR2 can be enabled on this interface to gain visibility into application traffic running over the VPN connections to the AWS VPCs Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 45 of 105

46 Procedure 2: View AVC data via the CLI Step 1. Run the "show ip nbar protocol-discovery" command from the CLI. You can gain application visibility via the CLI using the "show ip nbar protocol-discovery" command executed at the exec-level command interface of an ASR 1000 Series router. An example of the output is shown below: ASR1K-1#show ip nbar protocol-discovery GigabitEthernet0/0/2 Last clearing of "show ip nbar protocol-discovery" counters 7w2d Input Output Protocol Packet Count Packet Count Byte Count Byte Count 30sec Bit Rate (bps) 30sec Bit Rate (bps) 30sec Max Bit Rate (bps) 30sec Max Bit Rate (bps) http ipfix ssl snmp ping ssh Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 46 of 105

47 eigrp icmp isakmp ipsec unknown bgp ntp dns Total Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 47 of 105

48 Aggregate packet counts and byte counts since the last time the counters were cleared are displayed for each application identified by NBAR2. Incremental bit rates and maximum bit rates are also shown. The interval over which incremental bit rates are shown is based on the interface-level "load-interval" command. By default, the load interval is set to 5 minutes. You can adjust this. In the example above, the load interval has been adjusted down to 30 seconds. In addition, the output from the "show ip nbar protocol-discovery" command shows the total packet counts and byte counts since the last time the counters were cleared, as well as incremental total bit rates and maximum bit rates. Optional Procedure 3: View AVC data through the web interface You can gain application visibility via the web interface of Cisco ASR 1000 Series or Cisco CSR 1000V routers, as an alternative to the CLI. In order to access the web interface, you must first enable the secure web server within the ASR 1000 Series or CSR 1000V router. Step 1. Enable secure web access: ip http secure-server transport-map type persistent webui https-webui secure-server exit transport type persistent webui input https-webui These commands enable the built-in web server in the ASR 1000 Series or CSR 1000V router and enables access via the HTTPS protocol. You should always implement a cryptographically secure protocol, such as SSH for CLI access and HTTP for web access, to network infrastructure devices. Note: This deployment guide assumes that RSA key pairs for cryptographically secure protocols have already been generated via the crypto key generate rsa command, since this is necessary for accessing the ASR 1000 Series or CSR 1000V router via the SSH protocol for CLI access. The cryptographically secure protocols, including IPsec, require time synchronization of network devices. This deployment guide assumes that time synchronization has already been configured through one or more ntp server commands configured on the ASR 1000 Series or CSR 1000V routers. Step 2. View top applications from the ASR 1000 Series or CSR 1000V dashboard. The following figure shows an example of the dashboard screen that is presented after you log into an ASR 1000 Series router through the built-in web server Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 48 of 105

49 Figure 13. Cisco ASR 1000 Series web dashboard The dashboard includes a panel that shows the top applications seen, in terms of usage in bytes, by the overall ASR 1000 Series router over the past two hours. Step 3. View Application Visibility screen. For additional detail, you can click on Monitoring Services Application Visibility in the navigation panel on the left side of the screen, to bring up the Application Visibility screen. An example is shown in the following figure. Figure 14. Cisco ASR 1000 Series Application Visibility screen 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 49 of 105

50 The web interface provides you somewhat different application information than the CLI interface. You can specify the traffic direction through the drop-down menu at the top of the screen. The choices are ingress, egress, and both. This affects the graphical display directly below, as well as the textual information within the table further below. For example, when you select both directions, the usage percentage for each application is relative to the total of both ingress and egress traffic. However, if you select the ingress direction, the usage percentage for each application is relative to all ingress traffic only. In contrast, the CLI interface does not provide any percentage utilization statistics for applications. The time interval over which statistics are displayed within the web interface has three options the last two hours, the last 24 hours, or the last 48 hours. Statistics are presented as bytes sent, received, or total, as well as percentage usage. In contrast, statistics from the CLI include rates, specified in bits per second, based on the load-interval command configured for the interface, which can be set from between 30 and 600 seconds only. CLI statistics also include total byte and packet counts for applications viewed since the counters were last cleared. Step 4. Enable the NBAR2 HTTP-based Visibility Dashboard feature. The NBAR2 HTTP-based Visibility Dashboard feature enables a task on the ASR 1000 Series or CSR 1000V router that periodically collects NBAR2 discovery data. This feature is enabled through the following configurationlevel command: ip nbar http-services Step 5. View graphical data from the NBAR2 HTTP-based Visibility Dashboard. Graphical data from the NBAR2 HTTP-based Visibility Dashboard feature is visible through the following URL: or hostname>/flash/nbar2/home.html Figure 15 provides an example of the graphical output, which is similar to the Application Visibility screen shown in Figure 14: 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 50 of 105

51 Figure 15. Example output from the NBAR2 HTTP-based Visibility Dashboard Procedure 3: Configure flow monitoring and data export Although displaying application-level statistics on the Cisco ASR 1000 Series or the Cisco CSR 1000V router itself can prove useful, additional value can be gained by collecting and exporting application-level flow data in the form of Flexible NetFlow (FNF), Performance Monitor (PerfMon), and/or Application Response Time (ART) data to a centralized collector. All three types are supported by the ASR 1000 Series and CSR 1000V router. Note: Enabling AVC, ART, and PerfMon, along with Flexible NetFlow (FNF) data export will decrease the overall performance/throughput of the ASR 1000 Series or CSR 1000V router, due to the increased CPU required for these services. Enabling PerfMon with FNF data export has the greatest impact on performance. You should first determine if the application visibility needs of your deployment require enabling PerfMon or ART on the ASR 1000 Series or CSR 1000V routers functioning as VPN gateways, or whether just FNF with AVC is all that is required. The NetFlow collector used for this guide is provided by a Cisco partner, LiveAction ( through their LiveNX network performance and analytics platform. The LiveNX platform can automatically provision the FNF, PerfMon, and ART configurations onto various network equipment, including ASR 1000 Series and CSR 1000V routers. This guide assumes that LiveNX is already running within the network and that the ASR 1000 Series and CSR 1000V routers have been discovered. LiveNX Enterprise Edition, Version 7.0.1, was used for this guide. Export of flow information was enabled only for the ASR 1000 Series routers in the private network for this deployment guide Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 51 of 105

52 Step 1. Navigate to the flow configuration screen for the desired ASR 1000 series router. From the main window of the LiveNX client, highlight the ASR 1000 Series router for which you want to enable flow monitoring. Right-click on the device to bring up the drop-down menu, and select Flow. Right-click again and select Configure Flow from the submenu (see Figure 16). Figure 16. Navigating to flow configuration on LiveAction Note: Enable Flow Polling must be checked for the device to receive flow information. This will bring up a window similar to that shown in Figure 17. You can then select which interfaces to configure for flow monitoring and the type of flow monitoring desired. Figure 17. Enabling flow monitoring on a device 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 52 of 105

53 Step 2. Select the interfaces to be monitored. From the Flow Configuration screen, select the interfaces to be monitored, then select the type of monitoring you want. Your choices are as follows: Traffic Statistics Flexible NetFlow (FNF): This will configure a traditional FNF flow-record and flow-monitor configuration to the ASR 1000 Series platform. It will apply the flow monitor in both the inbound and outbound directions on the interface(s). A flow exporter pointing to the LiveAction server is also automatically provisioned. Application Response Time (AVC): This will configure an AVC-specific flow-record and flow-monitor configuration to the ASR 1000 Series platform and apply the PerfMon policy in both the inbound and outbound directions on the interface(s). A flow exporter pointing to the LiveAction server is also automatically provisioned. Voice/Video Performance (Medianet): This will configure a media-specific flow-record and flow-monitor configuration to the ASR 1000 Series platform and apply the PerfMon policy in both the inbound and outbound directions on the interface(s). A flow exporter pointing to the LiveAction server is also automatically provisioned. Note: Selecting either Application Response Time (AVC) or Voice/Video Performance (Medianet) causes the other to also be selected. A combined unified PerfMon policy with both policies is created and applied in the inbound and outbound directions on the interface or interfaces. Step 3. Save the flow configuration. Click the Save to Devices button when you are done selecting interfaces and flow-monitoring configuration. This will automatically generate the NetFlow and PerfMon configuration for the ASR 1000 Series router and provision it to the router. Step 4. Copy the running configuration to the startup configuration You will be prompted whether you want to save the running configuration, with the new commands provisioned on the platform, to the startup configuration. Select Yes to save the flow configuration to the startup configuration. An example of the configuration provisioned to an ASR 1000 Series router when selecting all three types of flow monitoring within LiveNX is shown in Appendix C. Procedure 4. View flow-monitoring information via the LiveNX client To view flow information collected from the ASR 1000 Series routers, select the name of the router from within the LiveNX client. Then select the Flow tab for the device (see Figure 18) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 53 of 105

54 Figure 18. Displaying flow information within LiveAction The bottom part of the display shows a high-level overview of the flows passing through the ASR 1000 Series router. The top part displays information regarding individual flows. The LiveNX client provides various tabs that allow you to narrow down to the types of flows of interest to you, and to pause the display while investigating an individual flow. You can inspect individual flows by highlighting the source and destination endpoints from the graphical display (see Figure 19) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 54 of 105

55 Figure 19. Highlighting an individual flow within LiveAction For example, application response time information for individual client flows reaching individual servers can be collected by enabling Application Response Time (AVC) monitoring within LiveAction. Such information may be useful when positioning web-based services in the AWS VPC that are accessed from inside an organization. Procedure 5. View flow-monitoring information via the LiveAction web interface Flow information can also be viewed by running ad hoc reports from the LiveAction web interface. Step 1. Log into the LiveAction server through a browser. When you login to the LiveAction server through the web interface, you will be taken to the main dashboard. Step 2. Navigate to the View Reports screen View Reports screen. An example is shown in Figure 20 below: e 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 55 of 105

56 Figure 20. LiveAction View Reports screen Step 3. Generate an ad-hoc application report. Select Flow Applications Application to bring up the report template. An example is shown in Figure 21. Figure 21. LiveAction application report template 2018 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 56 of 105

57 Step 4. Select the report parameters. From the report template, select the time range, the devices, the interfaces, and the flow direction for the report. For this example, the time range selected is 15 minutes. The device selected is ASR1K-1, since traffic between the private network and the VPCs traverse this VPN gateway. The interface selected is the back-side interface (GigabitEthernet0/0/2), since traffic is unencrypted on this interface. The flow direction is selected through the Advanced settings to be both inbound and outbound. Step 5. Execute the report. Click on the Execute button at the bottom of the report template screen to execute the ad-hoc report. An example of the output of the report is shown in Figure 22. Figure 22. Example output from applications ad hoc report The ad-hoc application report shows the total bytes and packets seen over the time interval for each application in the table at the bottom of the report. Additionally, the average bit rate and average packet rate over the time interval are also displayed in the table. The graph at the top of the report displays the bit rate of each application in smaller increments over the time range of the report, as opposed to just an average bit rate. Appendix A: Design considerations The following sections discuss some of the considerations and choices behind the design for this deployment guide. Positioning of the VPN gateway in the private network You have multiple choices regarding the positioning of the Cisco ASR 1000 Series Routers (functioning as dedicated VPN gateways) in the private network. For example, the VPN gateways can be positioned in the Internet edge of the private network, as shown in the figure below. Note: Redundancy is not shown in order to simplify the following figures Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 57 of 105

58 Figure 23. VPN gateway deployed in the Internet edge An advantage of this design is that all Internet-related connectivity is isolated to the Internet edge module of the private network. The Internet edge firewall can be used to control inbound access to the VPN gateway from the Internet. Optionally, unencrypted traffic from the back-side of the VPN gateway can be sent through the Internet edge firewall for stateful inspection and access control. Alternatively, you can position the VPN gateway in the data center of the private network, as shown in the following figure. Figure 24. VPN gateway deployed in the data center With the positioning of the VPN gateway in the data center, Internet connectivity could still be provided at the Internet edge. Alternatively, separate Internet connectivity could be provisioned directly in the data center, used solely for establishing IPsec VPN connectivity to the AWS VPC. The organization s security policy may determine whether all Internet connectivity passes through the Internet edge, or if separate Internet connectivity, provisioned directly to the data center, is allowed. When positioning the VPN gateway within either the Internet edge or the data center, the organization s security policy may also determine whether the Cisco ASR 1000 Series Routers must be placed behind a stateful firewall that may also implement Network Address Translation (NAT). Alternatively, the security policy of the organization may allow the ASR 1000 Series routers to be placed in parallel to the stateful firewall, and therefore may not require NAT Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 58 of 105

59 The design implemented within this deployment guide, shown in Figure 2 above, implements a pair of ASR 1000 Series routers, functioning as dedicated VPN gateways, in the data center of the private network. Separate Internet connectivity within the data center is not implemented for this deployment guide. Instead, the VPN gateways utilize the Internet connectivity provisioned in the Internet edge of the private network to establish IPsec VPN connections to the AWS VPC. Because of the design choice in this deployment guide, NAT is implemented at the Internet edge firewall. Network Address Translation (NAT) Because the VPN gateways are placed behind the Internet Edge firewall, the design followed in this deployment guide uses 1:1 NAT from the front-side interface (GigabitEthernet0/0/0) IP addresses of the Cisco ASR 1000 Series Routers to public IP addresses (IP addresses routable on the Internet) implemented on the stateful firewall in the Internet edge. The NAT configuration of the firewall is not shown in this deployment guide. The inbound access-control policy of the outside interface of the stateful firewall prevents any inbound connections initiated from outside the private network. The stateful firewall allows reverse connections (User Datagram Protocol [UDP] 500 for the Internet Key Exchange (IKE) protocol and UDP 4500 for NAT-Traversal [NAT-T]) initiated from the ASR 1000 Series routers within the private network, back into the private network. High availability You have multiple design options for implementing high availability in the private network and the AWS VPC, including redundant VPN gateways and redundant IPsec VPN connections. Depending on the choice of VPN gateway platform, these include active/active or active/standby configurations. This section discusses the specific high-availability design implemented for this deployment guide. High availability in the private network data center Although a single Cisco ASR 1000 Series Router can provide redundancy in case of a single IPsec VPN tunnel failure, it cannot provide high availability in case of the failure of the ASR 1000 Series router itself. For this deployment guide, high availability is implemented through a redundant pair of active/active ASR 1000 Series routers, dedicated as VPN gateways, deployed in the private network data center. Figure 25 shows the details regarding the logical connectivity of the ASR 1000 Series routers to the data center distribution-layer switches and firewalls. A pair of Cisco Nexus 5000 Series Layer 3 switches (N5K-1 and N5K-2) serves as the data center distribution-layer switches. An active/standby pair of ASAv firewalls serve as data center firewalls, dedicated for the VPC connectivity Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 59 of 105

60 Figure 25. High availability details in the private network data center Each of the ASR 1000 Series routers is configured to establish two IPsec VPN connections, each via a virtual tunnel interface (VTI) configuration to the AWS VGW. This is because each AWS VPN connection automatically establishes two IPsec VPN connections to the same IP address corresponding to an AWS customer gateway, and each ASR 1000 Series router functions as an AWS customer gateway. In other words, each ASR 1000 Series router supports a level of resiliency through two IPsec connections. Both IPsec VPN tunnels (Tunnel1 and Tunnel2) on each ASR 1000 Series router are normally established and up (however, AWS may periodically perform maintenance that results in one of the two IPsec VPN tunnels being temporarily down). Since both ASR 1000 Series routers are active, a total of four IPsec VPN tunnels between the private network data center and the AWS VPC are normally established and operational. BGP routing is configured between the tunnel interfaces of the ASR 1000 Series routers to the AWS VPC. Additionally, each of the ASR 1000 Series routers is configured to establish one IPsec VPN connection, via a VTI configuration (Tunnel100), to one of the CSR 1000V routers in the AWS Transit VPC. Since both ASR 1000 Series routers are active, a total of two IPsec VPN tunnels between the private network data center and the AWS Transit VPC are normally established and operational. Again, BGP routing is configured between the tunnel interfaces of the ASR 1000 Series routers to the AWS Transit VPC Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 60 of 105

61 Each of the ASR 1000 Series routers has two GigabitEthernet interfaces. The front-side interfaces (GigabitEthernet0/0/0) are configured as the source interfaces for the IPsec protected VTIs to AWS. The front-side interfaces of both ASR 1000 Series routers are each connected to one of the Cisco Nexus 5000 Series Switches via a Layer 2 interface. The Layer 2 interfaces are part of the same VLAN (VLAN 20) that extends across the two Cisco Nexus 5000 Series Switches via an EtherChannel connection consisting of two TenGigabitEthernet interfaces in a logical port-channel configuration. A Hot Standby Router Protocol (HSRP) group (20) is configured between the two Layer-3 switched virtual interfaces (SVIs) that provide the Layer 3 connectivity for the VLAN, configured on the Cisco Nexus 5000 Series switches. The priority of N5K-1, which connects to ASR1K-1, is increased, so that N5K-1 is the active router in the HSRP pair. This is because ASR1K-1 serves as the preferred path to the AWS VPC, as discussed in the following section. Active routing is not configured on the front-side interfaces of the ASR 1000 Series routers. Instead, each ASR 1000 Series router is configured with static routes corresponding to the publically routable IP addresses of the AWS VPN gateways and the CSR 1000Vs in the Transit VPC, to which each ASR 1000 Series router establishes IPsec connections. All static routes on each ASR 1000 Series router point to the shared virtual IP address of the HSRP group configured on the Cisco Nexus 5000 Series routers. This configuration also ensures that traffic within the private network is never accidently routed through the ASR 1000 Series routers. The back-side interfaces (GigabitEthernet0/0/2) of each of the ASR 1000 Series routers are where the unencrypted traffic from AWS enters the private network. The back-side interfaces of both ASR 1000 Series routers are each connected to a VLAN (VLAN 60) that spans both Cisco Nexus 5000 Series distribution-layer switches via the EtherChannel trunk. Also connected to the VLAN is the outside (GigabitEthernet0/0) interface of an active/standby pair of ASAv firewalls. No SVI interface is defined on the distribution-layer Nexus 5000 Series switches. Hence, VLAN 60 on the Nexus 5000 Series switches provides Layer 2 connectivity between the back-side interfaces (GigabitEthernet0/0/2) of each of the ASR 1000 Series routers and the outside (GigabitEthernet0/0) interface of the active/standby ASAv firewall pair. The inside (GigabitEthernet0/1) interface of the active/standby pair of ASAv firewalls also connect to the Nexus 5000 Series switches via another VLAN (VLAN 70). Each Nexus 5000 Series switch is configured with an SVI that provides the Layer 3 connectivity for the VLAN. Under normal conditions the active firewall is connected to N5K-1 whereas the standby firewall is connected to N5K-2. This is because ASR1K-1 serves as the preferred path to the AWS VPC, as discussed in the following section. Note: For this deployment guide, both ASAv firewalls were deployed on a single VMware ESXi 6.5 server. Therefore, the failover interface (GigabitEthernet0/8) is internal to the ESXi server, and not connected to the Nexus 5000 Series distribution-layer switches. In a production deployment, you should deploy each ASAv firewall on a separate ESXi server. This may require the failover connection to also be trunked via a separate VLAN between the Nexus 5000 Series distribution-layer switches Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 61 of 105

62 Active routing, using the Enhanced Interior Gateway Routing Protocol (EIGRP), is implemented across the Nexus 5000 Series switches, the ASAv active/standby firewall pair, and the back-side (GigabitEthernet0/0/2) interfaces of both ASR 1000 Series routers, and the rest of the private network. Routes learned via BGP from the AWS VPC are redistributed into EIGRP on both ASR 1000 Series routers. The routing metrics for routes redistributed into EIGRP are tuned such that ASR1K-1 advertises a path to the AWS VPC, which is always preferred over the path advertised by ASR1K-2. This provides both redundancy and ensures that the path to the AWS VPC is always through ASR1K-1 to meet the requirements for symmetric routing. Routes learned via EIGRP from the private network are redistributed into BGP. A route map is used to filter the routes that are redistributed from EIGRP to BGP. High availability in the AWS VPC and Transit VPC High Availability within the AWS VPC using the native IPsec VPN service is implemented by provisioning two AWS VPN connections. Each AWS VPN connection maps to a unique publicly routed IPv4 address, logically represented as an AWS customer gateway. Each ASR 1000 Series router corresponds to one of the two AWS customer gateways. However, since the ASR 1000 Series routers sit within the private network, they only have private IPv4 addresses. Because there are two ASR 1000 Series routers, port address translation (PAT) cannot be implemented at the Internet Edge firewall. Therefore, static 1:1 NAT translations are configured, mapping the frontside interfaces (GigabitEthernet0/0/0) of each of the ASR 1000 Series routers to publicly routed IPv4 addresses on the Internet Edge firewall. In the VPC with native IPsec VPN service, AWS provides hardware redundancy. Each AWS VPN connection consists of two IPsec tunnels. Each IPsec tunnel originates from a unique publicly routable IPv4 address owned by AWS. The combination of two IPsec tunnels per AWS VPN connection, along with two AWS VPN connections, results in a total of four IPsec tunnels between the VPN and the private network. In the AWS Transit VPC, redundancy is achieved through two CSR 1000V routers. Each of the CSR 1000V routers has a public IP address. IPsec VPN connections are established from each ASR 1000 Series router in the private network to each of the CSR 1000V routers in the AWS Transit VPC. In turn, each CSR 1000V router in the Transit VPC established two IPsec VPN connections to the spoke VPCs, which implement the AWS-native IPsec VPN service. Symmetric routing for AVC You should note that multiple paths do not necessarily mean load-balancing of traffic across both ASR 1000 Series routers, or across both of the IPsec VPN connections from each ASR 1000 Series router, when using BGP. BGP routing will select a single path between autonomous systems (in each direction) unless BGP multipath is supported and configured. A single path in each direction does not necessarily mean symmetric routing either. As shown in the figure below, traffic to the VPC could pass through one ASR 1000 Series router while return traffic from the VPC could pass through the other ASR 1000 Series router Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 62 of 105

63 Figure 26. Asymmetric routing across VPN connections Asymmetric routing is not necessarily a bad thing. However, in some situations it is undesirable. In particular, Application Visibility and Control (AVC) running on the ASR 1000 Series routers may not function correctly with asymmetric routing. In order to correctly classify some applications, AVC must have visibility into the first few packets of the flow in both directions. You can ensure symmetric routing through several methods. For this deployment guide, when redistributing routes learned via BGP into EIGRP, the routing metrics are modified in order to make one of the ASR 1000 Series routers more favorable from the viewpoint of EIGRP. EIGRP is the interior gateway protocol (IGP) implemented in the private network for this deployment guide. All internal routers and Layer 3 switches will then send traffic destined to the VPC to only one of the ASR 1000 Series routers. In order to make the path to the routes available through one of the ASR 1000 Series routers more favorable to the AWS VGW at the VPC and to the CSR 1000V routers in the Transit VPC, BGP AS prepending is implemented on the ASR 1000 Series routers. The effect of this technique is shown in Figure 27, below Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 63 of 105