VMware Cloud on AWS Networking and Security. 5 September 2018 VMware Cloud on AWS

Transcription

1 VMware Cloud on AWS Networking and Security 5 September 2018 VMware Cloud on AWS

2 You can find the most up-to-date technical documentation on the VMware website at: If you have comments about this documentation, submit your feedback to VMware, Inc Hillview Ave. Palo Alto, CA Copyright 2018 VMware, Inc. All rights reserved. Copyright and trademark information. VMware, Inc. 2

3 Contents About VMware Cloud on AWS Networking and Security 4 1 Determining Whether Your SDDC Networking is Backed by NSX for vsphere or NSX-T 5 2 Features Supported with NSX for vsphere and NSX-T 6 3 About VMware Cloud on AWS Networking With NSX for vsphere 7 Use the Configure MGW VPN Wizard to Configure a Management VPN and Gateway 7 Configuring Compute Gateway Networking 17 Using AWS Direct Connect with VMware Cloud on AWS 32 4 About VMware Cloud on AWS Networking with VMware NSX-T 37 Configure NSX Roles 37 Configuring VMware Cloud on AWS Networking Using NSX-T 37 Configure Connectivity to the On-Premises Data Center 38 Configure Management Gateway Networking 43 Configure Compute Gateway and Workload Networking 46 Configure Monitoring and Troubleshooting Features 58 VMware, Inc. 3

4 About VMware Cloud on AWS Networking and Security The VMware Cloud on AWS Networking and Security provides information about configuring networking and security for VMware Cloud on AWS. This information guides you in how to configure networking for both NSX for vsphere and NSX-T based SDDCs. Intended Audience This information is intended for anyone who wants to use VMware Cloud on AWS to create an SDDC that has the basic features required to run workloads in the cloud and can serve as a starting point for your exploration of additional features and capabilities. The information is written for readers who have used vsphere in an on-premises environment and are familiar with virtualization concepts. In-depth knowledge of vsphere or Amazon Web Services is not required. VMware, Inc. 4

5 Determining Whether Your SDDC Networking is Backed by NSX for vsphere or NSX-T 1 Your VMware Cloud on AWS SDDC uses networking backed by either VMware NSX for vsphere or VMware NSX-T. The networking configuration steps and capabilities differ between these two versions. You can determine which type of networking your SDDC uses by logging into the VMC Console, and clicking on the card for the SDDC. If there is a Network tab present as shown below, the SDDC uses NSX for vsphere. If there is a Networking & Security tab present as shown below, the SDDC uses NSX-T. VMware, Inc. 5

6 Features Supported with NSX 2 for vsphere and NSX-T SDDCs backed by NSX for vsphere and those backed by NSX-T support different sets of features. Table 2 1. Features supported with NSX for vsphere and NSX-T. Feature or Solution NSX for vsphere NSX-T Policy-based IPsec VPN Yes Yes Route-based IPsec VPN No Yes Direct Connect for All Traffic No (ESXi management and vmotion traffic only) Yes L2 VPN Yes Yes Edge Firewall Yes Yes Logical Networks, DHCP, DNS, NAT Yes Yes Distributed Firewall No Yes IPFIX, Port Mirroring No Yes Management Appliance and ESXi access to and from the overlay network and AWS VPC No Yes Multiple Clusters Yes Yes Multiple Availability Zone Stretched Clusters Yes Yes Bi-directional migration with vmotion Yes No VMware Site Recovery Manager Yes Yes VMware Hybrid Cloud Extension Yes Yes Horizon Yes No 3rd Party Solutions - Storage Partners Yes No 2nd Party Solutions - vra, vrops Yes Yes VMware, Inc. 6

7 About VMware Cloud on AWS Networking With NSX for vsphere 3 Information in this section explains how to configure networking for an SDDC based on NSX for vsphere. Your SDDC might be provisioned with networking features based on either NSX for vsphere or NSX-T. This section describes the configuration of SDDCs whose networking is backed by NSX for vsphere. This chapter includes the following topics: Use the Configure MGW VPN Wizard to Configure a Management VPN and Gateway Configuring Compute Gateway Networking Using AWS Direct Connect with VMware Cloud on AWS Use the Configure MGW VPN Wizard to Configure a Management VPN and Gateway A new SDDC includes a logical network (the management network) and an NSX Edge gateway that controls access to the network. To provide secure communications between this network and your onpremises management network, use the Configure MGW VPN wizard to create virtual private networks (VPNs) in each location, and configure the management gateway to connect them. The wizard guides you through the steps to create a VPN in the SDDC, configure the management gateway with firewall rules, and specify DNS server addresses for the management network. Your networking team can configure the on-premises end of the management VPN using information you download from the SDDC, then connect it to the SDDC through the management gateway and test network connectivity Note In addition to creating a management VPN, you can also create a compute VPN and an AWS Direct Connect connection between your on-premises data center and AWS services. For information about how to create these connections, see the Networking and Security Guide. Set Management Gateway Firewall Rules By default, the firewall for the management gateway is set to deny all inbound and outbound traffic. Add additional firewall rules to allow traffic as needed. VMware, Inc. 7

8 If you have configured a management gateway VPN, you can use the Firewall Rules Accelerator to create the firewall rules necessary for communication over the VPN. See Use the Firewall Rules Accelerator to Set Up Firewall Rules. Note In order to access vcenter Server in your SDDC, you must set a firewall rule to allow traffic to the vcenter Server. When access to vcenter Server is blocked, the topology diagram on the Network tab shows a dotted line between the internet and the management gateway. After you have added a firewall rule to allow access to vcenter Server, the diagram shows a solid line between the internet and the management gateway. VMware, Inc. 8

9 2 Click View Details on the SDDC card. 3 Click Network. 4 Under Management Gateway, click Firewall Rules. 5 Click Add Rule. 6 Enter the rule parameters. Option Rule Name Action Source Enter a descriptive name for the rule. The only action available for management gateway firewall rules is Allow. Enter or select one of the following options for the source: An IP address, IP address range, or any to allow traffic from that address or address range vcenter to allow traffic from your SDDC's vcenter Server. ESXi Management Only to allow traffic from your SDDC's ESXi management. Workload VM IP address or address range to allow traffic from management VMs. Management VM IP address or address range to allow traffic from workload VMs. Destination Enter or select one of the following options for the destination: An IP address, IP address range, or any to allow traffic to that address or address range vcenter to allow traffic to your SDDC's vcenter Server. ESXi Management Only to allow traffic to your SDDC's ESXi management. Workload VM IP address or address range to allow traffic to management VMs. Management VM IP address or address range to allow traffic to workload VMs. Service Select one of the following to apply the rule to: Any (All Traffic) ICMP (All ICMP) HTTPS (TCP 443) - applies only to vcenter Server as a destination. SSO (TCP 7444) - applies only to vcenter Server as a destination. Provisioning (TCP 902) - applies only to ESXi Management Only as a destination. Remote Console (TCP 903) applies only to ESXi Management Only as a destination. Ports The port that the selected service uses for communication. 7 Use the up and down arrow icons to change the order of the firewall rules. Firewall rules are applied in order from top to bottom. The following graphic shows an example firewall rule that allows all traffic to reach vcenter Server from a particular IP address. VMware, Inc. 9

10 See Example Management Gateway Firewall Rules for more examples of firewall rules for specific use cases. Example Management Gateway Firewall Rules Some common firewall rule configurations include opening access to the vsphere Client from the internet, allowing access to vcenter Server through the management VPN tunnel, and allowing remote console access. Commonly Used Firewall Rules The following table shows the Service, Source, and Destination settings for commonly-used firewall rules. Table 3 1. Commonly-Used Firewall Rules Use Cases Service Source Destination Provide access to vcenter Server from the internet. Use for general vsphere Client access as well as for monitoring vcenter Server HTTPS public IP address vcenter Provide access to vcenter Server over VPN tunnel. HTTPS IP address or CIDR block from on-premises data center vcenter Required for Management Gateway VPN, Hybrid Linked Mode, Content Library. Provide access from cloud vcenter Server to onpremises services such as Active Directory, Platform Services Controller, and Content Library. Any vcenter IP address or CIDR block from on-premises data center. Provisioning operations involving network file copy traffic, such as cold migration, cloning from on-premises VMs, snapshot migration, replication, and so on. Provisioning IP address or CIDR block, either public or from an onpremises data center connected by a VPN tunnel ESXi Management VMRC remote console access Required for vrealize Automation Remote Console IP address or CIDR block, either public or from an onpremises data center connected by a VPN tunnel ESXi Management VMware, Inc. 10

11 Table 3 1. Commonly-Used Firewall Rules (Continued) Use Cases Service Source Destination vmotion traffic over VPN Any ESXi Management IP address or CIDR block from on-premises data center Ping traffic to vcenter Server for network troubleshooting. ICMP (All ICMP) IP address or CIDR block, either public or from an onpremises data center connected by a VPN tunnel vcenter Ping traffic to ESXi management network for network troubleshooting ICMP (All ICMP) IP address or CIDR block, either public or from an onpremises data center connected by a VPN tunnel ESXi Management Firewall Rules That Enable Access to Workload VMs from Management VMs The following table shows the Service, Source, and Destination settings for firewall rules that enable access to workload VMs from VMs on the management network. Table 3 2. Firewall Rules to Enable Traffic Between Management VMs and Workload VMs Use Cases Service Source Destination Provide access to vcenter Server from a workload VM. HTTPS TCP ICMP Workload VM IP address or address range vcenter Provide access to a workload VM from vcenter Server. Any vcenter Workload VM IP address or address range Provide access to ESXi from a workload VM. HTTPS ICMP Workload VM IP address or address range ESXi TCP port 902 (Provisioning) TCP port 903 (Remote Console) TCP port 8000 (vmotion) Provide access to a workload VM from ESXi. Any ESXi Workload VM IP address or address range VMware, Inc. 11

12 Table 3 2. Firewall Rules to Enable Traffic Between Management VMs and Workload VMs (Continued) Use Cases Service Source Destination Provide access to NSX manager from a workload VM. HTTPS Workload VM IP address or address range NSX Provide access to a workload VM from NSX manager. Any NSX Workload VM IP address or address range Important AWS imposes a limit of 50 custom routes for workload logical networks in a VPC. Because these routes are allocated on a first-come, first-served basis, an SDDC can exceed the AWS custom route limit for its VPC. When this happens, workload access to management network addresses can fail with an error of the form: vmcd: [ERROR] errorcode:vmcd00327 [ ] Aws route table limit exceeded when adding /24 in rtb-02382b7a (routetables.onpremtable) Set Management Gateway DNS Set a DNS server to allow the management gateway, ESXi hosts, and management VMs to resolve fullyqualified domain names (FQDNs) to IP addresses on the management network. Unless you intend to use only static routing, the management network requires a DNS service that can resolve IP addresses on both sides of the management gateway to VM FQDNs. You must specify the IP address of at least one DNS server when you configure the management gateway. If you specify an optional backup DNS server, be sure that both servers are configured identically. 2 Click View Details on the SDDC card. 3 Click Network. VMware, Inc. 12

13 4 (Optional) Modify default DNS settings for the management VPN. The management VPN is created with two DNS servers configured to resolve names to addresses on the public Internet. You can change the DNS server addresses and the name resolution scope. a b Under Management Gateway, click DNS. Modify the DNS server addresses. Click Edit and enter the IP addresses for DNS Server 1 and, optionally, DNS Server 2. c Choose a scope for DNS name resolution. By default, the gateway DNS is configured to resolve names to addresses on the public Internet (Public IP resolvable from Internet). To limit the scope to addresses on the management VPN. Select Private IP resolvable from VPN and click Save. This configuration change applies to both DNS Server 1 and DNS Server 2. IPsec VPN Settings Reference The on-premises end of any IPsec VPN must be configured to reflect the settings you specified for the SDDC end of that VPN. Information in the following tables summarizes the available SDDC IPsec VPN settings. Some of the settings can be configured. Some are static. Use this information to verify that your on-premises VPN solution can be configured to match the one in your SDDC. Choose an on-premises VPN solution that supports all the static settings and any of the configurable settings listed in these tables. Phase 1 Internet Key Exchange (IKE) Settings Table 3 3. Configurable IKE Phase 1 Settings Attribute Allowed Values Recommended Value Protocol IKEv1, IKEv2 any Encryption Algorithm AES-256, AES-GCM, AES any Hashing Algorithm SHA-1, SHA-256 any Diffie Hellman DH Groups 2, 5, DH Group 14 Table 3 4. Static IKE Phase 1 Settings Attribute ISAKMP mode ISAKMP/IKE SA lifetime IPsec Mode IKE Authentication Value Main mode (Disable aggressive mode) seconds Tunnel Pre-Shared Key VMware, Inc. 13

14 Phase 2 Settings Table 3 5. Configurable IKE Phase 2 Settings Attribute Allowed Values Recommended Value Encryption Algorithm AES-256, AES-GCM, AES any Perfect forward secrecy (PFS) Enabled, Disabled any Diffie Hellman DH Groups 2, 5, DH Group 14 Table 3 6. Static IKE Phase 2 Settings Attribute Hashing Algorithm Tunnel Mode SA lifetime Value SHA-1 Encapsulating Security Payload (ESP) 3600 seconds (one hour) On-Premises IPsec VPN Configuration From the Network tab of your SDDC under Management Gateway, you can download a Remote VPN Config File that lists all settings of the SDDC side of the management VPN. Use the settings in that file to configure the on-premises side of the management VPN. Note Do not configure the on-premises side of the VPN to have an idle timeout (for example, the NSX Session idle timeout setting). On-premises idle timeouts can cause the VPN to become periodically disconnected. Mapping NSX Parameters to VMC Console VPN Parameters The table below matches terms for VPN parameters used in NSX Edge configuration to the terms used in the VMC Console. NSX Property Name Name Peer ID Peer Endpoint Peer Subnets Local ID Local Endpoint Local Subnets Encryption Algorithm Perfect Forward Secrecy Authentication VMC Console Property Name VPN Name On-prem Gateway IP On-prem Gateway IP On-prem Network Uplink SNAT (not a user-entered value) Uplink IP (not a user-entered value) Local Network Encryption Perfect Forward Secrecy PSK (not a user-entered value) VMware, Inc. 14

15 NSX Property Name Diffie Hellman Group Pre-Shared Key Enabled VMC Console Property Name Diffie Hellman Pre-Shared Key True (not a user-entered value) Create a Management VPN in your SDDC To create the management VPN, configure an IPsec VPN in the SDDC and another one in your onpremises datacenter. The management gateway connects these two VPNs and provides a common set of firewall rules and DNS services. 2 On Network tab of your SDDC, click ACTIONS > Configure Management Gateway. 3 Complete the Management Gateway VPN configuration. Parameter VPN Name Remote Gateway Public IP Remote Gateway Private IP Remote Networks Local Gateway IP Local Network Encryption Perfect Forward Secrecy Diffie Hellman Enter a name for the VPN. Enter the IP address of your on-premises gateway. If your on-premises gateway is behind NAT, provide the private IP address of the gateway. Enter the address of your on-premises management network. Displays the public IP address of the management gateway. This is not an editable field. Displays the CIDR block of the management subnet for the management gateway. This is not an editable field. Select AES-256. Select Enabled Select a Diffie Hellman group. Ensure that you use a group that your on-premises VPN gateway supports. Pre-Shared Key Enter a pre-shared key. The key is a string with a maximum length of 128 characters that is used by the two ends of the VPN tunnel to authenticate with each other. Click SAVE to save this configuration and create the VPN. After the system creates the VPN in the SDDC, you can click ACTIONS to Edit or Disable the VPN. When the VPN has a status of Connected, you can click VPN Status Detail to view VPN tunnel status and statistics. 4 Download the SDDC management VPN configuration details. Under Remote VPN Config File, click Download to download a configuration file that you can use when you configure the on-premises side of this VPN. VMware, Inc. 15

16 What to do next Configure the on-premises side of the management VPN. Use the Firewall Rules Accelerator to Set Up Firewall Rules The Firewall Rule Accelerator helps create appropriate firewall policies in the management gateway. This enables communication over the IPsec VPN tunnel with key management infrastructure components such as vcenter Server and ESXi from your on-premises data center. After you set up an IPsec VPN for the Management Gateway, you can use the Firewall Rules Accelerator to quickly set up the firewall rules. Setting these rules is a prerequisite for using Hybrid Linked Mode, performing workload migrations, and many other tasks. Prerequisites Configure a Management Gateway VPN. See Create a Management VPN in your SDDC. 2 Navigate to the Network tab of your SDDC. 3 Under Management Gateway, click IPsec VPNs. 4 Click Firewall Rule Accelerator. The Firewall Rules Accelerator opens. 5 From the VPN (Remote Network) drop-down menu, select the remote (on-premises) network that you want to create firewall rules for. The Firewall Rules Accelerator displays the rules that will be created. 6 Click Create Firewall Rules to create these rules. After the firewall rules are created, they are shown in the Management Gateway Firewall Rules list. You can edit or delete any rules as needed. If you change your remote VPN network, you can use the Firewall Rules Accelerator to create new firewall rules, but it does not update any already existing rules. You must delete or modify those rules manually. Change the Management Gateway FQDN Resolution You can change how the Management Gateway performs FQDN resolution. You can use a private IP, resolvable from the VPN you set up, or to use a public IP from the Internet. Prerequisites Set up the VPN for the Management Gateway. See Create a Management VPN in your SDDC. VMware, Inc. 16

17 2 Navigate to the Networking tab of your SDDC. 3 Under Management Gateway, click DNS and then Edit. 4 Select Private IP resolvable from VPN or Public IP resolvable from Internet and click Save. Configuring Compute Gateway Networking The compute gateway handles network traffic for your workload VMs. You can configure firewall rules, inbound NAT, VPN connections, DNS, and public IP addresses for your compute gateway. Create a Logical Network Create logical networks to provide network access to workload VMs. VMware Cloud on AWS supports two types of logical networks, routed and extended. Routed networks are the default type. These networks use the SDDC compute gateway as the default gateway. Routed networks have connectivity to other logical networks in the same SDDC and to external network services such as the SDDC firewall and NAT. Extended networks require a layer 2 Virtual Private Network (L2VPN), which provides a secure communications tunnel between an on-premises network and one in your cloud SDDC. Your SDDC starts with a single default logical network, sddc-cgw-network-1. You can use the HTML5 vsphere Client to create additional logical networks. 1 Log in to the vsphere Client for your SDDC. You cannot create logical networks using the vsphere Web Client. 2 Select Menu > Global Inventory Lists. 3 Select Logical Networks. 4 Click Add. 5 In the Name text field, enter a name for the logical network. VMware, Inc. 17

18 6 Select whether to create a routed network or an extended network. Option Routed Network A routed network is used for communication over an IPsec VPN or the internet. Set the following options: a In the CIDR Block text field, enter a CIDR block in xxx.xxx.xxx.0/yy format. b c Prefix length should be between 22 and 30, because your logical network must have no more than 1000 ports. (Optional) Select Enabled to enable DHCP. If you enable DHCP on a logical network and you have configured an onpremises DNS server, you must edit your compute gateway VPN to enable DNS queries to be correctly forwarded over the VPN. Select cgw-dnsnetwork as one of the local networks for the VPN. If you enabled DHCP, enter the domain name to use with VMs attached to this logical network in the DNS Domain Name text box. Extended Network A VMware Cloud on AWS extended network uses a layer 2 Virtual Private Network (L2VPN) to extend an on-premises network to one in your cloud SDDC. This extended network is a single subnet with a single broadcast domain, so you can migrate VMs to and from your cloud SDDC without having to change their IP addresses. See "Configure an Extended Network and Layer 2 VPN" in VMware Cloud on AWS Networking and Security. Important Workload logical networks must not overlap with the management network CIDR block. 7 Click OK. What to do next After you have created the logical network, you can attach VMs to it. See Attach a VM to or Detach a VM from a Logical Network. Attach a VM to or Detach a VM from a Logical Network You can connect and disconnect a single or multiple VMs from a logical network. 1 Log in to the vsphere Client for your SDDC. 2 Select Menu > Global Inventory Lists. 3 Select Logical Networks. 4 In the vcenter Server drop down menu, select the vcenter Server that manages the logical network you want to use. VMware, Inc. 18

19 5 Click next to the logical network name to select it. 6 Select whether to attach or detach VMs. Click Attach VM to attach VMs to the selected network. Click Detach VM to detach VMs from the selected network. 7 Select the virtual machine(s) you want to attach or detach, click >> to move them to the Selected Objects column, and click Next. 8 For each VM, select the virtual NIC you want to attach and click Next. 9 Click Finish. Set Compute Gateway Firewall Rules By default, the firewall for the compute gateway is set to deny all inbound and outbound traffic. Add additional firewall rules to allow traffic as needed. 2 Click View Details on the SDDC card. 3 Click Network. 4 Under Compute Gateway, click Firewall Rules. 5 Click Add Rule. 6 Enter the rule parameters. Option Rule Name Action Give the rule a descriptive name. Select Allow or Deny. VMware, Inc. 19

20 Option Source Destination Service Select the source for the network traffic. Enter an IP address, an IP address range, or Any if you want the rule to apply to all traffic. Select All Internet and VPN if you want the rule to apply to all traffic from the internet and the compute gateway VPN but not to traffic from the connected Amazon VPC. Select All Connected AWS VPC if you want the rule to apply to traffic from the connected Amazon VPC but not to traffic from the internet and the compute gateway VPN. Select the destination for the network traffic. Enter an IP address, an IP address range, or Any if you want the rule to apply to all traffic. Select All Internet and VPN if you want the rule to apply to all traffic to the internet and the compute gateway VPN but not to traffic to the connected Amazon VPC. Select All Connected AWS VPC if you want the rule to apply to traffic to the connected Amazon VPC but not to traffic to the internet and the compute gateway VPN. Select one of the following: Select Any to create a rule that applies to all traffic, regardless of protocol or port used. Select a specific service to create a rule that applies to that protocol and port. Select Custom TCP, Custom UDP, or Custom ICMP to create a rule that applies to a service and/or port that is not available in the dropdown menu. Ports If you selected a custom TCP, UDP, or ICMP service, enter the port number used by this service. 7 Use the up and down arrow icons to adjust the ordering of the firewall rules. Firewall rules are applied in order from top to bottom. Create a Compute VPN Configure a compute VPN to allow VMs in your SDDC to communicate securely with VMs in an onpremises data center or within an Amazon VPC. Create a compute gateway VPN allows you to deploy hybrid application architectures in which some VMs in the application are in your on-premises data center or on Amazon EC2, while others are in your cloud SDDC. Prerequisites Configuring a compute VPN requires the following: An on-premises router or firewall capable of terminating an IPsec VPN, such as Cisco ISR, Cisco ASA, CheckPoint Firewall, Juniper SRX, NSX Edge, or any other device capable of IPsec tunneling. The router or firewall should be configured with cryptography settings as described in IPsec VPN Settings Reference. VMware, Inc. 20

21 If your on-premises gateway is behind another firewall, allow IPsec VPN traffic to pass through the firewall to reach your device by doing the following: Open UDP port 500 to allow Internet Security Association and Key Management Protocol (ISAKMP) traffic to be forwarded through the firewall. Set IP protocol ID 50 to allow IPsec Encapsulating Security Protocol (ESP) traffic to be forwarded through the firewall. Set IP protocol ID 51 to allow Authentication Header (AH) traffic to be forwarded through the firewall. 1 Configure the Compute Gateway side of the tunnel. a b c d Log in to the VMC Console at Navigate to the Networking tab of your SDDC. Under Compute Gateway, click IPsec VPNs and then Add VPN. Complete the Compute Gateway VPN configuration. Parameter VPN Name Remote Gateway Public IP Remote Gateway Private IP Remote Networks Local Gateway IP Local Network Encryption Perfect Forward Secrecy Diffie Hellman Enter a name for the VPN. Enter the public IP address of your on-premises gateway. If your gateway device is behind NAT, enter the private IP address of your onpremises gateway. Enter the address of your on-premises compute network. Displays the IP address of the SDDC compute gateway. This is not an editable field. Select the logical network to connect to using this VPN. If the logical network uses DHCP and you have configured an on-premises DNS server, also select the cgw-dns-network to allow DNS requests to travel over the VPN. Select AES-256. Select Enabled. Select a Diffie Hellman group. Ensure that you use the same group in your onpremises VPN gateway settings. Pre-Shared Key Enter a pre-shared key. The key is a string with a maximum length of 128 characters that is used by the two ends of the VPN tunnel to authenticate with each other. e (Optional) Under VPN Peer Configuration, click Download to download a configuration file listing the configuration parameters needed to configure your on-premises gateway. VMware, Inc. 21

22 2 Configure the on-premises side of the tunnel. a Consult the documentation for your gateway or firewall device to learn how to configure it to match the VPN settings you've configured. Configuration of the gateway device in your on-premises data center might need to be performed by a member of your networking team. b If you selected as Local Network a non-default logical network that uses DHCP, configure the on-premises side of the tunnel of connect to local_gateway_ip/32 in addition to the Local Gateway IP address. This allows DNS requests to be routed over the VPN. When the VPN tunnel is configured, you should be able to verify connectivity in the VMC Console. Create a VPN Connection Between the Compute Gateway and an Amazon VPC If you need to connect VMs in your SDDC with resources in an Amazon VPC that isn't connected to your account using a cross-vpc ENI, you can create a VPN connection between your compute gateway and that VPC. If the Amazon VPC is connected to your VMware Cloud on AWS, you don't need to create this VPN connection to access it. Prerequisites To create this VPN connection, you need: A working SDDC in VMware Cloud on AWS An AWS account 2 Click View Details on the SDDC card. 3 Click Network. VMware, Inc. 22

23 4 Note the public IP address of the compute gateway as shown in the network system diagram. 5 Note the CIDR block for the logical network you want to connect to the VPN. 6 In another browser tab, log in to your AWS account. 7 If you don't already have a VPC and subnet you want to use, create them. a b c Go to and select Your VPCs. Click Create VPC. Enter a name and an IPv4 CIDR block for the VPC and click Yes, Create. d Click Subnets and click Create Subnet. VMware, Inc. 23

24 e f Enter a name for the subnet. Select the VPC for the subnet and click Yes, Create. 8 Create a Customer Gateway. a b c d Under VPN Connections, select Customer Gateways. Click Create Customer Gateway. Enter a name for the gateway. For the IP address, enter the IP address of your SDDC compute gateway that you noted in Step 4. 9 Create a Virtual Private Gateway and attach it to your VPC. a b c d Click Virtual Private Gateways and click Create Virtual Private Gateway. Enter a name for the Virtual Private Gateway, and click Yes, Create. Make sure that the Virtual Private Gateway is selected and click Attach to VPC. Select the VPC to attach the gateway to. 10 Create the VPN tunnel. Option Name tag Enter a name for the VPN connection. Virtual Private Gateway Select the Virtual Private Gateway you created in Step 9. Customer Gateway Select Existing and then select the Customer Gateway you created in Step 8 Routing Options Select Static. Static IP Prefixes Enter the CIDR block for the SDDC logical network that you noted in Step 5. VMware, Inc. 24

25 11 Click Yes, Create and then click Download Configuration. Option Vendor Platform Software Select Generic. Select Generic. Select Vendor Agnostic. 12 Open the configuration file and copy the Pre-Shared Key and the Virtual Private Gateway IP address. 13 In the VMC Console, create a VPN connection to the AWS Virtual Private Gateway as described in Create a Compute VPN. Include the Virtual Private Gateway IP and Pre-Shared Key as indicated in the screenshot below. 14 Verify that the tunnel comes up on the SDDC side by looking for the Connected status. 15 Verify that the tunnel comes up on the AWS side. a b c Go to and select VPN Connections. Select the VPN. Click Tunnel Details and check that the status is UP. 16 Add a route to your SDDC from the AWS console. a b c d e f Log in to the AWS console and select VPC. Select the route table for your VPC and click the Routes tab. Click Edit. Click Add another route. In the Destination text box, enter the CIDR block range for the logical network in your SDDC. In the Target field, select the Virtual Private Gateway you created. Configure an Extended Network and Layer 2 VPN A VMware Cloud on AWS extended network uses a layer 2 Virtual Private Network (L2VPN) to extend an on-premises network to one in your cloud SDDC. This extended network is a single subnet with a single broadcast domain, so you can migrate VMs to and from your cloud SDDC without having to change their IP addresses. In addition to supporting datacenter migration, an on-premises network extended with an L2VPN is useful for disaster recovery and dynamically engaging off-premise compute resources to meet an increase in demand (something often referred to as "cloud bursting"). VMware, Inc. 25

26 An L2VPN on the Compute Gateway can extend up to 25 of your on-premises networks. VMware Cloud on AWS uses NSX to provide L2VPN features in your cloud SDDC. Using NSX as your on-premises L2VPN solution can simplify configuration and operation, but you can use any supported layer 2 VPN solution in your on-premises datacenter. The VMware Cloud on AWS L2VPN feature supports extending VLAN and VXLAN networks. While there is no requirement for you to increase the MTU on the WAN segment of an L2VPN, L2VPN protocols require SSL (and TCP), so latency between your on-premises datacenter and your cloud SDDC can impact performance, as can the reduced bandwidth typical of L2VPN-extended networks. L2VPN is used to extend Virtual Machine networks. Your L2VPN is independent of the VMkernel networks used for migration traffic (ESXi management or vmotion), which use IPsec VPN or DX protocols. Important You cannot bring up an L2VPN tunnel until you have configured the L2VPN client and server and created an extended network that specifies the tunnel ID you assigned to the client. 1 Configure the L2 VPN Server in the SDDC The Compute Gateway in your cloud SDDC acts as the Layer 2 VPN server. Use the VMC Console to configure the server. 2 Configure and Enable the L2 VPN Client Configure a Layer 2 VPN client in your on-premises data center. 3 Create an Extended Network in Your SDDC and Bring Up the L2VPN Tunnel Before you can bring up an L2VPN tunnel, you must create an extended network that uses the tunnel ID you specified when configuring the L2VPN client. Configure the L2 VPN Server in the SDDC The Compute Gateway in your cloud SDDC acts as the Layer 2 VPN server. Use the VMC Console to configure the server. An L2VPN on the Compute Gateway can extend up to 25 of your on-premises networks. Use this procedure to configure the L2 VPN server from the VMC Console. 2 Navigate to the networking tab of your SDDC. 3 Under Compute Gateway, click L2 VPN and then Add VPN. 4 Configure Layer 2 VPN server settings. Option VPN Name Encryption Enter a name for the VPN. Select On (AES-128) to use AES128-GCM-SHA256 encryption or select Off (Null) to use NULL_SHA256. VMware, Inc. 26

27 Option Username Password Set the username that the L2 VPN client will use to connect to the VPN. Set the password that the L2 VPN client will use to connect to the VPN. The password must meet the following complexity requirements: The length must be a minimum of 12 characters. The length must be a maximum of 255 characters. The password must contain at least one of each of the following types of characters: upper case letters, lower case letters, numbers, and special characters. 5 Click Save. It might take some time to save the server configuration. Configure and Enable the L2 VPN Client Configure a Layer 2 VPN client in your on-premises data center. Prerequisites Ensure that the following requirements are met in your on-premises data center. Your on-premises data center must be running vsphere 5.0 or later. vsphere 5.1 or later is recommended. The source NSX Edge providing L2VPN client services must be NSX or later. NSX is recommended. Note The L2 VPNs section of Compute Gateway includes an option to download an installation package for a standalone NSX Edge appliance. Click Download under Remote Standalone Edge to open the Download VMware NSX for vsphere Standalone Edge page. Click the Documentation link on that page to access installation and configuration guidance for adding the appliance to your onpremises datacenter. VM networking can be configured to use a vsphere Standard Virtual Switch or Distributed Virtual Switch. A vsphere Distributed Virtual switch is recommended. If you use vsphere Distributed Virtual Switch, version or later is required. An uplink IP address is required for the NSX Edge instance that serves as the L2 VPN client. This address must be the Compute Gateway public IP. Create a firewall rule to allow HTTPS traffic from this IP address to the cloud SDDC. VMware, Inc. 27

28 u In your on-premises data center, configure an NSX Layer 2 VPN client. If your on-premises environment uses NSX Manager, follow these steps to configure a managed NSX Edge L2VPN client. a b c Add a Sub Interface to support the NSX Edge Add a VLAN Trunk to the sub interface. Configure the NSX L2VPN Client If your on-premises environment does not use NSX Manager, follow these steps to configure a standalone NSX Edge L2VPN client on either a Distributed Virtual Switch (DVS) or Standard Virtual Switch (VSS). a Configure a vsphere virtual switch for use by the standalone NSX Edge L2VPN client. The configuration procedure depends on the type of virtual switch associated with the NSX Edge. Type of Virtual Switch Distributed Virtual Switch (DVS) Standard Virtual Switch (VSS) Configuration Configure VLAN Tagging Add a Virtual Machine Port Group b c Configure a Sink Port Configure the NSX L2VPN Client Create an Extended Network in Your SDDC and Bring Up the L2VPN Tunnel Before you can bring up an L2VPN tunnel, you must create an extended network that uses the tunnel ID you specified when configuring the L2VPN client. Extended networks require a layer 2 Virtual Private Network (L2VPN), which provides a secure communications tunnel between an on-premises network and one in your cloud SDDC. Each end of this tunnel has an ID. When the tunnel ID matches on the cloud SDDC and the on-premises side of the tunnel, the two networks become part of the same broadcast domain. Extended networks use an onpremises gateway as the default gateway. Other network services such as DHCP and DNS are also provided on-premises. You can change a logical network from routed to extended or from extended to routed. For example, you might configure a logical network as extended to allow migration of VMs from your on-premises data center to your cloud SDDC. When the migration is complete, you might then change the network to routed to allow the VMs to use VMware Cloud on AWS networking services. 1 Log in to the vsphere Client for your SDDC as a user with cloud administrator privileges. 2 Select Menu > Global Inventory Lists. 3 Select Logical Networks. VMware, Inc. 28

29 4 Click Add. 5 In the Name text field, enter a name for the logical network. 6 Select Extended Network. 7 In the Tunnel ID text box, enter the same tunnel ID that you specified when configuring the L2 VPN client. What to do next Verify that the tunnel is up. 2 Select the SDDC card and click Networking. 3 Under L2 VPN, click the refresh icon next to the VPN tunnel status. If the tunnel is up, the status icon shows green and the status is listed as Up. Common issues that can prevent the tunnel from coming up include failure of the network connection (incorrect ports or addresses specified) or failure of SSL authentication (certificate validity or excessive time skew between the L2VPN client and server). View VPN Tunnel Status and Statistics The VMC Console provides status and statistics for Management Gateway and Compute Gateway IPsec VPNs and for Compute Gateway L2VPNs. 2 Navigate to the Networking tab of your SDDC. 3 Under Compute Gateway, click either IPsec VPNs or L2 VPNs and then VPN Status Detail. You can retrieve status and statistics for any tunnel that is up. Operation Icon Click the Information icon to display a Status Detail message that provides more information about channel (IKE Phase 1 negotiation) and tunnel status. For a VPN with a Status of Disconnected, the Status Detail tab displays any relevant log messages. You can use these messages in conjunction with the Tunnel Statistics and Error Counts to help understand channel or tunnel failures. Click the Refresh icon to refresh tunnel statistics. All VPN statistics are reset to 0 when the tunnel is disabled or re-enabled. What to do next For more information about troubleshooting VPN connection issues, see Troubleshooting Virtual Private Networks in the NSX for vsphere documentation. VMware, Inc. 29

30 Set Compute Gateway DNS Set a DNS server to allow the compute gateway and workload VMs to resolve fully-qualified domain names (FQDNs) to IP addresses. 2 Click View Details on the SDDC card. 3 Click Network. 4 Under Compute Gateway, click DNS. 5 Click Edit and enter the IP addresses for DNS Server 1 and, optionally, DNS Server 2. Note Both DNS servers must be able to resolve all intended FQDNs. Do not add one public DNS server and one private (on-premises) DNS server. If you do, FQDN resolution becomes unpredictable. What to do next If you have configured private DNS servers and you are using a non-default logical network for the compute gateway that uses DHCP, configure your compute gateway VPN to allow DNS requests over the VPN tunnel. Select cgw-dns-network as one of the local networks for the VPN. See Create a Compute VPN. Request Public IP Address You can request public IP addresses to assign to workload VMs to allow access to these VMs from the internet. VMware Cloud on AWS will provision the IP address from AWS. Prerequisites Before you create a public IP address, you should assign your VM a static IP address from its logical network. 2 Click View Details on the SDDC card. 3 Click Network. 4 Under Compute Gateway, click Public IPs. 5 Click Request Public IP. 6 Enter any notes that you want to make about the IP address. VMware, Inc. 30

31 7 Click Save. After a few moments, the Public IP address is provisioned. What to do next After the Public IP address is provisioned, you must configure NAT to direct traffic from the public IP address to the internal IP address of a VM in your SDDC. See Configure NAT Settings. Configure NAT Settings Inbound Network Address Translation (NAT) allows you to map internet traffic to a public-facing IP address and port to a private IP address and port inside your SDDC's compute network. When configuring NAT rules, you have the option of configuring either one-to-one NAT or one-to-many NAT. Use one-to-one NAT when you want to map a single public IP address and port to a single internal IP address and port. For example, a public IP of and port 443 is mapped to and port 443. In some cases, you might choose to map a source port to a different destination port. For example, and port 80 might be mapped to and port Use one-to-many NAT when a single public IP address and port is mapped to one internal IP address and multiple ports, or to multiple internal IP addresses and ports. Prerequisites Before you can assign a public IP address to a virtual machine, you must assign the virtual machine to a logical network and give it a static IP address. 2 Click View Details on the SDDC card. 3 Click Network. 4 Under Compute Gateway, click NAT. 5 Click Add NAT Rule. 6 Enter the NAT parameters. Option Public IP Service Enter a description for the NAT rule. Select the Public IP address you have provisioned for the VM. Select one of the following. Select Any for a rule that applies to all inbound traffic. Select a particular service to create a rule that applies only to traffic using that protocol and port. Select Custom TCP, Custom UDP, or ICMP (All ICMP) to create a rule that applies to a service and/or port that is not available in the dropdown menu. Public Ports If you selected a custom TCP or UDP, enter the port to use for that service. VMware, Inc. 31

32 Option Internal IP Internal Ports Enter the internal (private) IP address to direct the traffic from the public address to. If you selected a custom TCP or UDP, enter the port to use for that service. 7 Click Save. Using AWS Direct Connect with VMware Cloud on AWS AWS Direct Connect is a service provided by AWS that allows you to create a high-speed, low latency connection between your on-premises data center and AWS services. Direct Connect traffic travels over one or more virtual interfaces that you create in your customer AWS account. There are two types of virtual interfaces, private and public. Table 3 7. Characteristics of Private and Public Virtual Interfaces Interface Type Functionality Traffic Types Use Cases Private virtual interface Establishes a private connection between your onpremises data center and a single Amazon VPC. vmotion and ESXi management traffic only Speed up cold migration and migration with vmotion between your on-premises data center and your cloud SDDC. Public virtual interface Establishes a public connection to all AWS public IP addresses in a given region Any traffic that travels through the Internet Gateway Speed up access to AWS public services such as S3 buckets and EC2 public IP addresses. Speed up management and compute gateway IPsec VPN traffic. You can use either type of interface alone, or use both types at the same time. You can create multiple interfaces of each type to allow for redundancy and greater availability. Set Up an AWS Direct Connect Connection To set up an AWS Direct Connect connection, you must place an order through the AWS console. Refer to Getting Started with AWS Direct Connect for information about how to request an AWS Direct Connect connection. Prerequisites Request your Direct Connect access in a region where VMware Cloud on AWS is available. What to do next After your AWS Direct Connect connection is established, create a private virtual interface to connect to your VMware Cloud on AWS SDDC. VMware, Inc. 32

33 Create a Private Virtual Interface for vmotion and ESXi Management Traffic The private virtual interface allows ESXi management traffic and vmotion traffic to flow over the Direct Connect connection between your on-premises environment and your SDDC. Create one virtual interface for each Direct Connect link you want to make to your SDDC. For example, if you want to create two Direct Connect links for redundancy, create two virtual interfaces. Direct Connect BGP sessions in VMC environment use the following local VMC ASNs: in the Asia Pacific (Singapore) region, in the Asia Pacific (Tokyo) region, 9059 in the EU (Ireland) region, and 7224 ain other regions. Prerequisites Ensure that you meet the prerequisites for virtual interfaces as described in Prerequisites for Virtual Interfaces. Determine the AWS account ID for your VMC AWS account. This is displayed under the Direct Connect section of the Networking tab of your SDDC. 1 Log in to the AWS Console. 2 Click Direct Connect and then click Virtual Interfaces. 3 Click Create Virtual Interface. 4 Enter the parameters for the virtual interface. Option Private Virtual Interface Name Virtual Interface Owner Account ID Select Private to create a private virtual interface. Enter a name for the virtual interface. Select Another AWS Account. Enter the AWS account ID for your VMC AWS account. 5 Complete the other settings as described in Create a Hosted Virtual Interface. 6 Accept the virtual interface in the VMC Console. Before you accept the virtual interface connection, it is visible to all SDDCs in your environment. After you accept the virtual interface in a particular SDDC, it is available only in that SDDC. a b c d Log in to the and go to the Networking tab of your SDDC. Under Direct Connect, select Virtual Interfaces. Next to the virtual interface you created, click Attach. Select I understand that I will be responsible for data transfer charges incurred for the interface and click Accept Virtual Interface. VMware, Inc. 33

34 It can take up to 10 minutes for the BGP session to become active. When the connection is ready, the State shows as "Attached" and the BGP Status as "Up" in the VMC Console. 7 If you currently have an IPsec VPN configured, update the VPN configuration. After Direct Connect is enabled, the traffic from theesxi management subnet and vmotion subnet travels over Direct Connect rather than the IPsec VPN. This leaves only management appliance subnet traffic on the IPsec VPN. VPN settings in the VMC Console are updated automatically. You must reconfigure the on-premises gateway for the IPsec VPN to use the management appliance subnet as the remote network. a Refresh the VPN settings in the VMC Console. The management subnet in the network topology diagram updates to show the management appliance subnet address. The Appliance Subnet value in the network topology diagram shows the address for the management appliance subnet. b Update the remote network in your on-premises gateway to the value for the management appliance subnet. After the private virtual interface is attached, any management gateway firewall policies configured for ESXi host traffic are not enforced, because such traffic is routed through the Direct Connect connection. What to do next Ensure the vmotion interfaces are configured to use Direct Connect. See Configure vmotion Interfaces for Use with Direct Connect. Configure vmotion Interfaces for Use with Direct Connect If you are using a Direct Connect connection between your on-premises data center and your cloud SDDC, you must configure the vmotion interfaces for your on-premises hosts to route vmotion traffic over the Direct Connect connection. Prerequisites Configure Direct Connect and create a private virtual interface. VMware, Inc. 34

35 1 Select one of the following methods to configure the vmotion interface on each host in your onpremises environment. Option Override the default gateway (works for vsphere 6.5 hosts only) Configure the vmotion TCP/IP stack For each host, edit the VMkernel adapter used for vmotion traffic, and select the option to override the default gateway. Enter an IP address in your on-premises vmotion subnet that is capable of routing traffic to the on-premises side of the Direct Connect connection. See Edit a VMkernel Adapter Configuration. For each host: a Remove any existing vmotion VMkernel adapters. b Create a new VMkernel adapter and select the vmotion TCP/IP stack. See Place vmotion Traffic on the vmotion TCP/IP Stack of an ESXi Host. c Edit the host vmotion TCP/IP stack to change the routing to use an IP address in your on-premises vmotion subnet that is capable of routing traffic to the on-premises side of the Direct Connect connection. See Change the Configuration of a TCP/IP Stack on a Host. 2 (Optional) Test connectivity between an on-premises host and a cloud SDDC host using vmkping. See for more information. Create a Public Virtual Interface for VPN Traffic You can configure a public virtual interface to access AWS public services such as S3 to travel over the Direct Connect connection. You can also establish the management and compute gateway VPN tunnels over the Direct Connect public virtual interface. Create one virtual interface for each Direct Connect link you want to make to your SDDC. For example, if you want to create two Direct Connect links for redundancy, create two virtual interfaces. Prerequisites Ensure that you meet the prerequisites for virtual interfaces as described in Prerequisites for Virtual Interfaces. 1 Log in to the AWS Console. 2 Click Direct Connect and then click Virtual Interfaces. 3 Click Create Virtual Interface. 4 Enter the parameters for the virtual interface. Option Public Virtual Interface Name Virtual Interface Owner Select Public to create a public virtual interface. Enter a name for the virtual interface. Select My AWS Account. VMware, Inc. 35