aviatrix_docs Documentation

Transcription

1 aviatrix_docs Documentation Release Mark Ennamorato Jan 26, 2018

2

3 Getting Started 1 Aviatrix Overview 3 2 AWS 9 3 Azure 13 4 Google 19 5 Virtual Appliance CloudN 23 6 Test Drive CloudN on Your Laptop 45 7 Frequently Asked Questions 61 8 Onboarding and Account FAQs 69 9 Account IAM Role Azure ARM GCP Credentials Admin Users and Duo Sign in Aviatrix Companion Gateway V Reserve For On-Prem Use Quick Tour Gateway Transit VPC Workflow Instructions Peering Encrypted Transitive Peering 109 i

4 21 Cluster Peering Multi Cloud: Connecting Azure to AWS and GCP Site2Cloud CloudN for Site2Cloud Transit Network with BGP Setup Instructions Encryption over DirectConnect/ExpressRoute How to Build Simple and Scalable Transit VPC Solution Site2Cloud between Azure VPN Gateway and Aviatrix Gateway OpenVPN Aviatrix OpenVPN FAQs Aviatrix OpenVPN Feature Highlights OpenVPN Design for Multi VPCs OpenVPN for Geo Locations UDP LoadBalanced VPN using DNS Okta Authentication Duo Authentication OpenVPN with SAML Client on Okta IDP Anonymous Internet Surfing Developer s Sandbox Tag Based Security Policy FQDN Whitelists IPmotion Setup Instructions IPmotion Early Customer Trial Instructions Migrating VMs with Aviatrix IPMotion and AWS Migration Hub Service IPmotion Design Patterns IPmotion Dependency Discovery Service Chaining Environment Stamping Docker Swarm Cluster Installation Docker Container Access 255 ii

5 51 Migration from AWS Marketplace Licensing Model to BYOL Licensing Model Controller Backup and Restore Controller HA in AWS Inline Software Upgrade Logging Logs Diagnostics Error Messages How to Troubleshoot Azure RM Gateway Launch Failure FlightPath Aviatrix FlightPath Deployment Guide REST API Example Aviatrix Controller API CloudN Aviatrix VPN Client Release Notes Aviatrix VPN Client Changelog Migrating a Join deployment to Site2Cloud deployment Auto Booting CloudN VM Using ISO File Customize AWS-IAM-Policy for Aviatrix Controller Hybrid Network Load Balancing (NLB) Datadog Integration Transit VPC Network - CSR1000v vs. Aviatrix Aviatrix Terraform Provider Launch Aviatrix Controller Manually Using Aviatrix to connect from one site to another site with IPsec VPN Extending Your vmware Workloads to Public Cloud How to Build a Zero Trust Cloud Network Architecture with Aviatrix AWS Global Transit Network Setup Transit VPC Solution using Aviatrix Terraform Provider 411 iii

6 81 Enterprise Cloud Adoption Journey: Technical Challenges 417 iv

7 All Aviatrix product documentation can be found here. If you cannot find what you need, us at Hats off to all who helped fix typos and mistakes. You can do that too by clicking the Edit on Github button on the top right corner of any document. Please also visit our main website for more information regarding use cases and upcoming events. While all content is searchable, the site is organized into the following sections: Getting Started Onboarding and Accounts Gateway Transit VPC Peering Site2Cloud OpenVPN Security IPmotion Advanced Config Settings Troubleshoot REST APIs Downloads Release Notes Tech Notes Solutions Datasheets Whitepapers Getting Started 1

8 2 Getting Started

9 CHAPTER 1 Aviatrix Overview 1.1 What Do We Do? Aviatrix is a cloud networking company. We focus on solving networking problems in areas relevant to public clouds. These areas are: datacenter to cloud (Transit network solution) cloud to cloud (Encrypted peering connectivity in a cloud and multi cloud ) user to cloud (OpenVPN based SSL VPN solution for developers) site to cloud (Branch and customer sites to cloud) We also provide security features for workloads/application in the cloud: gateway inline L4 stateful firewall. gateway inline L7 FQDN filter for Internet egress traffic. In addition, we have specific network solutions for cloud migration and agile datacenter extension to cloud for vmware workloads. The Aviatrix product consists of two components, controller and gateway. Gateways are launched from the controller browser console by using your cloud account credentials with cloud provider APIs. The Controller image is available in AWS Marketplace, Azure Marketplace and GCloud. We are not a SaaS company, our product runs in your account and environment. We have no access to your data or credentials. 1.2 Why Should You Consider Us? Customers find the most compelling value of our product is ease of use, both at configuration time and operation time. Ease of use is easier said than done in networking as it is by nature complex, how can one achieve that? Here is how we do it: 3

10 Abstraction Abstraction is key to achieve ease of use at configuration time. Abstraction is about hiding layers and layers of complex network protocols, it is also about use-case-driven at presentation layer by combining multiple networking components and features. APIs and Terraform templates also benefit from this abstraction as fewer of them need to be managed. External Integration We integrated and developed applications for all popular logging services to forward events, alerts and a turn key dashboard to achieve ease of use at operation time. Centrally Managed A single pane of glass to manage all your cloud network scatterd in different regions and clouds. For example, we hide the platform differences between AWS, Azure and GCP, so that you have the same experience when networking to any of them or between them. Another example, we hide the complexity of building IPSEC so that you have the same experience when you build an IPSEC tunnel as you would with AWS native peering: a couple of clicks or a couple of APIs. Beyond ease of use, Aviatrix solutions solve many problems better than other products in each of the areas. This document summarizes these problems. Links to configuration documents are listed at the end of each section. Our goal is to become your goto tool for all thing cloud networking. 1.3 Datacenter to Cloud: Global Transit Network Solution Aviatrix Global Transit Network solution solves many problems when connecting datacenters to the growing number of VPCs. These problems are listed below: 1. Change Control Each time a new VPC is stood up, a change control process has to take place to modify the edge router for Direct Connect or IPSEC over Internet. This is not agile and the risk of errors in configuration is not acceptable. 2. BGP The AWS Global Transit solution runs VGW in each spoke VPC which runs a BGP session to Transit hub. This is operationally challenging to manage and troubleshoot. The BGP in VGW is a black box and invisible to outside. 3. Not Secure All spoke VPCs in The AWS Global Transit solution have connectivity to each other through BGP route propagation. There is no network segmentation. The blast raduis is my entire cloud network and datacenters. Not acceptable by security team. 4. Reach Route Limit AWS has route entry limits of 100 per each routing table. Combining the number of VPC CIDRs and the list of on-prem CIDRS, this route limit is fast approaching or already a problem. 5. Extra Charge In the CSR based solution, traffic from one spoke VPC to another spoke VPC traverses through one transit and sometimes two transit hub, resulting in 2x or 3x egress charge. 6. Too Complex The CloudOps is a team of 6 engineers managing 34 AWS services, the skill set and resources it takes to manage the CSR based Transit network is beyond what we want to handle. Follow this self qualification process to help your team decide if Aviatrix is the right solution for you. For how to setup the solution, follow up with this doc. 1.4 Cloud to Cloud Peering Aviatrix encrypted peering solution builds IPSEC tunnels to connect two VPC/Vnet. It solves these problems: 4 Chapter 1. Aviatrix Overview

11 1. Regulation My industry and regulations require packet in motion to be encrypted. AWS intra peering has no encryption. AWS inter region peering has one shared key. Not acceptable. 2. Reach Route Limit AWS has route entry limits of 100 per each routing table. Combining the number of VPC CIDRs and the list of on-prem CIDRS, this route limit is fast approaching or already a problem. 3. Multi Cloud My workloads in AWS need connectivity to works loads in Azure or Google. 4. Defense in Depth My CloudOps tools communicate to instances with data that is not encrypted. I need encryption for traffic between Shared Service VPC to workload VPC. Aviatrix peering solution can be found here. 1.5 User to Cloud Access Giving developers, contractors and partners around the globe direct access to VPC/VNet is the best way to reduce access latency and improve productivity. How to make it secure, high performance and manageable are key to the solution. Aviatrix user to cloud solution is based on OpenVPN. The solution solves these problems: 1. Bastion Station Bastion Station or Jump Host is a hack and insecure to allow developers to access cloud. Not acceptable. 2. Too Many Certs If each VPC runs a SSL VPN gateway and there are 50 VPCs, each developer needs to carry 50 VPN certificates and must learn which certificate to use to access which VPC. Not acceptable. 3. Large Group We have over 500 developers, need a VPN solution that scales beyond a single instance. 4. OKTA We are looking for a VPN solution that integrates with OKTA or DUO. 5. Blocked by Firewall We have a Linux machine in the office that needs to behave like a VPN client. We need a VPN solution that runs on TCP port 443 to allow this machine to go through the corporate firewall. 6. Global Workforce We have developers in multiple geo locations, cannot have them all land in the cloud in the same region. Latency will kill user experience. 7. SAML Client We are looking for a OpenVPN based VPN solution with SAML client support. Aviatrix user VPN solution cab be found on this link. One feature in the solution that customers like the most is Profile Based Access Control. 1.6 Site to Cloud Connectivity over Internet If you run a SaaS service that needs to securely move data from your customer sites to the cloud, or your enterprise has hundreds of branch offices that need to connect to the cloud, building secure tunnel to the cloud directly over Internet is the most economical way as you leverage the Internet infrastructure already in place. In this case, the cloud provider s native VPN solution falls short by a long shot. Aviatrix site2cloud solution solves these problems: 1. AWS/Azure VPN Gateway Limitation Native cloud provider VPN solution typically can support 30 connections per VPN gateway. I have more than 30 sites, the native solution is not usable. 2. No Manual I have to configure and manage hundreds or thousands of IPSEC tunnels, the manual way by using traditional vendors such as Cisco ASA and CSR is not possible User to Cloud Access 5

12 3. Overlapping IP addresses We run a SaaS operation, the CIDR blocks at your customer sites are not controlled by us. If a customer CIDR block overlaps with our operation VPC CIDR, we have to find a way to NAT the address. The cloud provider native solution is not usable in this case. 4. Encryption Algorithm Mismatch As SaaS operators, we cannot control what VPN device a customer wishes to use. My end of VPN termination needs to have the flexibility to interoperate with customer equipment. The native solution does not have that flexibility. 5. Too Slow to Onboard a Customer VPN runs on UDP port 500/4500, my customers have to request corporate firewall ports to open, is there a way to run IPSEC tunnel on TCP 443? 6. Traffic Direction Problem My SaaS service requires traffic to be initiated from cloud to the customer site, AWS VPN gateway cannot support this traffic pattern. We have to setup a separate machine to constantly ping to keep the tunnel up! To learn how to setup Aviatrix site2cloud, follow up with this link. 1.7 Gateway Inline L7 FQDN for Egress Control This solution is about adding security control to private workloads or applications accessing Internet. AWS and Azure provide a NAT gateway or NAT service, but it is limited in scope. Traditional firewall is either too complex or too expensive to be deployed per VPC. Aviatrix L7 FQDN filter solves these problems: 1. Only IP Based Rules AWS provides security groups for its NAT gateway, but it is IP address based and limits to 50 rules. My application needs to make API calls to Office 365 and that site along resolves to hundreds of changing IP addresses. Using Security group is not an acceptable solution. 2. Firewall for Each VPC is Too Complex My cloud instances are workloads and programs, they make API calls to known destinations. Deploying a traditional firewall that requires certs and keys to decrypt every packet for inspection is too complex and an overkill. 3. Firewall for Each VPC is Too Expensive Traditional firewall of IDS/IPS is too expensive to be deployed per VPC. 4. Whitelisting All I need is to be able to white list or black list the well known destinations by specifying them as fully qualified domain names (FQDN) for my http and https traffic. Support wild card or regex is a bonus. Follow up with more details on Aviatrix FQDN filter solution. 1.8 Gateway inline L4 Stateful Firewall Whenever there is traffic going through Aviatrix gateway, you can apply IP address based stateful firewall policies. This reduces the need to have to configure security groups of each instances in the VPC for traffic between VPCs. There is no limit as to how many rules you can apply on Aviatrix gateway. Aviatrix solution solves these problems: 1. Security Rule Limits An cloud instance s security group has a limit of 50 rules. How do I get around that? 2. Enforce Security Policies Developers don t always follow the best practice when it comes to security, enforcing policies at the gateway takes that worry away. To learn how to setup the L4 firewall, follow the doc. 6 Chapter 1. Aviatrix Overview

13 1.9 Cloud Migration Current cloud migration practice is complex and time consuming. The root case is the requirements that migrating VM must change its IP address after the migration. Read how Aviatrix solves this problem Extending Workloads to Cloud Not all your workloads require the bandwidth and latency that calls for a Direct Connect transport. For your Dev and QA or many applications, an existing Internet connectivity is sufficient. Even better, Aviatrix provides a unique solution that you do not even need to make changes to the edge router. Learn how this solution works. OpenVPN is a registered trademark of OpenVPN Inc Cloud Migration 7

14 8 Chapter 1. Aviatrix Overview

15 CHAPTER 2 AWS The Aviatrix cloud network solution consists of two components, controller and gateway, both are AWS instances. Gateways are launched from the controller browser console by using your account IAM roles and AWS APIs. This guide helps you to launch the Controller instance in AWS. The Controller image is also available in Azure Marketplace and GCloud. 2.1 Create an AWS EC2 Account You need to have an AWS EC2 account to use the solution. Note that the Controller supports multiple accounts with each one associated with a different AWS IAM role or account, but there needs to be at least one to start with. This AWS account can be a root account, IAM role, IAM administrator account or IAM user account with access privileges required by the Aviatrix solution. We strongly recommend you to use IAM role for security reasons. 2.2 Subscribe to Aviatrix on AWS Marketplace You must subscribe to one of the Aviatrix AMIs on AWS marketplace prior to launch the Controller. Once you subscribe, return to this page and continue to the next section. Search aviatrix on AWS marketplace and accept the terms and conditions to use the software. After subscription, follow the instructions in the next sections to launch the Controller. If you choose the BYOL image, you need a customer ID (license ID) to use Aviatrix solution. Send an to support@aviatrix.com to obtain one. 9

16 2.3 DNS Server Connectivity Check If the VPC where the Controller is deployed in has a custom DNS server (via DHCP option), make sure the Controller instance can reach this DNS server. Warning: Any resources created by the Controller, such as Aviatrix gateways, route entries, ELB, SQS queues, etc, must be deleted from the Controller console. If you delete them directly on AWS console, the Controller s view of resources will be incorrect which will lead to features not working properly. 2.4 Launch Aviatrix Controller Controller must be launched on a public subnet of a VPC Launch from CloudFormation script If you select the Aviatrix BYOL AMI, the recommended way to launch the Controller is by our CloudFormation script. Follow the instruction for Aviatrix QuickStart Cloudformation Script to launch a controller instance in a selected region Launch Utility AMIs manually For utility AMIs, you need to launch the utility AMIs controller manually described in this document. 2.5 Access the Controller After the Controller instance is in running state in AWS, you can access the Controller via a browser by where Controller_public_EIP is the Elastic IP address of the Controller. The initial password is the private IP address of the instance. Follow the steps to go through an initial setup phase to download the latest software. After the latest software is downloaded, re-login again to go through the onboarding process. 2.6 Onboarding The purpose of onboarding is to help you setup an account on Aviatrix Controller that corresponds to an IAM role with policies so that the Controller can launch gateways and build networks using AWS APIs. If you launched the Controller via CloudFormation script, the required IAM roles and policies are already setup, follow this instruction to complete account creation. Note you can create a single Aviatrix account that corresponds to AWS, Azure and GCloud account credentials. This is a multi cloud platform. To create a Global Transit Network, click Transit VPC on the main navigation bar to start. 10 Chapter 2. AWS

17 2.7 Setup for Operations If this Controller is for your production, we strongly recommend you to enable Controller Backup/Restore feature. This allows you to backup configurations on the Controller to an S3 bucket sothat you can recover the configurations in a disaster situation. 2.8 Controller HA To enable Controller HA in AWS, follow the instructions here. 2.9 Controller Monitoring If Controller HA is not enabled, we recommend you to use AWS CloudWatch to configure alarms and actions to reboot the controller when it fails Status Check Key Use cases Inter region and inter cloud peering Global Transit Network Client VPN or OpenVPN For support, send to support@aviatrix.com. Enjoy! OpenVPN is a registered trademark of OpenVPN Inc Setup for Operations 11

18 12 Chapter 2. AWS

19 CHAPTER 3 Azure The Aviatrix cloud network solution consists of two components, controller and companion gateway, both are Azure VMs. Gateways are launched from the controller console to specific VNets. This guide helps you to launch the controller VM in Azure. Make sure you follow the instructions to also subscribe Aviatrix Companion Gateway described in this guide Subscribe to Aviatrix Controller Go to Azure Marketplace to subscribe to one Aviatrix image Subscribe to Aviatrix Companion Gateway Aviatrix companion gateway needs to be subscribed as programmable. In order to launch Aviatrix gateway from the controller, you must also subscribe to Aviatrix Companion Gateway which is free in Azure marketplace. Follow the steps in this doc to subscribe Launch the Controller Create an Azure Account Create an Azure account if you do not already have one Launch Controller VM from Azure marketplace portal 1. Launch from marketplace, select the license type and click Create Virtual Machine, as shown below. If you select a BYOL image, you need a Customer ID. Send to support@aviatrix.com or info@aviatrix.com to request a Customer ID. 13

20 2. Select Create at the next screen. 3. At Basics column, fill in the VM name, user name, password and Resource group, click OK. 4. At Choose a size, select the VM size, click Select. 5. At Settings, Click Network security group (This is a critical configuration step) 6. Create a new security group, add an Inbound Rule for HTTPS port 443 for Inbound Traffic, Allow, as shown below. Make sure Source is Any, Service is HTTPS, Protocol is TCP, Port range is 443 and Action is Allow. 14 Chapter 3. Azure

21 7. After the new security rule is added, click OK. 8. Finish launching the VM. 9. Find the VM s public IP address, as shown below: 10. Use a browser to access the controller VM. In this example, it is At the login page, enter admin as username. Initial password is the internal IP address of the VM, as shown below Launch the Controller 15

22 12. Go through the login process. 13. Start with onboarding tab at the console. Warning: Any resources created by the controller, such as Aviatrix gateways, Azure routing entries, subnets, etc, must be deleted from the controller console. If you delete them directly on Azure console, controllers view of resources will be incorrect which will lead to features not working properly Access the Controller After the Controller instance is in running state in AWS, you can access the Controller via a browser by where Controller_public_IP is the static public IP address of the Controller. The initial password is the private IP address of the instance. Follow the steps to go through an initial setup phase to download the latest software. After the latest software is downloaded, re-login again to go through the onboarding process Onboarding The purpose of onboarding is to help you setup an account on Aviatrix Controller that corresponds to Azure account with policies so that the Controller can launch gateways using Azure API. Follow the instructions here to create an Aviatrix account that corresponds to your Azure account credential. Note you can create a single Aviatrix account that corresponds to AWS, Azure and GCloud account credentials. This is a multi cloud platform Gateway Troubleshoot If the Controller fail to launch Aviatrix gateway in Azure RM, check out this troubleshooting guide. 16 Chapter 3. Azure

23 Enjoy! Gateway Troubleshoot 17

24 18 Chapter 3. Azure

25 CHAPTER 4 Google The Aviatrix cloud network solution consists of two components, controller and gateway, both are GCloud instances. The gateway is launched from the controller browser console. This guide helps you to launch the controller instance in GCloud. Important note: a GCloud project corresponds to an Aviatrix cloud account or an AWS (IAM) account with its own credentials. A network in a GCloud project is logically equivalent to a VPC in AWS, but with a few significant differences, for example, a network in GCloud project can have disparate subnets and a subnet can connect across regions. 4.1 Prerequisite Get a Customer ID from Aviatrix Currently Aviatrix Controller for GCloud is only available via community image for BYOL license. Send to info@aviatrix.com or support@aviatrix.com with your organization name to request a customer ID. We offer a 30 day free trial license Create a Google Cloud Platform (GCloud) account Aviatrix Cloud Connect is a software product that is launched in your own GCloud account. The controller and the gateways created from the controller console are all in your own network perimeter and completely under your control. Create a GCloud account ( Go on to the next step if you have already done so. Note that the controller supports multiple accounts with each one associated with a different GCloud projects, but there needs to be at least one to start with Create a GCloud Project log in to your GCloud account and go to project page: 19

26 Create a project. Go on to the next step if you have already created one. Note the project ID will be used in referencing to this project by Aviatrix controller. (As an example, we created a project Aviatrix-UCC, the project ID is aviatrix-ucc-1214) Copy Aviatrix Controller Image to Your Project At your GCloud console ( select the project where you want to launch your controller. Click the 3 bars at the top left corner. At the drop down menu, select Compute Engine, then select Images. At the top screen, click [+] CREATE IMAGE, make sure: Select the project where you want to launch your Aviatrix Controller. Fill in the image name, for example, aviatrix-ucc Fill in the description. At Source, select Cloud Storage File. At Cloud Storage file, paste in the following text string: aviatrix100/aviatrix-cloud-services-gateway byol.tar.gz Click create, as shown below (Optional) Create Networks This step creates a network in the project created in the previous step. When a new project is created, a default network is created. You may skip this step if do not need to customize the network address range by creating a new network, or go on to the next step if you have done so. Note Aviatrix Controller handles a GCloud network like a VPC in AWS. Whenever a network configuration is mentioned for GCloud, the term VPC is used. (The VNet is used for Azure.) At GCloud console, select the project that you have copied the Aviatrix controller image to. Click the 3 bars. At the drop down menu, select Networking. Click [+] Create Network. Note: if you plan to have multiple projects, we suggest you plan your subnets so that the network addresses do not overlap. Select Custom to create subnets. 20 Chapter 4. Google

27 4.2 Launch the Aviatrix Controller At GCloud console, select the project that you just copied the Aviatrix controller image to. Click the 3 bars. At the drop down menu, select the Aviatrix controller image, click [+] Create Instance. Fill in Name for the instance, Zone and Machine type for the instance. Make sure the Machine type is n1-standard-2 or larger. For Identity and API access, select Allow full access to all Cloud APIs. Alternatively, At Access scopes, select Set access for each API, and then * Select Enabled for Cloud Pub/Sub. * Select Read Write for Compute. At Firewall, click Allow HTTPS Traffic, as shown below. Click Create. 4.3 Access the Aviatrix Controller After the instance is created, click the controller instance name, and note its External IP address and Internal IP address. Go to At the login prompt, type admin for username, type the internal IP address for password, as shown below: 4.2. Launch the Aviatrix Controller 21

28 Follow the initial setup process to setup admin address, password and install the latest software. Log in again with your new admin password Warning: Any resources created by the controller, such as Aviatrix gateways, GCP routing tables, subnets, LB, etc, must be deleted from the controller console. If you delete them directly on AWS console, controllers view of resources will be incorrect which will lead to features not working properly. 4.4 Onboarding If no GCloud account has been setup, you will be guided through the onboarding process. It takes only a few steps. Once that is done, follow the quick tour guide to start launching gateways. For onboarding instructions on GCloud, click this link. 4.5 Support Check out Help menu for Frequently Asked Questions (FAQs), Reference Design and Release Notes. All features have descriptions embedded and should be self-explanatory. An alert message will be displayed on the Dashboard menu when a new release becomes available. For support, send to support@aviatrix.com Enjoy! 22 Chapter 4. Google

29 CHAPTER 5 Virtual Appliance CloudN Aviatrix CloudN virtual appliance that is deployed in an on-premise datacenter or co-location facility. CloudN supports REST API that allows further automation and third party software integration. REST API document can be found at this link. For an example of how to use REST API, check out this link. CloudN performs three major functions: Datacenter Extension Extend your datacenter to multi cloud (Datacenter Extension or DCCX). Read How to build agile DevOps document for instructions. Site2Cloud Build encrypted tunnel to existing VPC/VNets (on-prem gateway for Site2Cloud). Read How to build ssite2cloud for instructions. IPmotion Build connectivity that makes it possible to migrate on-prem VMs to cloud while preserving their IP addresses. Read How to setup IPmotion for instructions. The following guide provides step by step instructions for deploying the virtual appliance. Read carefully as there are specific instructions for each of the above three use cases Download the Image Virtual appliance CloudN image can be downloaded from the download link Pre-Installation Check List AWS EC2 Account for Datacenter Extension and IPmotion Note: If CloudN is deployed for Site2Cloud function, you do not need to setup an EC2 account. Skip section

30 If you intend to launch VPC in AWS, you need to have an AWS account. You need to have an AWS account in order to use most of the commands on CloudN. Note that CloudN support multiple CloudN cloud accounts with each one associated with a different AWS account or IAM account, but there needs to be at least one to start with. The AWS account can be a root account, an IAM user in Administrator Group or an IAM user with full access permission to EC2, VPC, S3, SQS, SNS, CloudTrail and Route 53. For security reasons, we strongly recommend you use IAM user account. During onboarding, you will have opportunity to copy and paste a custom policy required by Aviatrix to your AWS IAM account IAM Administrator The following steps show you how to add a user to Administrator Group in AWS. Step 1. Login to Step 2. Click Users, select the user that needs to be added to Administrative privilege, click Add User to Groups Step 3. Add joe_smith to admin group which was created previously via Groups tab on the console IAM User If you are an IAM user, make sure you have full access to EC2, VPC, S3, SQS, SNS and CloudTrail service. Refer to this link on how to setup an IAM access policy required by CloudN. During the onboarding process, we will guide you through on setting up this IAM customer policy. 24 Chapter 5. Virtual Appliance CloudN

31 Microsoft Azure Account for Datacenter Extension Note: If CloudN is deployed for Site2Cloud function, you do not need to setup an Azure account. Skip this section. To create credentials for Azure, follow this instructions Deploy CloudN as a virtual router (Site2Cloud function) You can deploy CloudN as a virtual router and in a remote site for Site2Cloud function. In this deployment, CloudN functions as a router and it is deployed anywhere inside a datacenter and it does not require a public IP address. What is required is that the default gateway of the subnet where CloudN is deployed has a static route configured that routes traffic destined to the VPC CIDR where this remote site wish to connect to the CloudN Deploy CloudN for Aviatrix Datacenter Extension If you plan to use CloudN for IPmotion, skip section Cloud address planning and allocation When used for datacenter extension (DCCX) function, CloudN manages your entire cloud address space. You need to identify or create a subnet where CloudN is deployed. CloudN is deployed on a private subnet anywhere on your network. CloudN does not take a public IP address. Make sure this subnet is reachable by other subnets where traffic is originated from. CloudN should be deployed on a subnet (or VLAN) where CloudN is the only virtual machine on the VLAN. CloudN VM s IP address is determined by CloudN software during installation time Pre-Installation Check List 25

32 The default gateway for the VLAN should either have the lowest address or highest address for the VLAN. For example, if the VLAN where CloudN is deployed is /16, the default gateway IP address for this VLAN should be either or The size of this subnet or VLAN should be large enough to allow the creation of the desired number of VPCs. For example, a network with /16 prefix can support 15 VPC/VNets with each VPC/VNet contains /24 subnet in AWS or Azure. CloudN allocates 4 bits or 16 subnets in each VPC. By default, two subnets, one private and one public subnets are created in each available zone. A user can customize and create additional subnets Deploy on Subnets larger than /24 If you deploy a CloudN in a /23 subnet, only two VPC/VNet can be created. This VPC/VNet can support 8 subnets. It is recommended that you deploy CloudN in a subnet size between /16 and /22. Below is the table that describes the subnet size and the maximum number of VPCs Deploy on a Class C Subnet Deploying CloudN in a /24 subnet is a special case. It is handled differently from any other size of subnets. In this case, there is only one public subnet and 2 private subnets with each in a different availability zone created for a VPC Container. Up to 2 VPCs can be launched. Since not every AZ (Availability Zone) is covered in subnet creation, applications that require subnets in each AZ would not work. Deploying on /24 subnet is best used for POC projects. If you have local machines on the subnet where CloudN is deployed, you need to make sure all local machines including the default gateway and CloudN are in one sub segmented area, as illustrated below: 26 Chapter 5. Virtual Appliance CloudN

33 Leaving local machines outside the address range of /26 can result in duplicate IP addresses. Each VPC has 1 public subnet and 2 private subnets Network Interfaces CloudN local gateway is installed as a VM host with two network interfaces. Make sure the two interfaces are on the same VLAN or subnet. If CloudN runs on VMware ESXi host, follow the instruction in the next chapter to enable promiscuous mode and forged transmit mode for both interfaces. If CloudN runs on Microsoft Hyper-V, you do not need to configure the network interfaces as they are pre-configured as part of VHD image Internet Connectivity CloudN needs to have Internet connectivity to perform most its functions Proxy Settings If there is proxy server on-prem for Internet access, contact IT administrator to obtain proxy server IP address, proxy port, and if there needs to have username and password for authenticating by the proxy Binding to CloudN Private IP address to a Single NAT Public IP Address Note: If you select TCP as tunnel type for either datacenter extension or site2cloud function, the constraints in this section do not apply Pre-Installation Check List 27

34 If your organization has more than one public IP addresses as the NAT address, you must bind CloudN s private IP address to one of the public IP addresses. That is, CloudN will always be translated to one static public IP address for its outbound traffic. For example, on Cisco ASA, you can configure the following to bind a private IP address to one public IP: Step 1 Create a network object for the internal servers. hostname(config)# object network mywebserv hostname(config-network-object)# range Step 2 Configure NAT to map servers from to to a static public IP ( ) hostname(config-network-object)# nat (inside,outside) static Outbound TCP/UDP Ports CloudN requires the following TCP/UDP outbound ports open. TCP port 443. (optional) UDP ports 4500 and 500. Note: Aviatrix CloudN supports encrypted tunnels over TCP port 443. If you select TCP as the tunnel type for datacenter extension or site2cloud function, no UDP ports 500/4500 are required to be open. The advantage of selecting TCP as the tunnel type is to reduce deployment friction when building hybrid connectivity. In the current release for IPmotion, only UDP mode is supported. If you choose to reduce the scope of above ports, you can limit them to only AWS owned public IP address blocks. All AWS public IP addresses can be found in this link. Since CloudN operates in a client-server mode where the CloudN local gateway is the client, there is no restriction or requirement to open any known TCP/UDP port for inbound traffic Time Service CloudN uses extensively Amazon Web Service (AWS) APIs and Azure REST APIs. These APIs checks timestamp for each API call. CloudN is pre-configured to synchronize its time with Host (please double check on the VM advanced option to make sure this is the case.) To ensure correct operation of CloudN, it is important that the Host where CloudN is installed has correct time. Most likely enterprise data center syncs VM time to host. However if your environment requires you to sync time to an NTP server, CloudN allows you to accomplish that. You can configure this at Settings -> Time Service Performance Consideration CloudN is a virtual appliance that runs on a hypervisor. The supported hypervisors are VMware hypervisor products, Microsoft Enterprise 8.1 Hyper-V and Oracle VirtualBox. 28 Chapter 5. Virtual Appliance CloudN

35 By default CloudN is packaged with 2 vcpu, 4GB of memory and 20GB of hard disk (SCSI storage or hard drive) as part of its image make up. You can always reconfigure the VM to take more CPU and memory. For maximum performance, it is recommended that the host CPU has support for Intel AES-NI, instruction set for hardware encryption. Intel processors Westmere, Sandybridge, Ivrybridge and Haswell all have AES-NI enabled. In test environments, TCP throughput (using iperf tool) in the vicinity of 880Mbps has been observed with CloudN running on a VMware ESXi host with an Intel Xeon CPU (E3-1220L 2.30GHz) Installation CloudN OVF image can be imported and installed on a VMware ESXi 5.0/5.1 host, VMware Workstation, Fusion and VMware Player. Once you have signed up as a Aviatrix customer, follow the instructions to download the zip file on your PC. CloudN OVF image usually takes the name cloudn-ovf-date where date is the time when the image was built. CloudN is recommended to run on ESXi 5.0 or later version. However you can install the software on VMware Player, VMware Workstation and Fusion for testing and evaluation purposes Installation on ESXi 5.0 or later After downloading and extracting the zip file, copy the folder to a location where you can import the virtual machine. For installation, follow the steps below. Step 1: In the vsphere Client, select File > Deploy OVF Template Step 2: Locate the folder where.ovf file is located Installation 29

36 Step 3: Click Next to proceed through the rest of the installation. Please refer to the page ESXi Admin for more detailed instructions Configure Network Adapter Properties for Note: If you deploy CloudN for Site2Cloud connectivity, CloudN network interfaces are not in promiscuous mode. Skip this section. CloudN has two network interfaces, both of them need to be on the same VLAN. After the installation is finished, follow these steps to enable promiscuous mode on the network adapter (below is an example): Step 1. Select (Highlight) ESXi host tab where CloudN is hosted (for example, ) and click on the Configuration tab 30 Chapter 5. Virtual Appliance CloudN

37 Step 2. In the Hardware section, click Networking and then properties Step 3. Select VM Network adapter for CloudN and click edit Installation 31

38 Step 4. Click the Security tab, from the Promiscuous Mode dropdown menu, click the box and select accept and click OK. If you are running ESXi 5.1 or later, you also need to set Forged Transmit Mode for the port group to Accepted. 32 Chapter 5. Virtual Appliance CloudN

39 For more information on configuring security policies on the network switch, please refer to the instructions in this link. For additional CloudN on ESXi configuration illustrations, check out this note Note: DCCX does not support NICteaming in active-active mode. When NICteaming is configured, only active-standby mode is supported, as shown below where the ESXi host has 4 Ethernet ports and VLAN220 is the port group CloudN Ethernet ports belong to Installation 33

40 Installation on Windows 8.1 Enterprise Edition CloudN VHD image can be deployed on Windows 8.1 Enterprise Edition, or Windows 2012 Server R2 Hyper-V. After downloading the zip file and decompressing it, copy the folder to a location where you can import the virtual machine. For installation, follow guide below. Step 1: Import the VHD Image 34 Chapter 5. Virtual Appliance CloudN

41 Step 2: Locate Folder Step 3: Copy the Virtual Machine Installation 35

42 Step 4: Connect to the Virtual Machine Step 5: Start the Virtual Machine 36 Chapter 5. Virtual Appliance CloudN

43 Step 6: Login into Virtual Machine User Name: admin Password: Aviatrix123# Enable MAC Address Spoofing for DCCX and IPmotion Note: If you deploy CloudN for Site2Cloud function, MAC Spoofing is not needed. Skip this section. Both Network Adapters associated with CloudN VM should have Enable MAC Address Spoofing turn on. This is accomplished by expand Network Adapter, select Advanced Feature and check the box Check MAC Address Spoofing, for each Network Adapter. As part of VHD image, this setting should already be configured and should not be changed Installation 37

44 NIC Teaming Support for DCCX and IPmotion Note: If you deploy CloudN for Site2Cloud function, active and active NIC team is supported. For DCCX, NIC teaming is only supported for active standby mode Booting Up and Initial Configuration This section and the following steps can be automated. Check out this vmware PowerCli script. Below description is how you can boot up in a manual way. After the virtual machine boots up, you must first login into the machine while still in hypervisor console. CloudN Login User Name: admin CloudN Login Password: Aviatrix123# After this initial login, if you see the screen the screen below. 38 Chapter 5. Virtual Appliance CloudN

45 Follow the instruction to type help at the prompt. Follow the steps to go through the boot up process. You can type help at any time to review the steps. Type? to view all available commands. For each command, type? to view syntax and parameters Booting Up and Initial Configuration 39

46 Step 1: Setup Interface Address There are two ways to give CloudN its IP adddress: auto-generate by CloudN itself or statically assign one Statically assign CloudN IP address (Recommended method) Command: setup_interface_static_address Syntax: setup_interface_static_address [static_ip_address] [net_mask] [default_gateway_ip_address] [primary_dns_server_ip_address] [secondary_dns_server_ip_address] [proxy {true false}] Below is an example where there is no proxy server. In such case, CloudN will configure the network interfaces, test Internet connectivity and download the latest Aviatrix software. Note: For DCCX deployment, choose CloudN IP to be next to the default gateway IP address of the VLAN or subnet where CloudN is deployed. This does not apply to IPmotion deployment Proxy Configuration If there is proxy server for Internet access, you must setup proxy configuration on CloudN to pass traffic to proxy correctly. Following is the command command: setup_network_proxy syntax: setup_network_proxy <action> < http_proxy> < https_proxy> where action is test or save. Example: 40 Chapter 5. Virtual Appliance CloudN

47 setup\_network\_proxy test --http\_proxy --https\_proxy setup\_network\_proxy save --http\_proxy --https\_proxy Note after proxy configuration is saved, CloudN VM will reboot to have the proxy take effect Auto-generate CloudN interface IP address All you need to do here is to provide information related to the subnet where CloudN is deployed. CloudN scans the subnet and find an IP address that is close to the default gateway (for example, if the default gateway is , CloudN will try ) and is available, CloudN will then assin itself this IP addres and CloudN software will be downloaded if configuration is successfully. Command setup_interface_address: Syntax: setup_interface_address [net_mask] [default_gateway_ip_address] [dns_server_ip_address_1] [dns_server_ip_address_2] [proxy {true false}] CloudN will identify an unused IP address in an iterative fashion and assign it to itself. As seen in the above example, the IP address generated is Once the IP address is generated, CloudN will start to download the latest CloudN software snippet Booting Up and Initial Configuration 41

48 If you see the above message, the download is completed Step 2: Display Interface Address Now you can use the cloudn IP address as URL to access CloudN Manager that manages CloudN. Note: The hypervisor console has only limited CLI for initial booting up purposes. Once Aviatrix software is downloaded, full commands are installed. User should use the GUI to access CloudN Console Troubleshooting If there is any error messages during installation, it is usually due to lack of Internet connectivity, incorrect DNS server IP address or unopened firewall ports. Type? to see all the commands that help you troubleshoot. 42 Chapter 5. Virtual Appliance CloudN

49 Use command *ping* and *traceroute* to check out Internet connectivity. Check your DNS server setting, consult your network and server admin to determine the cause of routing failure. After connectivity issue is resolved, use command download_cloudn_software to continue installation and finish. Or you can again type in command setup_interface_address Use a Browser to Access CloudN CloudN has a built in CloudN Console that let you run provisioning from a browser. Once IP addressed setup is complete, you can use any browser, type address of CloudN> and see a Login page. Login with: User Name: admin Password: private IP address of the VM After login, go through the initial setup process. For the first time user and initial setup, follow Onboarding to go through the initial set up and launch your first VPC/VNet. Warning: Any resources created by the controller, such as Aviatrix gateways, AWS/Azure routing tables, subnets, etc, must be deleted from the controller console. If you delete them directly on AWS console, controllers view of resources will be incorrect which will lead to features not working properly Onboarding After you login to the browser console, click Onboarding to go through a few steps of initial setup and start using Aviatrix. For all feature documentation, go to docs.aviatrix.com Onboarding 43

50 For support issues, send to Enjoy! 44 Chapter 5. Virtual Appliance CloudN

51 CHAPTER 6 Test Drive CloudN on Your Laptop CloudN can be installed on your laptop and test drive it for evaluation purpose. It runs on vmware Workstation, vmware Player, Fusion and virtual box. 6.1 Download CloudN Images Follow the instruction to download CloudN image. 6.2 Test Drive CloudN in NAT Mode One good configuration to test drive cloudn is to deploy it on your laptop on a private subnet in NAT mode (In Hyper-V, the network adapters are configured as Internal Network Wire). As an example, if your NAT mode subnet is /24, you can create a maximum 2 VPCs from CloudN deployed on this subnet. Suppose the default gateway IP address is You should configure CloudN to take as its IP address. In addition CloudN reserves IP address ranges from to (If you have other VMs running on this subnet, if their IP address fall in the same sub segment as CloudN, you can use one of these VMs as test VM.) Once you launch VPCs from this CloudN, the other VMs on the subnet should be able to run SSH, RDP, and SCP (file copy) to any instances in VPCs using the instance private IP address seamlessly, without any bastion station or landing VPC. Refer to How It Works section for more explanations. Note: If you install CloudN on a NAT subnet, make sure both Ethernet interfaces are changed to NAT mode (By default, CloudN is pre-configured and shipped with both Network Adapters in Bridged mode). Right click on the CloudN VM, click Settings. Change both Network Adapters to NAT mode, as shown below for VMware Workstation: 45

52 6.2.1 Test Drive on MAC with vmware Fusion After downloading the zip file and decompressing it, copy the folder to a location, where your Mac can access it. Perform the following steps to install CloudN. Step 1: From the VMware Fusion menu bar, select File > Import. 46 Chapter 6. Test Drive CloudN on Your Laptop

53 aviatrix_docs Documentation, Release Step 2: The Import Library window appears, along with a dialog box for browsing to the location of OVF file. Step 3: Browse to the.ovf file and click open 6.2. Test Drive CloudN in NAT Mode 47

54 Step 4: Type the name for the imported virtual machine in the Save As text box and indicate where to save it. Step 5: After the import is complete, the virtual machine appears in the virtual machine library. Click on Start Up to start the CloudN virtual machine. 48 Chapter 6. Test Drive CloudN on Your Laptop

55 Step 6: Change Network Adapters to NAT mode Select the VM, click Settings, click Network Adapter, select Share with my Mac, as shown below Test Drive on PC with VMware Workstation Click on File -> Open, as shown below Test Drive CloudN in NAT Mode 49

56 Then open the desired VM. Highlight the VM, right click, select Settings, click on Network Adapter, change both Network Adapter to NAT mode as shown below. 50 Chapter 6. Test Drive CloudN on Your Laptop

57 6.2.3 Test Drive on VirtualBox CloudN works on VirtualBox only in a bridged mode. After downloading and extracting the zip file, copy the folder to a location where you can import the virtual machine. For installation, follow the steps below. Step 1: From the VirtualBox menu bar, select File > Import Appliance 6.2. Test Drive CloudN in NAT Mode 51

58 Step 2: Navigate to the CloudN ovf file and click Next Step 3: In the next screen, click on Import to start the import process and wait for it to finish 52 Chapter 6. Test Drive CloudN on Your Laptop

59 Step 4: CloudN virtual machine installation is finished and it can be launched by selecting it and clicking on the Start button Configure Network Interfaces CloudN network interfaces should be configured in bridge mode as the NAT mode makes it impossible for guests to communicate with each other. In addition to this, both interfaces should be allowed to be in promiscuous mode. Execute the steps below to satisfy these requirements Test Drive CloudN in NAT Mode 53

60 Step 1: Select the CloudN VM and click on Settings Step 2: In the settings window, select Network and select Bridged Adapter in the drop down list for the Attached to field. Step 3: Click on Advanced to reveal advanced configuration options and select Allow All in the drop down list for Promiscuous Mode field. Repeat this procedure for Adapter 2 as well. 54 Chapter 6. Test Drive CloudN on Your Laptop

61 6.3 Booting Up and Initial Configuration CloudN supports browser based GUI Interface and REST APIs. After the virtual machine boots up, you must first login into the machine while still in hypervisor console. CloudN Login User Name: admin CloudN Login Password: Aviatrix123# After this initial login, if you see the screen below: 6.3. Booting Up and Initial Configuration 55

62 Follow the instruction to type help at the prompt. Follow the steps to go through the boot up process. You can type help at any time to review the steps. Type? to view all available commands. For each command, type? to view syntax and parameters. 56 Chapter 6. Test Drive CloudN on Your Laptop

63 6.3.1 Step 1: Setup Interface Address CloudN works by dividing the subnet where CloudN is deployed into sub-segment where each sub-segment becomes the VPC/VNet CIDR in the cloud. We recommend you deploy CloudN in its own subnet to maximize the number of VPC/VNets you can create Statically assign CloudN IP address You can statically assign an IP address to CloudN. Choose this approach if you use CloudN to connect to an existing VPC. In the use case where CloudN does not create a VPC and build encrypted tunnel, CloudN does not need to be deployed on a separate subnet. Command: setup_interface_static_address Syntax: setup_interface_static_address [static_ip_address] [net_mask] [default_gateway_ip_address] [primary_dns_server_ip_address] [secondary_dns_server_ip_address] [proxy {true false}] Below is an example where there is no proxy server. In such case, CloudN will configure the network interfaces, test Internet connectivity and download the latest Aviatrix software Proxy Configuration If there is proxy server for Internet access, you must setup proxy configuration on CloudN to pass traffic to proxy correctly. Following is the command command: setup_network_proxy syntax: setup_network_proxy <action> < http_proxy> < https_proxy> where action is test or save. Example: 6.3. Booting Up and Initial Configuration 57

64 setup\_network\_proxy test --http\_proxy --https\_proxy setup\_network\_proxy save --http\_proxy --https\_proxy Note after proxy configuration is saved, CloudN VM will reboot to have the proxy take effect Step 2: Display Interface Address Now you can use the cloudn IP address as URL to access CloudN Manager that manages CloudN. Note: The hypervisor console has only limited CLI for initial booting up purposes. Once Aviatrix software is downloaded, full commands are installed. User should use the GUI to access CloudN Console Troubleshooting If there is any error messages during installation, it is usually due to lack of Internet connectivity, incorrect DNS server IP address or unopened firewall ports. Type? to see all the commands that help you troubleshoot. Use command *ping* and *traceroute* to check out Internet connectivity. Check your DNS server setting, consult your network and server admin to determine the cause of routing failure. After connectivity issue is resolved, use command download_cloudn_software to continue installation and finish. Or you can again type in command setup_interface_address Use a Browser to Access CloudN CloudN has a built in CloudN Console that let you run provisioning from a browser. Once IP addressed setup is complete, you can use any browser, type address of CloudN> and see a Login page. 58 Chapter 6. Test Drive CloudN on Your Laptop

65 Login with: User Name: admin Password: private IP address of the VM After login, go through the initial setup process. For the first time user and initial setup, follow Onboarding to go through the initial set up and launch your first VPC/VNet. 6.4 Onboarding After you login to the browser console, click Onboarding to go through a few steps of initial setup and start creating the first VPC/VNet. Once you login, click on Help for Frequently Asked Questions (FAQs). All features have descriptions and should be self-explanatory. For support issues, send to support@aviatrix.com. For feedback and feature request, click Make a wish at the bottom of each page. Enjoy! 6.4. Onboarding 59

66 60 Chapter 6. Test Drive CloudN on Your Laptop

67 CHAPTER 7 Frequently Asked Questions Aviatrix product consists of a controller and gateways. When the product is deployed in the public cloud marketplace, what you launch is the controller instance and from controller console you launch gateways. When the product is deployed as a virtual appliance in a virtualized datacenter environment, the controller and gateway is bundled into one virtual image, such as OVF and VHD. The following FAQ discusses only the cloud deployment scenario. 7.1 Aviatrix Cloud Gateway Q1: What can it do for me? Aviatrix Cloud Gateway provides an end to end secure network solution for AWS, Azure and Google GCloud. The solution includes an enterprise OpenVPN access to VPC/VNet, encrypted routing among VPC/VNets and monitoring and logging of link status and latency. The solution enables you to build a secure private network spanning one or more public clouds where a user access any instance/vm with a private IP address directly. No more bastion stations and jump hosts, the solution gives user the seamless experience that they enjoy when using the on-prem network. Aviatrix Cloud Gateway supports encryption over AWS Direct Connect and Azure Express Route. In additioin, the product interoperates with any third party IPSEC capable devices, including AWS VGW and Aviatrix s own on-prem virtual appliance CloudN. Architecturally, Aviatrix solution is a centrally managed, loosely coupled and globally deployed platform built for the cloud from the ground up. Q2: What are the key features? Manage all your cloud networking requirements from a central controller. Peering Features point and click encrypted peering applied to multi VPCs, multi regions and multi clouds (AWS, Azure and GCloud) enables you to build a partial or full mesh network. Transitive peering enables you to build a hub and spoke network with ease. 61

68 Cluster peering enables you to build encrypted tunnel with a performance that scales to 10Gbps. Site2Cloud Features Secure connection to remote branch sites and interoperability with legacy router/firewall device. Encryption over AWS Direct Connect and Azure Express Route. OpenVPN Features Scalable and highly available OpenVPN solution. Integrated with cloud provider native ELB, the solution scales out to unlimited number of users and bandwidth. Supports multi factor authentication: DUO, LDAP and OKTA. User profile defined dynamic security access rules that allow administrator to determine access privilege of any given user to any resources at the network perimeter. Supports Geo VPN for a global VPN solution deployment where a VPN user automatically connects to a nearest VPC. Supports wide range of clients: Windows, OSX, Linux, Android, ios, and Chromebook. Supports event logging with SumoLogic, Logstash, Splunk and remote syslog server. Supports Split tunnel and full tunnel mode. No extra hop to other VPC/VNets. Policy based stateful firewall at the VPC level for both access and deny to apps. Environment Stamping solution for repeatable enterprise SaaS deployment. Create identical VPC environments with one click for each customer. Uniquely mapping and addressing instances for CloudOps and developers access. Integrate AWS Route 53 DNS name service for each accessing. Security Features Stateful firewall on each gateway that controls traffic in and out the VPC. Fully Qualified Domain Name (FQDN) whitelists control Internet bound egress traffic from instances on private subnets. Q3: How do I launch the product? The product consists of two components, the controller and one or more gateways. The gateway is launched from the controller. The controller provides a central console for all provisioning, monitoring and upgrades of the services. The controller is available in AWS and Azure marketplace. It is also available as a GCloud community image. For marketplace launch, search for Aviatrix in marketplace. Follow Getting Started instructions to launch controller. Q4: How do I access the controller? Once you have launched the instance, you access the Controller instance via a web browser. Login with username admin. The first time password is the private IP address of the controller instance. You are required to change the password at your first login. 62 Chapter 7. Frequently Asked Questions

69 Q5: How do I secure the controller? Only TCP port 443 needs to be opened for inbound traffic to the controller. If you wish to reduce the scope of source addresses by specifying custom IP address, you must include all gateway public IP addresses, in addition to your own public IP address. This is because gateways launched from the controller use its public IP address to communicate back to controller. Q6: Is Aviatrix Cloud Gateway a SaaS offer? No. Aviatrix Cloud Gateway is a software product that is deployed in your own network perimeter. 7.2 Onboarding Q1: Where do I start? The first time when you login, complete Onboarding process. It takes a few steps. If you have a BYOL license or use a community image, you need to have a customer ID provided by Aviatrix to be able to use the product. Contact support@aviatrix.com if you do not have a customer ID. Q2: What is an Aviatrix Cloud Account? An Aviatrix Cloud Account is specific and unique on the controller. It contains cloud credentials, for example, your AWS IAM Access Key ID and Secret Key. The controller uses these credential to launch Aviatrix gateways by using cloud APIs. An Aviatrix Cloud Account can correspond to multiple cloud account. For example, it can contain credentials for an AWS IAM account, Azure account and GCloud account. Q3: How do I upgrade software? Click Settings -> Upgrade. This upgrades to the latest release of the controller software. When a new release becomes available, an alert message appears on Dashboard. An will also be sent to the admin of the controller. Q4: Is there a reference design example? Check out docs.aviatrix.com. Q5: What is the support model? For support, send to support@aviatrix.com. We also offer premium customers with 24x7 support. To request a feature, click Make a wish button at the bottom of each page. 7.3 Scale Out VPN Solutions Q1: How do I launch a VPN gateway? Click Gateway -> + New Gateway The controller launches an Aviatrix gateway instance in AWS/Azure/GCloud. The gateway instance must be launched from a public subnet. You need to give it a name (The name is presented as a Gateway Name field), this name becomes part of the instance name with a prefix CloudOps. In the Create page, select VPN Access to enable OpenVPN server capability. There is a default VPN CIDR /24. But you can change it, make sure the CIDR is outside the existing and future VPC CIDR range. This VPN CIDR is where VPN server assign virtual IP address to each user when she connects Onboarding 63

70 You can select Save Template to save the gateway template. When you come to the page the next time, most of the fields are pre populated. You may change any of the fields. Q2: How do I scale out VPN solution? You can launch multiple VPN gateways in the same VPC at the Create Gateway time. While launching a gateway, select yes for Enable AWS ELB. This will automatically create an AWS ELB (for the first gateway) and register the gateway with the newly created load balancer. VPN traffic will be load balanced across these multiple gateways. It is required to have consistent gateway configuration when ELB is enabled. For example, authentication methods, tunnel modes and PBR configurations should be identical. Q3: How do I setup Okta authentication for VPN? Aviatrix vpn gateway integrates seamlessly with Okta. It can authenticate vpn users to Okta service using Okta s OpenVPN pluggin in module. Follow the link: How to setup Okta for Aviatrix VPN gateway Q4: How do I enable Geo VPN? If you have global workforce that needs to access the cloud, Geo VPN offers a superior solution. Geo VPN enables a VPN user to connect to a nearest VPC that hosts Aviatrix VPN gateway. To enable Geo VPN, go to OpenVPN -> GEO VPN. Also check out this link for help. Q5: How do I add a VPN user? After at least one gateway is created, you can add VPN users. Click OpenVPN -> VPN Users -> +Add New. When a user is added, an is sent to the user with instructions on how to download client software and connect to VPN server. If you like to assign user profile based policies, you need to create profiles first, see the next section. Q6: What user devices are VPN client software supported? Windows, MAC, Linux, Chromebook, Android and ios devices are supported. Q7: Is NAT capability supported on the gateway? Yes, you can enable NAT function at gateway launch time. When enabled, instances on the private subnet can access Internet directly. If full tunnel mode is selected, you may want to enable NAT to allow instances in the VPC to have direct Internet access. Q8: Is full tunnel mode supported on the gateway? Yes, both split tunnel and full tunnel modes are supported. You can specify the mode at the gateway launch time. Full tunnel means all user traffic is carried through the VPN tunnel to the gateway, including Internet bound traffic. Split tunnel means only traffic destined to the VPC and any additional network range is carried through the VPN tunnel to the gateway. Any Internet bound traffic does not go through the tunnel. Q9: Can the maximum number of simultaneous connections to VPN gateway be configured? Yes, you can set the maximum number of connections at the gateway launch time. 64 Chapter 7. Frequently Asked Questions

71 7.4 Site2Cloud VPN Q1: Occasionally my tunnel will show as down since there is no interesting traffic transversing, how do I keep it up? This problem can be solved by disabling dead peer detection on your edge firewall. 7.5 User Profile Based Security Policies Q1: What is user profile based security policy? In VPN access, a user is dynamically assigned a virtual IP address when connected to a gateway. It is highly desirable to define resource access policies based on the users. For example, you may want to have a policy for all employees, a different policy for partners and a still different policy for contractors. You may even give different policies to different departments and business groups. The profile based security policy lets you define security rules to a target address, protocol and ports. The default rule for a profile can be configured as deny all or allow all during profile creation. This capability allows flexible firewall rules based on the users, instead of a source IP address. Q2: How do I setup profile based security policies? When a user connects to a VPC, the security policies associated with the profile that the user is assigned to are applied to the VPN gateway instance that user logs in. This effectively blocks traffic from entering the network. Click OpenVPN -> Profiles -> +New Profile to create profiles, then click Edit Policies to add rules. You can add multiple of them, then click on Save. Q3: How do I assign a user to a profile? When you create a VPN user at OpenVPN -> VPN Users -> +Add New, you can select profile option to assign the user to a specific profile. You can also attach the user to a profile at a later time. Go to OpenVPN -> Profiles. Click Attach User on a specific Profile and select a user that is added to the VPN gateway. Q4: What if I want to change profile policies? You can change profile policies any time. However, the users who are currently active in session will not receive the new policy. The user need to disconnect and reconnect to VPN for the new policy to take effect. Q5: How do I change a user s profile programmatically? The controller provides a REST API which can be invoked to change a user s profile. Refer to API document under Help menu. During this operation, the user s existing VPN session will be terminated. The new profile policy will take effect when he or she logs in again. The use case for this feature is to allow administrator to quarantine a VPN user for security reasons. 7.6 User Authentication Q1: Is DUO multi-factor authentication supported? Yes. If your enterprise has a DUO account with multi-factor authentication, it can be integrated into the VPN solution. From Gateways tab, click Create. At two-step authentication drop down menu, select DUO, then enter your company Integration Key, Secret Key and API hostname Site2Cloud VPN 65

72 To obtain Integration Key, Secret key and API hostname, login to DUO website as an admin, click on the left panel Applications, click Protect an Application below. Scroll down the application list and select OpenVPN (click Protect this Application), the next screen should reveal the credentials you need to configure on the Aviatrix controller. For additional help, follow this instruction. Currently advanced feature such as Trusted Device and Trusted Networks are not supported. Send us a request if you like to integrate these features. Q2: How do I configure LDAP authentication? LDAP configuration is part of the Gateway creation when VPN Access is enabled. Enter the necessary parameters and click Enable button to enable LDAP authentication for VPN clients. If your LDAP server is configured to demand client certificates for incoming TLS connections, upload a client certificate in PEM format (This certificate should contain a public and private key pair). Q3: Can I combine LDAP and DUO authentication? Yes. With both LDAP and DUO authentication methods enabled on a gateway, when launching the VPN client, a remote user will have to enter his or her LDAP user credentials and then approve the authentication request received on a registered mobile device to login to VPN. Q4: Is OKTA supported? Yes. OKTA with MFA is also supported. Follow the instructions 7.7 Policy Based Routing Q1: How does Policy Based Routing (PBR) work? When PBR is enabled at gateway launch time, all VPN user traffic arrives at the gateway will be forwarded to a specified IP address defined as PBR default gateway. User must specify the PBR Subnet which in AWS must be in the same availability zone as Ethernet 0 interface of the gateway. When PBR feature is combined with encrypted peering capability, VPN user should be able to access any instances in the peered VPC/VNets. This helps build an end to end cloud networking environment. For details, check out our reference design. Another use case for Policy Based Routing is if you like to route all Internet bound traffic back to your own firewall device on Prem, or log all user VPN traffic to a specific logging device, PBR lets you accomplish that. 7.8 Logging and Monitoring Q1: How do I forward syslog events to my Logstash server? Click on Settings-> Logging ->LogStash logging and input the required parameters to enable forwarding of controller syslog events and all gateways syslog and auth log to a Logstash server. SUMO Logic, Splunk, DataDog and rsyslog are also supported. Q2: What are the monitoring capabilities? Encrypted tunnel (peering and site2cloud) status is monitored. When a tunnel status changes, an alert is sent to the controller admin. Active VPN users are displayed on the Dashboard. Click on any username, the user VPN connectivity history is displayed. 66 Chapter 7. Frequently Asked Questions

73 You can also disconnect a user from the dashboard. Q3: Can alert be sent to a different address? Yes, you can choose an alternative address to send alert messages. This is useful if the controller admin is different from the operation team. 7.9 Encrypted peering Q1: What can Aviatrix encrypted peering do? Aviatrix encrypted peering builds an encrypted tunnel between two VPC/VNet with a single click. In addition to build the encrypted connection, the controller also program the cloud infrastructure routing table so that you don t have to. The VPC and/or VNet can be across region and across cloud. The solution enables you to build a full mesh encrypted network. You can enable stateful firewalls on each VPC/VNet to add additional security measures. Q2: How do I configure encrypted peering? Step 1: Gateway -> +New Gateway in one existing VPC/VNet. VPN access may be disabled. Step 2: Repeat Step 1 with a different VPC ID or VNet Name. Step 3: At Peering -> Encrypted Peering -> +New Peering. Select the two gateway names and click OK Environment Stamping Networking Q1: What does Environment Stamping networking feature do? Environment Stamping (envstamping) takes advantage of the unique nature of Virtual Private Cloud (VPC) and offers a deployment architecture that is secure and scalable. envstamping provides a deployment solution where you can create identical environments such as identical VPC CIDRs and access instances in the VPC seamlessly and securely via encrypted tunnel, as shown in the picture below: 7.9. Encrypted peering 67

74 In the above picture, each managed VPC shares identical CIDRs, instances private IP addresses and security groups. CloudOps and developers access VPC instances by connecting to the gateway in the management VPC via Aviatrix VPN capability. Q2: Who should be deploying this model? This deployment model allows for infinite scale of deployment, it is suitable for SaaS providers, development and testing. With this model, SaaS provider can offer secure and single tenant to its enterprise customers, while being able to access instances for maintenance and support. For example, a SaaS provider can offer an enterprise customer its own AWS account and VPC environment. Customer data is completely isolated from others. Only authorized personal can access customer instances for maintenance and troubleshooting. Q3: What is the workflow to enable this feature? Refer to this link for workflow steps Administration Q1: Can there be multiple admins? Yes. Username admin is the default admin user. But you can create multiple users with admin privilege. Follow the instructions to learn more about setting up multiple admin users. Q2: Is there 2FA support to log in to the console? Yes. In addition to password login, DUO authentication and LDAP are supported. Q3: Can there be read only account for operation team? Yes. Accounts -> Account Users -> Add A NEW USER, at Account Name field, select read_only from the drop down menu. This user account will have views to all pages but cannot make changes to any configurations. OpenVPN is a registered trademark of OpenVPN Inc. 68 Chapter 7. Frequently Asked Questions

75 CHAPTER 8 Onboarding and Account FAQs 8.1 Where do I start? The first time when you login, complete Onboarding process. It takes a few steps. If you have a BYOL license or use a community image, you need to have a customer ID provided by Aviatrix to be able to use the product. Contact support@aviatrix.com if you do not have a customer ID. 8.2 What is an Aviatrix Cloud Account? An Aviatrix Cloud Account is specific and unique on the controller. It contains cloud credentials, for example, your AWS IAM Access Key ID and Secret Key. The controller uses these credential to launch Aviatrix gateways by using cloud APIs. An Aviatrix Cloud Account can correspond to multiple cloud account. For example, it can contain credentials for an AWS IAM account, Azure account and GCloud account. 8.3 How do I upgrade software? Click Settings -> Upgrade, select latest. This upgrades to the latest release of the controller software. When a new release becomes available, an alert message appears on Dashboard. An will also be sent to the admin of the controller. 8.4 Are there reference design examples? Check out docs.aviatrix.com. 69

76 8.5 What is the support model? For support, send to We also offer premium customers with 24x7 support. To request a feature, click Make a wish button at the bottom of each page. 70 Chapter 8. Onboarding and Account FAQs

77 CHAPTER 9 Account Aviatrix Controller is a multi cloud platform. The Controller uses your cloud API credentials to make APIs calls, for example, to launch an Aviatrix gateway instance, on behalf of your cloud credentials. One cloud credential is represented as an Aviatrix account on the Controller. The Controller supports multiple Aviatrix accounts. One Aviatrix account may represent multiple cloud credentials, one from each cloud. For exmaple, an Aviatrix account name DevOps can have an IAM role for AWS, Azure ARM credential and GCP credential. For AWS, the account information consists of IAM roles and policies. For Azure, the account information consists of Azure ARM credentials. For GCP (Google Cloud), the account information consists of GCP credentials. You can create an Aviatrix account during onboarding process. You can also create an Aviatrix account by clicking Accounts at the navigation bar on the left side of the Controller web console. 71

78 72 Chapter 9. Account

79 CHAPTER 10 IAM Role With the support of AWS IAM role, there is no need to enter AWS access key and secret key when creating a cloud account on Aviatrix controller. Instead, two IAM roles will be created. Aviatrix controller will use the dynamically obtained security credentials to request access to AWS resources. Role-based IAM cloud account helps to reduce the risk of AWS credentials being compromised. This document provides instructions to create the IAM roles and policies. If you like to customize the conditions of the policies published by Aviatrix, consult this link. To use IAM role, the Aviatrix Controller you launch must have IAM role enabled Aviatrix Controller Launched from CloudFormation If you launched the Aviatrix Controller from our CloudFormation script, both IAM roles aviatrix-role-app and aviatrix-role-ec2 and their assoicated policies have already been created at the CloudFormation stack launch time. When you create an Aviatrix cloud account on Aviatrix Controller console, simply follow these steps to retrieve the fields for aviatrix-role-app ARN and aviatrix-role-ec2 ARN. Login to AWS portal Go to Services -> IAM -> Roles, you should see two roles have been created, aviatrix-role-app and aviatrixrole-ec2. Click role aviatrix-role-app, copy the Role ARN string to fill in the field for aviatrix-role-app ARN when creating a cloud account. Click role aviatrix-role-ec2, copy the Role ARN string to fill in the field for aviatrix-role-app ARN when creating a cloud account. Done and may skip the rest of the guide. 73

80 10.2 Setup IAM policies and roles for your own account If you launched or plan to launch the Aviatrix Controller manually with IAM role from AWS marketplace portal, proceed to complete the following steps. Before you launch an Aviatrix Controller from AWS marketplace, create the two necessary IAM roles and its corresponding policies Step 1. Create two IAM custom policies 1.1 Create aviatrix-assume-role-policy : Log in in to AWS console with your own account. Go to Services -> IAM -> Policies -> Create Policy -> Create Your Own Policy Enter the policy name, aviatrix-assume-role-policy, copy and paste the policy text from this link. Click Valid Policy to validate the policy. Click Create Policy button. 1.2 Create aviatrix-app-policy : Log in to AWS console with your own account. Go to Services -> IAM -> Policies -> Create Policy -> Create Your Own Policy Enter the policy name, aviatrix-app-policy, copy and paste the policy provided by this link into Policy Document section. In this example, the policy name is aviatrix-app-policy, as shown below. Click Create Policy button Step 2. Create Two IAM Roles 2.1 Create aviatrix-role-ec2 role This role will be associated with the Aviatrix Controller. The role name MUST be exactly aviatrix-role-ec2. Go to AWS console -> IAM service -> Roles -> Create role 74 Chapter 10. IAM Role

81 Select AWS Service -> EC2 -> EC2 -> Next: Permissions Search Policy aviatrix-assume-role-policy, then select this policy. Click Next Review Setup IAM policies and roles for your own account 75

82 Enter Role name aviatrix-role-ec2 (must be exact) then click [Create] Search/Check the role. You should see something like this for Role ARN: arn:aws:iam::575xxxxxx729:role/aviatrix-role-ec2 Make a note of the above Role ARN string, it will be used for setup Aviatrix Cloud Account later 2.2 Create aviatrix-role-app role This role is to be assumed by a granted AWS account. The Aviatrix controller acquires the assume role capability authorized by its aviatrix-ec2-role role. It then assumes to this service role that is granted by its own AWS account or other AWS accounts to perform AWS APIs. Go to AWS console -> IAM service -> Roles -> Create Role Select Another AWS account, and enter your AWS account ID, then Click [Next:Permissions] 76 Chapter 10. IAM Role

83 Select aviatrix-app-policy IAM policy, then click [Next: Review] Enter a Role Name, in this case aviatrix-role-app. Click Create role You should see something like this for Role ARN: arn:aws:iam::575xxxxxx729:role/aviatrix-role-app Make a note of the above Role ARN string, it will be used to setup Aviatrix Cloud Account later Setup IAM policies and roles for a cross account Aviatrix supports multiple AWS account. To launch a gateway for a different AWS account, you must create the same IAM policies and roles listed above for the second account (or third, fourth, etc.). The only difference is that the IAM Setup IAM policies and roles for a cross account 77

84 role in the non-primary account must trust the primary account. Instructions: From the secondary account 1. Create the IAM policies and roles listed above (Setup IAM policies and roles for your own account). (a) Remember to note the ARN identifier for both roles. 2. Grant the primary account access to the aviatrix-role-app in the second account (a) AWS console -> IAM service -> Roles > aviatrix-role-app (b) Click Trust Relationships > Edit Trust Relationship (c) Edit the trust relationship as follow 3. Done (d) Click Update Trust Policy Repeat this procedure for each non-primary AWS account that will be managed by Aviatrix. 78 Chapter 10. IAM Role

85 CHAPTER 11 Azure ARM Overview This document helps you to setup API credentials on Azure ARM. Aviatrix Cloud Controller uses Azure APIs extensively to launch Aviatrix gateways, configure encrypted peering and other features. In order to use Azure API, you need to first create an Aviatrix Cloud Account on the Aviatrix Cloud controller. This cloud account corresponds to a valid Azure account with API credentials. The new Microsoft Azure (as opposed to Azure Classic) is significantly different in how applications are authenticated and authorized to interact with Azure Resource Manager APIs to manage resources, such as Virtual Machines, Network, Storage Accounts, etc. This document describes how to obtain the necessary information, specifically Application Client ID, Application Client Secret, and Application Endpoint to create an Aviatrix Cloud Account with step by step instructions. There are 3 sections, make sure you go through all of them Azure Permission Setup for Aviatrix Setting up Azure permission for Aviatrix involves three main steps. 1. Register Aviatrix Controller Application with Azure Active Directory 2. Grant Permissions 3. Get Application Client ID, Application Client Secret and Application Endpoint Important: Complete the follow steps in order Step 1 Register Aviatrix Controller Application Login to the Azure Portal. 79

86 *Register Aviatrix Controller* 1. From the Azure portal click on Azure Active Directory and then App registrations 2. Click + Add (a) Name = Aviatrix Controller (b) Application Type = Web app / API (c) Sign-on URL = (d) Click Create. 3. Done Step 2 Grant Permissions *Grant Permissions* 1. Login to the Azure portal 2. On the bottom left, click More services, search for Subscriptions 3. Copy the Subscription ID (to notepad or a convenient location) 4. Click on the Subscription ID 5. Then select Access control (IAM). 6. Click Add and then select the Contributor role. 7. In the User search field, type in Aviatrix. The Aviatrix Controller app should show up. Select this one and click Select towards to the bottom. 80 Chapter 11. Azure ARM

87 Step 3 Get Application Information *Get Application Information* 1. From the Azure portal, click More services and search for Azure Active Directory. a. Retrieve the Application Endpoint ID. i. Scroll down the Azure Active Directory panel and Click on Properties ii. Copy the Directory ID (this is the Application Endpoint ID) b. Retrieve the Application Client ID. i. Click App registrations ii. Copy the Application ID this is the Application Client ID) c. Retrieve the Application Client Secret i. Click App registration -> Keys ii. Enter in the following 1. Description = Aviatrix 2. Expires = Never expires iii. Click save iv. Copy the key value d. Add App permissions Azure Permission Setup for Aviatrix 81

88 i. Click App registration -> Required permissions -> Add ii. Select an API -> Windows Azure Service Management API iii. Select Access Azure Service Management as organization user iv. Done 2. Done At this point you should have the following information. Subscription ID From step 2 Application Endpoint ID From step 3 Application Client ID From step 3 Application Client Secret From step 3 82 Chapter 11. Azure ARM

89 CHAPTER 12 GCP Credentials Before creating a cloud account for GCloud/GCP on Aviatrix controller, go through the steps below to make sure you have the credentials setup for API calls Step 1: Create a GCloud Account Create a GCloud or GCP account ( Go on to the next step if you have already done so. Note that the controller supports multiple accounts with each one associated with a different GCloud projects, but there needs to be at least one to start with Step 2: Create a GCloud Project Login to your GCloud account and go to project page: Create a project. Go on to the next step if you have already created one. Note the project ID will be used in referencing to the project by Aviatrix controller. (As an example, we created a project Aviatrix-UCC, the project ID is aviatrix-ucc-1214) 12.3 Step 3: Enable Compute Engine API Enable Compute Engine API on the selected project, 1. go to your Google Cloud Platform console, at the upper left corner left to Google Cloud Platform signage, click the 3 bars. A drop down menu will appear. 2. Select API Manager, at Google Cloud APIs, select Compute Engine API. 3. click Enable. 83

90 12.4 Step 4: Enable GCloud Messaging Service Aviatrix controller uses GCloud Pub/Sub messaging service to communicate with the gateways. To enable Pub/Sub on the selected project, 1. go to your Google Cloud Platform console, at the upper left corner left to Google Cloud Platform signage, click the 3 bars. A drop down menu will appear. 2. Select API Manager, at Google Cloud APIs, click more to expand. select Cloud Pub/Sub API, as shown below, then click Enable Step 5: Create Credential File When you create a cloud account for GCloud, you are asked to upload a GCloud Project Credentials file. Below are the steps to download the credential file from Google Developer Console. 1. Open the Credential page 2. Select the project you are creating credentials for. 3. At Credentials, Click Create credentials, select Service account key, as shown below 84 Chapter 12. GCP Credentials

91 4. At the Service account dropdown menu, select Compute Engine default service account, select JSON. 5. Click Create. The credential file will be downloaded to your local computer. 6. Upload the Project Credential file to Aviatrix controller at GCloud account create page Troubleshooting Tips If cloud account creation fails, check the error message at the Aviatrix controller console and try again with the steps provided in this document. For additional support, send to Troubleshooting Tips 85

92 86 Chapter 12. GCP Credentials

93 CHAPTER 13 Admin Users and Duo Sign in 13.1 Objectives This document describes a reference design using Aviatrix Cloud console s user management and Duo authentication capability to manage multiple users with admin privilege. As the cloud Ops team continues to expand to manage more cloud deployments, it is often required to give each member in the team her own username and password with admin privilege. In addition to username and password for log in credentials, a 2FA authentication can be added for enhanced security to manage cloud controller. DUO authentication is one of the supported methods. When enabled, it requires the user to accept a push message on the user s mobile device from DUO service in addition to username and password at the user log in time. The following diagram illustrates the user relationship in a typical cloud Ops department. In this example, the Ops team has created two cloud accounts. A cloud account is associated with one or more distinct cloud provider s API credentials. Typically, a cloud account corresponds to an IAM account of a distinct AWS and/or Azure account with a credit card. A default user admin is created by the system. In the picture below, admin has created two cloud accounts, each with a default user names bearing the cloud account name. Additional users in admin and cloud account are added by the admin or admin users. Note that the default user created from the cloud account can only access information specific to its account from the console. The default user cannot be changed in the Users page described below Configuration Workflow for Admin Users Add a new admin user The page can be found at Accounts -> Account Users -> +New User. After the first cloud account is created, additional admin users can be added from this page. The page can only be accessed by the admin or an admin user. Initially, the default users will be displayed. Once a new admin user is added, it will be added to the list. Only the new users added by this page can be changed or deleted. 87

94 When an admin user is created, changed, or deleted, an is sent to the admin s address as a record for bookkeeping purpose. After the user is added, the user can log into the console with the specified user name and password. The user then has full access to the console like the admin. When the user login to the console, the admin username will be displayed on the top of the right hand corner Delete an admin user The same page can be used to delete an admin user when the user leaves the group or the user role changes. After the delete button is clicked, a confirmation is sent to the admin s address. Note that an admin cannot be deleted by himself or herself though the user has the full console access. Typically, the admin user is added or deleted by the special username admin Change admin user s password Again, the admin user s password can be changed at the same page. An notification is sent to the admin s address after the change is successfully done Configuration Workflow for Duo Authentication Getting DUO API credentials Follow the instruction in to setup DUO API credentials on DUO Security website Create Duo Authentication To enable DUO, go to Settings -> Setup 2FA Login Enter Duo integration key, secret key, and API hostname of your account in DUO website described earlier. Currently only DUO push is supported. Once it is created successfully, the Duo push login applies to all users, including user admin. Every user (listed in settings -> Manage Accounts -> Users) who wishes to login to the system must have a matching user name in DUO account Remove Duo Authentication The Duo authentication setup can be removed completely by clicking the Remove button on the same page Disable/Enable Duo Authentication The authentication can be disabled or enabled without deleting the DUO credential configuration API Server Check This button can be used to troubleshoot Duo API server connectivity when the API failure is occurring. 88 Chapter 13. Admin Users and Duo Sign in

95 CHAPTER 14 Aviatrix Companion Gateway V2 If you need to launch a gateway in Azure ARM, you must subscribe to Aviatrix Companion Gateway V2 in Azure Marketplace. This model removes the requirement to download the Aviatrix gateway image into your Azure account which typically takes more than 30 minutes, thus greatly reduces the deployment time. The Aviatrix Companion Gateway V2 in Azure marketplace is free of charge. The following steps describe how to subscribe Aviatrix Companion Gateway V2 in Azure marketplace Step 1: Select Aviatrix Companion Gateway V2 Go to Azure Marketplace, search aviatrix, select [aviatrix-companion-gateway-v2], as shown below: 89

96 14.2 Step 2: Deploy Programmatically After you click Create Virtual Machine for Aviatrix Companion Gateway V2, click Want to deploy programmatically? Get started -> at the bottom of the page, as shown below: 14.3 Step 3: Enable subscription In the next step, select [Enable subscription], click [Save], as shown below: 90 Chapter 14. Aviatrix Companion Gateway V2

97 That s it! For support, send to support@aviatrix.com Step 3: Enable subscription 91

98 92 Chapter 14. Aviatrix Companion Gateway V2

99 CHAPTER 15 Reserve For On-Prem Use The Datacenter Cloud InterConnect (DCCX) feature works by dividing a VLAN or subnet into sub segments. Each sub segment becomes a CIDR block for VPC/VNet. If you want to reserve some of the sub segments for on-prem use, i.e., to launch VMs on these subnets, you can do so by reserving some CIDR blocks. One use case for this feature is for cloud burst. The VMs launched on a reserved subnet will treat instances in VPC/VNet as if they are on the same VLAN. If you have an application that requires the on-prem resource and in the cloud resource to be on the same subnet/vlan, this deployment will satisfy that. Note: This feature is available for R2.6 and later. 93

100 94 Chapter 15. Reserve For On-Prem Use

101 CHAPTER 16 Quick Tour 16.1 Scale out remote user VPN Solution No more bastion stations and jump hosts. Provide your employees with the ability to seamlessly access instances with private IP addresses by using our user VPN capability. To configure Cloud VPN: 1. At Gateway menu, create a gateway with VPN access enabled. 2. Repeat the above step for multiple gateways if ELB is enabled to create a scale out VPN solution. 3. (Optional) At OpenVPN -> Profiles, define VPN user profiles and access policies for each profile that will be dynamically enforced as user connects to the cloud at the network perimeter. 4. At OpenVPN -> VPN Users, add VPN users. 5. For single VPC user vpn solution, check out this link. 6. For a multi VPC user vpn solution, check out this reference design 16.2 Encrypted Peering 1. At Gateway menu, create a gateway in an existing VPC/VNet. 2. Repeat the step 1 for a different VPC/VNet. 3. At Peering -> Encrypted Peering, click New Peering to peer the two gateways. 4. For a complete end to end solution, check out this reference design Geo VPN If you have a global work force and would like to give your employees the best user experience accessing the services in the cloud, Geo VPN is the right solution for you. Go to Open VPN -> Geo VPN to enable Geo VPN. Check out this 95

102 reference design Developer s Sandbox If keeping your production environment secure while giving your developers isolated environment to learn and experiment new technologies is a challenge for you, Developer s Sandbox maybe a feature you want to explore Transitive Peering Use transitive peering reference design to see how to connect to your on-prem or co-location Site2Cloud Solution If you need to connect to your partner or customer sites to a VPC/VNet but do not want to replace the edge routers or firewalls that is already deployed at these sites, check out our Site2Cloud reference design Docker Container Access To learn how you can use Aviatrix to access containers remotely in the cloud, check out this reference design Environment Stamping If you wish to provide a differentiated and more secure SaaS services to your enterprise customers, Environment Stamping solution is the right one for you. Environment Stamping enables you to deploy identical and repeatable VPCs environment while providing unique access to all instances in VPC. Never have to manage VPC CIDRs, security policies and instances addresses any more. At VPC/VNet -> Environment Stamping, go to Read Me First to learn about this capability. Check out this reference design Help Under Help menu, check out FAQs and additional implementation guides. Send to support@aviatrix.com to get immediate support. OpenVPN is a registered trademark of OpenVPN Inc. 96 Chapter 16. Quick Tour

103 CHAPTER 17 Gateway 17.1 Launch a gateway Click Gateway at navigation panel. Click New to launch a gateway. To launch a gateway with OpenVPN capability, refer to this link Select Gateway Size When selecting the Gateway Size, note the following guidelines of IPSEC performance based on iperf tests conducted between two gateways of the same size: t2 series throughput is not guaranteed; it can burst up to 130mbps. m3 series are in the range mbps m4.xlarge or c4.xlarge: approximately 500mbps c3.2xlarge or m4.2xlarge: approximately 1Gbps c3.4xlarge: approximately 1.2Gbps c4.2xlarge: 1.2Gbps - 1.5Gbps if you need IPSEC performance beyond 1.2Gbps, refer to Cluster Peering Specify a Reachable DNS Server IP Address Aviatrix gateway is launched with a default public DNS server IP address to make sure the gateway has access to AWS public resources such as SQS for Controller and gateway communication. If you want to change to a different DNS server, select the box for Specify a Reachable DNS Server IP Address to enter an alternative DNS IP address. 97

104 17.4 Enable NAT Aviatrix gateway performs NAT function when this option is selected Allocate NEW EIP When this optioin is selected, Aviatrix gateway allocates a new EIP for the gateway from AWS. When this optino is unchecked, the gateway select one allocated but unassociated EIP from the AWS account from which the gateway is launched VPN Access When this option is selected, Aviatrix gateway is used for SSL VPN termination. It supports OpenVPN client and Aviatrix client. For more details, check out this link Add/Edit Tags Aviatrix gateway is launched with a default tag name avx-gateway@private-ip-address-of-the-gateway. This option allows you to add additional AWS tags at gateway launch time that you can use for automation scripts Designated Gateway If a gateway is launched with Designated Gateway enabled, the Aviatrix Controller programs the RFC1918 address ranges in the route table to point to the gateway instance. These routing entries are /8, /16 and /12. The Controller will not add additional route entries that is within this RFC1918 range when configuring Transit VPC, site2cloud or encrypted peering. However, if the address range is outside the RFC1918 the Controller will add these routes to point to the gateway Security Policy Starting Release 3.0, gateway security policy page has been moved Security -> Stateful Firewall. Check out this guide High Availability There are two types of high availabiity on Aviatrix: Gateway for High Availability and Gateway for High Availability Peering. Gateway for High Availability Peering is used for Enable HA field at Encrypted Peering configuration. Gateway for High Availability is used when you need HA for a Transitive Peering. 98 Chapter 17. Gateway

105 17.11 Gateway Size You can change Gateway Size if need to change gateway throughput. The gateway will restart with a different instance size. OpenVPN is a registered trademark of OpenVPN Inc Gateway Size 99

106 100 Chapter 17. Gateway

107 CHAPTER 18 Transit VPC Workflow Instructions This workflow provides you with a step by step instructions to build a Global Transit Network. It abstracts and combines multiple existing Aviatrix features, such Encrypted Peering, Transitive Peering and Site2Cloud to bring you a wizard like experience so that you do not have to go to multiple pages on the Controller console when building the Transit group. This Global Transit Network consists of a Transit gateway and a set of Spoke gateways, to faciliate communication between Spoke VPC EC2 instances and on-prem network. Note: For description purpose, gateway and GW are used interchangebly. The Global Transit Network diagram is described as below. 101

108 18.1 Planning and Prerequisites 1. Identify a VPC, call it Transit VPC, in a region where you want to launch the Transit GW. 2. Create a VGW in the same region. The VGW can be either attached to the Transit VPC or detached. This VGW can be connect to on-prem either over Direct Connect or over Internet. 3. If this is your first time using Aviatrix, make sure you go through the Aviatrix Controller on-boarding process to create Aviatrix account that corresponds to an IAM role. For instructions on how to launch an Aviatrix Controller, check out this link Login to the Aviatrix Controller Open a browser and navigate to Public IP address>/. Once authenticated, click on Transit VPC in the left navigation bar. Follow the steps below to set up Transit VPC network Launch a Transit Gateway The Transit GW is the hub gateway, it servers to move traffic between a Spoke VPC and on-prem network (Optionally) Enable HA for the Transit Gateway When HA is enabled, a second Transit GW will be launched. Note both Transit GWs will be forwarding traffic in any event of tunnel failure between a Spoke VPC and Transit VPC, and between the Transit GW and VGW Connect the Transit GW to AWS VGW This step builds a site2cloud IPSEC tunnel with VGW and establishes BGP session with VGW to exchange routes between on-prem and the cloud. 102 Chapter 18. Transit VPC Workflow Instructions

109 Launch a Spoke Gateway (Optionally) Enable HA for the Spoke Gateway Join a Spoke GW to Transit GW Group This step attaches a Aviatrix encrypted peering and transitive peering between the Spoke GW and the Transit GW. The Controller also instructs the Transit GW to start advertise the Spoke VPC CIDR to VGW via the established BGP session Launch a Spoke Gateway 103

110 Remove a Spoke GW from a Transit GW Group This step removes the Aviatrix encrypted peering and transitive peering between the Spoke GW and the Transit GW built in the previous step. The Controller also instructs the Transit GW to stop advertising the Spoke VPC CIDR to VGW. Note the Spoke GW is not deleted and you can go to step 6 to join the Transit GW group again. To delete a Spoke GW, go to Gateway on the main navigation tab, select the gateway and click Delete Add More Spoke VPCs Repeat step 4 to 6 to add more Spoke VPCs to the Transit GW group. 104 Chapter 18. Transit VPC Workflow Instructions

111 View the Network Topology You can view the network topology by going to the Dashboard. Click on the Map View to switch to Logical View. In the Logical View, each gateway is represented by a dot. You can rearrange the initial drawing by moving the dot, zoom in or zoom out, move the graph around. After you are done moving, click the Save icon Remove Transit GW to VGW Connection You can remove the BGP and IPSEC connection to VGW via this step. You can go to Step 3 to build the connection again Troubleshoot BGP Under Advanced Config on the main navigation bar, click BGP. The Transit GW will have BGP Mode as Enabled. Click the Transit GW and click Details to see Advertised Networks and Learned Networks. Learned Networks are network CIDR blocks that BGP learned from VGW. Advertised Networks are Spoke VPC CIDRs. You can also click Diagnostics. Select one of the show commands or type in yourself if you know the commands to see more BGP details View the Network Topology 105

112 106 Chapter 18. Transit VPC Workflow Instructions

113 CHAPTER 19 Peering Aviatrix provides a point and click solution to create an encrypted tunnel between two VPCs. The two VPCs could be in the same region, in different regions (inter region) and in different clouds (inter cloud) This guide helps you configure an encrypted peering. For cluster peering, refer to this doc Encrypted Peering 1. At Gateway menu, create a gateway in an existing VPC/VNet. 2. Repeat the step 1 for a different VPC/VNet. 3. Select Enable HA if you wish to build a backup encrypted tunnel for HA. Note that you must first create two respective backup gateways prior to this step. To launch backup gateways, go to Gateway page, select the gateway, click Edit, At Gateway for High Availability Peering field, select one public subnet and click Create. 4. At Peering -> Encrypted Peering, click New Peering to peer the two gateways. 5. Note Over AWS Peering field currently only works when used in conjunction to transitive peering. When this field is selected, the controller does not program the AWS routing table to points peer VPC CIDR routes to the gateway. 6. For a complete end to end solution, check out this reference design. 107

114 108 Chapter 19. Peering

115 CHAPTER 20 Encrypted Transitive Peering As DevOps and applications are now run in AWS, it makes sense to have your employees access cloud directly with the following highlighted benefits: Lower latency. Rather than having your employees connect via VPN to your corporate office first and then access the cloud, provide a cloud VPN where they can access AWS directly. Better Security. Traditional VPN servers do not support modern multi factor authentication methods such as a combination of DUO Security, LDAP and OKTA. Few hardware gears to manage. However, your business may require hosting some critical applications in wide spread co-locations. As a cloud infrastructure engineer, you need to access these sites to check on the health of your servers and applications. The challenge is to setup a system to enable secure accessing abilities to both the cloud and co-locations Solution Our solution is to leverage Aviatrix s encrypted peering and encrypted transitive peering capability to setup an end to end secure network. In this example, a datacenter or co-location hosts some critical customer facing applications. It connects to AWS VPC for additional processing, such as data analytics. The data center connects to a AWS VGW with IPSEC tunnel. Employees and developers access VPC-1 and VPC-2 directly via Aviatrix CloudVPN and encrypted peering configuration. The cloud infrastructure engineers need to access the servers in the datacenter or co-location for maintenance and monitoring purpose. They do so via an Aviatrix encrypted tunnel and Aviatrix encrypted transitive tunnel configuration. The solution diagram is shown below. 109

116 20.2 Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. We assume here that you have created a management VPC-main /16, its corresponding VPN gateways with ELB enabled. For more information for this part of configuration, check out this reference design. If you configure split tunnel mode for VPN gateways, make sure to include the co-location CIDRs in the additional CIDR field. The encrypted transitive peering configuration workflow is as follows, with major steps highlighted. 1. Create a gateway in VPC-2 Go to Gateway -> New Gateway, make sure (a) The gateway has NAT enabled, VPN disabled (as you don t need to enable VPN capability) 2. Create an encrypted peering between VPC-main and VPC-2 Go to Peering -> Encrypted Peering -> New Peering, make sure: (a) At VPC Name 1 drop down menu, select the peering gateway launched in VPC-main (note, this peering gateway is different from the VPN gateway). (b) At VPC Name 2 drop down menu, select the gateway launched in VPC-2. (c) Click Add. 2. Create an encrypted transitive peering 110 Chapter 20. Encrypted Transitive Peering

117 Go to Peering -> Transitive Peering -> New Peering, make sure: (a) At Source VPC drop down menu, select the peering gateway launched in VPC-main (the same VPC gateway selected in the previous step) (b) At Next Hop VPC drop down menu, select the gateway launched in VPC-2 (the same gateway for VPC-2 selected in the previous step) (c) At Destination CIDR, fill in the destination CIDR of the co-location. For example, /24. Note this address should be unique across your network. 3. Repeat the above step 3 for more co-locations. 4. For support, send to support@aviatrix.com. 5. For feature request and feedback, click Make a wish at the bottom of each page. 6. Enjoy! Configuration Workflow 111

118 112 Chapter 20. Encrypted Transitive Peering

119 CHAPTER 21 Cluster Peering 21.1 Performance Challenges Today encrypted peering (IPSEC tunnel) between two VPCs is carried out by two gateways (EC2 instance based), one in each VPC. This limits IPSEC tunnel packet throughput to the throughput of a single instance. For example, AWS C4.4xlarge provides up to 1.5Gbps for an iperf test with TCP. There is no solution for use cases that requires more than that throughput with one gateway instance. AWS infrastructure has its own performance limitation. Traffic leaving a VPC has a bandwidth limit of 5Gbps for one direction and 10Gbps for bi-directional. This limitation applies to both intra-region VPC traffic and Internet bound traffic. For example, running an iperf test between two instances in two VPCs in the same region yields 5Gbps one way throughput and 10Gbps bi-directional traffic Encrypted Cluster Peering Solution Aviatrix has developed a scale out IPSEC capability. A VPC can deploy a cluster of gateways. Encrypted peering between two VPCs is carried out by two clusters of gateways in each VPC. The deployment diagram is shown below. Aviatrix supports both inter-region cluster peering and intra-region cluster peering. In the first case, the encrypted cluster peering is over the Internet through IGW. In the second case, the encrypted cluster peering is over native AWS peering. The deployment diagrams are described below for both cases. 113

120 In this example, three Aviatrix gateways are deployed for encrypted peering between VPC-1 and VPC-2. A demux gateway is used to distribute user instance session traffic to 3 gateways. The distribution algorithm guarantees no packet for the same TCP stream is delivered out of order to the peering VPC Performance Benchmark and Analysis Below is the performance benchmark for cluster peering using the iperf tool. The results are collected with encryption over AWS peering in a same region. The Aviatrix Gateway size is C4.8xlarge. The demux gateway size is C4.8xlarge. As the results shown, with 4 or 5 gateways in a cluster, performance reaches AWS VPC line rate. Adding more gateways does not improve the performance. Note if the gateway size is C4.xlarge, more gateways are needed to achieve AWS line rate. For information on how to run multi stream iperf tests, check out our Github project. AviatrixSystems/PerformanceTest/blob/master/PerformanceTest.txt Chapter 21. Cluster Peering

121 21.4 High Availability The AWS Controller monitors the health of the peering gateways and demux gateway. When heartbeat information from any gateway fails, the Controller will restart the failing gateways. The detection to failover is under 30 seconds Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message (New!) appears, click New! to download the latest software. We assume you already know how to deploy the Aviatrix solution, if you need help, check out this reference design. The Cluster Peering work flow is as follows, with major steps highlighted. 1. Create a gateway in VPC-1 Go to Gateway -> New Gateway to create a gateway in VPC-1 2. Create 2 more gateways in VPC-1 Repeat the above step to create 2 more gateways in VPC-1. Note all gateways instances must be in the same subnet. 3. Create a cluster in VPC-1 Go to Peering -> Cluster Encrypted Peering -> + New Cluster. Make sure you highlight and select all 3 gateways at the Highlight and Select Gateways field. 4. Repeat the above 3 steps for VPC High Availability 115

122 5. Create Cluster Encrypted Peering Go to Peering -> Cluster Encrypted Peering -> Cluster Peering -> +New Peering, enter the two clusters you created in the previous steps. Special Notes. Select Over AWS Peering if the two VPCs are in the same region. Note when this option is selected, you must have AWS peering routing PCX programmed in the routing table only for the subnet where cluster gateway instances are deployed. You must NOT program PCX for routing tables whose associated subnets are where your application EC2 instances are deployed. 1. Once peering configuration is completed, you can view it in dashboard. Cluster peering is drawn with a thicker green line. 2. Note if you wish to add more gateways once a cluster peering has been created, you need to unpeer the cluster peering first. Add more gateways in each VPC, then create cluster peering again. 3. You can create multiple clusters in a VPC. A gateway may also belong to different clusters. 4. For support, send to support@aviatrix.com. 5. Enjoy! 116 Chapter 21. Cluster Peering

123 CHAPTER 22 Multi Cloud: Connecting Azure to AWS and GCP 22.1 Overview Companies are relying more and more on multiple cloud (multi cloud) providers. However, setting up the connectivity between those providers is difficult. And, maintaining and monitoring the tunnels is time-consuming and cumbersome to troubleshoot. Aviatrix simplifies this by providing simple, point-and-click tunnel creation between cloud providers. Additionally, Aviatrix gives you a single, centralized location from which to troubleshoot and monitor your connections. 117

124 22.2 Getting Started The Aviatrix Controller automates, monitors, and reacts to events in each cloud environment on your behalf. In order to do this, we ll need to configure a few things in each cloud to support this. We ll walk through these steps in the following sections. Once complete, you can connect to one or both cloud providers. Start by logging into the Azure Portal Step 1. Install Aviatrix Controller from the Azure Marketplace The first step is to install the Aviatrix Controller from the Azure Marketplace. Select the Aviatrix Cloud Gateway to AWS and GCP from the Marketplace. Configure the new VM to meet your preferences and requirements. Be sure to allow inbound connections on port 443. Once ready, launch the new VM and continue to the next step Step 2. Prepare your Azure Account While the VM is being deployed in the selected region, configure the following items: Register Aviatrix with Active Directory 1. Go to the Azure Active Directory (available from the left navigation panel or More Services) 118 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

125 2. Click on Properties (available under Manage on the left inner naviagation bar) Important: Endpoint. Copy and save the Directory ID for later use. It will be referred to again as the Application 3. Click on App registrations (available under Manage on the left inner naviagation bar) 4. Click on the + New application registration along the top 5. Populate the fields as follows: Field Name Application type Sign-on URL Value Aviatrix Controller Web app / API 6. Click the Create button at the bottom of the page Getting Started 119

126 Add a Key 1. Find and select the application you just registered in the list displayed Important: Copy and save the Application ID for later. It will be referred to again later in this document as 120 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

127 Application Client ID 2. Click on the Keys in the Settings pane on the right 3. Enter a new row: Field Key description Expires Value Aviatrix Never expires 4. Click Save 5. Copy the displayed key value and save it for later Getting Started 121

128 Important: Save this value. It will be referred to again later in this document as Application Client Secret 6. Close the Keys window using the X in the upper right corner. Add Required Permissions 1. Select the Aviatrix Controller application registration again (you may already be on it) 2. Click on the Required permissions just above Keys 3. Click + Add button 4. Click Select an API (on the right) 5. Find and select Windows Azure Service Management API 6. Click Select 7. In the Enable Access panel, click on Access Azure Service Management as organization users (preview) (checkbox next to it will become checked) 122 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

129 8. Click Select 9. Click Done 10. Close the Required Permissions panel by clicking on the X in the upper right corner. Grant Permissions to Aviatrix Controller 1. Go to the Subscriptions service (available from the left navigation panel or from More Services) 2. Click on the subscription where Aviatrix Controller is installed Important: Copy and save the Subscription ID for later 3. Click on Access Control (IAM) 4. Click + Add 5. Populate the fields as follows: Getting Started 123

130 Field Role Assign access to Select Value Contributor Azure AD user, group, or application Aviatrix Controller 6. Click Save 7. Close the Access control (IAM) panel by clicking on the X in the upper right corner Step 3. Configure Aviatrix Your Aviatrix Controller should be up and running by now. Go back to the Microsoft Azure portal and find the newly created instance. Open it and copy the Public IP address. Open a browser and navigate to ip address>/. Tip: You may receive a warning about the certificate not matching. You can safely ignore this and continue to the page. When you arrive at the login prompt, login with the Username admin. The password is the private IP address of the Azure instance. Tip: The Private IP address can be found on the instance page by clicking on the Networking navigation link. 124 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

131 After logging in, you will be prompted to provide your address. This is used for alert notifications as well as for password recovery. Enter your address and click OK. Set the admin password to something you will remember and click Save. If you require a proxy for this instance to get to the internet, enter that now. Otherwise, click Skip. Finally, the software will be upgraded. Click the Run button and the latest version of the Controller will be downloaded and installed. This will take a few minutes. Once complete, the login prompt will appear. Login with the username admin and the new password Getting Started 125

132 22.3 Azure After logging in, click on the Azure ARM button to connect Aviatrix to your Azure account Create Account Fill out the fields as follows: Field Account Name Password Confirm Password ARM Subscription ID Application Endpoint Application Client ID Application Client Secret Expected Value The login/username for users who will have admin access to Azure resources. For example, AzureOpsTeam. The address for this team. Password for login to the controller The Subscription ID you saved in a previous step. The Application Endpoint (i.e., the Directory ID) retrieved earlier. The Client ID (i.e., the Application ID ) saved earlier. The Client Secret (i.e., the key value) displayed earlier. Once complete, click the Create button at the bottom of the form. 126 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

133 Accept License Agreement Before you can automate launching an Aviatrix Gateway, you must first subscribe to the Aviatrix Companion Gateway in the Azure Marketplace. 1. Search for aviatrix companion gateway 2. Select the result 3. Click on the link at the very bottom titled Want to deploy programmatically? Get started Azure 127

134 4. Click on the Enable status button. 5. Click Save Create Gateway The controller can now automate creating a Gateway within Azure. Switch back to the browser tab or window with the Aviatrix Controller. Click on the Gateway in the left navigation bar: 128 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

135 Next, click on the + New Gateway button. Populate the Gateway Name and select the appropriate Region, VNet, and Public Subnet. The Gateway Size can be left at the smallest size. It can be scaled up (and out) later if needed. Click OK to create the Gateway automatically. This will take a few minutes as it creates the instance in the selected region and sets up the appropriate route table entries, etc. Once complete, click X Close. Now you have a Gateway in Azure that can connect to either (or both) AWS or GCP AWS Create Account 1. Go to the Onboarding section on your Controller AWS 129

136 2. Click on AWS Fill out the fields as follows: Field Account Name Password Confirm Password AWS Account Number IAM role-based AWS Access Key ID AWS Secret Key Expected Value The login/username for users who will have admin access to AWS resources. For example, AWSOpsTeam. The address for this team. Password for login to the controller You can find your account number on the AWS billing page Leave this unchecked for now. For production use, you ll want to use IAM roles with specific permissions. An admin user s AWS access key ID An admin user s AWS secret key Once complete, click the Create button at the bottom of the form. 130 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

137 Deploy a Gateway in AWS Head back over to the Gateways section in the Aviatrix Controller and click on + New Gateway button. 1. Select AWS for Cloud Type 2. Enter a Gateway name 3. Select the appropriate values for Region, VPC ID, and Public Subnet. 4. Keep the default Gateway Size at t2.micro. 5. Check Allocate New EIP so a new Elastic IP will be allocated on creation. 6. Click OK when ready. Tip: Create a new VPC for testing Peer the Gateways 1. Click on the Peering navigation link on the Controller. 2. Click on + New Peering AWS 131

138 3. Select the AWS Gateway and the Azure Gateway 132 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

139 4. Click OK Complete That s it. Your Azure VNet instances can now talk to your AWS instances over a secure tunnel. You will soon receive an notification that the tunnel is up. You ll receive additional notifications if the tunnel goes down GCP Prepare your Google Cloud Account The Aviatrix Controller requires a few settings to be enabled in order for it to be able to interact with your Google Cloud account. 1. Find the Project ID From the Google Cloud Console Dashboard, copy and save the Project ID GCP 133

140 2. Enable GCloud Messaging Service The Controller relies on Google Cloud Pub/Sub APIs to communicate with the Gateways in GCP. Enable these APIs by going to the APIs & services Dashboard for the selected project. Click the Enable APIs and Services link at the top of the page. Select Google Cloud Pub/Sub API from the list. Then, click Enable. 134 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

141 3. Create Credentials File Navigate back to the APIs & services Dashboard and select Credentials (or click here). Click Create credentials drop down and select Service account key GCP 135

142 Select the Compute Engine default service account for the Service account and select JSON for Key type. Then, click Create. A file will be downloaded to your computer. Find it and store it in a safe location. Then, click Close. 136 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

143 You are now ready to connect the Aviatrix Controller to your Google Cloud Platform account Create Account 1. Go to the Onboarding section on the Aviatrix Controller UI. 2. Click on Gcloud Fill out the fields as follows: Field Account Name Password Confirm Password GCloud Project ID GCloud Project Credentials Expected Value The login/username for users who will have admin access to Google Cloud resources. For example, GCPOpsTeam. The address for this team. Password for login to the controller The Project ID saved earlier Select the credentials file created in an earlier step. Once complete, click the Create button at the bottom of the form GCP 137

144 Deploy a Gateway in GCP Head back over to the Gateways section in the Aviatrix Controller and click on + New Gateway button. 1. Select the Cloud Type to be GCloud. 2. Enter a Gateway name. 3. Select a VPC ID, and Public Subnet. 4. Keep the default Gateway Size of f1-micro. 5. Click OK when ready. 138 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

145 Peer the Gateways 1. Click on the Peering navigation link on the Controller. 2. Click on + New Peering 3. Select the AWS Gateway and the Azure Gateway GCP 139

146 4. Click OK Complete That s it. Your Azure VNet instances can now talk to your GCP instances over a secure tunnel. You will soon receive an notification that the tunnel is up. You ll receive additional notifications if the tunnel goes down Summary If you peered your Azure account with both AWS and GCP, then you should see something like this on your Aviatrix Controller Dashboard: 140 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

147 Now that you have the accounts established, you can easily add connectivity to other VPCs in either AWS or GCP. And, of course, you can also connect AWS to GCP Summary 141

148 142 Chapter 22. Multi Cloud: Connecting Azure to AWS and GCP

149 CHAPTER 23 Site2Cloud 23.1 The Problem Traditionally enterprises host their IT applications in their own datacenter or at a co-location. Remote sites typically connect to the datacenter via an Internet based IPSec VPN tunnel or MPLS based private network. Such a hub and spoke architecture has been prevalent in the last 15 years. A problem with this deployment architecture is long latency or unstable Internet connectivity suffered by remote sites, especially between those in different continents. Such problems cause application time out, resulting in lost productivity and unhappy user experience. The solution to this pain point has been to deploy some form of WAN optimization gears in both the remote sites and datacenter to reduce application latency and reduce data bandwidth. These gears are complex, expensive and not every enterprise can afford them, and in some cases, they don t always work well Solution: Bring Application to User With the many regions in the world available brought by public cloud providers, such as AWS and Azure, the application latency issue can now be solved in a brand new way. By placing applications in a region of public cloud that your remote sites are closer to than to the datacenter, the long latency issue is eliminated all together. In addition, by moving servers to the cloud, you can reduce remote sites footprint and the amount of hardware to manage, thus reducing cost for ongoing maintenance. The comparison between the two deployment architecture is described below: 143

150 In the diagram above, remote sites or branch offices connect to headquarter datacenter via IPSec tunnels. International sites across continents can experience hundreds or more milliseconds in latency and in some countries, connectivity to headquarter is unstable at times. The first step in deploying application close to user is to build a new network architecture as shown in the right side of the diagram above. A remote site now connects via IPSec tunnel to a closest Aviatrix gateway in a VPC or VNet in a region closest to the site. Different remote sites may connect to different Aviatrix gateways. For example, sites in China connect to Aviatrix gateways in Azure China region and sites in Europe connect to Aviatrix gateway in a VPC in AWS eu-west-1 region. After the new network is deployed, you can now replicate Active Directory to VPC/VNet, and deploy applications such as ERP in the cloud too. The AD authentication latency and application latency can be reduced to tens of milliseconds. In addition, the remotes are simpler with fewer hardware equipment to manage Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. The Site2Cloud configuration workflow is as follows, with major steps highlighted. 1. Create a gateway in a VPC where you like to connect to sites. Go to Gateway -> New Gateway. The gateway may have VPN Access disabled. 2. (Optional) Create a secondary gateway in the same VPC for HA. Go to Gateway -> New Gateway. The gateway may have VPN access disabled. 3. Create a connection to a remote site Go to site2cloud -> Add New, make sure, (a) Select the VPC/VNet Name where Aviatrix gateway for encryption is launched. (b) If HA is not enabled: i. At Gateway field, select a gateway launched in the earlier step. (c) Else if HA is enabled: i. At Primary Gateway field, select a gateway launched earlier as primary gateway. ii. At Backup Gateway field, select a gateway launched earlier as backup gateway. 144 Chapter 23. Site2Cloud

151 (d) Input the connection with a unique name, for example, NewYork-site. (e) at Remote Gateway Type, select AWS VGW if the remote site is a VPC with AWS VGW VPN gateway; select Aviatrix if the remote site is on-prem Aviatrix gateway; select Generic if the remote site gateway is a third party router or firewall. (f) At Remote Gateway IP Address, enter the public IP address of the edge router for the remote site. Note if the Remote Gateway Type is Aviatrix, the Remote Gateway IP address is the public IP address of the site. (g) At Remote Subnet, enter network CIDR of the remote/customer site. If there are multiple subnets, enter each one separated with comma. For example, you may enter /24, /24 without the quote. (h) Pre-shared Key is an optional field. If you leave it blank, Aviatrix will auto generate a pre-shared key. You can paste your own pre-shared key if you prefer. (i) Do not select Private Route Encryption. (This feature is for overlay encryption on a AWS Direct Connect or Azure Express Route) (j) If you leave Local Subnet blank, Local Subnet will be the VPC/VNet CIDR. You can add more Local Subnet CIDR blocks, separate by comma. Make sure you include the VPC/VNet as well. These Local Subnets are advertised to Remote Subnets that the site2cloud connection can reach. You can change this settings later. (k) Algorithms field is prepopulated with default values. Click the field if you need to customize the algorithms. (l) Click OK to create a connection. 4. Generate remote site configuration template Go to site2cloud (a) Select the connection you just created, a EDIT panel will appear. (b) Click Download Configuration. (c) If your remote site device is not listed in the dropdown menu, simply select an available one in the menu. (d) Click Yes, Download to download a template file that contains the gateway public IP address, VPC CIDR, pre-shared secret and encryption algorithm. Incorporate the information to your remote router/firewall configuration. If the remote gateway is a Aviatrix CloudN, go to site2cloud and simply import the downloaded configuration file and click OK Troubleshooting To check a tunnel state, go to Site2Cloud, the tunnel status will be displayed in a pop up window. To troubleshoot a tunnel state, go to Site2Cloud -> Diagnostics Troubleshooting 145

152 146 Chapter 23. Site2Cloud

153 CHAPTER 24 CloudN for Site2Cloud CloudN can be deployed on-prem as a virtual router. This guide helps you to configure Site2Cloud IPSEC tunnels on CloudN that connects to an Aviatrix Gateway in AWS VPC, Azure VNet, Google Cloud VPC. (CloudN can also connects to any third party router or firewall for IPSEC tunnel.) 24.1 Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. The Site2Cloud on CloudN configuration workflow is very simple. 147

154 1. If the remote cloud gateway is an Aviatrix gateway, you should already have a configuration text file for this connection. If you need help to get this file, check out this link. (a) Click Site2Cloud on the left navigation panel, click +Add New. (b) Click Import (located at the right corner of the screen) (c) Click OK. You are done. (d) Refresh the screen, the tunnel should be up. (e) Add a static route on the default gateway where CloudN is deployed to point to CloudN as the next hop to reach the remote site. 2. If the remote side is NOT an Aviatrix gateway: (a) Click Site2Cloud -> +Add New (b) Enter Connection Name. For example: store1-to-cloud (c) At Remote Gateway IP Address, fill the public IP address of the remote gateway. For example, (d) Enter Pre-shared Key. (e) Enter Remote Subnet CIDR blocks. For example, /24 (f) Enter Local Subnet CIDR blocks. For example, /24 (g) Click OK. (h) Add a static route on the default gateway where CloudN is deployed to to point to CloudN as the next hop to reach the remote site Troubleshooting To check a tunnel state, go to Site2Cloud, the tunnel status will be displayed in a pop up window. To troubleshoot a tunnel state, go to Site2Cloud -> Diagnostics. 148 Chapter 24. CloudN for Site2Cloud

155 CHAPTER 25 Transit Network with BGP Setup Instructions 25.1 Introduction Aviatrix Services Architecture builds automated and scalable network architecture for the cloud, as shown in the diagram below. Key characteristics in this architecture: Spoke VPC to Spoke VPC networking is direct without going through the Transit VPC and is orchestrated by the central controller. Spoke VPCs do not run BGP protocol. BGP runs between the gateway in the Transit VPC and AWS VGW to faciliate communication between Spoke VPC and on-prem. The idea is you need to configure on-prem connectivity to VGW once and there is no need again when new Spoke VPC is stood up. 149

156 This guide provides instructions on how to enable BGP for a Transit VPC solution. Aviatrix gateway deployed in Transit VPC exchanges routes with a VGW that connects to on-prem by Direct Connect or Internet. Review the Best Practice section before you proceed Deployment Steps Establish BGP between Aviatrix Gateway and VGW in Transit VPC This step launches an Aviatrix gateway in Transit VPC and builds a IPSEC connection to VGW with BGP enabled. 1. At AWS Console create a VGW (the VGW is not attached to a VPC) which we will use to connect to on-prem over Direct Connect or Internet. For information on how to connect a VGW to Direct Connect, follow the steps for details. For IPSEC configuration, refer to this doc for IPSEC over Internet configuration guide. 2. From Aviatrix Controller console, launch an Aviatrix Gateway in the Transit VPC. This Aviatrix Gateway in the Transit VPC is the Customer Gateway (CGW) from VGW point of view. 3. At AWS Console, create Customer Gateway (CGW) in Transit VPC with the following configuration: Routing: Dynamic IP Address: Public IP of Aviatrix Gateway in Transit VPC. 4. At AWS Console create AWS VPN Connection in Transit VPC with the following configuration: Virtual Private Gateway: VGW in Transit VPC Customer Gateway: CGW created above Routing Options: Dynamic (requires BGP) 5. At AWS Console, download configuration template from AWS VPN Connection for Generic vendor (Referred as Configuration Template below). 6. At AWS Console, detach VGW from Transit VPC (if it was attached). 7. At Aviatrix Controller console, create Site2Cloud tunnel on Aviatrix Gateway to work with AWS VGW with the following configuration: VPC ID/VNet Name: Transit VPC ID Connection Type: Unmapped Connection Name: Any name Remote Gateway Type: AWS VGW Tunnel Type: UDP Algorithms: Deselected Encryption over ExpressRoute/DirectConnect: Deselected BGP: Selected Remote AS Number: IPSec Tunnel #1 -> Border Gateway Protocol (BGP) Configuration -> Virtual Private Gateway ASN from Configuration Template 150 Chapter 25. Transit Network with BGP Setup Instructions

157 CGW Inside IP Address: IPSec Tunnel #1 -> Tunnel Interface Configuration -> Inside IP Addresses - > Customer Gateway from Configuration Template VGW Inside IP Address: IPSec Tunnel #1 -> Tunnel Interface Configuration -> Inside IP Addresses - > Virtual Private Gateway from Configuration Template Deployment Steps 151

158 Advertise Network: Transit VPC CIDR Enable HA: Deselected Primary Cloud Gateway: Aviatrix Gateway in Transit VPC Remote Gateway IP Address: IPSec Tunnel #1 -> Tunnel Interface Configuration -> Outside IP Addresses -> Virtual Private Gateway from Configuration Template Pre-shared Key: IPSec Tunnel #1 -> Internet Key Exchange Configuration -> Pre-Shared Key from Configuration Template 152 Chapter 25. Transit Network with BGP Setup Instructions

159 8. At Aviatrix Controller console, Advanced Config -> BGP: Edit Local AS Num if required Enable BGP 9. At Aviatrix Controller s Site2Cloud page: Make sure site2cloud tunnel is up and working View Remote Subnet, this is on-prem network obtained through route exchange between Connect Spoke VPC to on-prem 1. At Aviatrix Controller console, launch an Aviatrix Gateway in a spoke VPC. 2. At Controller console, Peering -> Encrypted Peering, create peering between Aviatrix Gateways at spoke VPC and Transit VPC. 3. At Controller console, Peering -> Transitive Peering, create transitive peering from spoke VPC to on-prem via Transit VPC. Transitive Peering configuration:i Source Gateway: Spoke VPC Gateway Nexthop Gateway: Transit VPC Gateway Destination CIDR: on-prem network displayed at Site2Cloud -> Remote Subnet 4. At Controller s Site2Cloud page, select the Site2Cloud connection created above by Aviatrix gateway at Transit VPC with BGP. At BGP Advertised Networks field, append Spoke VPC s CIDR to the list. 5. Repeat the above section for each Spoke VPC connected to Transit VPC Building HA Transport Links There are multiple patterns to build HA in the transport link. AWS VGW can be used to create two Direct Connect links, two IPSEC over Internet links and one Direct Connect and one IPSEC over Internet links. Refer to this doc for Building HA Transport Links 153

160 details Best Practice Plan your cloud address space when designing a Transit VPC network. Best practice is to allocate a network address space from which the spoke VPC CIDRs are created. Make sure this network address space is unique and not overlapping with any on-prem network. For example, allocate /16 as your cloud address space. The spoke VPC CIDRs would be /24, /24, etc. With this approach, you just need advertise one prefix /16 once. When a new spoke VPC come up, you do not need to modify advertise network at the site2cloud page. Edit BGP Advertise Network after BGP has learned the on-prem network prefixes. When creating the Site2Cloud connection, leave the Advertised Networks blank. After Site2Cloud connection is created, go to Advanced Config to enable BGP. Go back to Site2Cloud connection, if you see list of subnets under Remote Subnet, it implies BGP has come up. At this point, click the connection to Edit BGP Advertised Networks. Enter the entire cloud address space as suggested above. This approach helps you see the list of the on-prem network prefixes to make sure you do not enter overlapping addresses BGP Troubleshooting Aviatrix BGP is implemented based on Quagga open source software. You can get debugging information at Controller console. Advanced Config -> BGP -> Diagnostic Release 3.0 Limitations You need to edit each Spoke VPC Transitive Peering settings when on-prem network is changed. The changed network can be viewed from the Controller Advanced -> BGP page. 154 Chapter 25. Transit Network with BGP Setup Instructions

161 CHAPTER 26 Encryption over DirectConnect/ExpressRoute 26.1 The Problem AWS Direct Connect and Azure ExpressRoute provides a private routed circuit to AWS VPC and Azure VNet. Aviatrix site2cloud feature provides encryption over Direct Connect or ExpressRoute. This document describes how to implement the feature over Express Route. The same method applies to AWS. The VNet VPN gateway that terminates the ExpressRotue connects VNet virtual machines with the on-prem servers in a traditional routing domain. While Azure ExpressRoute provides a private link between customer s on-prem network and Azure VNet without going through Internet, packets between on-prem edge and VNet travel through exchange points and third party provider networks and are not encrypted. If encryption is a requirement for security and compliance reasons, this is a problem Aviatrix Solution for Encryption over ExpressRoute Aviatrix site2cloud solution can be applied to encrypt traffic over ExpressRoute, as shown below. 155

162 In the diagram above, an encrypted IPSec tunnel is established between Aviatrix gateway and customer s edge router. Aviatrix gateway is deployed in a separate subnet from the subnets where user virtual machines are launched. (The controller is not drawn.) This is necessary as the Aviatrix gateway is the router for user subnets to reach Enterprise datacenter. Aviatrix gateway can be deployed in a 1:1 redundancy fashion where a backup gateway is ready to take over should the primary IPSec tunnel goes down due to gateway VM hardware/software failure Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click!new to download the latest software. For the network design, you need to decide if you want to enable HA for the gateway. The configuration workflow is as follows, with major steps highlighted. 1. Create a gateway in a VNet where you like to connect to enterprise datacenter. Go to Gateway -> Create, make sure: The gateway is launched in different subnet from the user subnets. In this example, the gateway is deployed on Subnet1. The gateway may have VPN access disabled 2. (Optional) If HA is enabled, create a backup gateway in the same VNet. Go to Gateway -> Create, make sure: The gateway is launched in different subnet from the user subnets. In this example, the gateway is deployed on Subnet Chapter 26. Encryption over DirectConnect/ExpressRoute

163 The gateway may have VPN access disabled. 3. Create a connection to the Enterprise datacenter Go to site2cloud -> Add New, make sure, a. Select the VPC/VNet Name where Aviatrix gateway for encryption is launched. b. If HA is not enabled: i. At Gateway field, select a gateway launched earlier for encryption. c. Else if HA is enabled: i. At Primary Gateway field, select a gateway launched earlier as primary gateway. ii. At Backup Gateway field, select a gateway launched earlier as backup gateway. d. Input the connection with a unique name, for example, FirstExpressRoute e. At Remote Gateway IP Address, enter the public IP address of the edge router for Enterprise datacenter. f. At Remote Network, enter network CIDR of the Enterprise datacenter. If there are multiple subnets, enter each one separated with comma. g. Check Encryption over ExpressRoute/DirectConnect. h. At Route Table To Modify field, select the route table(s) associated with subnet2 and subnet3. 4. Download the configuration template Go to site2cloud, a. Select the connection. b. Click Download Configuration. c. If your remote edge device is not listed in the dropdown menu, simply select an available one in the menu. d. Click Yes, Download to download a template file that contains the gateway public IP address, VPC CIDR, pre-shared secret and encryption algorithm. Incorporate the information to your remote router/firewall configuration. 5. At the Enterprise datacenter or remote site, configure encryption on the edge device. Make sure your peer network is Subnet2 and Subnet3, as shown in this example Configuration Workflow 157

164 158 Chapter 26. Encryption over DirectConnect/ExpressRoute

165 CHAPTER 27 How to Build Simple and Scalable Transit VPC Solution Solution Overview Aviatrix provides a Transit VPC solution that is centrally managed and simple to deploy, as documented in this link.. The solution requires no CCIE and crypto skills for maintenance and troubleshooting the network connectivity. One friction in this Transit VPC solution is that each time when a spoke VPC is stood up, the IPSEC tunnel between the transit VPC and on-prem needs to be modified to include the new spoke VPC CIDR. This modification of IPSEC tunnel invovles on-prem network team and can take up a few weeks of time. This document guides you to build a large and scalable Transit VPC network over Internet that requires minimum modification to the on-prem edge router or firewall devices. The idea can also be applied to the case where the connectivity between transit VPC and on-prem is over AWS Direct Connect. The idea of this scalable Transit VPC solution is to configure the IPSEC tunnel once between the transit VPC and onprem edge router or firewall. Subsequent spoke VPC connectivity to on-prem requires no change to this edge router or firewall. This solution enables CloudOps team to be self sufficient in building and operating the hybrid cloud network Cloud Address Planning The first step is to obtain from your network admin the on-prem address space in the most summerized form. For example, the on-prem network consists of /16. The next step is to work with your on-prem network admin to carve out one or a set of consective network address space that is not used anywhere by your company and reserve that as your cloud address space. For example, the address space could be /8. All spoke VPC CIDRs and Transit VPC will be subset of the reserved cloud address space (e.g /16, /16 and etc). 159

166 Transit VPC to on-prem IPSEC Tunnel The second step is to use this carved out cloud address space to build just one IPSEC tunnel between your on-prem network and the transit VPC. What you need to do is to specify the local network as the carved out and non-used address space. The remote network addresses should be your on-prem network address Spoke VPC to on-prem IPSEC Tunnel Once you have built the Transit VPC to on-prem IPSEC tunnel, you no longer need to modify edge routers or firewalls for any spoke VPC to on-prem IPSEC tunnels. Aviatrix transitive routing feature takes care of each new spoke VPC when it needs to connect to on-prem. You simply configure an encrypted peering between the spoke VPC to the transit VPC and then configure transitive peering from the spoke VPC to your On-Prem network through the Transit VPC Configuration Workflow Pre Configuration Checklist Before configuring VPC peering, make sure the following prerequisites are completed. Pre Configuration Check List 1. Deploy the Aviatrix Controller 2. Check VPC Settings These prerequisites are explained in detail below Deploy the Aviatrix Controller The Aviatrix Controller must be deployed and setup prior to configuring VPC and site peering. Please refer to Aviatrix Controller Getting Started Guide for AWS on how to deploy the Aviatrix Controller. Aviatrix Controller Getting Started Guide Check and make sure you can access the Aviatrix Controller dashboard and login with an administrator account. The default URL for the Aviatrix Controller is: ip of Aviatrix Controller> Check VPC Settings The VPC must have at least one public subnet to deploy the gateway. This means one subnet must be associated with a route table that has an IGW as its default route. If your Transit VPC and Spoke VPCs are in the same region and you like to route the traffic over AWS peering, go to AWS console and configure the necessary AWS peering between the two VPCs. 160 Chapter 27. How to Build Simple and Scalable Transit VPC Solution

167 Configuration Steps Make sure the pre-configuration steps in the previous section are completed before proceeding. The instructions in this section will use the following architecture. The CIDR and subnets may vary depending on your VPC setup; however, the general principals will be the same. In this example we have four Cloud VPCs: 1 Transit VPC, 3 Spoke VPCs and a corporate data center. The network will be configured such that all spoke nodes and on-prem will be able to communicate with each other via the Transit VPC Step a Deploy Gateways The first step is to deploy Aviatrix gateways in each VPC. Instructions: a.1. Login to the Aviatrix Controller Console a.2. Click on Gateway -> New Gateway Setting Cloud Type Account Name Region VPC ID Gateway Name Public Subnet Gateway Size Enable NAT VPN Access Value Choose AWS Choose the account name Choose the region where your VPC is located Choose the VPC This name is arbitrary (ex. gw01) Select a public subnet where the gateway will be deployed t2.micro is fine for testing. Uncheck this box Uncheck this box a.3. Click OK. It will take a few minutes for the gateway to deploy. Do not proceed until the gateway is deployed. a.4. Repeat steps a.2 and a.3 for the additional 3 VPCs in this example. a.5. Done Configuration Workflow 161

168 Step b Connect Spoke VPC to Transit VPC This step explains how to connect a Spoke VPC to the transit VPC. Instructions: b.1. From the Aviatrix Controller Console b.2. Click Peering -> Encrypted Peering b.3. Click New Peering b.4. Select the Transit VPC #0 gateway (Aviatrix GW #0) and Spoke VPC #1 gateway (Aviatrix GW #1) for the peering. Note: If the two VPCs are in the same region, you can check the box over AWS Peering. This would allow the encrypted peering to route traffic over native AWS peering, resulting in 10 times bandwidth saving. b.5. Click OK b.6 Repeat steps b.4 and b.5 for more scalable Spoke VPCs as Spoke VPC #2 gateway (Aviatrix GW #2) and Spoke VPC #3 gateway (Aviatrix GW #3) in this example. b.7 Done Step c Connect Corporate Data Center to Transit VPC This step explains how to connect the corporate data center to the Transit VPC. Instructions: c.1. From the Aviatrix Controller Console c.2. Click Site2Cloud -> Add New Setting VPC ID/VNet Name Connection Type Connection Name Remote Gateway Type Tunnel Type Algorithms Encryption over DirectConnect Enable HA Primary Cloud Gateway Remote Gateway IP Address Pre-shared Key Remote Subnet Local Subnet Value Choose Transit VPC ID Unmapped This name is arbitrary (ex. corpdatacenter) Aviatrix (in this example) UDP Uncheck Uncheck Uncheck Choose Transit VPC gateway Public IP address of On-Prem gateway Optional /16 (in this example) /8 (in this example) c.3. Click button OK c.4. View List, click the row of Transit VPC ID and Connection Name (ex. corpdatacenter) from above. c.5. Check Vendor, Platform and Software of On-Prem gateway on Corporate Data Center. Note: If your On-Prem gateway is: 1. a On-Prem Aviatrix gateway -> select Aviatrix (in this example) 162 Chapter 27. How to Build Simple and Scalable Transit VPC Solution

169 2. a Cisco ASA -> select Cisco 3. a third party router or firewall -> select Generic c.6. Click button Download Configuration c.7. If the On-Prem gateway is a Aviatrix CloudN as in this example, go to site2cloud page of CloudN website and simply import the downloaded configuration file and click OK. c.8. This template file contains the necessary information to configure the On-Prem gateway. Once the On-Prem gateway is configured, the tunnel will automatically come up. c.9. Done Step d Configure Transitive Routing This step explains how to configure transitive routing so that every spoke and on-prem node can communicate with each other via the transit VPC. Instructions: d.1. From the Aviatrix Controller Console d.2. Click Peering -> Transitive Peering d.2.1. For Spoke VPC #1: 1. Click + New Peering 2. Source Gateway: Aviatrix GW #1, Next Hop VPC: Aviatrix GW #0 (Transit VPC), Destination CIDR: /16 3. Click OK d.2.2. For Spoke VPC #2: 1. Click + New Peering 2. Source VPC: Aviatrix GW #2, Next Hop VPC: Aviatrix GW #0 (Transit VPC), Destination CIDR: /16 3. Click OK d.2.3. Repeat steps d.2.1 for more scalable Spoke VPCs as Spoke VPC #3 gateway (Aviatrix GW #3) in this example. d.3. Done 27.6 Troubleshooting To check a tunnel state, go to Site2Cloud, the tunnel status will be displayed at status column. To troubleshoot a tunnel state, go to Site2Cloud -> Diagnostics Troubleshooting 163

170 164 Chapter 27. How to Build Simple and Scalable Transit VPC Solution

171 CHAPTER 28 Site2Cloud between Azure VPN Gateway and Aviatrix Gateway This guide helps you to configure Site2Cloud IPSEC tunnels between an Aviatrix gateway and an Azure Virtual network gateway 28.1 Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. The Site2Cloud on CloudN configuration workflow is very simple. 1. At Aviatrix Controller, go to Gateway page to create one non-vpn gateway. 2. At Azure portal, go to Virtual network gateways page. Fill in the following information to create a new virtual network gateway: Name: Enter an Azure VPN gateway name (e.g. Azure-VPN-GW) Gateway type: VPN VPN type: Policy-based SKU: Basic Location: Select a desired location Virtual network: Select a desired VNet 3. Once the virtual network gateway is provisioned, record its public IP address 4. At Aviatrix Controller, go to Site2Cloud page. Fill in the following information to create a site2cloud connection: VPC ID/VNet Name: Select the VPC/VNet where your Aviatrix gateway is created at Step 1 Connection Type: Unmapped Connection Name: Enter a site2cloud connection name 165

172 Remote Gateway Type: Select Azure VPN Algorithms: Uncheck this box Encryption over ExpressRoute/DirectConnect: Uncheck this box Enabled HA: Uncheck this box Primary Cloud Gateway: Select the gateway created at Step 1 Remote Gateway IP Address: Enter the public IP of your virutal network gateway (collected at Step 3) Pre-shared Key: Enter your own pre-share key or leave it blank so that Controller will generate one Remote Subnet: Enter the CIDR of the VNet, in which your virtual network gateway is created at Step 2 Local Subnet: Enter the CIDR of the VPC/VNet, in which your Aviatrix gateway is created at Step 1 5. Once the site2cloud connection is created, select the same connection at Site2Cloud page. Select the following values for each specific field and click Download Configuration button. Vendor: Generic Platform: Generic Software: Vendor Independent 6. Collect the following information from the downloaded configuration template: Pre-Shared Key from #1: Internet Key Exchange Configuration Aviatrix Gateway Public IP from #3: Tunnel Interface Configuration Cloud Network(s) from Subnets section of #3: Tunnel Interface Configuration 7. At Azure portal, go to Local network gateways page. Enter the following information to create a local network gateway: Name: Enter a local gateway name (e.g. AVX-GW) IP address: Enter the Aviatrix gateway s public IP collected at Step 6 Address space: Enter Cloud Network CIDR collected at Step 6 Configure BGP settings: uncheck 8. At Azure portal, go to Virtual network gateways page and select the gateway created at Step 2 9. Select Connections from Settings. Enter the following information to create a connection: Name: Enter a VPN connection name (e.g. Azure-AVX-S2C) Connection type: Select Site-to-site (IPsec) Virtual network gateway: Select the VPN gateway created at Step 2 Local network gateway: Select the local gateway created at Step 7 Shared key (PSK): Enter the pre-shared key collected at Step Send some interesting traffic between Aviatrix gateway s VPC/VNet and Azure VPN gateway s VNet to bring up the site2cloud connection 166 Chapter 28. Site2Cloud between Azure VPN Gateway and Aviatrix Gateway

173 28.2 Troubleshooting To check a tunnel state, go to Site2Cloud, the tunnel status will be displayed in a pop up window. To troubleshoot a tunnel state, go to Site2Cloud -> Diagnostics Troubleshooting 167

174 168 Chapter 28. Site2Cloud between Azure VPN Gateway and Aviatrix Gateway

175 CHAPTER 29 OpenVPN Aviatrix provides a cloud native feature rich client VPN solution. The solution is based on OpenVPN and is compatible with all OpenVPN client software. In addition, Aviatrix provides its own client software that supports SAML authentication directly from the client. A summary of Aviatrix client VPN solution is described in the diagram below, to learn all Aviatrix OpenVPN features, check out this document. Note that only AWS VPC is drawn in the diagram, but the network diagram applies equally to Azure and Google Cloud. This guide helps you build a basic client vpn solution. If you like to learn how to build a user vpn solution with multiple VPCs, refer to this link. 169

176 29.1 Configuration Workflow Tips: Mouse over the fields to see its definition. Do a software upgrade if an upgrade alert message appears on your dashboard page. The description in the steps below provides critical fields you need to select; it may not include all fields. Make sure you have the correct VPC ID and its region for the VPC ID field and region in each step. 1. Launch a gateway with VPN capability in a VPC (a) Click Gateway. Click +New Gateway. (b) In the dropdown, select Cloud Type. (c) Provide a unique Gateway Name, such mgmt-gw. (d) Select one account you created earlier and a region, VPC ID, Public Subnet on which the vpn gateway will be launched and user will land. (e) Select VPN Access. More fields will appear. If you just want a basic user vpn solution without multi-factor authentication, you can skip the rest of the vpn related fields and click OK to launch a vpn gatewway. By default, ELB will be enabled, meaning you can create more vpn gateways that are load balanced by the ELB. (ELB will be automatically created by Aviatrix.) (f) Use the default VPN CIDR Block. The VPN CIDR Block is the virtual IP address pool that VPN user will be assigned. (g) If you use a DUO or Okta for multi factor authenication, select one of them at Two-step Authentication, more fields will appear. For details on Okta authentication, check out this link. (h) If you select Split Tunnel Mode, only the VPC CIDR traffic will go through the tunnel. If you specify Additional CIDRs, then these and the VPC CIDR will go through the vpn tunnel. You can modify Split tunnel settings later when more VPCs are created. (Go to OpenVPN -> Edit Config -> MODIFY SPLIT TUNNEL to make changes. Make sure you specify all the CIDRs, separated by comma.) You can leave Nameservers and Search Domains blank if you don t have one. Note: If you plan to support Chromebook, you must configure full tunnel mode as Chromebook only supports full tunnel. (i) ELB is enabled by default. If you disable ELB, your vpn traffic runs on UDP port When ELB is enabled, your vpn traffic runs on TCP 443. TCP 443 makes it easier to go through corporate firewall. (j) Click LDAP if vpn user should be authenticated by AD or LDAP server. After you fill up the LDAP fields, make sure you run Test LDAP Configuration to test your configuration is valid. (k) If you wish to create more of such vpn gateways (for example, behind ELBs for load balancing), click Save Template, which will save your LDAP and multi-factor authentication credentials. 2. Add Users and Profiles A profile is defined by a list of access policies with allow or deny for policy. When a vpn user is connected to vpn gateway, the user s profile is pushed dynamically to the vpn gateway and the user can only access resources defined in the profile. When a vpn user disconnects from the gateway, the policies are deleted. If a vpn user has no profile associatioin, the user has full access to all resurces. Note you can modify a user profile at any given time. (a) (Optionally) Go to OpenVPN -> Profiles to create as many profiles as you please. The target field can be FQDN (DNS names or fully qualified domain name). (b) Go to OpenVPN -> VPN Users to add as many user as you please. Associate each user with a profile. Note if no profile is associated, user has full access to all resources. When a user is added to the database, an with.ovpn file or.onc (for Chromebooks) will be sent to the user with detailed instructions. 170 Chapter 29. OpenVPN

177 OpenVPN is a registered trademark of OpenVPN Inc Configuration Workflow 171

178 172 Chapter 29. OpenVPN

179 CHAPTER 30 Aviatrix OpenVPN FAQs 30.1 How do I launch a VPN gateway? Click Gateway -> + New Gateway The controller launches an Aviatrix gateway instance in AWS/Azure/GCloud. The gateway instance must be launched from a public subnet. You need to give it a name (The name is presented as a Gateway Name field), this name becomes part of the instance name with a prefix CloudOps. In the Create page, select VPN Access to enable OpenVPN server capability. There is a default VPN CIDR /24. But you can change it, make sure the CIDR is outside the existing and future VPC CIDR range. This VPN CIDR is where VPN server assign virtual IP address to each user when she connects. You can select Save Template to save the gateway template. When you come to the page the next time, most of the fields are pre populated. You may change any of the fields How can I avoid managing multiple VPN user certs? If you have multiple VPCs, launching a VPN gateway in each VPC and create VPN users is not the correct way to manage. It forces your developers to carry multiple.ovpn certs and learn when to use which one when connecting to a VPC. Leverage VPC to VPC connectivity to build a scalable solution How do I scale out VPN solution? You can launch multiple VPN gateways in the same VPC at the Create Gateway time. While launching a gateway, select yes for Enable AWS ELB. This will automatically create an AWS ELB (for the first gateway) and register the gateway with the newly created load balancer. VPN traffic will be load balanced across these multiple gateways. 173

180 It is required to have consistent gateway configuration when ELB is enabled. For example, authentication methods, tunnel modes and PBR configurations should be identical How do I setup Okta authentication for VPN? Aviatrix vpn gateway integrates seamlessly with Okta. It can authenticate vpn users to Okta service using Okta s OpenVPN pluggin in module. Follow the link: How to setup Okta for Aviatrix VPN gateway 30.5 How do I enable Geo VPN? If you have global workforce that needs to access the cloud, Geo VPN offers a superior solution. Geo VPN enables a VPN user to connect to a nearest VPC that hosts Aviatrix VPN gateway. To enable Geo VPN, go to OpenVPN -> GEO VPN. Also check out this link for help How do I add a VPN user? After at least one gateway is created, you can add VPN users. Click OpenVPN -> VPN Users -> +Add New. When a user is added, an is sent to the user with instructions on how to download client software and connect to VPN server. If you like to assign user profile based policies, you need to create profiles first, see the next section What user devices are VPN client software supported? Windows, MAC, Linux, Chromebook, Android and ios devices are supported Is NAT capability supported on the gateway? Yes, you can enable NAT function at gateway launch time. When enabled, instances on the private subnet can access Internet directly. If full tunnel mode is selected, you may want to enable NAT to allow instances in the VPC to have direct Internet access Is full tunnel mode supported on the gateway? Yes, both split tunnel and full tunnel modes are supported. You can specify the mode at the gateway launch time. Full tunnel means all user traffic is carried through the VPN tunnel to the gateway, including Internet bound traffic. Split tunnel means only traffic destined to the VPC and any additional network range is carried through the VPN tunnel to the gateway. Any Internet bound traffic does not go through the tunnel. 174 Chapter 30. Aviatrix OpenVPN FAQs

181 30.10 Can the maximum number of simultaneous connections to VPN gateway be configured? Yes, you can set the maximum number of connections at the gateway launch time What is user profile based security policy? In VPN access, a user is dynamically assigned a virtual IP address when connected to a gateway. It is highly desirable to define resource access policies based on the users. For example, you may want to have a policy for all employees, a different policy for partners and a still different policy for contractors. You may even give different policies to different departments and business groups. The profile based security policy lets you define security rules to a target address, protocol and ports. The default rule for a profile can be configured as deny all or allow all during profile creation. This capability allows flexible firewall rules based on the users, instead of a source IP address How do I setup profile based security policies? When a user connects to a VPC, the security policies associated with the profile that the user is assigned to are applied to the VPN gateway instance that user logs in. This effectively blocks traffic from entering the network. Click OpenVPN -> Profiles -> +New Profile to create profiles, then click Edit Policies to add rules. You can add multiple of them, then click on Save How do I assign a user to a profile? When you create a VPN user at OpenVPN -> VPN Users -> +Add New, you can select profile option to assign the user to a specific profile. You can also attach the user to a profile at a later time. Go to OpenVPN -> Profiles. Click Attach User on a specific Profile and select a user that is added to the VPN gateway What if I want to change profile policies? You can change profile policies any time. However, the users who are currently active in session will not receive the new policy. The user need to disconnect and reconnect to VPN for the new policy to take effect How do I change a user s profile programmatically? The controller provides a REST API which can be invoked to change a user s profile. Refer to API document under Help menu. During this operation, the user s existing VPN session will be terminated. The new profile policy will take effect when he or she logs in again. The use case for this feature is to allow administrator to quarantine a VPN user for security reasons Can the maximum number of simultaneous connections to VPN gateway be configured?175

182 30.16 Is DUO multi-factor authentication supported? Yes. If your enterprise has a DUO account with multi-factor authentication, it can be integrated into the VPN solution. From Gateways tab, click Create. At two-step authentication drop down menu, select DUO, then enter your company Integration Key, Secret Key and API hostname. To obtain Integration Key, Secret key and API hostname, login to DUO website as an admin, click on the left panel Applications, click Protect an Application below. Scroll down the application list and select OpenVPN (click Protect this Application), the next screen should reveal the credentials you need to configure on the Aviatrix controller. For additional help, follow this instruction. Currently advanced feature such as Trusted Device and Trusted Networks are not supported. Send us a request if you like to integrate these features How do I configure LDAP authentication? LDAP configuration is part of the Gateway creation when VPN Access is enabled. Enter the necessary parameters and click Enable button to enable LDAP authentication for VPN clients. If your LDAP server is configured to demand client certificates for incoming TLS connections, upload a client certificate in PEM format (This certificate should contain a public and private key pair) Can I combine LDAP and DUO authentication? Yes. With both LDAP and DUO authentication methods enabled on a gateway, when launching the VPN client, a remote user will have to enter his or her LDAP user credentials and then approve the authentication request received on a registered mobile device to login to VPN Is OKTA supported? Yes. OKTA with MFA is also supported. Follow the instructions How does Policy Based Routing (PBR) work? When PBR is enabled at gateway launch time, all VPN user traffic arrives at the gateway will be forwarded to a specified IP address defined as PBR default gateway. User must specify the PBR Subnet which in AWS must be in the same availability zone as Ethernet 0 interface of the gateway. When PBR feature is combined with encrypted peering capability, VPN user should be able to access any instances in the peered VPC/VNets. This helps build an end to end cloud networking environment. For details, check out our reference design. Another use case for Policy Based Routing is if you like to route all Internet bound traffic back to your own firewall device on Prem, or log all user VPN traffic to a specific logging device, PBR lets you accomplish that. 176 Chapter 30. Aviatrix OpenVPN FAQs

183 30.21 What are the monitoring capabilities? Active VPN users are displayed on the Dashboard. Click on any username, the user VPN connectivity history is displayed. You can also disconnect a user from the dashboard. OpenVPN is a registered trademark of OpenVPN Inc What are the monitoring capabilities? 177

184 178 Chapter 30. Aviatrix OpenVPN FAQs

185 CHAPTER 31 Aviatrix OpenVPN Feature Highlights This document highlights Aviatrix OpenVPN features. For how to setup Aviatrix OpenVPN for developers and remote workers to access public cloud, consult this link VPN Management Centrally Managed A single pane of glass allows you to manage all VPN users, VPN certificates and VPN user visibility. OpenVPN Compatible Built on OpenVPN and is compatible with all OpenVPN client software. Split Tunnel Supports split tunnel mode where only specified CIDRs ranges go through the VPN tunnel. Full Tunnel Supports full tunnel mode where all user IP sessions including Internet browsing go through the VPN tunnel. PKI Managment Supports Bring Your Own (BYO) PKI management system. Force Disconnect Any admin can force disconnect a VPN user from the controller console. Dashboard View all active VPN users and their connection history from the controller console dashboard. REST API Support REST API for all managment activities Authentication Options LDAP/AD Integration Authenticates VPN user from Aviatrix gateways in addition to VPN certificate authentication. DUO Integration Authenticates VPN user from Aviatrix gateways in addition to VPN cert authentication. OKTA Integration Authenticates VPN user from Aviatrix gateways in addition to VPN cert authentication. MFA Integration Combines LDAP and DUO for multi-factor authentication. 179

186 Shared Certificate Support a shared certificate arrangment among VPN users. (When this option is is selected, you should enable additional authentication options to ensure secure access.) Client SAML Integration Authenticates VPN user directly from Aviatrix VPN client to any IDP via SAML protocol Authorization Profile-Based Access Control Each VPN user can be assigned to a profile that is defined by access privileges to network, host, protocol and ports. The access control is dynamically enforced when a VPN user connects to the public cloud via an Aviatrix VPN gateway Scale Out Performance TCP-based VPN You can use Aviatrix integrated ELB to load balance multiple Aviatrix VPN gateways. When ELB is used, OpenVPN client software runs on TCP port 443. TCP-based VPN requires no special corporate firewall rules when VPN client is on-prem. UDP-based VPN You can use Aviatrix integrated AWS Route53 round robin routing to load balance multiple Aviatrix VPN gateways. When Route53 round robin routing is used, OpenVPN client software runs on UDP port UDP-based VPN has improved file transfer performance. Geo VPN For TCP-based VPN, you can use Aviatrix integrated AWS Route53 latency-based routing to load balance clients residing in different geographic locations Logging Integration VPN User VPN user connection history and bandwidth usage can be logged to Splunk, SumoLogic, ELK, Remote Syslog and DataDog. User Activitiy Each VPN user TCP/UDP session can be logged to Splunk, SumoLogic, ELK, Remote Syslog and DataDog Client Software OpenVPN Client Software All OpenVPN client software is supported. The supported clients are macos, Windows, ios, Android, Chromebook, Linux and BSD. Aviatrix VPN Client Aviatrix VPN Client supports macos, Windows and Linux Debian distribution and BSD distribution. Choose Aviatrix VPN Client if you require SAML authentication directly from VPN client software. OpenVPN is a registered trademark of OpenVPN Inc. Note: This guide references AWS for illustration purposes and also applies to Azure (VNet) and Google (VPC). 180 Chapter 31. Aviatrix OpenVPN Feature Highlights

187 CHAPTER 32 OpenVPN Design for Multi VPCs This reference design helps you build an end to end secure cloud network, from accessing the network (AWS VPC) by users to routing packets among the VPCs, such that once a user is connected via VPN, she can access any private resources in the cloud no matter where that resource is. There are 3 use cases covered, from simple to more complex ones. You can read and decide which one suits you or combine parts from different ones to create a network that meet your requirements. You can easily build a full mesh network Multiple VPCs in one region The network you have in mind is shown below where all VPCs are in the same region. The Aviatrix controller instance can be in the same or a different VPC. 181

188 Assume you have created 4 VPCs in the same region (us-west-2 in this case). You like to use the VPC with CIDR /16 to host gateways where users connect to. After a user connects to this VPC via SSL VPN, she should be able to access any instances in the other VPCs as long as her profile allows, without having to connect to each VPC with SSL VPN. Another requirement is split tunnel mode, that is, only traffic destined to the cloud go through the SSL tunnel. If a user does general browsing to Internet or watch movies from Hulu, traffic should be routed via WI-FI to ISP to Internet. You do not wish to pay AWS for this type of compute and network costs Configuration Workflow Tips: Mouse over the fields to see its definition. Do a software upgrade if an upgrade alert message appears on your dashboard page. The description in the steps below provides critical fields you need to select; it may not include all fields. Make sure you have the correct VPC ID and its region for the VPC ID field and region in each step. 1. Launch a gateway with VPN capability in VPC /16. (a) Go to Gateway menu and create. Make sure: (b) At Gateway Name field, give it a distinct and convenient name. For example, mgmt-vpn-1 (c) VPN Access is selected. (d) Use the default VPN CIDR Block. (e) Split Tunnel Mode is selected. i. In the Additional CIDRs field under Split Tunnel, enter othervpcs/vnet or any network CIDRs you wish to reach beyond the VPC you are connecting to (in this case /16 is the connecting VPC). In the example shown, you should enter /16, /16, /16. It is a good idea to do some planning to include future VPCs or network address ranges. (In a case where you never have to worry about connecting to your corporate VPN, you may consider enter the entire private network address range in the Additional CIDRs range field, separating by comma: /12, /8, /16. Doing so afford you not to have to reconfigure the gateway if you need to add more VPCs for networking with different CIDR range in the future.) ii. (Optional) For the Nameservers and Search Domain field under Split Tunnel, enter your DNS server IP addresses and search domain if you have setup to use DNS names to access instances inside VPCs. Leave it blank if you do not know what they are. If you use AWS Route 53 private zone records for your host names, make sure the Nameserver is the DNS server of the VPC. In this case, you should enter (a) Enable AWS ELB is selected. (b) Save Template is selected. This Template saves you from entering repeated fields if you wish to create more gateways with the same configuration. 7. Repeat Step 1 to create more gateways with VPN enabled. Note each gateway must have a different VPN CIDR Block and name. You may select different AZs for the Public Subnet field. 8. Configure AWS peering. (a) Enter AWS console and select the region in which the VPCs were created. Select Services -> VPC - > Peering Connections. Click Create VPC Peering Connection button to make AWS peering. In this case, we need to make the following three AWS peering connections. All these peering connections should have one peer at the VPC terminating your SSL VPN connections (VPC1 in this case). i. pcx-xxxxxxx1: VPC1 (CIDR /16) <-> VPC2 (CIDR /16) 182 Chapter 32. OpenVPN Design for Multi VPCs

189 ii. pcx-xxxxxxx2: VPC1 (CIDR /16) <-> VPC3 (CIDR /16) iii. pcx-xxxxxxx3: VPC1 (CIDR /16) <-> VPC4 (CIDR ) (a) Modify the route tables of each VPC to add routes to its peer s subnets. In this case, the following route(s) should be added to each VPC s route table: i. VPC1 ( /16) route table: 1. Destination /16 -> Target pcx-xxxxxxx1 2. Destination /16 -> Target pcx-xxxxxxx2 3. Destination /16 -> Target pcx-xxxxxxx3 ii. VPC2 ( /16) route table: 1. Destination /16 -> Target pcx-xxxxxxx1 iii. VPC3 ( /16) route table: 1. Destination /16 -> Target pcx-xxxxxxx2 iv. VPC4 ( /16) route table: 2. Add Users and Profiles 1. Destination /16 -> Target pcx-xxxxxxx3 (a) Go to OpenVPN -> Profiles to create as many profiles as you please. The target field can be FQDN (DNS names or fully qualified domain name). (b) Go to OpenVPN -> VPN VPN Users to add as many user as you please. Associate each user with a profile. Note if no profile is associated, user has full access to all resources. When a user is added to the database, an with.ovpn file or.onc (for Chromebooks) will be sent to the user with detailed instructions. 3. Launch VPN connections from remote users to VPC1 ( /16). Once the SSL VPN connection is established, this VPN user should be able to reach all instances (in all VPCs) to which he/she has access permission. 4. Done Multiple VPCs in multi regions, split tunnel The network you have in mind is shown below where VPCs are in different regions. The Aviatrix Controller instance can be in the same or a different VPC Multiple VPCs in multi regions, split tunnel 183

190 Assume you have created 4 VPCs. You like to use the VPC with CIDR /16 in us-west-2 to host gateways where users connect to. After a user connects to this VPC via SSL VPN, she should be able to access any instances in the other VPCs as long as her profile allows, without having to connect to each VPC with SSL VPN. Another requirement is split tunnel mode, that is, only traffic originated from the user and destined to resources in VPCs is routed through SSL VPN tunnel. The traffic to Internet will be routed through ISP instead of SSL VPN tunnel Configuration Workflow Tips: Mouse over the fields to see its definition. The description in each step does not include all fields. Make sure you have the correct VPC ID and its region for the VPC ID field and region in each step. 1. Launch a gateway with VPN capability in VPC /16. (a) Go to Gateway menu and click create. (b) At Gateway Name field, give it a distinct and convenient name. For example, mgmt-vpn-1 (c) VPN Access is selected. (d) Use default VPN CIDR Block. (e) Split Tunnel Mode is selected. i. For the Additional CIDRs field under Split Tunnel, enter other VPC/VNet or any network CIDRs you wish to reach beyond the VPC you are connecting to. In the example shown, you should enter /16, /16, /16. It is a good idea to do some planning to include future VPCs or network address ranges. (In a case where you never have to worry about connecting to your corporate VPN, you may consider enter the entire private network address range in the Additional CIDRs range field, separating by comma: /12, /8, /16. Doing so afford you not to have to reconfigure the gateway if you need to add more VPCs for networking with different CIDR range in the future.) ii. (Optional) If you like to use private DNS name to access instance, you can fill Nameservers and Search Domain field under Split Tunnel. Enter your private DNS name and search domain. If you use AWS Route 53 private hosted zone and records for your host names, make sure the Nameserver is the DNS server of the VPC. In this case, you should enter Chapter 32. OpenVPN Design for Multi VPCs

191 (f) Enable AWS ELB is selected. (g) Save Template is selected. This Template saves you from entering repeated fields if you wish to create more gateways with the same configuration. 2. Repeat Step 1 to create more gateways with VPN enabled. You may select different AZs for the Public Subnet field. 3. Build encrypted routing networks to reach other VPCs. (a) Launch a gateway without VPN capability in VPC /16. This is the routing gateway, make sure: i. At Gateway Field, give it a distinct and convenient name. For example, dev-east-1, or teamkardashian-east-1 for the Kardashian game project. ii. VPN Access is not selected. iii. Enable NAT is NOT selected (since step 1 has enabled NAT function for this VPC) iv. Save Template is not selected. (so that you don t overwrite the hard work of entering the fields of gateways with VPN enabled) (b) Repeat step 3 for VPC /16, /16 and /16. Select Enable NAT if you want instances in these 3 VPCs to be able to reach Internet directly. (c) Configure encrypted peering. Go to Peering -> New Peering. Note each VPC is represented by one or more gateways. Make sure you want to peer between two gateways without VPN capability. 4. (Optional) Setup Stateful Firewall Rules at VPC level Go to Gateway, select the gateway you just created to edit Security Policies to add any policies for each VPC. 5. The above steps complete the network infrastructure setup. 6. Add Users and Profiles 7. Done (a) Go to OpenVPN -> Profiles to create as many profiles as you please. The target field can be FQDN (DNS names or fully qualified domain name). (b) Go to OpenVPN -> VPN Users to add as many user as you please. Associate each user with a profile. Note if no profile is associated, user has full access to all resources. When a user is added to the database, an with.ovpn file or.onc (for Chromebooks) will be sent to the user with detailed instructions Multiple VPCs in multi regions, full tunnel, your own firewall The network you have in mind is shown below where VPCs are in different regions. The Aviatrix Controller instance can be in the same or a different VPC Multiple VPCs in multi regions, full tunnel, your own firewall 185

192 Assume you have created 4 VPCs. You like to use the VPC with CIDR /16 in us-west-2 to host gateways where users connect to. After a user connects to this VPC via SSL VPN, she should be able to access any instances in the other VPCs as long as her profile allows, without having to connect to each VPC with SSL VPN. Another requirement is full tunnel mode, that is, all traffic originated from the user is routed through SSL VPN. Your organization requires to run its own firewall function for any Internet bound traffic Configuration Workflow Tips: Mouse over the fields to see its definition. The description in each step does not include all fields. Make sure you have the correct VPC ID and its region for the VPC ID field and region in each step. 1. Launch a gateway with VPN capability in VPC /16. (a) Go to Gateway menu and click create. (b) At Gateway Name field, give it a distinct and convenient name. For example, mgmt-vpn-1 (c) The VPN CIDR Block must be a subnet that is outside your current VPC CIDR range and your laptop or device subnet range. In the example above, you may enter /24. (d) Full Tunnel Mode is selected. (e) Enable AWS ELB is selected. (f) Enable Policy Based Routing (PBR) is selected. i. Note PBR Subnet must be a subnet that is in the same AZ as the primary subnet (Public Subnet where the gateway is launched). Enter the AWS subnet default gateway for PBR Default Gateway field. For example, if PBR Subnet is /20, the default Gateway field is ii. (optionally) you can enable NAT Translation Logging to log every user s each activity to every server and site. This is useful to auditing and compliance. i. Save Template is selected. This Template saves you from entering repeated fields if you wish to create more gateways with the same configuration. 186 Chapter 32. OpenVPN Design for Multi VPCs

193 2. Repeat Step 1 to create more gateways with VPN enabled. You may select different AZs for the Public Subnet field. 3. (Optional) If you have own your routing network to route between the VPCs and one of your own backbone routers can route traffic to your own firewall for Internet bound traffic, you can skip this step and the next two steps (step 4 and 5). (a) Launch a gateway without VPN capability in VPC /16. This is the routing gateway, make sure: i. At Gateway Field, give it a distinct and convenient name. For example, dev-east-1, or teamkardashian-east-1 for the Kardashian game project. ii. Enable NAT is not selected. iii. VPN Access is not selected. iv. Save Template is not selected. (so that you don t overwrite the hard work of entering the fields of gateways with VPN enabled) 4. (Optional) Repeat step 3 for VPC /16, /16 and /16. Select Enable NAT if you wish the instances in these VPCs to be able to reach Internet directly. 5. (Optional) Configure encrypted peering. Go to VPC/VNet Encrypted Peering -> Add. Note each VPC is represented by one or more gateways. Make sure you want to peer between two gateways without VPN capability. 6. The above steps complete the network infrastructure setup. 7. Add Users and Profiles 8. Done (a) Go to OpenVPN -> Profiles to create as many profiles as you please. The target field can be FQDN (DNS names or fully qualified domain name). (b) Go to OpenVPN -> VPN Users to add as many user as you please. Associate each user with a profile. Note if no profile is associated, user has full access to all resources. When a user is added to the database, a with.ovpn file or.onc (for Chromebooks) will be sent to the user with detailed instructions. OpenVPN is a registered trademark of OpenVPN Inc Multiple VPCs in multi regions, full tunnel, your own firewall 187

194 188 Chapter 32. OpenVPN Design for Multi VPCs

195 CHAPTER 33 OpenVPN for Geo Locations If you have a global work force that needs to access the cloud with the best user experience, building a cloud network with Geo VPN access capability is the right solution for you. Geo VPN combines our scale out VPN solution with latency based routing to dynamically route VPN users to the nearest VPN access gateway based on the latency between the user and the gateways. In this reference design we also enable split tunnel mode, that is, only traffic destined to the cloud go through the SSL VPN tunnel. If a user does general browsing to Internet or watch movies from Hulu, traffic should be routed via WI-FI to ISP to Internet. You do not wish to pay AWS for this type of compute and network costs. You may combine this reference design with other capabilities and reference designs to build out a network that meets your requirements Network Diagram The network diagram is shown below, where there are two sets of VPN access gateways, one in us-west-2 and another in eu-central-1. When a VPN user access the cloud, the network returns a VPN server IP address (the ELB DNS name) based on which region is closer to the user. 189

196 33.2 Configuration Workflow Tips: Upgrade to the latest software if there is an alert message on the controller dashboard. Mouse over the labels to get help. The description in each step does not include all fields. Make sure you have the correct VPC ID and its region for the VPC ID field and region in each step. 1. Create a VPN gateway cluster in VPC /16. Go to Gateway menu and click create. Make sure: (a) At Gateway Name field, give it a distinct and convenient name. For example, vpn-west2-1. (b) Enter VPC ID for /16 (c) Enable NAT is selected (d) VPN Access is selected. (e) The VPN CIDR Block must be a subnet that is outside of all your current and future VPC CIDR range. In the example above, you may enter /24 (say you ll never plan to configure a VPC in the /16 range). (f) Split Tunnel Mode is selected. i. For the Additional CIDRs field under Split Tunnel, enter other VPCs CIDR or other network CIDRs you wish to reach beyond the VPC you are connecting to. In the example shown, you should enter /16, /16, /16. It is a good idea to do some planning to include future VPCs or network address ranges. (In a case where you never have to worry about connecting to your corporate VPN, you may consider enter the entire private network address range in the Additional CIDRs range field, separating by comma: /12, /8, /16. Doing so afford you not to have to reconfigure the gateway if you need to add more VPCs for networking with different CIDR range in the future.) 190 Chapter 33. OpenVPN for Geo Locations

197 ii. (Optional) For the Nameservers and Search Domain fields under Split Tunnel, enter your private DNS server IP addresses and search domain if you have setup to use DNS names to access instances inside VPCs. Leave it blank if you do not know what they are. If you use AWS Route 53 private hosted zone and records for your host names, make sure the Nameserver is the DNS server of the VPC. In this case, you should enter (g) Enable AWS ELB is selected. (h) Save Template is selected. This Template saves you from entering repeated fields if you wish to create more gateways with the same configuration. (i) Repeat Steps a-g to create more gateways with VPN enabled. You may select different AZs for the Public Subnet field so that your gateways are load balanced between AZs. 2. Create a VPN gateway cluster in VPC /16 Repeat the procedures in step 1 to create a second VPN gateway cluster in eu-central-1 3. Enable Geo VPN (a) Go to OpenVPN -> Configuration -> Geo VPN, select Enable (b) For Domain Name, enter a public domain name that is registered on AWS Route 53 as a public hosted zone. For example, aviatrixvpn.com (c) Enter any name you like for VPN Service Name, for example, OpsVPN. The VPN Service Name combines with Domain Name forms the Geo VPN server name. (d) Select one ELB from the drop down menu for ELB DNS Name. Click OK. (e) Click Add to add the second ELB from the drop down menu. (f) If you have more ELB VPN gateway clusters in the future, you can add more later. 4. Build encrypted routing networks to reach other VPCs. (a) Launch a gateway without VPN capability in VPC /16. This is the routing gateway, make sure: i. At Gateway Field, give it a distinct and convenient name. For example, dev-east-1, or teamkardashian-east-1 for the Kardashian game project. ii. VPN Access is not selected. iii. Enable NAT is NOT selected (since step 1 has enabled NAT function for this VPC) iv. Save Template is not selected. (so that you don t overwrite the hard work of entering the fields of gateways with VPN enabled) (b) Repeat the above procedure for VPC /16. (c) Repeat the above procedure for VPC /16, /16. Select Enable NAT if you want instances in these 3 VPCs to be able to reach Internet directly. (d) Configure encrypted peering. Go to VPCs menu and Encrypted Peering -> Add. Note each VPC is represented by one or more gateways. Make sure you want to peer between two gateways without VPN capability. 5. (Optional) Setup Stateful Firewall rules at VPC Go to Gateway, select a Gateway, click Edit. Click Security Policies to add any policies for each VPC. 6. The above steps complete the network infrastructure setup. 7. Add Users and Profiles Configuration Workflow 191

198 (a) Go to OpenVPN -> Profiles to create as many profiles as you please. The target field can be FQDN (DNS names or fully qualified domain name). (b) Go to OpenVPN -> VPN Users to add as many user as you please. i. When Geo VPN is enabled, VPC ID association is no longer relevant. You can select any VPC ID. ii. Associate each user with a profile. Note if no profile is associated, user has full access to all resources. When a user is added to the database, an with.ovpn file or.onc (for Chromebooks) will be sent to the user with detailed instructions Troubleshooting If Enabling Geo VPN fails, make sure the Domain Name you enter is a registered name under AWS Route 53 in a public hosted zone. In addition, this Domain name must be hosted in the account that you have access privilege. If the domain name is hosted by another account, you will not be able to add DNS record. To register a public domain name under your account in AWS, go to AWS management console portal. Under Services, select Route 53 Management Console. Under Domains, select Registered domains, then click Register Domain. OpenVPN is a registered trademark of OpenVPN Inc. 192 Chapter 33. OpenVPN for Geo Locations

199 CHAPTER 34 UDP LoadBalanced VPN using DNS This feature is available from version 2.7. AWS does not allow you to create loadbalancers for TCP VPN gateways. To circumvent this issue, we utilize Route53 services of AWS to create a round robin based UDP LoadBalanced VPN Gateways. Note: UDP based OpenVPN provides higher packet throughput than TCP based VPN solution. UDP based VPN solution runs on UDP If you plan to deploy this solution for on-prem users, make sure your corporate firewall is open on UDP 1194 for outbound traffic Configuration Workflow Tips: Upgrade to the latest version. Make sure you are running Create VPN Gateways from Gateways Page. Make sure you have VPN Enabled and ELB disabled. 2. Create DNS Loadbalancers a. Go to OpenVPN ->Advanced->UDP Loadbalancer b. Click +New button (a) Select cloud type and account (Currently only supported on AWS) (b) Enter the hosted zone name ( This must exist in your AWS Route53) (c) VPN Service name is a unique identifier fot the Loadbalancer. For example a service name vpn1 and hosted zone aviatrix.com will create a DNS entry vpn.aviatrix.com (d) Select the Gateways that need to be added to the Loadbalancer. If you dont see any gateways, you may not have created non ELB VPN Gateways. i. Hit OK and this creates the UDP LoadBalancer 22. Add VPN Users. a. Add VPN Users directly to the LoadBalancer by going to OpenVPN ->VPN Users page b. In the VPC ID/LB /DNS select the loadbalancer created in Step 2. ( Example:vpn.aviatrix.com ) c. Populate the username and fields and the VPN user is created 193

200 23. (Optional) Edit DNS LoadBalancer a. You can add or delete gateways to the load balancer after it has been created OpenVPN is a registered trademark of OpenVPN Inc. 194 Chapter 34. UDP LoadBalanced VPN using DNS

201 CHAPTER 35 Okta Authentication Aviatrix VPN gateway supports Okta authentication as part of multi-factor authentication for OpenVPN access. Following are the steps. 1.0 Log in into your Okta account as Super Admin. This allows the privilege to create a Token for API access by Aviatrix gateway. 1.1 Go to Security -> API -> Create Token. Give the token a name, for example, Aviatrix, and copy the token string. You ll need the token string for Aviatrix gateway API access to Okta. 2.0 If you have not created users for VPN access, go to Directory -> People to create an account for VPN user. In this example, the account name is demoaviatrix@aviatrix.com 195

202 3.0 At Aviatrix Controller, go to Gateway to create a gateway with VPN Access enabled. Select Okta for Two-step Authentication and enter Okta related fields as following: URL: Your Okta account login URL. (For example, Token: Token string copied from Step 1. Username Suffix (Optional): In this example, aviatrix.com was entered. If Username Suffix is provided, users should enter their account ID without the domain name when loggin in from the VPN Client. For example, if your Okta account is demoaviatrix@aviatrix.com and aviatrix.com as Username Suffix, you should enter demoaviatrix as your VPN username when prompted for username by OpenVPN Client. If Username Suffix is not provided, you must enter demoaviatrix@aviatrix.com, as shown below. 196 Chapter 35. Okta Authentication

203 OpenVPN is a registered trademark of OpenVPN Inc. 197

204 198 Chapter 35. Okta Authentication

205 CHAPTER 36 Duo Authentication Aviatrix OpenVPN solution provides DUO authentication integration. This document helps you to set up DUO API. For how to configure OpenVPN, check out this link Getting DUO API credentials You need to first have a DUO account setup. If you company already has a DUO account and you have admin privilege, log in to the DUO Admin Panel, Click Applications. Click one of the applications, in the Details panel is the API credential you need to configure DUO on the controller. If this is your first time creating a DUO account, Log in to the Duo Admin Panel and navigate to Applications. Click Protect an Application and locate OpenVPN in the application list. Click Protect this application to view Integration key, secret key and API hostname. This information is needed to configure DUO on the controller. For more details, see for more details. At least one user has to be created in DUO account like below before the Duo authentication on the controller is setup so the user can log into the console. 199

206 If you cannot login to the controller because of DUO account is not setup properly, you can always change DUO policy to Bypass mode to disable push approval authentication. OpenVPN is a registered trademark of OpenVPN Inc. 200 Chapter 36. Duo Authentication

207 CHAPTER 37 OpenVPN with SAML Client on Okta IDP Overview Aviatrix user VPN is the only OpenVPN based remote VPN solution that provides a vpn client that supports SAML authentication. This guide provides an example on how to use Aviatrix SAML client to authenticate Okta IDP. When SAML client is used, Aviatrix controller acts as the identity service provider (ISP) that redirects browser traffic from client to IDP, in this case, Okta, for authentication Pre-Deployment Checklist Before configuring the SAML integration between Aviatrix and Okta, make sure the following is completed. Pre Installation Check List 1. Aviatrix Controller is setup and running. 2. Have a valid Okta account with admin access. 3. Download and install the Aviatrix SAML client These prerequisites are explained in detail below Aviatrix Controller If you haven t already deployed the Aviatrix controller, follow the below instructions on how to deploy the Aviatrix controller. Instructions here. 201

208 Okta Account A valid Okta account with admin access is required to configure the integration. If you don t already have an Okta account, please create one with the following link from Okta. Okta create account Aviatrix SAML Client All users must use the Aviatrix SAML client to connect to the system. Download the client for your OS here. For Linux users, do this:: tar -xvzf AVPNC_linux.tar.gz; cd AVPNC_setup;./install.sh to install type AVPNC in the terminal to run Configuration The integration configuration consists of 4 parts. 1. Create an Okta SAML App for Aviatrix 2. Retrieve OKta IDP metadata 3. Launch Aviatrix Gateway 4. Create Aviatrix SAML SP Please complete the configuration in the following order Create an Okta SAML App for Aviatrix This step is usually done by the Okta Admin. 1. Login to the Okta Admin portal 2. Click Admin 3. Click Applications 4. Click Add Application 5. Click Create New App (a) Platform = Web (b) Sign on method = SAML Chapter 37. OpenVPN with SAML Client on Okta IDP

209 6. General Settings (a) App Name = Aviatrix Dev (arbitrary) 7. SAML Settings (a) Single sign on URL* = (b) Audience URI(Entity ID)* = (c) Default RelayState* = Create an Okta SAML App for Aviatrix 203

210 (d) Name ID format = Unspecified (e) Application username = Okta username These values are also available in the controller OpenVPN ->Users page after step 3.4 The aviatrix_controller_hostname is the hostname of the Aviatrix controller(if no DNS is used, this is the public IP). The aviatrix_username is an arbitrary identifier. Note this value as it will be needed when configuring SAML from the Aviatrix controller. Please contact your Aviatrix admin if you do not have the Aviatrix controller s public IP address. (a) Attribute Statements i. FirstName -> Unspecified -> ${user.firstname} ii. LastName -> Unspecified -> ${user.lastname} iii. -> Unspecified -> user. 204 Chapter 37. OpenVPN with SAML Client on Okta IDP

211 8. Done Retrieve Okta IDP metadata This step is usually completed by the Okta admin. After the above application is created, click on Sign On and then View Setup Instructions Retrieve Okta IDP metadata 205

212 Look for the section titled IDP metadata to your SP provider. 206 Chapter 37. OpenVPN with SAML Client on Okta IDP

213 Note this information. This information will be used to configure the SAML configuration on the Aviatrix controller Launch Aviatrix Gateway This step is usually completed by the Aviatrix admin. 1. Login to the Aviatrix controller 2. Click Gateway -> Add New 3. Select the appropriate Account, region, vpc, subnet and gateway size 4. Check VPN Access and then Enable SAML 5. Default settings for everything else Launch Aviatrix Gateway 207

214 6. Click OK to launch the gateway Create Aviatrix SAML SP This step is usually completed by the Aviatrix admin. 1. Login to the Aviatrix Controller 2. Click OpenVPN -> VPN Users -> Add New 3. Select the VPC where the above gateway was launched 4. Username = aviatrix_username (this is the username that you choose during the Okta SAML configuration) 5. User = any valid address (this is where the cert file will be sent 6. IPD Metadata type = Text 7. IDP Metadata Text = paste in the IDP metadata from the Okta configuration 8. Entity ID = Hostname 9. Done OpenVPN is a registered trademark of OpenVPN Inc. 208 Chapter 37. OpenVPN with SAML Client on Okta IDP

215 CHAPTER 38 Anonymous Internet Surfing Solution Overview Normally when you surf an Internet website, the website administrator can easy identify where the user is located. This is done by identifying the source IP address contained in the packets (public IP address assigned to your location). Sometimes, business needs arise when your employee s internet browsing and online research needs to be anonymous or needs to appear to originate from some other place. Such needs arise when analysis of competitors is required or when avoiding countries firewalls for better performance and access. This document describes how to setup anonymous browsing from client machine by routing internet traffic through an AWS-based gateway in a different region Configuration Workflow Pre Configuration Checklist Before configuring VPC Site to Cloud peering, make sure the following prerequisites are completed. Pre Configuration Check List 1. Deploy the Aviatrix Controller 2. Create AWS VPCs and Check Settings These prerequisites are explained in detail below Deploy the Aviatrix Controller The Aviatrix Controller must be deployed and setup prior to configuring VPC and site peering. Please refer to Aviatrix Controller Getting Started Guide for AWS on how to deploy the Aviatrix Controller. Aviatrix Controller Getting Started Guide 209

216 Check and make sure you can access the Aviatrix Controller dashboard and login with an administrator account. The default URL for the Aviatrix Controller is: ip of Aviatrix Controller> Create AWS VPCs and Check Settings Create 2 VPCs - VPC #1 (in Region 1) with CIDR /16 and VPC #2 (in Region 2) with CIDR /16 In VPC #1, create 2 public subnet in same Availability Zone /24 and /24. This means both subnets must be associated with a route table whose default route points to IGW. In VPC #2, create 1 public subnet /24. This means one subnet must be associated with a route table whose default route points to IGW Configuration Steps Make sure the pre-configuration steps in the previous section are completed before proceeding. The instructions in this section will use the following architecture. The CIDR and subnets may vary depending on your VPC setup; however, the general principals will be the same Step a Deploy Gateways The first step is to deploy Aviatrix gateways in each VPC. Instructions: a.1. Login to the Aviatrix Controller Console a.2. Create Aviatrix Peering Gateway #1 in Subnet1 of VPC #1 (in Region 1) a.3. Click on Gateway -> New Gateway 210 Chapter 38. Anonymous Internet Surfing

217 Setting Value Cloud Type Choose AWS Gateway Name This name is arbitrary (e.g. vpc-01-avx-gw) Account Name Choose the account name Region Choose the region of VPC #1 VPC ID Choose the VPC ID of VPC #1 Public Subnet Select a public subnet where the gateway will be deployed (e.g /24) Gateway Size t2.micro is fine for testing Enable NAT Uncheck this box (IMPORTANT) VPN Access Uncheck this box Designated Gateway Uncheck this box Allocate New EIP Uncheck this box Save Template Uncheck this box a.4. Click OK. It will take a few minutes for the gateway to deploy. Do not proceed until the gateway is deployed. a.5. Create Aviatrix VPN Gateway in Subnet2 of VPC #1 (note that VPN Gateway is in a different subnet of Peering Gateway) a.6. Click on Gateway -> New Gateway Setting Value Cloud Type Choose AWS Gateway Name This name is arbitrary (e.g. vpc-01-avx-vpn) Account Name Choose the account name Region Choose the region of VPC #1 VPC ID Choose the VPC ID of VPC #1 Public Subnet Select the public subnet where the VPN gateway will be deployed (e.g /24) Gateway Size t2.micro is fine for testing. Enable NAT Uncheck this box VPN Access Check this box Designated Gateway Uncheck this box Allocate New EIP Uncheck this box Enable SAML Uncheck this box VPN CIDR Block (e.g /24) MFA Authentication Optional (Disable is fine for testing) Max Connections 100 is fine for testing Split Tunnel Mode No Enable ELB Yes ELB Name Leave blank is fine for testing Enable Client Cert. Sharing No Enable PBR Check this box PBR Subnet Select the subnet where Aviatrix Peering Gateway is located (e.g /24) PBR Default Gateway Select the private IP of Aviatrix Peering Gateway (e.g ) NAT Translation Logging Uncheck this box Enable LDAP Optional (Uncheck this box is fine for testing) Save Template Uncheck this box a.7. Click OK. It will take a few minutes for the gateway to deploy. Do not proceed until the gateway is deployed. a.8. Create Aviatrix Peering Gateway #2 in VPC #2 a.9. Click on Gateway -> New Gateway Configuration Workflow 211

218 Setting Value Cloud Type Choose AWS Gateway Name This name is arbitrary (e.g. vpc-02-avx-gw) Account Name Choose the account name Region Choose the region of VPC #2 VPC ID Choose the VPC ID of VPC #2 Public Subnet Select a public subnet where the gateway will be deployed (e.g /24) Gateway Size t2.micro is fine for testing Enable NAT Check this box (IMPORTANT) VPN Access Uncheck this box Designated Gateway Uncheck this box Allocate New EIP Uncheck this box Save Template Uncheck this box a.10. Click OK. It will take a few minutes for the gateway to deploy. Do not proceed until the gateway is deployed. a.11. Done Step b Establish Site to Cloud peering connection This step explains how to establish a Site to Cloud (S2C) connection between two Aviatrix Gateways in VPC #1 and VPC #2. Instructions: b.1. From the Aviatrix Controller Console b.2. Click Site2Cloud -> Site2Cloud b.3. Click +Add New to establish S2C connection from Aviatrix Peering Gateway #1 (in VPC #1) to Aviatrix Peering Gateway #2 (in VPC #2). Setting Value VPC ID/VNet Name Choose VPC ID of VPC #1 Connection Type Unmapped Connection Name This name is arbitrary (e.g. vpc01-s2c-vpc02) Remote Gateway Type Aviatrix (in this example) Tunnel Type UDP Algorithms Uncheck this box Encryption over DirectConnect Uncheck this box Enable HA Uncheck this box Primary Cloud Gateway Select Aviatrix Peering Gateway #1 in VPC #1 (e.g. vpc-01-avx-gw) Remote Gateway IP Address Public IP of Aviatrix Peering Gateway #2 in VPC #2 Pre-shared Key Optional Remote Subnet /0 Local Subnet IP of eth1 of Aviatrix VPN Gateway #1 (e.g /32) b.4. Click button OK b.5. From S2C connection table, select the site2cloud connection created above (e.g. vpc01-s2c-vpc02) b.6. Select Aviatrix from Vendor drop down list b.7. Click button Download Configuration then save it 212 Chapter 38. Anonymous Internet Surfing

219 b.8. Click +Add New to establish Site2Cloud connection from Aviatrix Peering Gateway #2 b.9. Choose VPC ID of VPC #2 from VPC ID/VNet Name drop down list. Click button Import to upload the downloaded configuration saved in Step b.7. b.10. This template file contains the necessary information to configure the new S2C connection. Setting Value VPC ID/VNet Name Choose VPC ID of VPC #2 Connection Type Unmapped Connection Name This name is arbitrary (e.g. vpc02-s2c-vpc01) Remote Gateway Type Aviatrix Tunnel Type UDP Algorithms Check this box Phase 1 Authentication SHA-1 Phase 2 Authentication HMAC-SHA-1 Phase 1 DH Groups 2 Phase 2 DH Groups 2 Phase 1 Encryption AES-256 Phase 2 Encryption AES-256 Encryption over DirectConnect Uncheck this box Enable HA Uncheck this box Primary Cloud Gateway Aviatrix Peering Gateway #2 (e.g. vpc-02-avx-gw) Remote Gateway IP Address Public IP of Aviatrix Peering Gateway #1 Pre-shared Key (automatically created) Remote Subnet IP of eth1 of Aviatrix VPN Gateway #1 (e.g /32) Local Subnet /0 Notes: IP of eth1 of Aviatrix VPN Gateway can be acquired from AWS console b.11. Click button OK b.12. Done Step c Create an OpenVPN user This step explains how to create a OpenVPN user. Instructions: c.1. From the Aviatrix Controller Console c.2. Click OpenVPN -> VPN Users c.3. Click button +Add New Setting Value VPC ID Choose the VPC ID of VPC #1 LB/Gateway Name Choose the ELB in VPC #1 User Name This name is arbitrary (ex. vpn-user) User address Profile Uncheck this box is fine for the testing c.4. Click button OK c.5. Check your to receive an ovpn file Configuration Workflow 213

220 c.6. Done Step d Start anonymous browsing This step explains how to establish an OpenVPN connection and surf network as anonymous. Instructions: d.1. Enable an OpenVPN client tool d.2. Establish an OpenVPN connection with the ovpn file which has received in d.3. Confirm the connectivity to public network d.3.1. Ping to d.3.1. Check public IP address (ie. d.3.2. Check IP location (ie Troubleshooting To check a tunnel state, go to Site2Cloud, the tunnel status will be displayed at status column. To troubleshoot a tunnel state, go to Site2Cloud -> Diagnostics. OpenVPN is a registered trademark of OpenVPN Inc. 214 Chapter 38. Anonymous Internet Surfing

221 CHAPTER 39 Developer s Sandbox 39.1 Objective As a gatekeeper to manage a functional network for your production environment in the cloud, security must be one of your top concerns. How to keep your environment secure while giving your developers the freedom to experiment and learn new services offered by AWS is a challenge. We are here to help you. This reference design leverages the multi tenants capability of Aviatrix controller to build sandboxes for your developers. While the developer has full administrative authority to her sandbox, the sandbox itself is isolated from your main production environments. The network diagram is shown below, 215

222 where the Aviatrix controller instance can be in the same or a different VPC, and two developer s sandboxes are shown. John: /16 and Sam: /16. In this configuration, assume you want to the VPN to be in split tunnel mode, that is, only traffic destined to the cloud go through the SSL tunnel. If a user does general browsing to Internet or watch movies from Hulu, traffic should be routed via her device WI-FI to ISP to Internet. You do not wish to pay AWS for this type of compute and network costs Solution The solution is to give John and Sam each their own AWS accounts, link their accounts to your corporate root account, using the Consolidated Billing feature offered by AWS. With John s own AWS account and API credentials, you create a corresponding Cloud Account on the controller. Use this Cloud Account, you can create VPC for John. John may have more privileges in his VPC Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. We assume here that you have created a management VPC /16, its corresponding VPN gateways and John has been added as a VPN user. For more information for this part of configuration, check out this reference design: 216 Chapter 39. Developer s Sandbox

223 The configuration workflow is as follows, with major steps highlighted. 1. Create a Cloud Account for John Go to Accounts -> Cloud Account -> New Account, make sure: (a) Account Name is unique to the controller, for example, JohnSmith. (b) Account Password can be used to login in with Account Name. (c) An will be sent for this account created. (d) Add AWS credentials for this account. 2. Create a VPC and Gateway for John Go to Advanced Config -> Create VPC Pool -> Create: (a) Account Name: JohnSmith (b) Pool Name: John (c) Number of VPCs: 1 (d) VPC Size: the gateway size. A t2.micro maybe all you need. A t2.micro Aviatrix gateway performance is between 40mbps to 80mbps. (e) Launch Gateway: check (f) Custom CloudFormation Script: a URL that points to your custom CloudFormation script in S3. Note only VPC ID is taken as input parameter. After gateway is launched, a CloudFormation stack will be created. One use case for this script is security groups and policies if you have a standard one. (g) Public Subnets: check. This will create subnets whose default gateway is IGW. (h) Enable NAT: check. If this is checked, NAT function is integrated on the gateway. 3. Build Encrypted Peering Go to Peering -> Encrypted Peering -> New Peering Note each VPC is represented by one or more gateways. Make sure you want to peer between two gateways without VPN capability. In this example, the peering is between John and the peering gateway in the management VPC /16 (no the VPN gateway) 4. Repeat the above two steps for other developers or projects. 5. Add users If you have not done so, add VPN user John to the cloud network. Go to OpenVPN, Use Profile to control which user can access what cloud instance/application/ports. 6. Done. OpenVPN is a registered trademark of OpenVPN Inc Configuration Workflow 217

224 218 Chapter 39. Developer s Sandbox

225 CHAPTER 40 Tag Based Security Policy Aviatrix gateway security policies is implemented at each gateway, key features are: It is a L4 stateful firewall that filters on CIDR, protocol and port. Each policy is assoicated with Allow or Deny action. A Base policy for Allow or Deny for the gateway can be used as a catch all rule. All security policy events as well as packets can be logged to Splunk, SumoLogic, Syslog, ELK and Datadog. Starting with release 3.0, A tag mechanism has been introduced to make the security policy specification more user friendly. You can associate an IP address or a subnet with a name tag and use it as a shorthand to specify the source and destination for your security rules Define a Tag You give a tag a name and a list of one or more network addresses. The network address can be a subnet or a host IP address. Security -> Stateful Firewall -> Tag Management -> Add New A tag is a global resource to the Aviatrix Controller and can be applied to any gateway Edit a Tag Once a tag is created, you can Edit the tag. Editing is about adding a name to a CIDR address (network address or host address). Multiple Name<->CIDR pair can be added. When you are done Editing, click Update to take effect Apply Policy Click Security -> Stateful Firewall -> Policy 219

226 You should see a list of gateways that the Controller manages. Highlight a gateway and click Edit. To configure security policies, select a Base Policy. Base Policy is the always attached as the last rule as a catch all policy. Select Enable Packet Logging if you want to forward logs to well known log management systems, such as Splunk, Sumo Logic, Elastic Search and remote syslog. If you click Add New, you can specify Source by manually entering a specific CIDR; you can also click the table icon to select one of the tags you created earlier. Both Source and Destination can be configured either manually or by using tags. Once a rule is Saved, you must remember to click Update for the policy to take effect. Note: For the Destination field, if a host name is specified either manually or with a tag, the IP address of the host name will be resolved when programming the security policy. A host name is not suitable if it is a public web site, such as To filter on public host names, refer to FQDN Whitelists View Policy and Tags To view the names in a tag, select Tag Management, highlight a tag and click Edit. To view the policies of gateway, select Policy, highlight a gateway and click Edit Example Use Case Say you have a group of EC2 instances or a group of AWS Workspace instance. You like to setup policies to allow them to access a database which itself consists of a group of nodes. You can create a tag, name it my-app, configure the list of IP addresses assoicated with each instance with a name. You can then create a second tag, name it my-database, configure the list of IP addresses assoicated with each instance with a name. You then can simply apply one policy at the gateway that says my-app to my-database is allowed. The Controller will automatically push the policies to the gateway. 220 Chapter 40. Tag Based Security Policy

227 CHAPTER 41 FQDN Whitelists 41.1 Why FQDN Whitelists feature are needed? Aviatrix Security Policy feature is enabled at a gateway for a stateful firewall filter at layer 4 level. You specify an action for each rule, allow or deny, for each packet as it passes through the gateway. The rules are based on network, IP addresses, protocol and ports. This feature is useful to firewalling different private networks. For Internet bound egress traffic, specifying at IP address level is not sufficient as the domain names of a site can be translated to many different IP addresses. The egress filtering needs to happen at Layer 7. On the other hand, workloads in AWS are mostly applications where it is deterministic which outbound APIs the application program calls. For example, the application runs API queries to for data retrieving; the application also runs API queries to for app authentication. In these cases, making sure only these sites are allowed for egress traffic is sufficient from security point of view. Note this is very different from onprem situation where end user traffic and application traffic are mingled together, you may need a full fledged firewall for Internet bound traffic What does Aviatrix FQDN feature do? Aviatrix Fully Qualified Domain Name (FQDN) Whitelisting is a security feature specially designed for workloads in public cloud. It filtersinternet bound egress traffic initiated from workloads in a VPC. Aviatrix FQDN Whitelisting filters on HTTP and HTTPS traffic and allows only the destination host names (whitelist) specified in the list to pass and drop all other destinations. Each host name is specified as fully qualified domain name. For example, if you only allow Internet bound traffic to you can list the domain name in the whitelist. It also supports wild card, such as *. In this example, you can specify *.salesforce.com to allow traffic to any domain names that ends salesforce.com. Note the gateway must have NAT enabled if you want to turn on FQDN whitelists. 221

228 41.3 How does it work? This features works for HTTP and HTTPS traffic to public Internet. The function is carried out inline without requiring any certificate or keys to decrypt the traffic. A tag is defined as a list of FQDNs. One or more gateways is attached to a tag. Any updates to a tag automatically triggers updates to all gateways attached to the tag. Multiple tags can be defined for the controller. The domains in the tag are the destinations that are allowed for traffic to pass Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message (New!) appears, click New! To upgrade to the latest software. 1. To configure, go to Advanced Config -> FQDN Filter. 2. Create a tag with a name. Click Enable. 3. Edit the tag by adding FQDN hostname part of URLs (e.g. or *.google.com). 4. Attach Gateway. One or more gateways can be attached to a tag. 5. Note: Step 2, 3 and 4 can be done first without enabling the tag. Once the tag is enabled, HTTP and HTTPS traffic to these FQDN will be allowed, and any destination outside the FQDN will be denied. 6. For support, send to support@aviatrix.com 7. Enjoy! 222 Chapter 41. FQDN Whitelists

229 CHAPTER 42 IPmotion Setup Instructions Aviatrix IPmotion (IP Motion) is a technology that connects the same two subnets between on-prem and in the VPC. The technology is useful when migrating an on-prem VM to public cloud while preserving its IP address. It can also be used for mission critical application HA to public cloud. The technology is described in the diagram below, where an on-prem VM with IP address is migrated to AWS while preserving its IP address. After migration, any on-prem VMs can continue to communicate with this migrated VM as if it still resides on-prem. Note the actual migration process is not included in this document. We assume you have tools, for example, AWS Migration Hub to migrate on-prem VMs to public cloud. We also provide an example that demonstrates how to migrate a VM by combining AWS Server Migration Service and IPmotion. 223

230 42.1 Planning and Prerequisites 1. Identify an on-prem subnet where you plan to migrate VMs. For example, the subnet is / Create a AWS VPC that has the same or larger CIDR block than the migrating subnet. 3. IPmotion builds an IPSEC tunnel using UDP ports 500 and Make sure these two UDP ports are open for outbound traffic. Inbound return traffic will also run on these two ports. The prots should be optn to AWS public IP address ranges. 4. Consider Design Patterns for IPmotion. 5. For simplicity, in this guide, we assume the cloud subnet is a public subnet and the migration is over Internet 6. Deploy Aviatrix virtual appliance CloudN in the on-premise subnet. Read this document on how to deploy the virtual appliance. AWS reserves five IP addresses on a given subnet, make sure CloudN IP address is not any one of them. For example on a /24 subnet, and are reserved. 7. Once the virtual appliance is deployed, go through on-boarding process and create an AWS account. 8. Take an inventory of IP addresses of all running VMs and unused IP addresses on this subnet, as shown in the example below. Note: For description purpose, a migrated VM that has the same IP address as its on-prem VM is called the migrated EC2 instance Login to the CloudN Controller Open a browser and navigate to IP address>/. Once authenticated, click on IP Motion in the left navigation bar. Follow the steps below to set up IP Motion for the selected subnet. 224 Chapter 42. IPmotion Setup Instructions

231 Specify on-prem IP Address List The on-prem IP address list of a subnet includes both the list of IP addresses of VMs that will be migrated and the list of IP addresses of VMs that will remain on-prem but need to communicate with the migrated VMs. One simple way to specifiy this address range is to provide the list of IP addresses of all running VMs, excluding CloudN IP address, since out of this list, some or all VMs will be migrated to cloud. For example, as shown in the above diagram, if the running VMs excluding CloudN on subnet /24 are in the range of , and you plan to move all running VMs to cloud, then specify this range for Step 1 as below Note: the on-prem IP address format could be a single IP address or a range of IP addresses using a - in the list. Specifiy multiple ranges of IP addresses by separting them with a comma. Example: , , Note the larger this list is, the larger IPmotion gateway instance size needs to be in the cloud (AWS). The reason is that IPmotion gateway needs to allocate private IP addresses from AWS for any on-prem VMs. You can optimize the list by making sure only the running VMs are being specified. For the above example, if is an IP address not assigned to any VM, you should skip this address and specify a multiple range separating by a comma: , , Currently the largest number of VMs that a CloudN can handle on a subnet is 202 which requires a c4.4xlarge IPmotion gateway instance size. This number of VMs can be expanded in the future release. (You can further optimize the list for the on-prem part by specifying only the dependent VMs. For example, the CloudN is deployed on subnet /24. On this subnet, IP addresses of VMs that are planed to be migrated are , IP addresses of VMs that are to remain on the subnet but need to communicate with migrated VMs are in the range then you should enter , , ) , , Reserve IPmotion Gateway IP Address List This field is about specifying 10 IP addresses that are not being used by any running VMs and reserve these addresses for Aviatrix IPmotion gateway. Again as an example displayed in the above diagram, are not used by any running VMs, you can reserve this range for IPmotion gateway. In another words, if you specify as IPmotion gateway reserved IP addresses, it means that these range of IP addresses are not currently used by any VM on the subnet, they are reserved by Aviatrix during migration phase. Note: AWS reserves the 5 IP addresses of a subnet in VPC. For example, if the VPC subnet is /24, the addresses , , , and are reserved by AWS. if you have on-prem VMs including CloudN that uses the first 3 IP addresses (excluding default gateway, DNS or any other infrastructure purpose) of a subnet, the IPmotion method will not work Specify on-prem IP Address List 225

232 Launch IPmotion Gateway This step launches an Aviatrix IPmotion gateway and builds an tunnel (IPSEC tunnel if the connection is over Internet, direct tunnel if the connection is over Direct Connect.) between the two subnets. Note the IPmotion gateway size reflects how many on-prem VMs can be supported, as the table shown below. IPmotion Gateway Size Max VMs can be migrated t2.micro 0 t2.small 2 t2.medium 9 m4.large 8 m4.xlarge 41 m4.2xlarge 41 m4.16xlarge 202 c3.large 17 c3.xlarge 41 c3.2xlarge 41 c3.4xlarge 202 c4.large 17 c4.xlarge 41 c4.2xlarge 41 c4.4xlarge 202 c4.8xlarge 202 The Migrate Subnet is the subnet that has the same CIDR as on-prem migrating subnet. IPmotion Gateway Subnet is the subnet where Aviatrix IPmotion gateway is deployed. Consult Design Pattern for IPmotion subnet choice IPmotion Move This step consists of two parts: Staging and Commit Staging Staging is the preparation step. After an IP address is moved to Staging state, you can power up the migrated EC2 instance with the same IP address as the on-prem VM for testing and staging. Note the migrated EC2 instance at this point cannot communicate with on prem. Highlight a specific IP address in on-prem panel and click the Staging button Undo Staging If you want to move any IP address in Staging state back to on-prem, select the IP address and click Undo. Note: if the migrated EC2 instance is already running, you must terminate the instance from AWS console before you can move its IP address back to on-prem state. 226 Chapter 42. IPmotion Setup Instructions

233 Commit Commit is to enable the migrated EC2 instance to communicate with any on-prem VM. Note: Before you commit an IP address, the on-prem VM that has been migrated must be powered down first. Commit the IP address implies that the migrated EC2 instance will be in operation. Hightlight a specific IP address and click the Commit button Undo Commit If migration fail after cut over, you can Undo the Commit by selecting the IP address from the cloud panel and click Undo. Undo function of Commit is to revert a committed IP address to Staging state. After reverting to Staging state, the communication between the migrated EC2 instance to on-prem is stopped and you can power up the on-prem VM and resume its operation Test Connectivity After an IP address is committed, you can test connectivity. Go to CloudN console, Troubleshoot -> Diagnostics -> Network -> Ping Utility. Enter the committed IP address and click Ping. Make sure the security group of the migrated EC2 has ICMP allowed. Also make sure the migrated EC2 instance responds to Ping request Troubleshooting Tips View Button click View button on Step 1 or Step 2 at any time to see what state an IP address is at. Reset Button If all things fail and you like to start over, first delete the IPmotion gateway by going to Gateway List, select the gateway and click Delete. After Delete is completed, go to Step 1 and click Reset. You can then start it over by going through Step 1 again. Get Support support@aviatrix.com for assistance Discover application dependencies After migrating one VM, you can use Aviatrix IPmotion gateway to discover application dependencies by following the dependancy map discovery Migrate more VMs on the same subnet Repeat Step 4 to migrate more VMs on this subnet Test Connectivity 227

234 Migrate VMs in a different subnet To migrate a VM in a different subnet, you need to launch a new virtual appliance CloudN on that subnet and repeat all the steps described in this document. For example, suppose you have created a VPC /16 and migrated subnet /24. Now you plan to migrate subnet /24. Follow these steps: Go to AWS console to create a second public subnet /24 in VPC /16. Launch Aviatrix virtual appliance CloudN on subnet /24. Repeat the steps listed in this document Post Migration Once you have migrated a few subnets to a VPC, you have the option to delete Aviatrix IPmotion gateway, delete the Aviatrix on-prem virtual appliance and remove the on-prem subnets that are now empty of any VMs. You can then connect the VPC to on-prem via Aviatrix site2cloud, AWS Direct Connect and other layer 3 connectivities Limitations There are a few known limitations in the current release. Cannot migrate any on-prem VMs whose IP addresses overlap with AWS reserved IP addresses on a given subnet. AWS reserves five IP addresses of a given subnet, if an on-prem VM overlaps with any of these three IP address, this solution cannot migrate this VM. VPC CIDR cannot be /16. In the range, the largest CIDR is /17. The maximum number of on-prem VMs can be migrated per subnet is 202. Aviatrix IPmotion solution is deployed on a per subnet bases, the maximum throughput per gateway is 1Gbps for IPSec performance. If connecting over private link such as Direct Connect, the performance is higher. 228 Chapter 42. IPmotion Setup Instructions

235 CHAPTER 43 IPmotion Early Customer Trial Instructions 1. Get a trial license Obtain a customer ID from Aviatrix support. to support@aviatrix.com 2. Read IPmotion Setup Instructions 3. Complete the Prerequisites in the above document that include download, install and bootup Aviatrix virtual appliance CloudN. 4. Download IPmotion beta software Login to the web console of CloudN. Go to Settings -> Maintenance - > Upgrade to Custom Release field, enter 3.0, click Upgrade to a custom release. This will download the IPmotion beta software. When it finishes, repeat this step to upgrade again. 5. Setup IPmotion Once upgrade is successful, relogin to the console, at the left navigation menu, click IPmotion, follow the step by step instruction to starting moving IP addresses! 229

236 230 Chapter 43. IPmotion Early Customer Trial Instructions

237 CHAPTER 44 Migrating VMs with Aviatrix IPMotion and AWS Migration Hub Service Solution Overview This document describes how to migrate an on-prem VM to AWS while preserving its IP address. The migration tools we use are AWS Migration Hub service (AWS Server Migration Service) and Aviatrix IPmotion, where Aviatrix IPmotion feature enables IP address preservation after an VM is migrated to AWS via AWS Server Migration Service. By preserving its IP address of an on-prem VM after migrating to AWS, dependencies of this VM to other on-prems are automatically preserved, thus there is no need to discover the dependencies for migration purpose. There is no need to update on-prem security rules, AD, DNS and Load Balancers Configuration Workflow The instructions in this section will use the following network diagram. The CIDR and subnets may vary depending on your network setup; however, the general principle will be the same. 231

238 Prerequisites Before setting up Aviatrix IPMotion for migration, make sure the following prerequisites are completed. 1. Plan the Cloud Address and create an AWS VPC 2. Setup AWS Server Migration Service (SMS) to create migrated AMIs 3. Deploy an Aviatrix Virtual Appliance CloudN in On-Premise These prerequisites are explained in detail below Plan the Cloud Address and create an AWS VPC First identify the on-prem subnet from which you plan to migrate VMs. In this example, the subnet is /16 with two On-Prem VMs ( and ) (In this illustration, the cloud subnet is a public subnet. There are other design patterns you can follow.) Then create an AWS VPC with a public subnet that has identical CIDR as the on-prem subnet where migration is to take place. For example, create a VPC CIDR /16 with a public subnet /16 in region Oregon. Note that it is not neccessary for the migrated VMs to have public IP addresses. AWS Example Setting Value Cloud Type AWS Region Oregon VPC CIDR /16 Public Subnet / Setup AWS Server Migration Service (SMS) to create a migrated AMI Please refer to AWS Server Migration Service Server Migration to the Cloud Made Easy! for detail. AWS Server Migration Service Server Migration to the Cloud Made Easy! Deploy the Server Migration Connector virtual appliance on On-Premise. vcenter Setting Example Setup networks /16 Configure the connector on On-Premise. Connector Setting AWS Region Example US West (Oregon) Import the server catalog on AWS SMS console AWS SMS Setting Example Replication job ID VM which will be migrated to cloud (e.g. VM with ip ) After completing the previous steps, a user is able to view and launch the migrated AMI in below console: i.) AWS -> Migration -> Server Migration Service 232 Chapter 44. Migrating VMs with Aviatrix IPMotion and AWS Migration Hub Service

239 ii.) AWS -> Compute -> EC2 -> Launch Instance Please confirm the migrated AMI is ready on AWS console. This document will describe how to integrate the migrated AMI with IPMotion feature in Step b Deploy an Aviatrix Virtual Appliance CloudN in On-Premise subnet The Aviatrix Virtual Appliance CloudN must be deployed and setup in the on-prem subnet where you plan to migrate VMs prior to configuring IPMotion. For example, the subnet is /16. Please refer to Virtual Appliance CloudN on how to deploy the Virtual Appliance CloudN. Virtual Appliance CloudN Check and make sure you can access the Aviatrix Virtual Appliance CloudN dashboard and login with an administrator account. The default URL for the Aviatrix Virtual Appliance CloudN is: ip of Aviatrix Virtual Appliance CloudN> Configuration Steps Make sure the pre-configuration steps in the previous section are completed before proceeding Configuration Workflow 233

240 Step a Deploy Aviatrix IPMotion gateway The first step is to deploy Aviatrix IPMotion gateway in AWS VPC. Please refer to IPmotion Setup Instructions for detail. IPmotion Setup Instructions Instructions: a.1. Login to the Aviatrix Virtual Appliance CloudN a.2. Click on IP Motion in the left navigation bar a.3. For section 1> Specify the on-prem IP Address List, enter both the list of IP addresses of VMs that will be migrated and the list of IP addresses of VMs that will remain on-prem. a.4. Click Specify. IPMotion Configuration Example On-prem Subnet IP List a.5. Click View to check those specified IPs and its status. Status Value ON-PREM IN-CLOUD-STAGING IN-CLOUD Notes IP of VM in On-Prem IP of VM in staging Mode IP of VM migrated to Cloud a.6. For section 2> Reserve IPmotion Gateway IP Address List, specify 10 IP addresses that are not being used by any running VMs and reserve these addresses for Aviatrix IPmotion gateway. a.7. Click View to check those reserved IPs. IPMotion Configuration Example IPmotion Gateway Reserve IP List a.8. For section 3> Launch an IPmotion Gateway in the AWS VPC, it launches an Aviatrix IPmotion gateway and builds an encrypted IPSEC tunnel between the subnet of On-Prem and AWS VPC. Setting Value Cloud Type Choose AWS Account Name Choose the account name Region Choose the region of VPC (e.g. us-west-2) VPC ID Choose the VPC ID of VPC Gateway Name This name is arbitrary (e.g. IPMotion-GW) Gateway Size t2.small is fine for testing. Gateway Subnet Select the public subnet (e.g /16) a.9. Click Launch. It will take a few minutes for the gateway to deploy. Do not proceed until the gateway is deployed. a.10. Done 234 Chapter 44. Migrating VMs with Aviatrix IPMotion and AWS Migration Hub Service

241 Note: Next Step b Integrate Aviatrix IPMotion with AWS AMI will explain how to utilize section 4> Let s Move! to coordinate IP migration with the migrated AMI created by AWS SMS Step b Integrate Aviatrix IPMotion with AWS AMI This step explains how to integrate Aviatrix IPMotion with the AMI that a user migrated from On-Premise VM to AWS via AWS SMS earlier. b.1. Click on IP Motion in the left navigation bar of GUI of Aviatrix Virtual Appliance CloudN b.2. Navigate to section 4> Let s Move! b.3. Select the IP of VM which will be migrated to cloud. (e.g ) b.4. Click Staging. This is the preparation step for a user to shutdown the On-Prem VM with the selected IP and power up its corresponding cloud VM with the same IP. b.4.1. Shutdown the On-Prem VM via vcenter. (e.g ) b.4.2. Power up the AWS EC2 instance with that selected IP. (e.g ) b Navigate to AWS -> Compute -> EC2 console b Click Launch Instance b Step 1: Choose an Amazon Machine Image (AMI) -> Click side bar My AMIs -> Click Select of the AMI which is created by AWS SMS b Step 2: Choose an Instance Type b Step 3: Configure Instance Details: b In first section, here is an example for the testing topology AWS Example Setting Value Number of instances 1 Purchasing Optional Uncheck this box is fine for testing Network Choose the VPC ID of the planned VPC Subnet Choose the Subnet ID of the planned Subnet Auto-assign Public IP Enable is fine for testing IAM role None is fine for testing Shutdown behavior Stop is fine for testing Enable termination protection Uncheck this box is fine for testing Monitoring Uncheck this box is fine for testing Tenancy Shared - Run a shared hardware instance is fine b (Important) In second section - Network interfaces, enter the selected IP (e.g ) Configuration Workflow 235

242 b Step 4: Add Storage: default settings is fine for testing. b Step 5: Add Tags: default settings is fine for testing. b Step 6: Configure Security Group -> Click Create a new security group. For this testing topology, adding a rule with Type of All traffic and Source of Custom /16 to allow all traffic between On-Prem VM and Cloud VM. User can further customize the firewall rules. b Step 7: Review Instance Launch -> Click Launch It will take a few minutes for the EC2 instance to deploy. Do not proceed until the EC2 instance is deployed. b.5. (Optional) Click View of section 1> Specify the on-prem IP Address List to check status. That IP status will change from status ON-PREM to IN-CLOUD-STAGING. b.6. Navigate back to the section 4> Let s Move! of IP Motion of GUI of Aviatrix Virtual Appliance CloudN b.7. Select IP > Click Commit b.8. (Optional) Click View of section 1> Specify the on-prem IP Address List to check status. That IP status will change from status IN-CLOUD-STAGING to IN-CLOUD. b.9. Done 236 Chapter 44. Migrating VMs with Aviatrix IPMotion and AWS Migration Hub Service

243 Step c Test Connectivity This step explains how to test the connectivity between the On-Prem VM to the migrated VM in the cloud. Instructions: c.1. Browse the GUI of Aviatrix Virtual Appliance CloudN c.1.1. Click side bar Troubleshoot -> Diagnostics -> Network -> Ping Utility. c.1.2. Enter the committed IP address -> click Ping. c.2. Test bi-directional end-to-end connectivity c.2.1. Login to the On-Prem VM (e.g ) c.2.2. Check ICMP protocal via command ping c.2.3. Login to the migrated EC2 (e.g ) c.2.4. Check ICMP protocal via command ping Note: Make sure the security group of the migrated EC2 has ICMP allowed. Also make sure the migrated EC2 instance responds to Ping request Troubleshooting 1. Click button View of section 1> Specify the on-prem IP Address List of IPMotion of GUI of Aviatrix Virtual Appliance CloudN to check what state an IP address is at. 2. Click button Reset if all things fail and you like to start over 2.1. First of all, delete the IPmotion gateway by navigating to side bar Gateway List 2.2. Select the gateway -> click Delete. It will take a few minutes to delete. Do not proceed until the gateway is deleted After deletion is completed, go back to section 1> Specify the on-prem IP Address List of IPMotion and click button Reset You can then start it over by going through Step a Deploy Aviatrix IPMotion gateway and Step b Integrate Aviatrix IPMotion with AWS AMI again. 3. Get Support support@aviatrix.com for assistance Troubleshooting 237

244 238 Chapter 44. Migrating VMs with Aviatrix IPMotion and AWS Migration Hub Service

245 CHAPTER 45 IPmotion Design Patterns This guide describes different design patterns, it assumes you have read IPmotion Setup Instructions. IPmotion connects an on-prem subnet and a cloud subnet with identical CIDR block, it is flexible to deploy. The cloud subnet can be a private subnet or a public subnet in AWS. The connections can be over Internet or private links such as Direct Connect. There maybe multiple cloud subnets in one VPC with each on cloud subnet connects one on-prem subnet IPmotion over Internet IPmotion over Internet is convenient as it requires no additional private link infrastructure such as Direct Connect, all you need is Internet access. This deployment model suits well for applications whose bandwith throughput requirement is less than 1Gbps. When IPmotion is deployed over Internet, packets are encrypted in flight with IPSEC. If cloud subnet is a public subnet, the IPmotion gateway subnet can be the same as migrate subnet. If cloud subnet is a private subnet, the IPmotion gateway subnet must be on a public subnet whose CIDR block does not overlap with any on-prem datacenter range, as displayed in the diagram below. 239

246 IPmotion over Private Links IPmotion over private link such as Direct Connect provides consistent bandwidth and latency as well as requlartory compliance for certain industry. When IPmotion is deployed over Direct Connect, data is not encrypted in flight. The IPmotion gateway subnet must be an non-overlapping subnet with on-prem datacenter. This subnet must be routable to on-prem via AWS VGW. The route propagation should be enabled. Below is a deployment diagram. 240 Chapter 45. IPmotion Design Patterns

247 Migrating multiple subnets to one VPC Multiple on-prem subnets can be migrated into one VPC with identical cloud subnets. In this case, you need to identify one subnet that is not overlapping with any on-prem datacenter CIDRs and use that as IPmotion gateway subnet. A deployment diagram is shown below IPmotion HA The Aviatrix virtual appliance CloudN should be deployed in a vmware HA cluster for on-prem HA protection. The IPmotion gateway is monitored by CloudN for gateway health. If the gateway becomes unreachable, CloudN will stop the gateway instance and start it again. The default gateway failure detection and failover time is 3 minutes. You can change this setting by going to CloudN console, Settings -> Advanced -> KeepAlive and change to different setting Simultaneously migrate multiple VMs on multiple subnets You can simultaneously migrate multiple VMs on multiple subnets. Deploy multiple Aviatrix virtual appliance CloudN, build connections. You can move on-prem IP addresses to Staging state before AWS AMI is ready, as long as you do not power down the on-prem corresponding VM (meaning the on-prem VM is still in operational state.) Migrating multiple subnets to one VPC 241

248 242 Chapter 45. IPmotion Design Patterns

249 CHAPTER 46 IPmotion Dependency Discovery Dependency discovery can be setup on Splunk to see all the dependencies after migrating your applications to cloud Setup instructions 1. Enable packet logging on all gateways for which you want to see dependency discovery. See instructions to enable packet logging. 2. Install and setup Aviatrix Splunk app using instructions mentioned here. 3. Install an another app named Sankey Diagram - Custom Visualization, which will be used to visualise Dependency discovery. To install this app, login to Splunk server. Go to Apps-> Find More Apps. Search for Sankey Diagram in search bar, and install the app named Sankey Diagram - Custom Visualization 4. Now Go to Aviatrix Splunk, and click on Dependency discovery dashboard to see dependencies across apps. 5. This dashboard lets you see network flow to/from servers across the network. It allows you to also filter on gateway, Source, Destination and Port. 243

250 244 Chapter 46. IPmotion Dependency Discovery

251 CHAPTER 47 Service Chaining Service Chaining is the capability to combine multiple services in tandem in a VPC. For example, if EC2-initiated traffic needs to be inspected before being sent to another VPC, you can do so by deploying a third-party firewall function along with Aviatrix peering gateway, as shown in the diagram below. In the diagram, the firewall (FW) has trusted interface and untrusted interface (WAN). User EC2 instances are on private subnets. These private subnets have default gateway points to the firewall s trusted interface (so that all egress traffic are sent through the firewall). To accomplish this, you need to go to AWS console, under VPC route tables create an route entry /0 to point firewall s untrusted interface. Note this untrusted interface should be on the same subnet as Aviatrix peering gateway Configuration Workflow Before you start, make sure you have the latest software by checking the Dashboard. If an alert message (!New) appears, click!new to download the latest software. We assume you already know how to deploy Aviatrix solution, if you need help, check out this reference design. Firewall function configuration is outside the scope of the reference design. 245

252 The Service Chaining configuration workflow is as follows, with major steps highlighted. 1. Create a gateway in VPC-1 Go to Gateway -> New Gateway to create a gateway in VPC-1. Note the gateway must be launched on the same subnet as the firewall untrusted interface. 2. Enable Service Chaining in VPC-1 Go to Advanced Config -> Service Chaining -> Add New. For Route Table ID field, select the route table that associates the subnet where firewall untrusted interface and gateway are deployed on. For Downstream IP field, type the private IP address of the firewall untrusted interface. 1. Repeat step 1 for VPC Repeat step 2 if VPC-2 also needs firewall function. 3. Create VPC Peering Go to Peering -> Encrypted Peering +New Peering, enter the two gateways to create the peering. 1. All EC2 initiated traffic from VPC-1 that is destined to VPC-2 will go through firewall function for inspection before they are sent to VPC Note: You can create more peering connections from VPC-1, all traffic will be inspected. 3. For support, send to support@aviatrix.com. 4. Enjoy! 246 Chapter 47. Service Chaining

253 CHAPTER 48 Environment Stamping 48.1 Objectives This reference design helps you build a repeatable deployment solution that scales indefinitely, as shown in the diagram below: 247

254 where the Aviatrix controller instance can be in the same or a different VPC. Each customer or managed VPC shares an identical VPC CIDR, security policies and instances. A user who connects to the management VPC should be uniquely address an instance in any given VPC by a private IP address or with a preferred name Configuration Workflow Before you start make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. The configuration workflow is as follows. It highlights the major steps. 1. Create a gateway in management VPC The mgmt-vpc is our management VPC with CIDR /16 in us-west-2. Click Gateway, then Create, make sure: (a) The VPC ID field is the AWS VPC ID where you launch the management gateway. (b) The Gateway Name in this example is mgmt-gw. (c) Enable NAT is not selected. (d) VPN Access is not selected. 2. Create an Access Address Pool 248 Chapter 48. Environment Stamping

255 This Access Address Pool is the address range from which instance addresses are mapped to. Accessing instance is done by accessing one mapped address from the pool. Go to VPC/VNet -> Environment Stamping -> Map Instance Addresses -> Address Pool. Make sure: (a) The Access Address Pool is big enough. In this example, we use /16 which gives you 16K unique IP addresses. (b) Select mgmt-vpc as the gateway choice. (c) Click Set. 3. (Optional) Setup Instance Names envstamping integrates Route 53 private hosted zone feature to enables you to access instances with DNS names and preferred (alias) names. Skip this step if you do not wish to use names to access instances. Go to VPC/VNet -> Environment Stamping -> Setup Instance Names -> Config. Enter a private domain name. For example, mydevops.com. Click Enable. 4. Create VPN gateways Create one or more VPN gateways in the management VPC for users to connect to AWS and access instances. In this example, we configure a split tunnel mode solution where only cloud bound traffic goes to the VPN tunnel. Among all fields you need to enter, make sure: a. Enable NAT is selected. b. VPN Access is selected. i. VPN CIDR Block must be an address range that is outside of management VPC and all other VPCs you intend to create. In this example, enter /24. ii. Split Tunnel Mode is Yes. 1. Additional CIDRs: enter the Access Address Pool CIDR. In this example, enter /16 2. (optional) Nameservers: enter the private DNS server of the management VPC if Setup Instance Names is enabled. In this example, enter (optional) Search Domains: The private hosted zone domain name if Setup Instance Names is enabled. In this example, enter mydevops.com c. Enable AWS ELB is Yes. d. Save Template: check to save the template. e. Repeat the above steps to create more VPN gateways to achieve scalability and resilience. 5. Create a managed VPC pool and its gateways This step creates a number of managed VPCs and gateways. If you already have existing VPCs, you should use Gateway tab to just create gateways. Make sure VPN access is disabled. (a) Go to VPC/VNet -> Environment Stamping -> Manage VPC Pool -> Create (b) Pool Name: a name for this VPC pool. Every VPC created in this pool will have a numeric number append to it. In this example, enter customer. (c) Number of VPCs: the number of VPCs. In this example, enter 3. (d) Check Launch Gateway i. Enable NAT: check this box if you like the gateway to also perform NAT function Configuration Workflow 249

256 6. Launch customer instances Once VPC and gateways are created, you can launch instances from AWS console or your own CloudFormation scripts. The pool of managed VPC may already have some instances. 7. Map instance addresses This step scans and maps instance private addresses in managed VPC to addresses from Access Address Pool, so that you can access these instances via Access Address Pool addresses. (a) Go to VPC/VNet -> Environment Stamping -> Map Instance Addresses -> Auto Mapping (b) Management VPC: select the gateway for management VPC. In this example, select mgmt-gw (c) Managed VPC: select one gateway from managed VPC. In this example, select customer001. Click Scan & Map. (d) Repeat the above step for all the remaining gateways in managed VPC. (e) Go to VPC/VNet -> Environment Stamping -> Map Instance Addresses -> List to view your instances and their mapped addresses. 8. Add users Add VPN users to the cloud network. Go to VPC/VNet -> VPN Access -> Users. Use Profile to control which user can access what cloud instance/application/ports. 9. Access Instances with Names When a user connects to management VPC, she can access instances in all managed VPCs. The instances can be accessed by its mapped Access Address, DNS name or nickname. When using DNS names and nicknames, make sure you include the domain name. For example, an instance with nickname webfrontend should be accessed as webfrontend.mydevops.com 10. For support, send to support@aviatrix.com. 11. For feature request, click Make a wish at the bottom of each page. 12. Enjoy! 250 Chapter 48. Environment Stamping

257 CHAPTER 49 Docker Swarm Cluster Installation 49.1 Introduction This document describes how to spin up a multi-host Docker swarm cluster built on a VXLAN overlay network in AWS, where a host is a AWS instance and multiple hosts may reside in the same VPC or different VPCs. If there is underlying network connectivity to connect VPCs securely (such as using Aviatrix encrypted peering), a swarm cluster can span across multiple VPCs and to Azure and Google. For a reference design on how to use Aviatrix OpenVPN capability to remotely access containers in a swarm cluster in the same manner as accessing instances from your laptop, for example, being able to use curl to a container running a web service, check out this link. To build a Swarm cluster, there needs to be a manager instance, a consul instance, and a few hosting instances. To simplify the topology, manager and consul are combined into one instance. There are many How to resources online on creating a swarm cluster with a VXLAN overlay network, the guide below is intended to be a reference Installation Steps Create one manager/consul instance and few container hosting instances. At AWS console, launch instances by using Amazon Linux AMI with Docker package, as shown below: 251

258 Install docker daemon For each of the above instances, do the following: a. ssh into each instance b. sudo yum update c. curl -ssl sh d. sudo service docker start e. sudo service docker stop f. sudo docker daemon -H tcp:// :2375 -H unix:///var/run/docker.sock& g. sudo usermod -ag docker ec2-user h. logout Start manager/consul On the manager/consul instance: a. ssh into the manager/consul instance b. ifconfig eth0 (to get the consul s eth0 IP address, for example, ) c. sudo docker run -d -p 8500:8500 name=consul progrium/consul -server -bootstrap d. docker run -d -p 4000:4000 swarm manage -H :4000 replication advertise :4000 consul:// : Setup Docker configuration file On each container hosting instance: a. ssh into each instance b. Follow the same procedures as described in Step 2 to install docker daemon c. sudo vi /etc/sysconfig/docker and add the following line: OPTIONS= default-ulimit nofile=1024:4096 -H tcp:// :2375 -H unix:///var/run/docker.sock cluster-advertise eth0:2375 cluster-store consul:// :8500 d. sudo service docker restart Join swarm cluster On each container hosting instance: 252 Chapter 49. Docker Swarm Cluster Installation

259 a. ssh into each instance b. Use ifconfig eth0 to get container hosting instance s own IP address, e.g Tell consul my IP address and to join the swarm cluster: sudo docker run -d swarm join advertise= :2375 consul:// : Create a VXLAN overlay network On the manager/counsul instance: a. ssh into each instance b. Create an overlay network my-overlay-network with network CIDR /24 : docker -H :4000 network create - -subnet= /24 my-overlay-network c. To list the network and you will see the newly created my-overlay-netork on each hosting instance joined the swarm cluster docker network ls Launch Ubuntu container On each container hosting instance: a. ssh into the host b. Launch a ubnutu container test01 within the overlay network my-overlay-network: sudo docker run -itd net my-overlay-network name test01 ubuntu /bin/bash c. Find out the overlay IP address for container test01 assigned by consul. There are at least three ways: 1. type command on the instance: sudo docker inspect test01 The above command returns a json output, look for IPAddress under my-overlay-network. 2. Type command on the instance: docker network inspect my-overlay-network, where my-overlay-network is the overlay network name. 3. Alternatively, use the following command to find out overlay IP address: docker inspect -f {{.Name}} - {{range.networksettings.networks}}{{.ipaddress}}{{end}} test01 where test01 is the container name Install Optional Tools (just for fun) 1. Access into the container and install some network tools if you like: sudo docker exec -ti test01 /bin/bash apt-get update yes apt-get install net-tools yes apt-get install iputils-ping yes 2. If you like to have ssh access to your container, follow these steps: apt-get install openssh-server apt-get install vim Installation Steps 253

260 sudo vi /etc/ssh/sshd_config and modify the following 2 lines to: PermitRootLogin yes #StrictModes yes Setup root password by typing command passwd Sudo service ssh restart ifconfig eth0 to get the IP address 3. In the container, do ping x to other containers you created To add more container hosting instances, repeat steps 1, 2, 4, 5 and 7. Note: You may need to modify Security Group of each instance and manager to allow the access to their ports. OpenVPN is a registered trademark of OpenVPN Inc. 254 Chapter 49. Docker Swarm Cluster Installation

261 CHAPTER 50 Docker Container Access 50.1 Introduction Project Skyhook by Aviatrix enables VPN users to access remote Docker containers in a multi-host Swarm cluster built on a VXLAN overlay network in the same manner to access remote cloud instance. (A host is a cloud instance.) With Aviatrix encrypted peering capability that connects VPC/VNets securely across regions and clouds, a multi-host Docker swarm cluster can span across multiple VPC regions and multiple clouds, such as AWS, Azure and Google. Users can use Aviatrix enterprise OpenVPN capability to connect to the cloud, then from your desktop access remote containers in a swarm cluster in the same manner as accessing instances. VPN users from desktop, for example, can use curl or run a browser session directly to a remote container running a web service. Without Aviatrix solution, it requires complex port mapping to access a remote Docker container. It is not possible today, from your desktop, to access a remote container in a VXLAN overlay network. In addition, administrators can leverage already built in multi-factor authentication and user profile defined access control to grant or deny access to a container or container application port. In this reference design, we are going to show you how to enable and use this capability. This document assumes some familiarity with Aviatrix Cloud Native Networking product, Docker Swarm cluster and VXLAN multi-host networking. If not, no worries, read on and proceed, we have compiled instructions for you Skyhook: Docker Container Access Aviatrix Docker Container Access solution can be deployed as shown below: 255

262 In the diagram above, the left most VPC ( /16), the VPN landing VPC, is the one hosting the Swarm primary/secondary managers, consul (Discovery backend) and a few Swarm nodes. Read this link on how to create a Swarm Cluster that you ll need later. Instances in the rest of the VPCs are part of the swarm cluster nodes that span across multiple VPC regions and to Azure and Google by using Aviatrix encrypted peering capability The Aviatrix Solution Benefits Aviatrix gateways are deployed and managed by an Aviatrix Cloud Connect Controller (the pink color instance in the diagram) which itself is a cloud instance or VM. Aviatrix benefits are highlighted below: Aviatrix solution enables users to remotely access swarm containers as well as instances. Once VPN in, users can use native desktop commands such as curl without complex port mapping and docker exec... type of commands. Multi-factor authentication and user profile based access control enable fine granular security. Aviatrix VPN gateways are supported by ELB for high availability and scalability. Extensive logging allows administrators to have complete visibility of network event and user browsing history. With Aviatrix encrypted peering, we can easily span Swarm cluster across different regions, and cloud providers (AWS, Azure, and Google GCE). The gateway is launched from a central controller web console with a few clicks Configuration Workflow Before you start, make sure you have the latest software by checking the Dashboard. If an alert message displays, click Upgrade to download the latest software. 256 Chapter 50. Docker Container Access

263 As a prerequisite, you must create a Swarm overlay network cluster first. You need to record the Docker Swarm Cluster Consul IP address, the Overlay Network Name (e.g. multi-host-overlay), and the Overlay Network Address (e.g /16). Please refer for instructions on how to create a Swarm Cluster. The configuration workflow is as follows, with major steps highlighted. 1. Setup secure VPC access and connectivity infrastructure This step setup a secure environment so that all your instances and containers can be accessed and communicated securely with private IP addresses. If you are going to start with all containers in one VPC ( /16 as shown in the diagram), launching an Aviatrix VPN gateway and create a VPN user for secure remote access to the instances in the VPC. On the other hand, if you like to try to run containers span across multiple VPCs, launch encrypted peering gateways and Aviatrix VPN gateways to create the necessary network infrastructure for secure access and secure connectivity among instances. Note you must launch separate peering gateway and VPN gateways. In either case, check out this reference design for instructions. 2. Create a Docker swarm cluster Follow the instructions to create a Docker swarm cluster and create some containers. First VPN to the landing VPC, then ssh into each swarm node (instance) with its private IP address. With Aviatrix VPN access capability and encrypted peering, your entire swarm cluster can be deployed on private subnets with private IP addresses. 3. Enable overlay network access if you have selected Split Tunnel mode when creating VPN gateways at step 1, you need to add the VXLAN overlay network /16 to allow your laptop to tunnel the address range to the VPC, Follow the steps below: Go to VPC/VNet -> Edit Configuration, click Modify Split Tunnel. At VPC/VNet Name field, select the landing VPC (the one with CIDR /16) At Additional CIDRs, add /16 to the CIDR strings separated with comma. (If you have Nameservers and Search Domains, fill in these fields so you can access containers with names.) Click Modify. 4. Enable Docker Container Access. Go to VPC/VNet -> VPN Access -> Skyhook:Docker Container Access Click on Enable for the gateway you just created (e.g. avx-vpngw01). Fill in the Docker Swarm Cluster Consul IP address, the Overlay Network Name (e.g. multi-host-overlay), and the Overlay Network Address (e.g /16). Click Enable to confirm the request. Important notes If there are more than one VPN gateways, make sure you enable Docker Container Access for each one and the same configuration should be applied to all VPN gateways. 5. Verify your setup Now you should be able to access your containers. Use your desktop VPN client to VPN into the VPC. You can try a few things. Note you need to use the container overlay IP address for accessing, in this reference design, all containers overlay IP address is in the /16 range Configuration Workflow 257

264 If you have one container that runs a web server, you should be able to access the web server from your desktop browser, run a command wget from a Linux machine, or run a command curl from a OSX machine. If one container has been loaded with ssh access capability, you can ssh directly into the container from your desktop. Ping the container overlay IP address. 6. Adding a new swarm node You can still add a new swarm node later, just follow the same instruction as described in this link. Important note: for a container on a Google GCE instance, you must enable IP forwarding when you launch the Google GCE instance Troubleshooting 1. If you failed to enable Docker Container Access for a gateway, make sure the Docker Swarm Consul IP address is reachable from your gateway. Check the security group associated with the instances. 2. If there are more than one VPN gateway, make sure you enable Docker Container Access for each one and the same configuration should be applied to all VPN gateways; otherwise, you may experience inconsistent behaviors. 3. After you disabled the Docker Container Access for a VPN gateway, if you try to enable it immediately, it may fail. This is because the Swarm Consul still has the node entry in the DB and it needs time to discover that the node is gone. Simply wait for a few minutes until the TTL expired and the key-value store cleans up the old entry automatically. For support, send to support@aviatrix.com. For feature request and feedback, click Make a wish at the bottom of each page. Enjoy! OpenVPN is a registered trademark of OpenVPN Inc. 258 Chapter 50. Docker Container Access

265 CHAPTER 51 Migration from AWS Marketplace Licensing Model to BYOL Licensing Model 51.1 Introduction Many customers start by trying our AWS Marketplace image that allows you to deploy 10 VPN Users or 5 Peering Tunnels. Those images are not flexible and cannot be extended beyond it s initial license. In order to exceed this limitations, the customer needs to move to a BYOL License model. This document outlines all the steps necessary to execute the migration Pre-requisites 1. Existing Aviatrix AWS Marketplace instance deployed 2. Contact your Aviatrix Sales Account Manager to acquire the appropriate BYOL License 3. All Aviatrix controller should be running v2.7 (or later) 51.3 Step 1 - Enable Backup On the Aviatrix Markeplace Controller, goto Settings > Maintenance and select the Backup & Restore tab. Create an S3 bucket and copy the name on the corresponding field. Click Enable. 259

266 Note: If you already have Backup enabled, please disable and re-enable to make sure the backup is executed. Double check on your S3 bucket that the file has been updated based on the timestamp Step 2 - Stop the Marketplace instance On the AWS console proceed to STOP the Aviatrix AWS Marketplace controller instance Step 3 - Disassociate EIP On the AWS console, goto EC2 >Network & Security > Elastic IPs, and disassociate the EIP from the Aviatrix AWS Marketplace controller instance. Note: Make sure browser cache is cleared before the next Step, to avoid connecting to an old stale session Step 4 - Launch BYOL Controller Launch new Aviatrix Controller using the BYOL License. 260 Chapter 51. Migration from AWS Marketplace Licensing Model to BYOL Licensing Model

267 51.7 Step 5 - Attach EIP On the AWS console, goto EC2 >Network & Security > Elastic IPs, associate the same EIP from step 3 to the new Aviatrix BYOL Controller Step 6 - Upgrade Controller Make sure your Aviatrix Controller is upgraded to version (or later) Step 5 - Attach EIP 261

268 51.9 Step 7 - Restore On the Aviatrix Controller, goto Settings > Maintenance and select the Backup & Restore tab. Insert the right S3 bucket name and execute a restore Step 8 - Install License On the on-boarding page, enter the customer id provided by your Aviatrix Sales Account Manager. 262 Chapter 51. Migration from AWS Marketplace Licensing Model to BYOL Licensing Model

269 51.11 Step 9 - Enjoy your new controller Optional: After confirming everything is running correctly, delete the previous Aviatrix AWS Marketplace controller instance. For support, send to support@aviatrix.com. For feature request and feedback, click Make a wish at the bottom of each page Step 9 - Enjoy your new controller 263

270 264 Chapter 51. Migration from AWS Marketplace Licensing Model to BYOL Licensing Model

271 CHAPTER 52 Controller Backup and Restore When deployed in a cloud environment, the Aviatrix controller, is not in the data path as packet processing and encryption is done by the Aviatrix gateways. When the controller is down or out of service, your network will continue to be operational and encrypted tunnels and OpenVPN users stay connected and are not affected. Since most of the data logs are forwarded from the gateways directly, the loss of log information from the controller is minimal. The only impact is that you cannot build new tunnels or add new OpenVPN users. This loosely coupled relationship between the controller and gateways reduces the impact of the availability of the controller and simplifies your infrastructure. Since the controller stores configuration data, it should be periodically backed up to the appropriate AWS/Azure/Google account. If a replacement controller is launched, you can restore the configuration data from your backup How to backup configuration 1. Settings -> Maintenance -> Backup & Restore -> Under BACKUP section 2. Select Cloud Type 3. Select Account Name (example: If AWS were selected as the cloud type, specify an S3 bucket name) The first time you enable this feature, the configuration will backed up to your specified location. After this, the configuration data is automatically backed up daily at 12am. If you want to force an immediate backup (e.g. for a configuration change) you can accomplish this by disabling the backup and then re-enabling it again. Note: Only the latest configuration data is stored (no versioning at this time). Each time the configuration is backed up, it overwrites the previous one. 265

272 52.2 How to restore configuration 1. Settings -> Maintenance -> Backup & Restore -> Under RESTORE section 2. Select the Cloud Type and other credentials to retrieve the backup configuration data. For example, for AWS, you need to specify the Access Key, Secret Key, Bucket Name and File Name. 3. Click Restore The new controller will then inherit the configuration data. OpenVPN is a registered trademark of OpenVPN Inc. 266 Chapter 52. Controller Backup and Restore

273 CHAPTER 53 Controller HA in AWS Aviatrix Controller HA in AWS leverages auto scaling group and Lambda function to perform monitoring, launching a new controller and restoring configuration when the active controller instance become unreachable. When a new controller is launched, the existing controller is terminated by auto scaling group and its EIP is associated to the controller. Existing configuration files are restored which gives you a seamless experience when failover happens. 1. Existing Controller with HA If you already have an existing controller running and would like to enable HA, follow this instruction. 2. New Controller with HA If you are launching a brand new controller and would like to have HA enabled, follow instructions here 267

274 268 Chapter 53. Controller HA in AWS

275 CHAPTER 54 Inline Software Upgrade Aviatrix software is released frequently every 6-8 weeks. When a new release becomes available, an alert will be sent to the controller admin. When upgrading a controller software, all gateways are upgraded with the new software at the same time. This is done by controller pushing new software to gateways directly and automatically How to upgrade software At the controller dashboard, a!new alert icon appears when a new software becomes available, click the!new icon to upgrade. Select UPGRADE TO THE LATEST to upgrade. Alternatively, go to Settings -> Maintenance -> Upgrade -> UPGRADE TO THE LATEST 54.2 Inline and hitless software upgrade Aviatrix software upgrade happens inline without taking down the controller. In addition, gateway upgrade is hitless, that is, all gateway encrypted tunnels stay up without going down. There is no packet loss when upgrading the software Upgrade impact on OpenVPN users Most upgrades do not impact connected OpenVPN users. In some cases, OpenVPN service needs to be restarted as part of software upgrade, for example, upgrade to a new SSL version for security patch. in these cases, connected OpenVPN users will experience an disconnect. They will need to connect again. When a release affects OpenVPN users, the Release Note will make a note of it. Make sure you read Release Notes before making upgrade. OpenVPN is a registered trademark of OpenVPN Inc. 269

276 270 Chapter 54. Inline Software Upgrade

277 CHAPTER 55 Logging Introduction Aviatrix Controller and all its managed gateways can be configured to forward logs to well known log management systems, such as Splunk, Sumo Logic, Elastic Search and remote syslog. In addition to standard information on syslog, Aviatrix also provides capability for user VPN connections, VPN user TCP sessions, security rule violation statistics, Gateway stats and FQDN filter violations. Log Management System can be used to sift through the Aviatrix logs and get the meaningful trend charts that helps monitor the network connectivity and user VPN sessions. The following sections provides a list of useful Aviatrix logs which can be parsed on Splunk, Sumo Logic and other log management systems to display relevant analytics of data collected from Aviatrix Controller and gateways Aviatrix Log Format for Log Management Systems Following types of Aviatrix log keywords can be identified by the Log Management System for further analysis: AviatrixVPNSession: This log is for gateways that have VPN enabled. To enable VPN, check VPN Access when launching a gateway. Logs sent by the Controller contains the information such as VPN user name, the VPN gateway IP address and name where the user connects to, client virtual IP address, connection duration, total received bytes, total transmitted bytes, and login and logout time. Two logs will be generated for each VPN connection. One is when the connection is established, the other when it s disconnected Example logs: Connect Log: 271

278 Aug 17 22:07:39 ip cloudx\_cli: AviatrixVPNSession: User=Splumo, Status=active, Gateway=splunksumo, GatewayIP= , VPNVirtualIP= , PublicIP=N/A, Login= :07:38, Logout=N/A, Duration=N/A, RXbytes=N/A, TXbytes=N/A Disconnect log: Aug 17 22:26:37 ip cloudx\_cli: AviatrixVPNSession: User=Splumo, Status=disconnected, Gateway=splunksumo, GatewayIP= , VPNVirtualIP= , PublicIP=N/A, Login= :07:38, Logout= :26:37, Duration=0:0:18:59, RXbytes=2.1 MB, TXbytes=9.03 MB AviatrixUser: This log is for gateways that have VPN enabled. To enable VPN, check VPN Access when launching a gateway. Logs with this prefix come from each VPN gateway managed by the controller. The log contains the information for the TCP session, such as Inbound and outbound interface, source IP address, destination IP address, TTL value, protocol name, and packet length. The log record is for each packet that passes through the VPN connection from the client to the destination Two example logs: Aug 17 22:15:47 ip kernel: \[ \] ***AviatrixUser***:IN= OUT=eth0 SRC= DST= LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=28916 DF PROTO=TCP SPT=50428 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0 Aug 17 22:15:47 ip kernel: \[ \] ***AviatrixUser***:IN= OUT=eth0 SRC= DST= LEN=66 TOS=0x00 PREC=0x00 TTL=254 ID=13309 PROTO=UDP SPT=64775 DPT=53 LEN= AviatrixLicenseVPNUsers: This log is for gateways that have VPN enabled. To enable VPN, check VPN Access when launching a gateway. Logs with this prefix come from Controller and can be used to moniter the license usage of active vpn users connected to all vpn gateways One example log: Sep 25 23:40:19 ip cloudxd: AviatrixLicsenseVPNUsers: users=2 Note: There is a typo in some versions (as noted in above example) that incorrectly shows this entry as AviatrixLicsenseVPNUsers instead of AviatrixLicenseVPNUsers. 272 Chapter 55. Logging

279 AviatrixRule: You need to configure security policies to see AviatrixRule log. Logs with this prefix come from each gateway managed by the controller. Any packet that triggers the security policy rule will generate a log record of this type with the first 100 bytes of the packet. It contains the information such as gateway IP address, inbound and outbound interface, MAC address, TTL value, protocol name, source IP address, destination IP address and packet length Two example logs: syslog:aug 4 21:46:28 ip kernel: \[ \] ***AviatrixRule*** A:IN=eth0 OUT=tun0 MAC=0a:06:c3:e3:d0:4f:0a:d2:ef:22:68:61:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0x00 TTL=62 ID=798 DF PROTO=TCP SPT=443 DPT=56505 WINDOW=231 RES=0x00 ACK FIN URGP=0 syslog:aug 4 21:46:32 ip kernel: \[ \] ***AviatrixRule*** A:IN=eth0 OUT=tun0 MAC=0a:06:c3:e3:d0:4f:0a:d2:ef:22:68:61:08:00 SRC= DST= LEN=334 TOS=0x00 PREC=0x00 TTL=40 ID=5452 DF PROTO=TCP SPT=443 DPT=56203 WINDOW=368 RES=0x00 ACK PSH URGP= AviatrixGwNetStats: Logs with this prefix come from each gateway managed by the controller. These logs are sampled every 1 minutes, and give details about gateway network interface Two example logs: May 17 00:23:17 ip gwmon.py: AviatrixGwNetStats: timestamp= :23: name=wing-aws-aws-use-2-gw0000 public\_ip= private\_ip= interface=eth0 total\_rx\_rate=3.47kb total\_tx\_rate=2.85kb total\_rx\_tx\_rate=6.32kb May 17 00:28:17 ip gwmon.py: AviatrixGwNetStats: timestamp= :28: name=wing-aws-aws-use-2-gw0000 public\_ip= private\_ip= interface=eth0 total\_rx\_rate=2.40kb total\_tx\_rate=2.10kb total\_rx\_tx\_rate=4.49kb AviatrixGwSysStats: Logs with this prefix come from each gateway managed by the controller. These logs are sampled every 1 minutes, and give details about gateway memory, cpu and disk load Two example logs: May 17 00:23:20 ip gwmon.py: AviatrixGwSysStats: timestamp= :23: name=wing-aws-aws-use-2-gw0000 cpu\_idle=100 memory\_free= disk\_total= disk\_free= Aviatrix Log Format for Log Management Systems 273

280 May 17 00:28:20 ip gwmon.py: AviatrixGwSysStats: timestamp= :28: name=wing-aws-aws-use-2-gw0000 cpu\_idle=100 memory\_free= disk\_total= disk\_free= AviatrixFQDNRule You need to configure FQDN Whitelists in order to see these logs. Logs with this prefix come from each gateway managed by the controller. Domain name filtering can be configured per gateway via controller. And every time a gateway tries to access a domain name, it will check if the domain name passes the configured filters. If it does, access will be allowed with state as MATCHED, otherwise it will be discarded with state as NO_MATCH Two example logs: May 24 10:54:40 ubuntu64-dev avx-nfq: AviatrixFQDNRule\[CRIT\]nfq\_ssl\_handle\_client\_hello() L\#137 P:7025 Gateway=bogusGw hostname= state=matched May 24 10:17:08 ubuntu64-dev avx-nfq: AviatrixFQDNRule\[CRIT\]nfq\_ssl\_handle\_client\_hello() L\#162 P:6138 Gateway=bogusGw hostname=clients2.google.com state=no\_match drop\_reason=no\_hostname\_match AviatrixTunnelStatusChange Logs with this prefix come from controller whenever a tunnel status changes. old_state means old state of the tunnel, and new_state is the new changed state of tunnel Example log: Jul 21 04:28:19 Controller cloudx_cli: AviatrixTunnelStatusChange: src_gw=oregon-devops-vpc(aws us-west-2) dst_gw=gcloud-prod-vpc(gcloud us-central1) old_state=down new_state=up latency= AviatrixCMD Logs with this prefix come from controller whenever a CLI command is issued. It contains information on the CLI command that was issued, the results of the execution, and reason message if there is a failure Example log: Nov 10 01:05:41 ip cloudxd: AviatrixCMD: action=add_time_server, argv=['--rtn_file', '/run/shm/rtn ', 'add_time_server', 'time2.google.com'], result=success, reason= 274 Chapter 55. Logging

281 Logging Configuration at Aviatrix Controller To enable logging at Aviatrix Controller, go to Settings->Logging page. Once logging is enabled, both Controller and all gateways will forward logs directly to the logging server. Two examples for Remote Syslog and Logstash Forwarder below Remote Syslog On the Aviatrix Controller: 1. Server: FQDN or IP address of remote syslog server 2. Port: Listening port of remote syslog server (6514 by default) 3. Cert: A compressed file in tgz format with both certificates (.crt format) of remote syslog server and CA 4. Protocol: TCP or UDP (TCP by default) On the Remote syslog server: 1. SSH into the remote syslog server 2. Go to /var/log/aviatrix directory 3. Find the directory of desired controller or gateway (a) Controller s directory name is in a format of Controller-public_IP_of_controller (b) Gateway s directory name is in a format of GW-gateway_name-public_IP_of_gateway 4. Each controller/gateway directory should have (a) auth.log (b) commmandlog.log (c) syslog Logstash Forwarder On the Aviatrix Controller: 1. Server Type: Remote or Local 2. Server: FQDN or IP address of logstash server 3. Port: Listening port of logstash server (5000 by default) 4. Trusted CA: CA certificate (.crt format) Note: If Local is selected for Server Type, Aviatrix Controller itself will be enabled as a logstash server. Before you do this, make sure your controller has at least 30GB of hard disk space. On the Logstash console: Log into the web page of your logstash server to access the logs. The Kibana interface is divided into four main sections: 1. Discover By default, this page will display all of your most recently received logs. You can filter through and find specific log messages based on Search Queries, then narrow the search results to a specific time range with the Time Filter. 2. Visualize The Visualize page is where you can create, modify, and view your own custom visualizations Logging Configuration at Aviatrix Controller 275

282 3. Dashboard The Dashboard page is where you can create, modify, and view your own custom dashboards. With a dashboard, you can combine multiple visualizations onto a single page, then filter them by providing a search query or by selecting filters by clicking elements in the visualization. 4. Settings The Settings page lets you change a variety of things like default values or index patterns Log management system Apps Aviatrix controller can be configured to forward logs to various log management systems. Aviatrix also provides apps with prebuilt dashboards for popular log management systems like Splunk and Sumo Logic Splunk App for Aviatrix Splunk app for Aviatrix can be downloaded from Splunkbase. Click here to check instructions on Github. Sample 276 Chapter 55. Logging

283 Sumo Logic App for Aviatrix Sumo Logic app installation guide is also available on Github. Sample Log management system Apps 277

284 278 Chapter 55. Logging

285 CHAPTER 56 Logs Upload tracelog. 279

286 280 Chapter 56. Logs

287 CHAPTER 57 Diagnostics Network Please refer to Debug gateway connectivity. Please refer to Packet capture. Gateway Please refer to Run diagnostics on a gateway. 281

288 282 Chapter 57. Diagnostics

289 CHAPTER 58 Error Messages 1. **Error message:** 'Legal terms have not been accepted for this item on this subscription. To accept legal terms, please go to the Azure portal... and configure programmatic deployment for the Marketplace item or create it there for the first time' If you see this error message when you launch an Azure ARM gateway, chances are you have not subscribed to Aviatrix gateway during Azure onboarding process. Either go back to onboarding page and follow the instructions there, or click this link for guidance. 2. **Error message:** `Error: Exception CloudxErrExt Context:message:EC2ResponseError: 401 Unauthorized AuthFailureAWS was not able to validate the provided access credentialsf67841bc-cb94-4cfd-a990-05d27d11f540` If you see this error message when launching an AWS gateway, chances are your access key or secret ID is not correct. Re-enter these two fields. If it still does not work, change the credential on AWS console and try again. 283

290 284 Chapter 58. Error Messages

291 CHAPTER 59 How to Troubleshoot Azure RM Gateway Launch Failure Before you launch an Aviatrix gateway in Azure RM, you must first subscribe to the Aviatrix Companion Gateway in Azure marketplace. To check if you have done so, following these steps. 1. Login in to Azure Portal. 2. Click More Services. 3. Click Subscriptions. 4. Click the subscription you wish to launch Aviatrix gateway. 5. Under Settings, click Programmatic deployment. 6. You should see aviatrix companion gateway is in Enable status. 7. If aviatrix companion gateway is in Disable state, click to enable. 8. If you do not see aviatrix companion gateway at all, follow this instruction When Aviatrix Controller fail to launch a gateway, there is a toaster error message on the Controller console. If this message does not help you understand the root cause, take the following steps to further troubleshoot Disable Rollback Function Typically when a gateway launch fails, the Controller rolls back all resources including the ones allocated from Azure. In this case, you want to disable the rollback function. Go to Troubleshoot -> Diagnostics -> KEEP GATEWAY ON ERROR. Enable it to make it True. Note this rollback only applies to the next gateway launch. Each time when you need to disable the rollback of gateway creation, you need to turn on this option Launch the Gateway and Observe Failure From Controller console, launch gateway again and observe the failure. 285

292 Check on Azure Portal Activity Log 1. Login in to Azure Portal. 2. Click More Services. 3. Click Resource Groups. 4. Click the resource group created by Aviatrix Controller. The resource group should have a prefix av-. Click Activity Log. 5. Click the error message in red color. 6. The specific error message should have a Summary tab and a JSON tab. 7. Click the JSON tab to examine the detailed error message, as shown below: Get Help from Aviatrix Support If you still cannot figure out, to support@aviatrix.com to get help. 286 Chapter 59. How to Troubleshoot Azure RM Gateway Launch Failure

293 CHAPTER 60 FlightPath FlightPath is a troubleshooting tool. It retrieves and displays, in a side by side fashion, AWS EC2 related information such as Security Groups, Route table and route table entries and network ACL. This helps you to identify connectivity problems What you need You do not need to launch Aviatrix gateways to use this tool, but you need to create Aviatrix accounts so that the Controller can use the account credential to execute AWS APIs to retrieve relevant information How to use it Click FlightPath icon on the onboarding page or Troubleshooting -> FlightPath. Select on Source side cloud type, account, region, vpc name and click one instance. Do the same for the Destination side. Run FightPath Test, a report will appear. The information is arranged in three sections, Security Groups, Route table entries and Network ACL. The Security Group is what is associated with the instance, both the route table and the Network ACL are associated with the subnet that the instance is deployed. Note: If you just need to check information on one instance, for example, check Internet reachability on an instance on a private subnet, select any instance in the Destination field, as it does not matter in this case Example Here is one example to show how FlightPath works. Say a developer from BusinessOps account filed a ticket that says one instance DevOps Server in Oregon region cannot run ssh into the Prod instance in California region. 287

294 From the Controller browser console, click FlighPath under Troubleshooting on the navigation menu. Specify the above info and you ll see something like the screenshot below. The highlights on each panel are the instances in question. Note the DevOps Server has IP address Now run FlightPath Test, you ll see the FlightPath Report. First check the routing table, it shows good connectivity: Scroll up and down the FlightPath Report to check other fields. Next check the Security Group. And of course, the California Prod instance does not have its ssh port open to the Oregon DevOps instance IP address Chapter 60. FlightPath

295 Problem Solved in minutes! Upon further inspection, you ll notice the complaining instance has a ssh open to the entire world. You may need to notify the ticket issuer to reduce the source address scope Example 289

296 290 Chapter 60. FlightPath

297 CHAPTER 61 Aviatrix FlightPath Deployment Guide The Aviatrix cloud network solution consists of two components, Controller and Gateway, both are AWS instances. Aviatrix Flight PathTool only requires the Aviatrix Controller to function. This guide helps you to launch the Aviatrix Controller instance in AWS Create an AWS EC2 Account You need to have an AWS EC2 account to use the solution. Note that the Controller supports multiple accounts with each one associated with a different AWS IAM role or account, but there needs to be at least one to start with. This AWS account can be a root account, IAM role, IAM administrator account or IAM user account with access privileges required by the Aviatrix solution. We strongly recommend you to use IAM role for security reasons Subscribe to Aviatrix FlightPath Tool on AWS Marketplace You must subscribe to the Aviatrix Flightpath AMI Free Tool - one of the Aviatrix AMIs on AWS marketplace prior to launch the Controller. Search Aviatrix Flight Path on AWS marketplace, click on Continue to Subscribe, Choose Manual Launch Tab, and accept the terms and conditions to use the software. After subscription, follow the instructions in the next sections to launch the Controller. 291

298 61.3 DNS Server Connectivity Check If the VPC where the Controller is deployed in has a custom DNS server (via DHCP option), make sure the Controller instance can reach this DNS server. Warning: Any resources created by the Controller, such as Aviatrix gateways, route entries, ELB, SQS queues, etc, must be deleted from the Controller console. If you delete them directly on AWS console, the Controller s view of resources will be incorrect which will lead to features not working properly Launch Aviatrix Controller Controller must be launched on a public subnet of a VPC. The recommended way to launch the Controller is by our CloudFormation script. Follow the instruction here to launch a controller instance in a selected region. To launch the controller manually, follow this document Access the Controller After the Controller instance is in running state in AWS, you can access the Controller via a browser by where Controller_public_EIP is the Elastic IP address of the Controller. The initial password is the private IP address of the instance. Follow the steps to go through an initial setup phase to download the latest software. After the latest software is downloaded, re-login again to go through the onboarding process Onboarding The purpose of onboarding is to help you setup an account on Aviatrix Controller that corresponds to an IAM role with policies so that the Controller can launch gateways and build networks using AWS APIs. If you launched the Controller via CloudFormation script, the required IAM roles and policies are already setup, follow this instruction to complete account creation. Note you can create a single Aviatrix account that corresponds to AWS, account credentials. 292 Chapter 61. Aviatrix FlightPath Deployment Guide

299 61.7 Setup for Operations If this Controller is for your production, we strongly recommend you to enable Controller Backup/Restore feature. This allows you to backup configurations on the Controller to an S3 bucket sothat you can recover the configurations in a disaster situation Controller Monitoring If Controller HA is not enabled, we recommend you to use AWS CloudWatch to configure alarms and actions to reboot the controller when it fails Status Check How to Use FlightPath for Connectivity Troubleshooting FlightPath is a troubleshooting tool. It retrieves and displays, in a side by side fashion, AWS EC2 related information such as Security Groups, Route table and route table entries and network ACL. This helps you to identify connectivity problems What you need You do not need to launch Aviatrix gateways to use this tool, but you need to create Aviatrix accounts so that the Controller can use the account credential to execute AWS APIs to retrieve relevant information. How to use it For support, send to support@aviatrix.com. Enjoy! Setup for Operations 293

300 294 Chapter 61. Aviatrix FlightPath Deployment Guide

301 CHAPTER 62 REST API Example 62.1 Introduction The APIs for Aviatrix can be used for the tasks that are done through the Web UI. The following is an example to utilize the APIs to create a VPC/VNet under Datacenter Extension. For the complete REST API documentation, check out this link. Datacenter Extension capability manages your cloud address range. It creates VPC/VNet, subnets, routing tables and create a IPSec tunnel to the virtual appliance (ACX), so that on-premise VMs and servers can communicate with instances in created VPC with packet encryption and private IP addresses Workflow for Datacenter Extension Make sure the latest version of Aviatrix software is installed or upgraded before you start. You should see the alert for software upgrade on the menu bar of the controller if a newer version is available. Click Upgrade and wait for the upgrade to complete. Here are the steps to successfully use the APIs to achieve the same result without the Web UI. 1. Log in to get the session ID 2. Enter the license (customer ID) 3. Set up the maximum number of VPC/VNet 4. Create a user account 5. Create a VPC/VNet for Datacenter Extension 295

302 62.3 Use the APIs to Create a VPC/VNet The APIs in this section are to demonstrate how to use them to accomplish the steps described above. The data used here is for the purpose of demonstration only. Replace the values in your case. For more information, refer to Cloud Services Gateway Controller API reference for details. You can retain a copy of this document under?help > API Reference on the menu bar after you log on the Web console. 1. Log in to get the session ID password=password Replace IP_Address_of_ACX with your own IP address of ACX. Replace the values of username and password with the credentials you use to log in the Web console. It should return a CID upon successful login. { "return": true, "results": "User login:admin in account:admin has been authorized successfully - Please check confirmation.", "CID": "584b4b57a42f2" } Note the value of CID for the API calls hereafter. 2. Enter the license Obtain a valid license (customer ID) from Aviatrix in advance then enter the value in the API CID=584b4b57a42f2&action=setup_customer_id&customer_id=carmelodev Replace the value of CID with the one in step 1. Replace the value of customer_id with your license. Make sure the license is successfully entered and it returns the license information correctly. { "return": true, "results": { "license_list": [ { "Lic ": { "Verified": 0, "Type": "c4.4xlarge", "Expiration": " ", "Allocated": 0, "IssueDate": " ", "Quantity": 20 } } ], "CustomerID": "carmelodev " } } 3. Set up the maximum number of VPC/VNet 296 Chapter 62. REST API Example

303 containers&vpc_num=4 Replace the value of CID with the one in step 1. Replace the value of vpc_num with the number you desire to set up. { } "return": true, "result": { "cidr_list": [ " \/19", " \/19", " \/19", " \/19" ] } 4. Create a User Account Before calling the API to set up an account that enables ACX to access the cloud, gather the account information from the cloud provider. AWS ( cloud_type = 1 ): Account Number, Access key and Secret Key Azure ( cloud_type = 2 ): Azure Subscription ID Azure RM ( cloud_type = 8 ): Azure Subscription ID, Application Endpoint, Application Client ID and Application Client Secret This API needs to use POST method of HTTP to send the account information. Use any tool of your preference to send the POST HTTP request POST Body { } "CID": "584b4b57a42f2", "action": "setup_account_profile", "account_name": "user2", "account_password": "12345", "account_ ": "user2@123abc.com", "cloud_type": "1", "aws_account_number": " ", "aws_access_key": "AKIAIQDAABCPKKKWQA", "aws_secret_key": "9ttSESnQvb\/OlWZKCjyPsbcdYgamthksK2+1G" The above example is to set up an AWS account (cloud_type is 1 ). The others are the account information from AWS. { "return": true, Use the APIs to Create a VPC/VNet 297

304 } "results": "An with instructions has been sent to 5. Create a VPC/VNet for Datacenter Extension Currently, two cloud types are available for Datacenter Extension. They are AWS and Azure ARM. Hence, it either to create a VPC or VNet. The CIDR of this VPC/VNet can only be one of the available CIDRs you set up in step 3. Enter the CIDR as the value of vpc_net in this API. POST Body { } "CID": "584b4b57a42f2", "action": "create_container", "cloud_type": "1", "account_name": "user2", "vpc_name": "dc-us-west-1", "vpc_reg": "us-west-2", "vpc_size": "t2.micro", "vpc_net": " \/19" The result is expected to return after a while. There are other options you can specify when you use this API to create a VPC/VNet. Refer to the reference document for more details about the options. 298 Chapter 62. REST API Example

305 CHAPTER 63 Aviatrix Controller API Click here for Aviatrix API documentation. 299

306 300 Chapter 63. Aviatrix Controller API

307 CHAPTER 64 CloudN CloudN comes with OVF, VHD and KVM images, to support VMware hypervisor, Microsoft Hyper-V and KVM. Latest vmware OVF image is CloudN-ovf , it can be downloaded from OVF image link. Latest KVM image is CloudN-kvm , it can be downloaded from KVM image link. Hyper-V VHD image is CloudN-vhd , it can be downloaded from VHD image link. 301

308 302 Chapter 64. CloudN

309 CHAPTER 65 Aviatrix VPN Client The Aviatrix VPN solution is the only VPN solution that provides SAML authentication from the client itself. The solution is built on OpenVPN. The Aviatrix VPN Client provides a seamless user experience when authenticating a VPN user through a SAML IDP. The client also supports password based authentication methods as well. The VPN Client can be installed on desktop platforms and is supported on various OS like Windows, Mac and Linux. Consult the VPN client user guide for how to use it. Latest version: (Dec ) Changelog Windows Windows client can be download from this link At the end of the installation, please install the TUN TAP driver if you haven t done so earlier. 303

310 65.2 Mac Mac client can be downloaded from this link If you have installed version or lower please uninstall before you install the newer version 65.3 Linux For the.deb files, if opening them using software center does not work, use sudo dpkg -i file.deb; sudo apt-get install -f (Dependencies)to install For the.tar files use tar -xvzf file.tar.gz; cd AVPNC_setup; sudo./install.sh to install If the icon is missing from the launcher, type AVPNC in the terminal to launch the app Debian/Ubuntu Ubuntu 16/Generic - Debian file, Tar file. Trusty/Ubuntu14 - Debian file, Tar file, Zesty/Ubuntu17 - Debian file, Tar file 65.4 FreeBSD FreeBSD client can be downloaded from- this link tar -xvzf file.tar.gz; cd AVPNC_setup; sudo./install.sh to install 65.5 Development version These are preview images for the next release. Windows, Mac, Linux tar, Debian file, Linux tar trusty, Debian trusty, Linux tar zesty, Debian zesty, FreeBSD OpenVPN is a registered trademark of OpenVPN Inc. 304 Chapter 65. Aviatrix VPN Client

311 CHAPTER 66 Release Notes 66.1 R3.0 (12/1/2017) Connectivity BGP Support BGP interoperability between Aviatrix gateway and AWS VGW. For use case details, check out the Transit Network with BGP Setup Instructions.. IPmotion For AWS migration and DR use case that allows on-prem VMs to migrate to AWS without changing their IP addresses. For use case and details, check out this link.. AWS ENA on Aviatrix gateway Security Tag your security policy to associate a CIDR with a name tag for a scalable and user friendly. For configuration detail, check this link. AES-GCM crypto algorithm. For IPSEC tunnel connectivity between two Aviatrix gateways, such as Aviatrix peering and IPmotion, the crypto algorithm has been upgraded to AES-GCM Controller Audit user actions on the Controller. All commands from web console or REST API are now logged to syslog and can be forwarded to integrated log services. Name your controller for ease of use. Click Your controller name goes here on the Controller console and start typing a new name. Hit return to save the name. On demand backup of the Controller configuration to cloud storage. To configure, go to Settings -> Maintenance -> Backup & Restore -> Backup Now 305

312 Backup multiple copies of Controller configuration file. You can choose to backup multiple copies of configuration file. To do so, go to Settings -> Maintenance -> Backup & Restore and select Multiple Backup. Up to 3 backup files are stored. You can select any one of them to restore. Migrate licenses from AWS Marketplace Utility image to BYOL. For details, check out this link Modular Configuration Transitive Peering supports multiple subnets being configured at the same time. Multiple subnets separated by comma can be added once when configuring transitive peering. Join Function now support the ability to delete all subnets at once in Join Function gatewway Troubleshooting FlightPath tool, an AWS EC2 to EC2 connectivity troubleshooting tool. In the first release, EC2 related resources, such as security groups, route table and Network ACLs are displayed side by side for easy visualization and troubleshooting Datacenter Extension Features non-rfc1918 on premise network range is now supported. To add, first launch a Datacenter Extension gateway, go to Gateway List, select the gateway and click Edit. At Edit Extended Public CIDR, add one or multiple non- RFC1918 CIDR blocks separated by comma. For example, /24, /24 Repair gateway to replace a gateway in a limbo state. At the Datacenter Extension page, click Replace of specific gateway R Controller Console Responsiveness improvements. Significant improvements in page responsiveness when using controller web console. Support third party signed certificate. You now can import a third party signed certificate to the controller. This should remove the Not Secure sign displayed by the browser. To configure, go to Settings -> Advanced -> Certificate -> CERTIFICATE IMPORT. First Enable Certificate Checking. The console will ask you to enter a domain name and generate a CSR file (Certificate Signing Request). Send this CSR to get singed, then import both CA and server certificate. Note if intermediate certifcate is one of the return files, use the intermediate certificate file for CA import Connectivity Support Site2Cloud tunnel on TCP. In addition to run IPSEC tunnel on UDP protocol, you can now run on TCP 443. This option removes the requirements of having to open site firewall ports on UDP 4500/500. To configure, go to Site2Cloud -> Add New. Select TCP for Tunnel Type selection. 306 Chapter 66. Release Notes

313 Scalability Support load balancing UDP based OpenVPN gateways. If your OpenVPN users experience slow terminal response or long file transfer time, use UDP based VPN gateway can help. This release allows you to create multiple UDP based VPN gateways and load balance them in a round robin fashion by leveraging AWS Route53. To configure, go to OpenVPN -> Advanced -> UDP Loadbalancer. Note with UDP protocol UDP port 1194 is used. When using from on-prem, firewall port UDP 1194 must be open. Support Designated Gateway. If you are planning to have a large set of tunnels going through a gateway or are hitting AWS route entry limit, this feature is for you. If Designated Gateway option is selected at the gateway launch time, the Controller programs 3 route entries based on RFC1918 for the gateway. Controller will not program additional route entries when configure a VPN tunnel that end on the Designated Gateway. Note if you currently do not have a Designated Gateway and you are hitting route entry limit, launch a new gateway with Designated Gateway enabled and configure future tunnels from the Designated Gateway. Note there can only be one Designated Gateway per VPC. Designated Gateway only supports Gateway HA Modular Configuration Allocate New EIP. When this option is selected at new gateway launch time, Controller always allocates a new EIP from AWS and associated it with the gateway. If this option is unchecked, Controller will first look at the EIP pool that belong to the account: if there is allocated but unassociated EIP, Controller will allocate EIP from this pool and associate it with the gateway, otherwise it will select one EIP from the pool and associate it with the gateway. Support resizing active Gateway without deleting its peering tunnel. You can resize an active gateway when there peering HA configured. The workflow should be: 1) Settings -> Gateways -> select the gateway, select Edit. 2) Select it desired gateway instance size, click Change. As the result of this function, the gateway will be stopped and tunnel switch to backup tunnel. 3) Go to Settings -> Peering, select the peer and click Force Switchover. Support resizing UDP based OpenVPN gateway instance NEW REST APIs Set VPC Access Base Policy. Update VPC Access Policy. Enable Packet Logging R Connectivity Run encrypted tunnel on TCP port 443. Aviatrix Virtual Appliance CloudN now offers a TCP based secure tunnel connectivity. With this new capability, you do not need to open outbound UDP port 500 and The encrypted tunnel runs on TCP 443. To configure, go to Datacenter Extension, select TCP for the field Tunnel Type. UDP based encrypted tunnel is still supported. Reserve on-prem segment for Datacenter Extension feature of CloudN. After deciding how many VPCs you wish to configure during on boarding, you can sub divide the segments to reserve some for on prem VM deployment. This allows you launch applications where some part of it (such as database) is on-prem and others parts of it (such as web front end) to be in VPC R

314 Google IDP support. Google IDP is now supported IDP for the Aviatrix SAML VPN solution Security FQDN blacklist. In addition to FQDN whitelist, FQDN whitelist is added as a base configuration for each FQDN tag. To configure, go to Advanced Config -> FQDN Filter. After you create a new tag, you can select either White List or Black List. With Black List, the URLs on the Black List will be rejected REST API New APIs are published. list active VPN users, edit Open VPN configuration, backup and restore, list vpc peers, list image. For API details, click this link. for details User Interface re-organize menu items on Settings. Under Settings -> Maintenance are Upgrade, Backup & Restore and Security Patches. Under Settings -> Controller are System Time, License information, settings and controller access method authentication LDAP or DUO configuration.ounder Settings -> Advanced are tunnel timeout and keepalive configuration, password change and certificate management. Make a wish. Customers can now send feedbacks on UI pages regarding features, usability and make a wish on new requirements and enhancements R Security improvements Provide security patch to upgrade OpenVPN server to v To apply the patch, go to Settings->Patches and select OpenVPN New Aviatrix VPN client (v1.3.9) for user VPN (Mac, PC and Unix). To download, go to this link. Hardened password management for forgot password. Additional ciphers for site to cloud tunnels for interoperability. To configure, go to Site2Cloud -> Add New -> Algorithms Public cloud specific features AWS China [available in the UCC version only] Restful API support for AWS China. For details of the complete APIs, refer to API Document. Aviatrix cluster peering over AWS peering. To enable it, go to Peering -> Cluster Encrypted Peering -> New Peering and select Over AWS Peering. Aviatrix backup/restore in Google Cloud. To configure back/restore, go to Settings -> Backup & Restore. Python script for Google Cloud Controller HA monitoring and restarting. Follow 308 Chapter 66. Release Notes

315 Usability enhancements Multiple enhancements on User Interface. Aviatrix product Doc site is now available at New browser support: IE Administration automation Cloud-init script to accept input parameters to launch Aviatrix Controller on premises. Automated Aviatrix Controller deployment in AWS using Cloudformation: GW Resizing API edit_gw_config. Support proxy setting modification through Advanced Config -> Proxy Settings. Frictionless install UX [Register Aviatrix on premises Gateway with UCC Controller at the time of install to auto-fetch initial configuration; available for AWS at this time] Configurable Aviatrix Gateway Failover/HA time Support configurable health check frequency between Aviatrix Controller and Gateways for customers to meet their HA failover time constraint. To change the health check frequency, go to Settings -> Keepalive. Select slow only when your network is unstable and gateways send too many status alerts Logs and troubleshooting Aviatrix for Splunk has been published on Splunkbase. To download, click this link. For instructions on how to use the app, click this link. Aviatrix for SumoLogic application is available. To download, click this link. Rsyslog over UDP for customers needing UDP based rsyslog. To configure, go to Settings -> Loggings -> Remote Syslog and select UDP for Protocol Configurable gateway debug level. To adjust the debug level, go to Troubleshot -> Diagnostics -> Gateway Debug Level and select the appropriate debug level for your gateway New Aviatrix OVF for VMWare Visit download.aviatrix.com 66.5 UserConnect Security First release to white list public Fully Qualified Domain Names (FQDN filtering) for egress HTTP and HTTPS traffic to Internet initiated by instances on private subnets in a VPC. The FQDNs can be specified with regex wild card, such as *.example.com. A tag is defined as a list of FQDNs and one or more gateways is attached to a tag. Any updates to a tag automatically triggers updates to all gateways attached to the tag. Multiple tags can UserConnect

316 be defined on the controller. This feature works together with Gateway Security Policy feature where private network, IP address, protocol and ports can be filtered. To configure, go to Advanced Config -> FQDN Filter. The workflow is 1) create a tag, 2) Click Enable to enable the tag, 3) Edit the tag by adding FQDN hostname part of URLs (e.g. or *.google.com), and 4) Attach Gateway. One or more gateways can be attached to a tag. Step 1), 3) and 4) can be done first and then Enable the tag. Once the tag is enabled, HTTP and HTTPS traffic to these FQDN will be allowed, and any destination outside the FQDN will be denied. Note the gateway with FQDN must have NAT enabled for Internet egress traffic. Caveat: in this release FQDN filter is not failover capable when peering HA is configured Monitor and Troubleshooting During UCC gateway launch, Controller now reports in text the progress of gateway creation in addition to the progress bar view. Dry Run for system upgrade. Dry Run performs health checks for the Controller and gateways to detect potential upgrade failure without executing the command. Go to Settings -> Upgrade. Optionally, click Dry Run. If it is successful, you may click Upgrade. Dashboard now displays a summary packet statistics per gateway. Click on a specific gateway, top 10 packet statistics of the gateway are also displayed. Support test network connectivity. This is useful to troubleshoot any firewall or security policy that blocks connectivity from the controller or gateway. To test, go to Troubleshoot -> Diagnostics -> Network Connectivity Utility. Select either Controller or one gateway and test if it can reach a specific port of a remote host. Capability has been added to log tunnel status change notification to syslog (in addition to an notification with the same content). Enhancement has been made for tunnel status alert mechanism by allowing users to configure tunnel down detection time. To change the detection time, go to Settings -> Tunnels. The default detection time is 60 seconds. Capability has been added to check the VPC settings of a specific gateway. VPC settings include security groups, route tables, subnets, Network ACLs, DHCP options. To configure, go to Troubleshoot -> VPC Diagnostics Splunk forwarder has been upgraded from version 6.2 to version Connectivity and High Availability Support multiple independent UDP based VPN gateways (without ELB) within the same VPC. These VPN gateways can have different attributes. For example, one gateway has split tunnel configured while the other one has full tunnel configured. Support API credential change on controller console for Azure ARM accounts when the credential becomes out of sync with the credential on cloud provider console. For example, the account credentials are changed by the cloud provider or user herself. HA support has been added to Service Chaining with AWS gateways in different zones. Support IAM role-based controller and cloud account for AWS GovCloud. The Controller must be in GovCloud to create GovCloud gateways with IAM role-based accounts. Site2Cloud HA support has been added with CloudN as the on-prem device. To configure it, launch two gateways in the same VPC/VNet with UCC Controller. Then go to Site2Cloud page to create a new connection. Check Enable HA and select Aviatrix from Remote Gateway Type list. After creating the site2cloud 310 Chapter 66. Release Notes

317 connection, select this connection and download configuration with Aviatrix as Vendor. Import the same configuration file at CloudN s Site2Cloud page Controller Administration Function has been added to notify admin via admin when a new release becomes available. Support has been added to enforce password complexity of account users. To enable it, go to Settings -> Security -> Password Management. Support read only (operator) role for Controller management. The read only account has dashboard view, status view and list view, but cannot make modification to any configuration. To create a read only user, go to Accounts -> Account Users -> New User. Select read_only from the dropdown list of Account Name. CloudN s console password can be changed from the default Aviatrix123#. To do so, type enable to enter config mode and then issue change_console_password command. Capability has been added for HTTPS certificate check for control traffic between Controller and gateways. To turn on this function, go to Settings -> Security -> Certificate Checking. The following APIs have been added. For details of the complete APIs, refer to API Document. list_vpcs_summary peer_ha_switch_over upload_cloudx_command_log upgrade 66.6 UserConnect First release of Service Chaining. Service Chaining capability allows multiple instance based functions to work in tandem to control traffic flow path within an AWS VPC. For example, a firewall instance can be service chained with Aviatrix gateway so that EC2 initiated traffic will first be sent to firewall for inspection before forwarding it to Aviatrix gateway for peering to another VPC. To enable the function, go to Advanced Config -> Service Chaining to select the route table and enter Downstream IP. Aviatrix gateway will only modify the selected route table to specify which outgoing traffic needs to go through itself and also route the incoming traffic to the Downstream IP address. Normally, the selected route table is associated with the subnet of your firewall s WAN (or untrusted) interface. The Downstream IP should be the IP address of your firewall s WAN interface. For details, check out this reference design. Within AWS, support has been added to allow deployment of the UCC Controller in VPC s private subnet. To enable this, during the Controller s initial setup, when prompted for If this controller is being launched on a private subnet, check the box below, otherwise, leave it blank, select private subnet and then click the save button. Please note that when the Controller is deployed in private subnet it can only create gateways in private subnets. We assume these private subnets in various VPCs can reach each other through AWS peering. For AWS, account diagnostics have been added. To run these diagnostics, go to Troubleshoot -> Diagnostics -> Account Diagnostics. This diagnostics command will validate the AWS account credentials and check the status of associated gateways and SQS queues. There is now support for adding multiple CIDRs separated by commas in Advanced Config -> Join Function -> Allow Subnet at CloudN. Tunnel HA for Azure ARM gateways can now be created through Advanced Config -> Join Function. To enable tunnel HA, select a particular gateway on the Gateway page and then go to Gateway for High Availability Tunnel/Peering to create a backup gateway UserConnect

318 Support has been added to allow the creation of two VPN gateways (without ELB) in the same VPC, one with SAML enabled and the other one with only certification authentication enabled (no MFA method supported on the 2 nd gateway). The Dashboard now displays the IPSec tunnels created by site2cloud connection. Support has been added for enabling NAT on CloudN Controller itself. To enable this, go to Troubleshoot -> Diagnostics -> NAT Configuration. With this release, both the actual public IP address of the Controller and the stored public IP address if it is different from the actual public IP are displayed. To view these public IP addresses, go to Troubleshoot -> Diagnostics -> Controller Public IP. Proxy server support has been added on the UCC Controller for initial download and ongoing communication. During the Controller s initial setup, when prompted for If the controller accesses the Internet through a proxy server, provide the following information, otherwise leave the fields blank, enter the server URLs for HTTP Proxy and HTTPS Proxy. If the proxy server issues a self-signed certificate, upload a CA certificate. The ability to setup proxy server setting for Internet connectivity in CloudN OVA has been added. To configure proxy server support, use setup_network_only {true false} for clish command setup_interface_address and setup_interface_static_address. Use clish command setup_network_options {test save cancel} to test/save/remove http/https proxy setting. Currently, Datacenter Extension and Join Function are not supported when proxy server is enabled. Traceroute support has been added on gateways. To run Trace Route, go to Troubleshoot -> Logs -> Traceroute Utility. For site2cloud, users can now select the route tables to be modified when Encryption over Express- Route/DirectConnect is enabled. Only subnets associated with the selected route tables will have tunnel connections to on-prem. To select route tables, go to Site2Cloud -> Add New and enable Encryption over ExpressRoute/DirectConnect. Available route tables will show up in the Route Tables to Modify field. The following APIs have been updated. For details of the complete APIs, refer to API Document. Added: update_profile_policy & add_admin_ _addr Deprecated: add_profile_policy & del_profile_policy Changed: connect_container & add_vpn_user In the Aviatrix VPN client release, Linux version AVPN client is now in the supported list. Linux version is only supported on Ubuntu UserConnect Add support for three additional AWS regions: Ohio (us-east-2), Canada (ca-central-1) and London (eu-west-2). Enable load balancer support for Azure ARM VPN gateway creation. Add packet capture support for both Controller (CloudN only) and gateways. To run Packet Capture, go to Troubleshoot -> Diagnostics. Select Local from Gateway list to capture packets on CloudN. Select a gateway name from Gateway list to capture packets on the particular gateway. The packet capture files are in.pcap format and can be downloaded for analysis. Add traceroute support on Controller (CloudN only). To run Trace Route, go to Troubleshoot -> Logs. Extend the Peering HA support initiated at release from AWS to GCloud and Azure ARM. To enable this feature, go to Gateway -> Gateway for High Availability Peering to create the backup gateway first and then go to Peering -> Encrypted Peering to create the peering with Enable HA selected. 312 Chapter 66. Release Notes

319 Add diagnostics tools for IPSec tunnels created through CloudN Join Function. Go to Advanced Config -> Join Function. Select the IPSec tunnel to run diagnostics on it. The following options are available: debug, ping, measure latency, restart services and check peering status. Allow to add VPN users to each individual gateway (with ELB disabled) instead of the whole VPC. Select the gateway name from LB/Gateway Name list at OpenVPN -> VPN Users -> Add New to add VPN users to that gateway. Support migrating the same CloudN from one public IP address to another address. Go to Troubleshoot -> Diagnostics -> Migrate to migrate CloudN from its old public IP address to a new one. Support Controller migration from the old CloudN to a new CloudN. Go to Settings -> Backup & Restore to run backup at the old CloudN. Launch a new CloudN with a different public IP. Go to Settings -> Backup & Restore to run restore at the new CloudN. The migration function will automatically update the new CloudN with its own public IP. Support LDAP for Controller login. To enable it, go to Settings -> Setup LDAP Login to enable LDAP login first. Then add users at Accounts -> Account Users with local passwords. These account users should exist at LDAP server also. With LDAP login enabled, these users can log into Controller with their LDAP passwords. If LDAP login disabled, these users can log into Controller with their local passwords. Allow credential change for AWS and GCloud accounts when the account credentials are changed by the cloud provider. Support Okta along with Client Certificate Sharing when creating VPN gateways. Select Okta from Twostep Authentication list and select Yes for Enable Client Certificate Sharing when launching a new gateway. In previous releases, Client Certificate Sharing can t be enabled when Okta is used. Allow users to customize the notification (both content and attachment file name) for VPN client. To configure it, go to OpenVPN -> Configuration -> User Defined Notification to edit the file name or content. The new format will be used when a VPN certificate is issued. Add support for the following new APIs. For details of the complete APIs, refer to API Document test_ldap_bind get_gateway_supported_size get_supported_region list_peer_vpc_pairs peer_vpc_pair unpeer_vpc_pair Aviatrix VPN client release 66.8 UserConnect Added search capability to the Gateway list page. You can now search for gateways by any of the gateway attributes, such as Name, gateway instance size, account name, etc. Added search capability to active VPN users list on dashboard. You can now search for active VPN users by all attributes, such as Name, Profile, Landing Gateway, etc. CloudN Join function HA support. Join capability allows you to connect to an existing VPC with an IPSec tunnel. To enable HA, go to the Gateway page, click the gateway, and enable HA. Remote Syslog enhancement. Enable remote syslog to optionally not be encrypted. To configure, go to Settings -> Loggings -> REMOTE SYSLOG, simply ignore the cert option UserConnect

320 Aviatrix SAML VPN client preview for GCloud. The new Aviatrix SAML client provides a seamless user experience when authenticating a VPN user through a SAML IDP. For customers who use SAML based Single Sign On (SSO) for a unified user authentication and access control to their applications, this new capability allows them to treat the Aviatrix VPN solution as another application that authenticates VPN users by an already established mechanism. This preview release has been tested on GCloud. Forgerock is the primarily tested IDP and Okta has been partially verified. The supported platforms for the Aviatrix SAML VPM clients are Mac OSX, Windows 10, and Windows UserConnect Scale out encrypted peering support for AWS. You can create a cluster in a VPC that consists of up to 7 gateways, peering between two clusters in two VPCs increases packet throughput. To enable cluster encrypted peering, click Cluster Encrypted Peering under Peering tab. Preliminary iperf performance test shows TCP packet throughput can reach up to 8.5Gbps with bi-directional traffic. For more information, check out Cluster Peering Reference Design Controller HA support. Create a standby controller in any region and any cloud (AWS, Azure ARM and GCloud). When the primary controller goes down, the standby controller takes over and becomes operational. To enable the feature, click Settings -> Controller HA -> Enable. Input the standby controller s public IP address. You also need to input standby controller s admin username and password for authentication purpose. Enhanced peering HA support. The new peering HA feature reduces failover to a backup peering to under 2 seconds. To enable the feature, click Peering -> Encrypted Peering and enable HA. Note the current gateway HA support will be phased out in the future. Transitive peering support for Azure ARM, Azure classic, GCloud and Azure China. Built on the earlier release of transitive peering support for AWS, this feature is now covered by all cloud types. This feature enables you to deploy a hub and spoke architecture of multiple VPCs in a simple point and click manner. To enable transitive peering, click Peer -> Transitive Peering. Peering Diagnostics support. Troubleshooting peering tunnel status is made easy. Click Diag of the specific peer. Options are debug, test latency, ping and restart the tunnel. Display the public IP address of the controller. This feature is useful for CloudN64 virtual appliance where its public IP address is needed for configuring Site2Cloud capability. To view the controller s public IP address, click Troubleshoot -> Diagnostics -> CONTROLLER PUBLIC IP. Support all Azure ARM regions. Support interoperability of Aviatrix gateway Site2Cloud to AWS VGW and Azure VPN Gateway. When configuring Site2Cloud, you can select the specific cloud provider VPN gateways to ensure encrypted tunnel work correctly. Add REST API for CloudN64 Join features: allow subnet to VPC and delete subnet to VPC. For the complete APIs, refer to API Document UserConnect Add Mumbai (ap-south-1) to AWS region support list. Support multiple Splunk indexers by importing Splunk config file. This enables Aviatrix controller and gateway logs to be integrated with multiple Splunk servers that many enterprises deploy. To configure, go to Settings -> Loggings -> Splunk. Select Import files to import a Splunk configuration file. You may also choose Manual Input, in this case each indexer must be listensing on the same port. 314 Chapter 66. Release Notes

321 Support DataDog agent for both controller and gateways. To enable, go to Settings -> Loggings -> DataDog, provide an API Key. Enhancement for VPN user profile editing: when adding a user to a specific profile, only display those who do not belong to the profile. When deleting a user to a specific profile, only displays users who belong to the profile. Support tooltip for many labels. Move mouse over a label, a detailed explanation displays for the label UserConnect Support encryption over AWS peering. This capability allows two same region VPCs to send encrypted traffic to each other without going over Internet by leveraging AWS peering as underlying infrastructure. This mechanism significantly reduces data transfer cost. To use this feature, customer must configure AWS peering from AWS console between the two VPCs in the same region. To enable this feature, go to Peering -> Encrypted Peering -> New Peering. Check Over AWS Peering. One use case for this feature is to enable NetApp OnTAP software to run in HA mode. Support Azure ARM North Europe region. Support Skyhook for Docker 1.12 release UserConnect Support site2cloud use case where the gateway imports a template configuration file from a different Aviatrix gateway that initiates the configuration. This capability is useful to build IPSEC tunnels between two enterprises where each has its own Aviatrix UCC controller. Support using Aviatrix CloudN as customer device for site2cloud connection. Follow these steps: 1) use UCC Controller to create a site2cloud connection by entering CloudN s public IP and subnet CIDRs for customer onprem network. 2) On UCC Controller, select Aviatrix as vendor choice to download this site2cloud configuration file. 3) go to CloudN s site2cloud page and import the downloaded configuration file to establish the site2cloud connection. Allow users to provide an optional IPSec pre-shared key when creating site2cloud connections. When the filled is left empty, UCC controller will automatically generate a pre-shared key. Support HA for GCloud gateways with a zone selection option. Update REST API to accommodate GUI 2.0 development UserConnect Support on GUI 2.0: Settings -> Change Password Settings -> Settings -> System Time OpenVPN -> Profiles -> Edit -> Add New. Users can select subnets from VPCs/VNets without typing these CIDRs manually UserConnect

322 Gateway -> Click + next to the gateway name. Users can display all VMs inside the gateway s VPC/VNet VPN User list displays user and associated profile information. Allow users to setup VPN user license threshold notification. When license usage exceeds the threshold, notification will be sent out to admin s account. Azure Aviatrix gateway image is available at marketplace. There is no need to download gateway image to your storage account before launching a gateway. Instead, users need to subscribe to the Aviatrix Companion Gateway in Azure marketplace. This new capability significantly reduces Azure gateway deployment time. The Aviatrix Companion Gateway is free of charge. Please refer to startup guide for details UserConnect GUI 2.0 becomes production. To access GUI 2.0, go to Note: Old GUI is still available at All the new features developed in this release are only available for GUi 2.0. (Known issue: After upgrading to UserConnect , the browser does not log out properly. You must type in to re-login) Allow users to specify their own ELB names when creating AWS/GCloud VPN gateways. If no ELB name specified by users, Controller will provide a default ELB name. Support AWS IAM role. When AWS IAM role is used, there is no need to enter AWS access key and secret key when creating a cloud account at Controller. Instead, two IAM roles will be created. Controller will use the role-based temporary security credentials to request access to AWS resources. Cloud account created by IAM role helps to reduce the risk of compromising AWS credentials. Please refer to Aviatrix IAM role Configuration Guide for details. Support AWS Geo VPN to include other cloud type s ELB DNS name. To configure, go to OpenVPN -> Configuration to enable AWS Geo VPN first. Then you can add ELB DNS names from other cloud types to Geo VPN. With this capability, VPN gateway in Azure and GCloud can be included as part of Geo VPN solution. Support gateway resizing without a need to terminate the old gateway and create a new one. This feature is available for AWS, Azure Classic, Azure ARM and GCloud but only on gateways without ELBs. To configure, go to Gateway, select the target gateway and desired size from Gateway Size dropdown list, click Change button. Support an option to select subnet/availability zone when enabling HA for AWS. To configure, go to Gateway, select the target gateway and the desired subnet from Backup Gateway Subnet dropdown list, click Enable HA button. Support an option to select ELB name when editing VPN gateway configuration. This feature is useful for GCloud network, which may have multiple ELBs, each in a different subnet. To configure, go to Advanced Config -> Edit Config and select the ELB from LB Name dropdown list. Support to map multiple real CIDRs to multiple virtual CIDRs for site2cloud connection mapped connection. The multiple CIDRs need to be separated by a comma. The numbers and masks of the real CIDRs and corresponding virtual CIDRs must match each other. A new Aviatrix IAM custom policy is provided with more restrictive rules and some additional rules to support role-based IAM. 316 Chapter 66. Release Notes

323 66.15 UserConnect GUI 2.0 for preview. To access GUI 2.0, go to Note: Old GUI is still available at GUI 2.0 doesn t support all the features available at the old GUI at this time. Note: GUI 2.0 requires the controller to run on a instance with at least 4GB of memory. If your current controller does not meet this requirement, follow the procedure below: AWS controller: stop the controller instance, change instance type to t2.medium or larger, start the controller instance again. Azure Classic and Azure ARM controller: you can change the instance dynamically to at least D2 without stopping the instance first. Google Controller: stop the controller instance, change instance type to n1-standard-2, start the controller instance again. Support site2cloud connection between customer network and cloud network where the two sides may have overlapping CIDRs. Only GUI 2.0 support this feature. To configure, select Mapped for Connection Type and assign different virtual subnets to both customer network and cloud network. GUI 2.0 dashboard displays IPSec tunnel status and link latency of an encrypted peering. When IPSec tunnel status of an encrypted peering flips between up and down, an notification will be sent to the admin. GUI 2.0 displays all VPN users added to the controller without selecting VPC ID/VNet name first. VPN users are sorted alphabetically for easy search UserConnect Project Skyhook release: Docker swarm cluster container access support. From your desktop, you now can access Docker containers in a multi-host Docker swarm cluster built on a VXLAN overlay network that may span AWS, Azure and Google. To enable this feature, go to VPC/VNet -> VPN Access -> Skyhook: Docker Container Access. This feature is available on VPN gateways created after upgrade to this release. (If you have enabled ELB, delete the existing gateways and create new one. VPN user database are not affected.) For reference design on how to use this feature to access Docker containers, check out this link. Key benefits: a) MFA and user profile based access control apply to containers in the same manner as for instances. b) use the familiar tools such as curl, vim and wget on container without resorting to docker exec type of commands UserConnect Enhance stability, manageability and debug ability for gateway launch and encrypted peering functions. Support one load balancer in each different subnet of the same GCloud network. Kernel support on new gateway launches UserConnect When VPN gateways are behind ELB, allow to import a new CRL URL without recreating VPN users/profiles or reissuing VPN certificates. To configure, delete all the VPN gateways first and then go to VPC/VNet -> VPN Access -> Certificate Management -> Import Certificates. Make sure that CA Certificate, Server Certificate and Server Private Key are the same as before. The new CRL URL can be entered in CRL Distribution Point URI field. After finishing certificate management, recreate the VPN gateways behind ELB UserConnect

324 Enhance Encrypted Peering by verifying IPSec tunnel connection state after creating the peering. Provide Test HA function for verifying VPC high availability. To test it, go to VPC/VNet -> VPC HA to enable HA for your gateway first and then click Test HA button to test HA function. Enhance gateway creation by only listing the cloud types enabled in cloud accounts. Allow to modify site2cloud connection and configuration template by editing Cloud Networks or Customer Networks CIDRs. To use this feature, go to VPC/VNet -> Site2Cloud -> List -> Edit. If changes need to be made for subnets/address spaces in VPC/VNet, select Cloud Networks to enter all VPC/VNet CIDRs. If changes need to be made for subnets in on-prem network, select Customer Networks to enter all on-prem CIDRs. This feature minimizes the configuration changes on customer sites by not having to delete the existing site2cloud connection UserConnect First release for Azure ARM cloud support. If you currently have deployments in Azure Classic, we recommend you skip this release. Azure ARM is the new Azure portal that is significantly different in how API works comparing with Azure Classic. Support launching gateways in Microsoft Azure Resource Manager (ARM) VNet. Follow the embedded Aviatrix s instructions to collect Application Endpoint, Application Client ID and Application Client Secret before creating a cloud account. The main feature supported by ARM in this release is Site2Cloud. Peering with ARM VNet is not supported in this release. Site2Cloud supports to generate a configuration template for generic customer gateway devices. Support security patches for both controller and gateways. To apply the software patch, go to Setting -> System -> Security Patches. The patch available for this release is glibc Vulnerability UserConnect Support launching gateways in Microsoft Azure China. Azure China account is required to launch the gateways in Azure China. Support launching gateways in Amazon AWS GovCloud. AWS GovCloud account is required to launch the gateways in AWS GovCloud. Support Site2Cloud null encryption. This feature allows you to create an IPSec tunnel without encrypting the packets. To configure, go to VPC/VNet -> Site2Cloud -> Add and then select Null Encryption UserConnect This release consists of a few significant features: GCloud Support, Modular Split Tunnel Configuration, Site to Cloud, Encryption for Azure ExpressRoute, Transitive Peering and VNet route diagnostics, as described below: Support Google Cloud (GCloud). The following major functions are available on GCloud for this release: Launch an Aviatrix Controller from GCloud directly. Follow the instructions to do so. From AWS/Azure/GCloud controller, you can now launch a gateway in GCloud. GCloud account creation, editing and deletion Multiple GCloud projects support 318 Chapter 66. Release Notes

325 GCloud gateway (with or without ELB, with or without VPN access) creation and deletion Gateway encrypted peering to other projects in GCloud and with AWS VPC and Azure VNets. Security policies at GCloud network level. Edit configuration (LDAP, DHCP, and Split Tunnel) on existing gateway Support the ability to edit the split tunnel mode on existing VPN gateways. Previously, to make any split tunnel related configuration changes, users have to delete the existing VPN gateways and re-create new ones. With this release, when you add a new VPC/VNet and your VPN users need to access them via VPN, you just modify the CIDRs at additional CIDRs field at split tunnel configuration without deleting any existing gateways. To configure, go to VPC/VNet -> Edit Configuration-> Modify Split Tunnel. Note all additional CIDRs (the CIDRs that are not the VPC/VNet CIDR where VPN gateways are deployed) must be entered all together, separated by comma. For example, you have two new VPCs, /16 and /16, and you like to access them via split tunnel VPN. You must enter at the Modify Split Tunnel field /16, /16 without the quote. In addition, you may need to add encrypted peering with the new VPCs in order for traffic to go through. The changes are effective immediately to the VPN gateway in the VPC/VNet. If there are multiple VPN gateways behind a load balancer, they are all updated at the same time. Active VPN users will be disconnected during this configuration time. Support Transitive Peering. Transitive Peering enables you to route traffic from instances in Source VPC, encrypted, through a NextHop VPC gateway to reach a destination. Before creating Transitive Peering, you need to make Encrypted Peering between Source VPC and NextHop VPC first. To create/delete Transitive Peering, go to VPC/VNet -> Encrypted Peering -> Transitive Peering. Support site to cloud IPSec VPN connections. Using this feature, you can create IPSec VPN connections linking your on-prem networks to VPC/VNets in the cloud. To configure, go to VPC/VNet -> Site2Cloud. After adding a site2cloud connection, you can download a configuration template file for your on-prem devices (Only Cisco ASA configuration template is available now). If High Availability (HA) function is enabled, one gateway serves as the primary VPN connection endpoint and the other one serves as the backup. In case on-prem device loses the VPN connection to the primary VPN gateway, it can switch to the backup gateway to recover the VPN connection. Some diagnostic tools for site2cloud are also provided. Support encryption for Azure ExpressRoute. This feature allows to run IPSec over Azure Express Route to ensure a higher security level. To enable it, first launch a gateway in a subnet dedicated for the gateway, then go to VPC/VNet -> Site2Cloud, click Add tab and select Private Route Encryption. Support VNet route diagnostics. Go to Settings -> Troubleshooting -> VNet Route Diagnostics to find various VNet routing related diagnostics tools UserConnect Support VPN certificates maintained by a third party PKI system. Third party PKI must be created before any gateway launch. To enable this feature, go to VPC/VNet -> VPN Access -> Certificate Management. Use this feature to import certificates and download VPN configuration files. Support the ability to edit the LDAP settings on existing VPN gateways. Previously, to make any LDAP related configuration changes, users have to delete the existing VPN gateways and re-create new ones. With this support, you can enable, disable, or modify LDAP configuration on existing VPN gateways without deleting them. To configure, go to VPC/VNet -> Edit Configuration-> Modify LDAP Configuration UserConnect

326 66.23 UserConnect Support remote syslog to a third party or Aviatrix syslog server. The feature allows 24x7 premium customer to forward both controller and gateway events to a customized Aviatrix syslog server for debugging and troubleshooting purpose. This feature improves customers network uptime. To enable this feature, go to Settings -> Setup loggings -> Remote Syslog. Support the ability to push down to VPN user client the DHCP settings made in AWS VPC Console Create DHCP Options Set menu. For example, if you wish to change DNS name after the gateway has been launched, you can use this feature to make changes. The active VPN users will be disconnected when this feature is executed. To configure, go to VPC/VNet -> Edit Configuration -> Reload DHCP Configuration UserConnect Support Sumologic logging collector. When enabled, syslog data from the controller and all gateways will be forwarded to a Sumologic account. To enable, click Settings -> Setup Loggings -> Sumologic Add LDAP user search capability when Test LDAP Configuration to further test drive the correctness of a LDAP configuration. Enable gateway High Availability capability with a pair of gateway instances in active and hot standby mode. To enable, go to VPC/VNet -> VPC HA. Add Help me! for a drop down display of VPC/VNet in a specific region and cloud account UserConnect Clean up onboarding messages and texts for Azure usage UserConnect Support Geo VPN feature where a VPN user is connected to a nearest VPC. To enable Geo VPN, go to VPC/VNet -> VPN Access -> Geo VPN UserConnect Bug fix to allow multi-az and PBR routing configuration scenario. Added AZ display along with subnet info at gateway create. Created Reference Designs UserConnect Support 2FA DUO authentication to console log in, in addition to password credentials. The configuration is at Settings -> System -> Setup 2FA Login. 320 Chapter 66. Release Notes

327 66.29 UserConnect Support multiple controller and gateway clusters in the same VPC UserConnect Support Okta authentication. Support integration of Elasticsearch on the controller. Support both allow and deny rules for each VPC security policies UserConnect Support PBR event syslog for NAT translation of every TCP/UDP/ICMP session. The log describes the VPN user virtual IP address, source port and the destination IP address and port. By correlating with VPN username and its assigned virtual IP address, IT admin can uniquely track and identify every VPN users access activity history to both internal resource and external resource. Support multiple users in admin privilege. Support multiple users in user account privilege UserConnect Added hard token authentication support on DUO security. Made DUO authentication configuration optional. When Token is configured as the Push Mode for all gateways, user must append the 6 digits token number to their password. **Note: ** 1. All active VPN users will be disconnected for this upgrade duo to VPN server restart. 2. You must log out and log back in again for new features to take effect. 3. You need to run upgrade command two times. Support VPN user certificate re-issuing. When existing VPN user certificate is re-issued, the current certificate of the user is revoked and a new certificate is sent to the user. Active VPN user on dashboard display is dynamically refreshed every 5 minutes UserConnect Support launch gateways in Microsoft Azure UserConnect Support backup DUO push. When both LDAP and DUO are enabled, user can type #push1 or #push2 appending to the password field to specify which phone in the DUO device list to be notified for approval. For example, if a user John Smith s password is johnsmith, he can type at password prompt johnsmith#push1 or johnsmith#push UserConnect

328 to specify the first phone or the second phone to be notified. If only password is typed in, the default phone (the first phone on the device list in DUO) will be notified. **Note: You must run upgrade command twice to have the upgrade take effect for this particular upgrade. All VPN users need to be deleted and added again as the existing certificates will not work with the new encryption algorithm. The first upgrade command may generate an exception, just ignore it and run upgrade again. ** Suggested upgrade procedure: delete all existing users. Upgrade once and upgrade again, and then add users back. Support enhanced encryption algorithms: AES-256-CBC, SHA512, TLS1.2 and TLS-AUTH. Detailed display of VPC/gateway on Dashboard. Clicking on the gateway name displays the complete configuration of the gateway. Support REST API for all CloudOps commands. Support the option to launch gateway when creating CloudOps VPC pool. Support CloudOps Access IP address map history and initiator (from Console or from REST API). Hash all password. Add confirmation check when deleting a VPC or gateway. Dynamically display controller logs on UI. Bug fixes for out of order gateway command delivery and multiple identical users on the same gateway display UserConnect Support for CloudOps VPC pool creation and CloudOps Read Me First. Support additional route push to VPN client when split tunnel is enabled. Disable password caching and credential saving in.onc file for Chromebook users. Display profile name instead of command name in VPN active user Dashboard. Fix typos in notification sent to VPN users. For UDP connections, send a disconnect message to VPN gateway immediately when the client terminates. Fix release version alert problem UserConnect Support Diagnostics on controller and gateways. Added DNS name service for CloudOps Networking feature. Dashboard performance improvement. Enhance Chromebook VPN ONC file connection name to be profiled based. Bug fix for logstash forwarder. 322 Chapter 66. Release Notes

329 66.37 UserConnect Support upgrades without terminating existing active VPN users unless specifically documented. Various bug fixings. General UI look and feel update UserConnect Integrate LDAP configuration with Gateway creation to streamline provisioning process. Display Profile fields in Active VPN User dashboard. Support logstash forwarder to forward all syslog and auth log to designated logstash server. Support software release version visibility UserConnect Support template generation at create gateway and configure VPN access. Support user activity history UserConnect Support operator account where operator can only access dashboard. Support disconnect user from dashboard page UserConnect Support capability to manage instances in overlapping CIDRs of VPCs. Support dashboard for active user display UserConnect LDAP client certificate import facility to support LDAP servers with TLS client verification Support configurable action parameter in user profile policy Support forwarding of syslog events to Logstash server UserConnect

330 66.43 UserConnect Support LDAP + Duo multi-factor combined authentication Support configurable base policy for user profiles REST API to change a VPN user s profile UserConnect Support Chromebook as a VPN client. Support DUO multi-factor authentication. Support syslog display with regex filtering capability for each VPN gateway UserConnect Support policy based routing on the VPN server to allow enterprise to re-direct traffic to its own backbone UserConnect Support user authentication via Google 2-Step Verification process. Support multiple domain names UserConnect Support setting the maximum number of connections for each gateway. Support NAT capability for each gateway. Support both split tunnel and full tunnel mode for each gateway. Support gateway size c4.2xlarge. Support add and delete members on the Profile page UserConnect Support user profile based security policies. Support scale out and highly available OpenVPN solutions for direct access to VPCs. Support LDAP integration. Support Windows, MAC OS and Chromebook clients. OpenVPN is a registered trademark of OpenVPN Inc. 324 Chapter 66. Release Notes

331 CHAPTER 67 Aviatrix VPN Client Changelog Dec FreeBSD support Configure reconnection behaviour on network disconnection Disable TLSv1 for client browser communication View log issue fix Oct Mac does not require admin password to run Mac icon fix Removed cert warning Bundled TAP driver for Windows Improved linux support. Fixed system tray. App mode Debian installation files Fixed viewing logs in Linux Aug Signed Mac application Parallel windows execution fix Jun Disconnection fixes Timeout fixes Connection profile is diplayed IE support for SAML Signed Windows application 325

332 1.2 - Mar HTTPS Version for SAML Multiple Profiles Linux version Connection status detection Unblock disconnection while connecting Retry prompt for LDAP Multi process feature for Mac/Linux. Removed VPN Lockdown Permissions fixes Fixes in logging Jan Settings window for troubleshooting Mac default application behavior Bug fixes for hangs In built resources Connection timeout issues fixed Kill other OpenVPN on start Connection status fix VPN lockdown feature Dec Initial release HTTP Version OpenVPN is a registered trademark of OpenVPN Inc. 326 Chapter 67. Aviatrix VPN Client Changelog

333 CHAPTER 68 Migrating a Join deployment to Site2Cloud deployment If you have deployed virtual appliance CloudN and used Join feature to connect your existing VPCs and would like to migrate to use Site2Cloud feature instead to connect to the same set of VPCs, the following steps can be a reference guide. You can choose to re-use the same CloudN for the on-prem gateway in Site2Cloud implementation or a different CloudN. For ease of reference, we call the VPC where the Join and Site2Cloud VPC gateway terminates migrating VPC. Note: This migration process will have tunnel down time. It is best practice to plan the migration during a maintenance window. 1. Launch an Aviatrix Controller in AWS or Azure. 2. From the Controller, launch an Aviatrix gateway in a migrating VPC. 3. From the original CloudN where Join function was implemented, delete all participating subnets. After all subnets are deleted, delete the corresponding gateway. 4. On the default routing gateway where the original CloudN is deployed, remove the routes that points to the original CloudN as the next hop to the migrating VPC. (This step is not needed if the new and the original CloudN are the same one.) 5. On the Aviatrix Cloud Controller, create a Site2Cloud connection on the migrating VPC. Download the configuration template. 6. On the new (this new could be the same original) CloudN, import the previously downloaded configuration template. 7. Make sure the tunnel comes up. 8. On the default routing gateway where the new CloudN is deployed, add a static route that points the new CloudN as the next hop to reach the migrating VPC. (This step is not needed is the new and the original CloudN are the same one.) 9. The VPC migration from Join function to Site2Cloud is done. 10. Repeat the above steps for more migrating VPCs. 327

334 328 Chapter 68. Migrating a Join deployment to Site2Cloud deployment

335 CHAPTER 69 Auto Booting CloudN VM Using ISO File This document provides one method to boot CloudN VM automatically without the initial manual configuration stage for interface address. The method is to use a customized ISO file when launching the virtual machine. Note CloudN can be downloaded from this link: Installation on vmware vsphere Client Create the customized configuration In order to boot CloudN that passes in interface address information, we need to create an ISO image contains both user-data and meta-data in ISO9660 format. Creating user-data file In the following example, CloudN is designed to boot up with a static ip address , netmask , gateway and dns-nameservers Please note that #cloud-config is not a comment but a directive to cloud-init. Sample contents of user-data: #cloud-config write_files - path: /etc/network/interfaces content: auto lo iface lo inet loopback auto eth0 iface eth0 inet static address

336 netmask gateway dns-nameservers Note: If CloudN VM were to be deployed in a proxy environment, we would need to include additional proxy settings in the user-data. In the following sample, is the IP address of the CloudN VM, is the proxy IP address with port 8080, as shown in the example below. Sample contents of user-data (with proxy settings): #cloud-config write_files: - path: /etc/sudoers.d/90-proxy content: #Aviatrix http/https proxy integration Defaults env_keep += "http_proxy https_proxy no_proxy" - path: /etc/network/interfaces content: auto lo iface lo inet loopback auto eth0 iface eth0 inet static address netmask gateway dns-nameservers bootcmd: - grep -q _proxy /etc/environment (echo "http_proxy= echo "https_proxy= echo "no_proxy= , ") >> /etc/environment - grep -q _proxy /etc/apache2/envvars (echo "export http_proxy= echo "export https_proxy= echo "export no_proxy= , ") > > /etc/apache2/envvars Create meta-data file instance-id: CloudN-local local-hostname: CloudN-local Create the ISO After user-data file and meta-data file are created, you can create the ISO by using this following command. 330 Chapter 69. Auto Booting CloudN VM Using ISO File

337 $ genisoimage -o cloudn iso -volid cidata -J -r user-data meta-data Verify the ISO (optional) ubuntu@ubuntu:~$ sudo mkdir /media/test_iso ubuntu@ubuntu:~$ sudo mount -o loop cloudn iso /media/test_iso mount: /dev/loop0 is write-protected, mounting read-only ubuntu@ubuntu:~$ cat /media/test_iso/user-data #cloud-config write_files: - path: /etc/network/interfaces content: auto lo iface lo inet loopback auto eth0 iface eth0 inet static address netmask gateway dns-nameservers ubuntu@ubuntu:~$ cat /media/test_iso/meta-data instance-id: CloudN-local local-hostname: CloudN-local ubuntu@ubuntu:~$ sudo umount /media/test_iso Deploy CloudN VM with the ISO Now you can deploy a CloudN VM with the cloudn iso attached as a CDROM to the VM. During the boot up process, the CloudN will be configured with the customized configuration in user-data and meta-data. Once the CloudN network is up, it will automatically download the latest CloudN software. We will be able to access the web UI directly without having to access the CloudN VM console to perform the initial interface setup Installation on vmware vsphere Client 331

338 After importing the CloudN ovf is completed, 332 Chapter 69. Auto Booting CloudN VM Using ISO File

339 Click on Edit virtual machine settings and select CD/DVD Drive under the Hardware section. Make sure the Device status Connect at power on option is checked Click on Use ISO image to browse to the cloudn iso. Click OK to complete the Virtual Machine Settings. Power on the CloudN virtual machine. The configuration in cloudn iso will be read by cloud-init during the installation process and CloudN will upgrade to default version when the network is up Installation on vmware vsphere Client 333

340 334 Chapter 69. Auto Booting CloudN VM Using ISO File

341 Once the CloudN login prompt is shown on the VM console, we can access the to complete the admin s and password initialization process Installation on vmware vsphere Client 335

342 Installation on Linux KVM The same methods previously described to create the cloudn iso can be applied to KVM virtualization environment Contents of user-data: #cloud-config write_files: - path: /etc/network/interfaces content: auto lo iface lo inet loopback auto eth0 iface eth0 inet static address netmask gateway dns-nameservers Chapter 69. Auto Booting CloudN VM Using ISO File

343 Note: If your enviornment has proxy server for accessing Internet, you need to include that as described in the vmware section Contents meta-data: instance-id: CloudN-local local-hostname: CloudN-local Create the ISO Image $ genisoimage -o cloudn iso -volid cidata -J -r user-data meta-data Deploy CloudN VM with the ISO Image Copy the CloudN qcow2 image and cloudn iso to the /var/lib/libvirt/images. cp /home/ubuntu/downloads/cloudn-ovf qcow2. cp /home/ubuntu/downloads/cloudn iso. ls -l CloudN-kvm qcow2 -rw-r--r-- 1 root root Mar 19 22:09 CloudN-kvm qcow2 root@ubuntu1:/var/lib/libvirt/images# ls -l cloudn iso -rw-r--r-- 1 root root Mar 19 22:11 cloudn iso In this example below, a bridge interface br1 is created and eno1 is assigned to this br1. ubuntu@ubuntu1:~$ ifconfig br1 Link encap:ethernet HWaddr 00:30:48:b3:59:92 inet addr: Bcast: Mask: inet6 addr: fe80::230:48ff:feb3:5992/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2060 errors:0 dropped:0 overruns:0 frame:0 TX packets:507 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (163.3 KB) TX bytes:74489 (74.4 KB) eno1 Link encap:ethernet HWaddr 00:30:48:b3:59:92 inet6 addr: fe80::230:48ff:feb3:5992/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2076 errors:0 dropped:0 overruns:0 frame:0 TX packets:559 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes: (201.5 KB) TX bytes:83977 (83.9 KB) Installation on Linux KVM 337

344 Interrupt:21 Memory:fe fe enp4s0 Link encap:ethernet HWaddr 00:30:48:b3:59:93 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Interrupt:19 Memory:fe fe lo Link encap:local Loopback inet addr: Mask: inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:656 errors:0 dropped:0 overruns:0 frame:0 TX packets:656 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1 RX bytes: (107.2 KB) TX bytes: (107.2 KB) virbr0 Link encap:ethernet HWaddr 00:00:00:00:00:00 inet addr: Bcast: Mask: UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) brctl show bridge name bridge id STP enabled interfaces br b35992 no eno1 virbr yes Create a new CloudN-1 by importing the CloudN-kvm qcow2 image with the customized cloudn iso root@ubuntu1:/var/lib/libvirt/images# virt-install --os-type linux --os-variant ubuntu import --disk path=./cloudn-kvm qcow2,bus=virtio,format=qcow2,size=20 --name CloudN-1 --ram vcpus 2 --disk path=./cloudn iso,device=cdrom --network bridge=br1,model=virtio --network bridge=br1,model=virtio --graphics spice Note: You may need to install virt-viewer package on your Linux machine in order to use the SPICE graphics. A Virt Viewer windows will pop up to show the installation process of CloudN. Once the CloudN login prompt is shown on the Virt Viewer console, we can access the to complete the admin s and password initialization process. 338 Chapter 69. Auto Booting CloudN VM Using ISO File

345 Installation on Linux KVM 339

346 340 Chapter 69. Auto Booting CloudN VM Using ISO File

347 When you close the Virt Viewer window, the CloudN VM is still running and you will notice that the Domain creation completed on the terminal that you executed virt-install command earlier. To shut down or delete the CloudN VM, you may use the Virtual Machine Manager or virsh commands like any other VMs supported by Linux KVM Installation on Linux KVM 341

348 342 Chapter 69. Auto Booting CloudN VM Using ISO File

349 CHAPTER 70 Customize AWS-IAM-Policy for Aviatrix Controller 70.1 Introduction Aviatrix provides the default Aviatrix-AWS-IAM-Policy for its solution. This document provides examples on how to customize these IAM policies. The customization reduces the scope of resource privileges and helps you meet your organization s security requirements. You can remove some of the policy rules by using this IAM-Policy if you only plan on using the following Aviatrix features Gateway creation without ELB (Elastic Load Balancer) 2. Encrypted-Peering 3. Transitive-Peering 4. Peering-HA (High Ability) 5. Site2Cloud 6. Controller Backup & Restore The next few sections provide examples on how to restrict policy rule scopes When to Modify AWS-IAM-Policy (aviatrix-app-role-policy) Before customizing the AWS-IAM-Policy for the Aviatrix Controller, follow the steps below Step 01: Use the original/default Aviatrix-AWS-IAM-Policy for every Aviatrix-Cloud-Account creation. The following screenshot is the account creation during the AVX controller Onboading process. 343

350 Step 02: After account creation, as administrator you can start editing/customizing the AWS-IAM-Policy, aviatrix-app-rolepolicy from your AWS-IAM-Policy section to increase the security level of your AWS environment/resources. Please see the following for more reference How to Modify AWS-IAM-Policy Step 01: Login to your AWS GUI console 344 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

351 Step 02: Go to IAM service Step 03: Click Policies and select the policy If you have not created aviatrix-app-policy, please see here How to Modify AWS-IAM-Policy 345

352 Step 04: Click Edit Policy Now you are ready to edit the policy! Please refer to the examples later in this document What Permissions are Required in App Role Policy and Why In the App role policy (example), it has different Actions to allow on certain resource. Your Aviatrix controller needs those policies to function. 1. ec2 to create/delete/list/modify VPCs, Aviatrix gateways, security groups, route tables, tags, start instance, stop instance, reboot instance, associate/de-associate IP address, etc. 2. elasticloadbalancing to create/configure/delete/modify ELB for Aviatrix VPN gateway 3. s3 to create/add/delete s3 buckets for save-and-restore and cloudtrail features 4. sqs to create/delete/list/send/get SQS and SQS messages for controller-to-gateway communication 5. sns to create/delete/list/subscribe/unsubscribe SNS and SNS topic for gateway HA feature 6. route53 to create/delete/list hosted zone, and change resource record for GeoVPN feature 7. cloudwatch to put/delete alarm for Aviatrix gateway HA feature 8. iam to support role based IAM account 346 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

353 70.5 How to reduce APP Role Policy : Default APP Role Based Policy Click here to see a default APP role based policy. In the default APP role based policy, it allows actions to apply to all resource. By changing Resource field from a wildcard * to a more specific resource ARN can limit the service the assumed role can do. The examples are described in the later sections : Use Condition to Allow Service Requests from Certain IP Addresses User can add Condition field to deny all requests not initiated from the Aviatrix Controller IP address or a range of CIDRs. The following policy only allows service requests from IP address /24, or /32, or /24. { } "Version": " ", "Statement": { "Effect": "Deny", "Action": [ "ec2:describeimageattribute", "ec2:describeimages", : : "ec2:describevpcpeeringconnections" ], "Resource": "*", "Condition": {"NotIpAddress": {"aws:sourceip": [ " /24", " /32", " /24" ]}} } We can also use Allow instead of using Deny in Effect element/key, and both ways have the same behavior. See the following... Syntax: { } "Effect": "Allow", "Action": [ "ec2:runinstances" ], "Resource": "*", "Condition": { "IpAddress": { "aws:sourceip": ["AVIATRIX-CONTROLLER-IP/32"] } } How to reduce APP Role Policy 347

354 Example: { } "Effect": "Allow", "Action": [ "ec2:runinstances" ], "Resource": "*", "Condition": { "IpAddress": { "aws:sourceip": [" /32"] } } NOTE: The method of specifying IP address of AWS instance(s) can apply to many AWS-API permissions, such as... ec2:describe*, elasticloadbalancing:describe*, route53:list*, route53:get*, sns:list*, s3:list*, s3:get*, etc... not only for ec2:runinstances Launch instances(aviatrix-gateway) on a specific subnet only from Aviatrix-Controller Syntax: { "Effect": "Allow", "Action": "ec2:runinstances", "Condition": { "IpAddress": { "aws:sourceip": [ "AVIATRIX-CONTROLLER-IP/32" ] } }, "Resource": [ "arn:aws:ec2:*:*:image/ami-*", "arn:aws:ec2:region:aws-account-id:subnet/subnet-id", "arn:aws:ec2:region:aws-account-id:instance/*", "arn:aws:ec2:region:aws-account-id:network-interface/*", "arn:aws:ec2:region:aws-account-id:volume/*", "arn:aws:ec2:region:aws-account-id:key-pair/*", "arn:aws:ec2:region:aws-account-id:security-group/*" 348 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

355 } ] Example: { } "Effect": "Allow", "Action": "ec2:runinstances", "Condition": { "IpAddress": { "aws:sourceip": [ " /32" ] } }, "Resource": [ "arn:aws:ec2:*:*:image/ami-*", "arn:aws:ec2:us-west-2: :subnet/subnet-abcd1234", "arn:aws:ec2:us-west-2: :instance/*", "arn:aws:ec2:us-west-2: :network-interface/*", "arn:aws:ec2:us-west-2: :volume/*", "arn:aws:ec2:us-west-2: :key-pair/*", "arn:aws:ec2:us-west-2: :security-group/*" ] : Launching instances on specific VPC(s) The policy can be modified to limit running gateways on certain VPCs only. In the following examples, we limit the role to launch Aviatrix Gateway on AWS account , region us-west-2, and vpc-873db7e2 and vpcfda23c98. Note, we can use wildcard * to replace region, account number, or VPC ID. { "Effect": "Allow", "Action": [ "ec2:runinstances" ], "Resource": "arn:aws:ec2:us-west-2: :subnet/*", "Condition": { "StringEqualsIgnoreCase": { "ec2:vpc": [ "arn:aws:ec2:us-west-2: :vpc/vpc-873db7e2", "arn:aws:ec2:us-west-2: :vpc/vpc-fda23c98" ] } } }, { "Effect": "Allow", "Action": "ec2:runinstances", "Resource": "arn:aws:ec2:*:*:image/ami-*" }, { "Effect": "Allow", How to reduce APP Role Policy 349

356 } "Action": "ec2:runinstances", "Resource": [ "arn:aws:ec2:*:*:instance/*", "arn:aws:ec2:*:*:volume/*", "arn:aws:ec2:*:*:network-interface/*", "arn:aws:ec2:*:*:key-pair/*", "arn:aws:ec2:*:*:security-group/*" ] Syntax { "Effect": "Allow", "Action": "ec2:runinstances", "Resource": "arn:aws:ec2:region:aws-account-id:subnet/subnet-*", "Condition": { "StringEquals": { "ec2:vpc": [ "arn:aws:ec2:region:aws-account-id:vpc/vpc-abcd1234" ] }, "IpAddress": { "aws:sourceip": [ " /32" ] } } }, { "Effect": "Allow", "Action": "ec2:runinstances", "Resource": [ "arn:aws:ec2:*:*:image/ami-*", "arn:aws:ec2:region:aws-account-id:instance/*", "arn:aws:ec2:region:aws-account-id:network-interface/*", "arn:aws:ec2:region:aws-account-id:volume/*", "arn:aws:ec2:region:aws-account-id:key-pair/*", "arn:aws:ec2:region:aws-account-id:security-group/*" ] } Example { "Effect": "Allow", "Action": "ec2:runinstances", "Resource": "arn:aws:ec2:us-west-2: :subnet/subnet-*", "Condition": { "StringEquals": { "ec2:vpc": [ "arn:aws:ec2:us-west-2: :vpc/vpc-abcd1234" ] }, 350 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

357 "IpAddress": { "aws:sourceip": [ " /32" ] } } }, { "Effect": "Allow", "Action": "ec2:runinstances", "Resource": [ "arn:aws:ec2:*:*:image/ami-*", "arn:aws:ec2:us-west-2: :instance/*", "arn:aws:ec2:us-west-2: :network-interface/*", "arn:aws:ec2:us-west-2: :volume/*", "arn:aws:ec2:us-west-2: :key-pair/*", "arn:aws:ec2:us-west-2: :security-group/*" ] } : AWS S3 Permissions/Policies The following S3 IAM-Policy examples demonstrate allowing AWS API which is to write/putobject AVX-Controller- Backup configuration file to a specified AWS-S3-Bucket and the command is issued only by your AVX controller. Syntax: { "Effect": "Allow", "Action": [ "s3:list*" ], "Resource": "arn:aws:s3:::*", "Condition": { "IpAddress": { "aws:sourceip": [ "AVIATRIX-CONTROLLER-IP-ADDRESS/32" ] } } }, { "Effect": "Allow", "Action": [ "s3:createbucket", "s3:deletebucket" ], "Resource": "arn:aws:s3:::*aviatrix*", "Condition": { "IpAddress": { "aws:sourceip": [ "AVIATRIX-CONTROLLER-IP-ADDRESS/32" ] } } How to reduce APP Role Policy 351

358 }, { "Effect": "Allow", "Action": [ "s3:putobject" ], "Resource": "arn:aws:s3:::your-s3-bucket-name/*", "Condition": { "IpAddress": { "aws:sourceip": [ "AVIATRIX-CONTROLLER-IP-ADDRESS/32" ] } } }, { "Effect": "Allow", "Action": [ "s3:get*" ], "Resource": "arn:aws:s3:::your-s3-bucket-name*", "Condition": { "IpAddress": { "aws:sourceip": [ "AVIATRIX-CONTROLLER-IP-ADDRESS/32" ] } } } Example: { "Effect": "Allow", "Action":[ "s3:list*" ], "Resource": "arn:aws:s3:::*", "Condition": { "IpAddress": { "aws:sourceip": [ " /32" ] } } }, { "Effect": "Allow", "Action": [ "s3:createbucket", "s3:deletebucket" ], "Resource": "arn:aws:s3:::*aviatrix*/*" "Condition": { "IpAddress": { "aws:sourceip": [ 352 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

359 " /32" ] } } }, { "Effect": "Allow", "Action": [ "s3:putobject" ], "Resource": "arn:aws:s3:::*aviatrix*/*" "Condition": { "IpAddress": { "aws:sourceip": [ " /32" ] } } }, { "Effect": "Allow", "Action": [ "s3:get*" ], "Resource": "arn:aws:s3:::*aviatrix*", "Condition": { "IpAddress": { "aws:sourceip": [ " /32" ] } } } AWS-Simple-Queue Permissions/Policies The following example(s) demonstrate allowing the IAM User/Role to access AWS-Simple-Queue object(s) only to the queues that the names start with the string aviatrix. Syntax: { "Effect": "Allow", "Action": [ "sqs:list*", "sqs:get*", ], "Resource": "arn:aws:sqs:*:aws-account-id:aviatrix-*" }, { "Effect": "Allow", "Action": [ "sqs:addpermission", "sqs:changemessagevisibility", "sqs:createqueue", How to reduce APP Role Policy 353

360 } "sqs:deletemessage", "sqs:deletequeue", "sqs:purgequeue", "sqs:receivemessage", "sqs:removepermission", "sqs:sendmessage", "sqs:setqueueattributes" ], "Resource": "arn:aws:sqs:*:aws-account-id:aviatrix-*" Example: { "Effect": "Allow", "Action":[ "sqs:list*", "sqs:get*", ], "Resource": "arn:aws:sqs:*: :aviatrix-*" }, { "Effect": "Allow", "Action":[ "sqs:addpermission", "sqs:changemessagevisibility", "sqs:createqueue", "sqs:deletemessage", "sqs:deletequeue", "sqs:purgequeue", "sqs:receivemessage", "sqs:removepermission", "sqs:sendmessage", "sqs:setqueueattributes" ], "Resource": "arn:aws:sqs:*: :aviatrix-*" } NOTE: We do not recommend using AWS-resource-IP checking mechanism to modify AWS-SQS API permissions EC2 Role Policy Examples Default EC2 Role Policy The Amazon EC2 role allows EC2 instances to call AWS services on your behalf. This policy allows action AssumeRole to ALL roles. The default EC2 role policy allows AWS EC2 instance to assume to any role. By changing the Resource field from a wildcard * to a more specific account number, role name or prefix of role name can limit the EC2 instance s role. 354 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

361 { } "Version": " ", "Statement": [ { "Effect": "Allow", "Action":[ "sts:assumerole" ], "Resource": "*" } ] : Example of EC2 Role Policy with More Specific Resource field The policy attached to the Amazon EC2 role can limit the role it can assume by specifying the 12-digit AWS account number, role name or prefix of the role name. In this example, the EC2 instance can assume role to any 12-digit AWS account with role name prefix HR-, or AWS account number with role name prefix aviatrix-, or AWS account number , role name developer. { } "Version": " ", "Statement": [ { "Effect": "Allow", "Action": [ "sts:assumerole" ], "Resource": [ "arn:aws:iam:: :role/aviatrix-*", "arn:aws:iam::*:role/aviatrix-role-app", "arn:aws:iam::*:role/hr-*", "arn:aws:iam:: :role/developer" ] } ] NOTE: Please refer to the policy example below. Aviatrix recommends our customers to add the ARN(Amazon Resource Name) of your APP-Role (aviatrix-role-app) into the Resource section. However, we do not recommend specifying any IP addresses such as your Aviatrix-Controller or Aviatrix-Gateway instances under the Condition section in order to avoid further unexpected issues. The best practice to specify which of your AWS instances are allowed to operate your AWS resources is to modify the APP-Role (aviatrix-role-app). Please see the examples under APP Role Examples section of this document. Recommended: { "Version": " ", "Statement": [ { "Effect": "Allow", "Action": [ EC2 Role Policy Examples 355

362 } ] } "iam:updateassumerolepolicy", "sts:assumerole" ], "Resource": ["arn:aws:iam:: :role/aviatrix-role-app"] Not Recommended: { } "Version": " ", "Statement": [ { "Effect": "Allow", "Action": [ "iam:updateassumerolepolicy", "sts:assumerole" ], "Condition": { "IpAddress": { "aws:sourceip": [" /32"] } }, "Resource": ["arn:aws:iam:: :role/aviatrix-role-app"] } ] 356 Chapter 70. Customize AWS-IAM-Policy for Aviatrix Controller

363 CHAPTER 71 Hybrid Network Load Balancing (NLB) 71.1 Balance Traffic between AWS and your Datacenter using AWS Network Load Balancer and Aviatrix Gateway Problem Description Operations teams are frequently managing infrastructure and services hosted both in the cloud and on-premise. Some common examples include: DR scenarios, centrally located shared services, and application and workload migration to the cloud. Establishing reliable and secure network connectivity for these hybrid use cases presents a challenge to most teams. Imagine one specific example: you have a critical internal web application hosted in remote offices around the globe as well as in AWS. In order to provide fault-tolerance for the application, you would like to setup a central load balancer that balances traffic between the remote sites and AWS. AWS recently released the Network Load Balancer that made this possible by adding the ability to specify an IP address as a load balancer target, in addition to instances. However, using the NLB to forward traffic to a target IP address outside of AWS will only work if you have Direct Connect between the remote site and the AWS region. An IPSEC tunnel built between AWS VGW and on-prem site does not work since in this case traffic is always initiated from the VPC. So, for most users this doesn t help. Aviatrix solves this for AWS customers without Direct Connect. In this document, we will demonstrate how to go from an empty AWS VPC and a remote, on-premise hypervisor to a working demo that balances web traffic between the two sites. 357

364 Demonstration This demo will involve two web servers hosting a basic website. One server will be located in a remote site and one will be hosted in AWS. We ll set up AWS NLB service to listen on port 80 and configure both of these servers as targets. This diagram represents the desired configuration: The webdemo hostname has been registered in DNS pointing to the NLB. When a user accesses the demo site (webdemo.aviatrix.com/index.html) from a browser, that request will be handled by the Network Load Balancer (the orange line in the diagram). The NLB will choose either the green route to the remote site or the blue route to the EC2 instance and the selected web server will respond to the user with the contents of the requested file. For the purposes of this demo, the contents of index.html will differ slightly on each server to include either Welcome to the Data Center or Welcome to AWS Prerequisites In order to complete the steps in this guide, you ll need: An AWS account, An Aviatrix license key ( to info@aviatrix.com if you don t have one) 71.2 Step 1: Create AWS Resources For AWS, we ll create a new VPC, EC2 instance, and enable the NLB service Step 1a: Create VPC There are a number of ways to create a VPC in AWS. We ll use the VPC Wizard, available in the VPC Dashboard. 358 Chapter 71. Hybrid Network Load Balancing (NLB)

365 Click the Start VPC Wizard button to launch the wizard. Then, select the VPC with a Private Subnet Only and Hardware VPN Access option. Finally, fill out the form that follows providing an appropriate CIDR block and VPC name Step 1b: Create EC2 Instance (Web Server) We ll create a T2-micro instance running Amazon Linux and Apache to handle the web server role. The steps we used to create the EC2 instance are shown below: Step 1: Create AWS Resources 359

366 360 Chapter 71. Hybrid Network Load Balancing (NLB)

367 Connect to the new instance via SSH. We temporarily associated an Elastic IP with this instance for convenience while configuring it. > ssh -i ~/aviatrix/demo/aws/aviatrix-demo.pem And, install the Apache package: > sudo yum install httpd Finally, create a simple index.html page in the doc root (/var/www/html/ for our installation): <html> <head> <title>welcome!</title> </head> <body> <h3>welcome to AWS</h3> </body> </html> Now, if we go directly to the instance EIP in a web browser we should see this: In the next step, we ll set up the NLB to route traffic to this instance so we will no longer need the EIP associated with this instance Step 1c: Configure the Network Load Balancer In the EC2 Dashboard, select Load Balancers, click the Create Load Balancer button, and finally select Network Load Balancer when prompted for the type: Step 1: Create AWS Resources 361

368 On Step 1 of the form that is displayed, give the NLB a name and select internet-facing for the Scheme. We ll only need one listener on port 80 for this demo, so the default configuration is sufficient. Under Availability Zones, select the VPC we created in step 1a and then check the only subnet in the table below that. 362 Chapter 71. Hybrid Network Load Balancing (NLB)

369 On Step 2, select New target group0 and provide a name. Be sure to change the Target type to ip instead of instance (we ll rely on this configuration later when accessing our remote site). Everything else will remain the default Step 1: Create AWS Resources 363

370 Step 3 requires us to select our target(s). For now, we only have one (our Linux EC2 instance that we created in the previous step. In the IP field, type in the private IP address of the EC2 instance that was created earlier. Keep the default port of 80 in the Port field and then click Add to list. 364 Chapter 71. Hybrid Network Load Balancing (NLB)

371 Review the configuration and click Create. Give the Load Balancer a few minutes to move out of the provisioning state into active. Once active, open a web browser and go to the public DNS name of the new load balancer Step 2: Create and Configure Remote Site Web Server The remote site can be any network not in AWS. For this demo, I ve provisioned an Ubuntu VM with Apache on my laptop s VMware Fusion environment. On this VM, I ve also added a simple index.html file: <html> <head> <title>welcome!</title> </head> <body> <h3>welcome to the Remote Site</h3> </body> </html> 71.4 Step 3: Set up Aviatrix in the Cloud Without a Direct Connect connection between the remote site and AWS, you won t be able to add this new VM to the NLB. However, Aviatrix can overcome this requirement with a few simple steps Step 2: Create and Configure Remote Site Web Server 365

372 Step 3a: Install and configure the Controller The Aviatrix Controller provides a single pane of glass to visualize all of your hybrid cloud networking connections. An example dashboard looks like this: Follow the installation instructions to get a Controller up and running in AWS. Once complete, open a browser and connect to the controller over https ( ec2 public IP>/). Login with the username admin. The password is the controller s private IP address. Follow the prompts to enter your address and click Run when prompted to upgrade the Controller to the latest version. When the upgrade is finished, login using admin/<private ip address>. Once you login, you will be prompted to change your password. After that you will see this screen: 366 Chapter 71. Hybrid Network Load Balancing (NLB)

373 Select AWS to configure your AWS account. And, then enter your Aviatrix customer ID and click Save: Finally, create an Aviatrix Controller account. You ll use this to login to the Controller. Aviatrix recommends selecting IAM role-based option for AWS access Step 3: Set up Aviatrix in the Cloud 367

374 Step 3b: Create a Gateway Next, follow the instructions to install an Aviatrix Gateway in this VPC. This will be where our remote site will connect. Once the Gateway is up, you should see it appear on the Controller s dashboard: 71.5 Step 4: Set up Aviatrix on your remote site Our final step is to add an Aviatrix Gateway at our remote site. Aviatrix provides a virtual appliance that can be downloaded from here. Download the appropriate appliance for your environment and spin up a VM. 368 Chapter 71. Hybrid Network Load Balancing (NLB)

375 Step 4a: Configure the Appliance At the prompt, enter help to see the options available. You ll want to set up a static IP address. The format of the command is: > setup_interface_static_address <static_ip> <netmask> <default_gateway> <primary_dns> <secondary_dns> proxy {true false} The configuration we used (on a VMware Fusion instance) looks like this: Once complete, open a browser and browse to the IP address you just configured for your controller. Follow the same initial steps as you did for the cloud (AWS) Controller. Once you get to Step 2 Datacenter Extension or Site2Cloud, stop and click on the Site2Cloud icon on the left Step 4: Set up Aviatrix on your remote site 369

376 Step 4b: Connect Remote Site to AWS In a separate browser window, log into the Aviatrix Controller hosted in AWS. Click on the Site2Cloud icon on the left and click + Add New button at the top. Select the correct VPC, enter a Connection Name, and change the Remote Gateway Type to Aviatrix. Finally, provide your edge router IP address for the Remote Gateway IP Address and populate the appropriate Remote Subnet. Then, click OK. 370 Chapter 71. Hybrid Network Load Balancing (NLB)

377 Once complete, select the connection from the table you just created. Click Download Configuration (NOTE: you may need to disable the popup blocker in your browser) Step 4: Set up Aviatrix on your remote site 371

378 Once downloaded, go back to the browser window with the Aviatrix Controller in the remote site. You should be on the Site2Cloud page. Click + Add New at the top. Then, scroll to the bottom and select Import. In the file open box, select the configuration downloaded in the previous step. Once complete, switch to the Aviatrix Controller hosted in AWS and go to the dashboard. You should see the 2 sites connected but with a red line. Once the link is established and the line representing the link turns green, we are all set. 372 Chapter 71. Hybrid Network Load Balancing (NLB)

379 One last step that we ll need to do is to tell the default gateway on the subnet where Aviatrix gateway is deployed that the next hop is the Aviatrix Gateway for traffic in AWS VPC private IP address range. The steps to make this change will depend on your individual router. You ll need to route all traffic destined for the AWS VPC private IP range ( /24 in my example) back to the Aviatrix Gateway Step 4c: Add Remote Site Web Server to the NLB Back in the AWS console, go to the Target Groups in the EC2 Dashboard. Click on the Target Group we created earlier and then click on Targets. You should have just one IP in the list right now. Click Edit and then click on the + icon at the top. Change the Network drop down to Other private IP address and then enter the private IP address of the Ubuntu Apache VM we set up earlier on the remote side. Click Add to list and then Register Step 4: Set up Aviatrix on your remote site 373

380 Once the remote VM is registered, verify that the NLB shows both targets as healthy. It may take a few seconds for the newly added IP to move from initial to healthy. After both target IP addresses are healthy, we are ready to test Step 5: Test First, let s open a browser window to the NLB s EIP. We should see the welcome message from one of the web servers. On my first attempt, I saw the remote site: 374 Chapter 71. Hybrid Network Load Balancing (NLB)

381 Next, let s turn off the web server on remote VM: > sudo systemctl status apache2 > sudo systemctl stop apache2 > sudo systemctl status apache2 The NLB target group reports the server as unhealthy quickly after: And, the browser, after refresh, shows the welcome message from AWS: Step 5: Test 375

382 Next, start Apache back up on the remote VM and wait for the target group to show both targets as healthy. Once both are healthy, shut down Apache on the AWS (or remove port 80 from the security group s allowed inbound ports): Wait for the NLB to show the AWS node as unhealthy: 376 Chapter 71. Hybrid Network Load Balancing (NLB)

383 Now, the browser, after refresh, shows the welcome message from the remote VM: Start Apache back up on the AWS instance (or add port 80 back to the security group): Step 5: Test 377

384 71.7 Conclusion Aviatrix makes balancing load between AWS and remote sites easy. But that s just the beginning. Aviatrix makes cloud and hybrid networking as simple, dynamic, and disposable as compute and storage. Read more about Aviatrix here. 378 Chapter 71. Hybrid Network Load Balancing (NLB)

385 CHAPTER 72 Datadog Integration 72.1 Summary The Datadog integration sends system metrics from Aviatrix Gateways and the Controller to your Datadog instance. Once enabled, all existing and new Gateways will send system metrics via an installed Datadog agent to the configured Datadog instance Prerequisites In order to complete the steps in this guide, you ll need: An Aviatrix License Key (CustomerID) Tip: if you don t have a license key A Datadog account and API Key Tip: Sign up for a Datadog account here. Once you have an account, you can create a new API Key from the Integrations~APIs menu. 379

386 72.3 Enable/Disable Integration Login to the Aviatrix Controller. Go to the Settings in the navigation bar and click on Logging. At the bottom of the page, find Datadog Agent: Change the status to Enabled and enter your Datadog API Key and finally click Enable What Data Is Collected Once enabled, the Controller will install and configure the Datadog agent on each of your Gateways and on the Controller automatically Host Name Metrics from Aviatrix Gateways will have a host name in this format aviatrix-gw-<gateway Name> The Aviatrix Controller will appear as: aviatrix-ucc-<controller Public IP Address> 380 Chapter 72. Datadog Integration

387 CHAPTER 73 Transit VPC Network - CSR1000v vs. Aviatrix 73.1 Introduction This document depicts a typical deployment of CSR1000v on AWS in a Global Transit VPC architecture. The limitations encountered on this architecture are explained in a series of videos. I hope this document helps you understand the advantages of Aviatrix over the CSR1000v Solution The setup To setup this demonstration we have followed the AWS Cisco-Based Transit network VPC. document. The AWS document runs a cloudformation script that deploys all the necessary CSR1000v Instances and Lambdas function to aid with the provisioning of new Spoke VPCs and route configurations Transit network VPC This video shows the Transit VPC resulting of running the AWS Cisco-Based Transit network VPC Oregon Spoke VPC This video shows how the Oregon spoke has been configured to work with CSR1000v Transit VPC Virginia Spoke VPC This video shows how the Virginia spoke has been configured to work with CSR1000v Transit VPC.. Feel free to skip as it mirrors Oregon s setup, main difference are the subnets. 381

388 73.3 Troubleshoot #1 - Troubleshooting route propagation This video shows how to enable route propagation. Route propagation is a step that is not documented but it is essential to make the Transit Architecture work. This video will save you a few hours of googling around to figure out why your transit VPC deployment is not working right off the box. In contrast the Aviatrix solution does not require any manual intervention, we will show how to create connectivity using Aviatrix in the next video Troubleshoot #1 - Troubleshooting route propagation using Aviatrix (Or lack of thereof) This video shows how to deploy gateways and create peering tunnels using Aviatrix Controller. Notice that Aviatrix install the necessary routes on the AWS Routing tables, without any need for manual interaction Ohio Spoke VPC This video shows the introduction of a third VPC, which by mistake utilizes the same ip range as the Virginia VPC Troubleshooting #2 - Making sense of the routing Now that the Ohio VPC has been created and it s not working due that it is using the same ip range as Virginia VPC. In this video we try to troubleshoot by trying to make sense of the routing tables both in the CSR1000v and the AWS side. Quickly we find that this is not an easy task nor it s scalable Troubleshooting #3 - IP overlap This video shows the worst case scenario: a wrongfully configured Spoke VPC taking over the ip range of a shared services VPC, bringing the whole network and it s services offline. 382 Chapter 73. Transit VPC Network - CSR1000v vs. Aviatrix

389 CHAPTER 74 Aviatrix Terraform Provider Aviatrix terraform provider is used to interact with Aviatrix resources. Click this link for details on setting up Aviatrix terraform provider on your system. The provider allows you to manage Aviatrix resources like account, gateway, peering, etc. It needs to be configured with valid Aviatrix UCC/CloudN s IP, and account credentials. Click this link < _ to read how to setup transit VPC using terraform Example Usage # Configure Aviatrix provider provider "aviatrix" { controller_ip = " " username = "admin" password = "password" } # Create a record resource "aviatrix_account" "myacc" { #... } 74.2 Resources Use the navigation to the left to read about the available resources, along with their examples. More resources will be added in future. 383

390 aviatrix_account Manages an Aviatrix cloud account. Example Usage provider "aviatrix" { controller_ip = " " username = "admin" password = "password" } # Create Aviatrix AWS account with IAM roles resource "aviatrix_account" "tempacc" { account_name = "username" account_password = "password" account_ = "abc@xyz.com" cloud_type = 1 aws_account_number = " " aws_iam = "true" aws_role_arn = "arn:aws:iam:: :role/aviatrix-role-app" aws_role_ec2 = "arn:aws:iam:: :role/aviatrix-role-ec2" } # Or you can create Aviatrix AWS account with access_key/secret key resource "aviatrix_account" "tempacc" { account_name = "username" account_password = "password" account_ = "abc@xyz.com" cloud_type = 1 aws_account_number = " " aws_access_key = "ABCDEFGHIJKL" aws_secret_key = "ABCDEFGHIJKLabcdefghijkl" } aviatrix_gateway Manages an Aviatrix gateway Example Usage provider "aviatrix" { controller_ip = " " username = "admin" password = "password" } resource "aviatrix_gateway" "test_gateway1" { cloud_type = 1 account_name = "devops" gw_name = "avtxgw1" vpc_id = "vpc-abcdef" vpc_reg = "us-west-1" vpc_size = "t2.micro" vpc_net = " /24" } 384 Chapter 74. Aviatrix Terraform Provider

391 aviatrix_tunnel Manages an Aviatrix tunnel Example Usage provider "aviatrix" { controller_ip = " " username = "admin" password = "password" } resource "aviatrix_tunnel" "test_tunnel1" { vpc_name1 = "avtxgw1" vpc_name2 = "avtxgw2" } aviatrix_transpeer Manages an Aviatrix transitive peering Example Usage provider "aviatrix" { controller_ip = " " username = "admin" password = "password" } resource "aviatrix_transpeer" "test_transpeer" { source = "avtxuseastgw1" nexthop = "avtxuseastgw2" reachable_cidr = " /16" } 74.3 Sample configuration to launch a full mesh network on AWS # Sample Aviatrix terraform configuration to create a full mesh network on AWS # This configuration creates a cloud account on Aviatrix controller, # launches 3 gateways with the created account and establishes tunnels # between each gateway. # Edit to enter your controller's IP, username and password to login with. provider "aviatrix" { controller_ip = "w.x.y.z" username = "admin" password = "Aviatrix123" } # Increase count default value to add more VPCs and subnets to launch more gateways together. variable "count" { default = Sample configuration to launch a full mesh network on AWS 385

392 } # Enter VPCs where you want to launch gateways. variable "vpcs" { description = "Launch gateways in different VPCs." type = "list" default = ["vpc-7a6b2513", "vpc-2ee4a147", "vpc-0d7b3664"] } # Enter Subnets within VPCs added above. variable "vpc_nets" { description = "Launch gateways in different VPC Subnets." type = "list" default = [" /24", " /24", " /24"] } resource "aviatrix_account" "test_acc" { account_name = "devops" account_password = "Aviatrix123" account_ = "abc@xyz.com" cloud_type = 1 aws_account_number = " " aws_iam = "true" aws_role_arn = "arn:aws:iam:: :role/aviatrix-role-app" aws_role_ec2 = "arn:aws:iam:: :role/aviatrix-role-ec2" } # Create count number of gateways resource "aviatrix_gateway" "test_gw" { count = "${var.count}" cloud_type = 1 account_name = "devops" gw_name = "avtxgw-${count.index}" vpc_id = "${element(var.vpcs, count.index)}" vpc_reg = "ap-south-1" vpc_size = "t2.micro" vpc_net = "${element(var.vpc_nets, count.index)}" depends_on = ["aviatrix_account.test_acc"] } # Create tunnels between above created gateways. resource "aviatrix_tunnel" "test_tunnel" { count = "${var.count * (var.count - 1)/2}" vpc_name1 = "avtxgw-${count.index}" vpc_name2 = "avtxgw-${(count.index+1)%3}" depends_on = ["aviatrix_gateway.test_gw"] } 386 Chapter 74. Aviatrix Terraform Provider

393 CHAPTER 75 Launch Aviatrix Controller Manually This guide walks you through how to launch the Controller manually from AWS Marketplace. Refer to AWS Startup Guide for a complete information. Before you launch the controller with IAM role, you must first create 2 IAM roles and its associated policies. Follow this link to have them setup. Then go to search for Aviatrix and select the image type you wish to launch. Note if you select the BYOL image, you need a customer ID from Aviatrix for launching gateways. Send to support@aviatrix.com or info@aviatrix.com to request a customer ID. Customer ID is not needed if you select utility images such as 5 Connections and 10 Connections. At the AWS marketplace console, select Manual Launch that takes you to EC2 console to launch with IAM role. Once you select Manual Launch, click at a region where you wish to launch the controller. 387

394 Once you are at AWS EC2 console, follow the steps below: 1. Select the instance size t2.large of 8GB of memory, which is the minimum instance required. 2. Select the VPC where the controller will be launched. 3. Subnet. Make sure the subnet you select is a public subnet with IGW as its default gateway, otherwise the controller will not be accessible as it won t have a public IP address. 4. Enable IAM role by selecting aviatrix-role-ec2 you created earlier, as shown below 5. Edit security groups to allow inbound TCP port 443 open to anywhere, as shown below: 388 Chapter 75. Launch Aviatrix Controller Manually

395 6. Use an Elastic IP address for the controller. 7. After launching the instance, note down the instance s Private IP address and Public IP. 8. Use a browser to log in to the console. Use a web browser, go to to access the controller console, as shown below. At the Sign In page, log in with username admin. The default password is the instance s Private IP address. You can retrieve the Private IP address from the AWS console instance panel, as shown below. 389

396 9. Once you are logged in, change your password for future accesses via the console. 10. Go through the initial installation of software. 11. After the installation is complete, log in again to the controller by typing at the browser: Troubleshooting tips: (a) If you experience Login timeout error, check your instance outbound security policy to make sure it opens on port 443. (b) If you cannot find your instance s public IP address, you may have launched the instance from a private 390 Chapter 75. Launch Aviatrix Controller Manually

397 subnet. The controller instance must be launched from a public IP address. (c) The controller needs to have its inbound port 443 open to AWS address ranges as Aviatrix gateways need to communicate to the controller on this port. For support, send to support@aviatrix.com. Enjoy! 391

398 392 Chapter 75. Launch Aviatrix Controller Manually

399 CHAPTER 76 Using Aviatrix to connect from one site to another site with IPsec VPN Aviatrix gateways can be used to connect one site to and other. This solution requires one Aviatrix gateway in each location that needs to be connected. These on-premise gateways can be deployed as virtual machines on VMware, KVM or Hyper-V Environment Requirements An Aviatrix Site to Site is accomplished by one gateway initialing an IPSec session with the other gateway. For this to work atleast one of the Gateway need to accessible via a public IP address. This can be accomplished by setting up a the public IP addres on the edge router in the premise and configuring NAT from that public IP address to the Aviatrix VM. The only ports that need to be forwarded from the edge router to the VM are UDP ports 500 and

400 On the other site, the second gateway does not need a public IP assigned to the Aviatrix Gateway. This second Gateway will reach outbound to the first Aviatrix GW (GW1) The last requirement is to configure static routes in the internal routers (default gateway of the Aviatrix VM) in both the sites. This static route should send traffic destined to the other site to the Aviatrix GW as the next hop. 394 Chapter 76. Using Aviatrix to connect from one site to another site with IPsec VPN

401 76.2 Steps to Configure IPSec connectivity Step 1: Install Aviatrix gateway in each site. Download and install the Aviatrix Gateways VMs by following instructions in this document Step 2: Configure Site2Cloud in Gateway 1 Note: In the Aviatrix terminology, Site2Cloud is the name of the feature that enables connections from one site (or datacenter) to other sites (including cloud environments). 1. Log into the Web UI of the first Gateway (GW1). 2. Click on Site2Cloud in the navigation pane. 3. Click on Add New Connection button Steps to Configure IPSec connectivity 395

402 4. Fill out the details in the Site2Cloud form as shown below. I. Remote Gateway IP as the public IP of the other Site II. Remote Subnet is the CIRD (or comma separated CIDRs) of the other site III. Local Subnet is the CIDRs in the local site. 5. Click OK. You will see the connection listed in the Site2Cloud UI. 396 Chapter 76. Using Aviatrix to connect from one site to another site with IPsec VPN

403 6. Click on the connection from the list. You will see Edit Site2Site options appear under the list. 7. Select Aviatrix in the Vendor dropdown. 8. Click on Download Configuration button. This will download a text file (.txt) to your local machine. 9. Log in to Gateway 2 s web UI on the other site (GW2). 10. Go to Site2Cloud page 11. Click on Add New Connection 12. Locate the Import button at the bottom of the screen. 13. Select the text file you downloaded from the other Gateway. This will auto populate the details in the form. 14. Click OK 15. This will start the IPsec negotiations between both gateways. You should see the connection status change to Up within a few minutes. Please reach out to info@aviatrix.com if you have any questions. doc version: Steps to Configure IPSec connectivity 397

404 398 Chapter 76. Using Aviatrix to connect from one site to another site with IPsec VPN

405 CHAPTER 77 Extending Your vmware Workloads to Public Cloud Overview Aviatrix Systems provides the next generation cloud networking solution built from the ground up for the public cloud. Aviatrix simplifies the way you enable site to cloud, user to cloud and cloud to cloud secure connectivity and access. The solution requires no new hardware and deploys in minutes. Aviatrix CloudN is a virtual appliance deployed in datacenter. Aviatrix Cloud Interconnect (ACX), also known as Datacenter Extension is a unique technology on CloudN. It manages your public cloud address space and allows rapid scaling of AWS Virtual Private Cloud (VPC) by removing the pain point of building secure connections to the VPCs. 399

406 ACX Key Benefits Manage Cloud Address Space No more spreadsheet to manage your cloud address space. Easy to Deploy Deployed without touching existing network infrastructure. Fast to Provision Provision a VPC with secure tunnel to datacenter in minutes. Simple to Use 1-click operation to create and delete VPC with secure tunnels. Rapid Scaling Creates multiple VPCs in any region with secure connectivity. Full Mesh Connectivity inter region VPC can be securely peered in minutes. IT Supported Self Service Work flow allow multiple users to create VPCs. Billing Visibility Support multiple AWS accounts for different departments, DevOps and projects Remote Access Capability. Built in VPN server allows remote workers to access VPC directly. Ideal for partners and remote workers How it Works Mix Layer 2 and Layer 3 Technologies CloudN uses a mixed Layer 2 and Layer 3 technologies whereas the CloudN virtual appliance behaves as a Layer 2 bridge and Gateway (launched by CloudN at VPC creation time) behaves as a Layer 3 router. The design of CloudN as a Layer 2 bridge makes it possible to build an overlay IPSec tunnel to AWS VPC without involving edge routers in 400 Chapter 77. Extending Your vmware Workloads to Public Cloud

407 the network. The design of Gateways as a Layer 3 router makes it possible for the VPC to fully utilize all AWS VPC underlying infrastructures and services without requiring any software agent reside in any of the instances. Instances within the VPC communicate with each other directly and transparently without involvement of Gateway. From the user s perspective, what CloudN creates is a standard VPC. CloudN views each VPC as the smallest autonomous environment, it allows you to create security policies to deny any subnet or hosts on premise to access any VPC. For example, you may want to block developers from accessing production VPC. By default, inter-vpc communication is blocked. By using VPC/VNet peering capability, you can establish direct secure tunnels among VPC in the same region or across different regions. Enterprise users can access instances seamlessly in all private and public subnets over the secure tunnel using instance private addresses. All instances on private subnets can reach back to enterprise. Optionally packets from instances on private subnets can reach Internet directly without being first sent back to the enterprise Dividing Subnets CloudN works by dividing the subnet where cloudn is deployed into sub segments (or smaller subnets). The VPC CIDRs created by cloudn are one of the sub segments. The mechanism is illustrated below. VPC in the below diagram could be replaced with a VNet. Where a local subnet /16 has a default gateway The subnet is divided into 4 sub segments. The default gateway and CloudN IP address fall into one segment. The rest of each segment is mapped to a VPC CIDR, in this case, the VPC CIDRs are /19, /19 and /19. If this subnet /16 is reachable from other network in the enterprise, then the instances inside each VPC takes private IP address as if they are on the local subnet /16. For users in the enterprise, it is as if they are communicating with hosts on the local network Pre Configuration Checklist AWS EC2 Account You need to have an AWS account to use most of the commands on CloudN. Note that CloudN support multiple cloud accounts with each one associated with a different AWS IAM account, but there needs to be at least one to start with Pre Configuration Checklist 401

408 Plan Cloud Address Space CloudN manages your cloud address space. Carve out an unused consective network address space in your datacenter. The CIDR block of this address can be determined by how many VPCs you will need and how big the address space you can allocate. For example, a CIDR block with /16 address range can create as many as 254 VPCs. Once you have created all the VPCs from the allocated address space, you can always allocate a new address space and launch a new CloudN virtual appliance Deploy the Aviatrix CloudN Virtual Appliance Reference the startup guide to deploy the virtual appliance. Check and make sure you can access the Aviatrix Controller dashboard and login with an admin account. The default URL for the Aviatrix Controller is: IP address of Aviatrix Controller> Configuration Steps Onboarding and create a cloud account Upon login to the controller for the first time, follow the onboarding process to create a cloud account that corresponding to an AWS IAM account. Aviatrix CloudN uses the account IAM credential to execute AWS REST APIs to create VPC and necessary resources Create a VPC and build an encrypted tunnel After going through onboarding steps, click ACX. Provide a name for the VPC you are about to create, select an AWS region, and click Launch. In a few minutes of time, a VPC, public subnet and private subnet in each AZ of the selected region, IGW and routing tables will be created; an Aviatrix Gateway will be launched and an encrypted tunnel will be created. You then can launch instances in the VPC and access the instances by their private IP addresses. Reapt the above step for more VPC with encrypted tunnel creations. 402 Chapter 77. Extending Your vmware Workloads to Public Cloud

409 CHAPTER 78 How to Build a Zero Trust Cloud Network Architecture with Aviatrix 78.1 What is Zero Trust network architecture? Zero Trust architecture came from the realization that perimeter security solutions such as edge firewalls are not sufficient to prevent data breaches. Lateral movement inside a network to scan and obtain target data has been the approach in the recent serious attacks. The idea of Zero Trust is to build walls inside the datacenter by network segmentation to prevent lateral movement and always authenticate and authorize users for all data access How to build a Zero Trust cloud network Classify data by network segmentation Separating production data from dev and test is the first step. Give them separate cloud accounts is the best practice to ensure isolation. Different business groups should have separate cloud accounts. The more the fine grained accounts the more micro segmentation goal is achieved. There should be zero connections among these networks by default. In public cloud such as AWS, using the above principles to build your cloud network results in isolated islands of VPCs. If one VPC is breached, it is impossible to gain access to other VPCs, thus significantly reduce attack surface. Aviatrix is a multi account platform that enables you to manage all cloud accounts from a single pane of glass Policy driven connectivity with stateful firewall rules The connectivity between VPCs and on-prem network should be policy driven. A network solution such as the AWS Global Transit Network with CSR is a opposite to Zero Trust architecture point of view as all VPCs and on-prem is built into a full mesh network. In contrast, 403

410 AWS Global Transit Network with Aviatrix meets Zero Trust architecture requirements where secure connection is established by organization policy. In addition to policy driven network connections, there must be firewall rules that govern data flow and reduce the connection scope. For example, you should consider place application and database in separate VPCs and setup a stateful firewall rule to only allow traffic initiated from application to access database, no the other way around. Aviatrix gateway stateful firewall enforces and logs all network events. Within a VPC, you can use AWS native security groups assoicated with instances to enforce policies for communications User access with authentication and authorization Users access cloud resources must be first authenticated. Certificate only based authentication is a weak solution as certificate can be stolen. Another insecure access method is Jump Host or Bastion stations. Multi factor authentication such as integrating with LDAP/DUO/OKTA and client SAML Single Sign On significantly improves authentication strengths. However authentication alone is not sufficient, Users access cloud resources must be authorized. The finer grained control you apply, the less literal movement a user can make even if accessing to the network is attained. With Zero Trust, you should only grant access to the required resources. User access activities must be fully audited. Every user initiated TCP session in the cloud network must be logged for audit and inspection. Aviatrix Enterprise OpenVPN Solution is the strongest secure client solution in the marketplace built for the public cloud Summary Zero Trust architecture is Never trust, always verify, a critical component to enterprise cloud adoption success. Aviatrix provides a rich set of capabilities that enables you to build a Zero Trust network for the public cloud. OpenVPN is a registered trademark of OpenVPN Inc. 404 Chapter 78. How to Build a Zero Trust Cloud Network Architecture with Aviatrix

411 CHAPTER 79 AWS Global Transit Network 79.1 AWS Reference Deployment Guide This document is published by AWS Answers for AWS Global Transit Network as Partner Offering Overview Aviatrix is a next generation cloud networking solution built from the ground up for the public cloud. For transit VPC design, Aviatrix provides one console for building, managing, monitoring and troubleshooting all aspects of your network connectivity. The console (controller) gives users the ability to implement Transit VPC design with a point-and-click (no CLI) as well as REST API. This configuration guide provides step by step instruction on how to build a highly available AWS Global Transit Network. Below is an architecture diagram of what a general AWS Transit VPC deployment looks like, where a Hub VPC (or Transit VPC) connects many Spoke VPCs to facilitate communication between the Spoke VPCs and on-prem network. 405

412 Pre Configuration Checklist Before configuring user VPC peering, make sure the following is completed. Pre Configuration Check List 1. Deploy the Aviatrix Controller 2. Check VPC Settings These prerequisites are explained in detail below Deploy the Aviatrix Controller The Aviatrix Controller must be deployed and setup prior to configuring VPC and site peering. Please reference the Aviatrix Controller getting started guide for AWS on how to deploy the Aviatrix Controller. Aviatrix Controller Getting Started Check and make sure you can access the Aviatrix Controller dashboard and login with an administrator account. The default URL for the Aviatrix Controller is: 406 Chapter 79. AWS Global Transit Network

413 ip of Aviatrix Controller> Check VPC Settings The VPC must have at least one public subnet to deploy the gateway. This means one subnet must be associated with a route table that has an IGW as its default route. If your hub VPC and spoke VPC are in the same region and you like to route the traffic over AWS peering, go to AWS console and configure the necessary AWS peering between the two VPCs Configuration Steps Make sure the pre-configuration steps in the previous section is completed before proceeding. The instructions in this section will use the following architecture. The CIDR and subnets may vary depending on your VPC setup; however, the general principals will be the same. In this example we have three VPCs: Transit VPC, spoke VPC in US-WEST1 and spoke VPC in US-EAST1. The corporate data center is located in California. The system will be configured such that all spoke nodes and sites will Configuration Steps 407