Running the vsan Witness Appliance in vcloud Air First Published On: 02-03-2017 Last Updated On: 10-10-2017 1
Table of Contents 1. Overview 1.1.Introduction 1.2.2 Node & Stretched Cluster Basic Requirements 1.3.Choosing vcloud Air for the Third Site 1.4.How can a vsan Witness be run in vcloud Air? 2. Uploading the vsan Witness Appliance to vcloud Air 2.1.The vsan Witness Appliance OVA & OVF 2.2.Appliance Size Considerations 2.3.Uploading the Witness Appliance OVF to vcloud Air 3. Networking in vcloud Air Required 3.1.Networking in vcloud Air 3.2.Data Site Configuration 4. Deploying the vsan Witness OVA in vcloud Air 4.1.What's Next? 4.2.Once the vsan Witness Appliance has been deployed 2
1. Overview Overview of running the vsan Witness Appliance in vcloud Air 3
1.1 Introduction A common question asked about vsan 2 Node and Stretched Clusters configurations, is Can I run a vsan Witness Appliance in vcloud Air? It is a great use case for a virtual machine that needs to run in a 3rd site, away from, in the case of vsan Stretched Clusters, 2 different sites, and from a single site when using vsan 2 Node for small deployments. This is now supported. 1.2 2 Node & Stretched Cluster Basic Requirements VMware vsan introduced 2 Node and Stretched Cluster capability with the release of vsan 6.1. Looking at a basic 2 Node & Stretched Cluster schematic, the Witness Appliance can be seen in a third site, with hosts in both the Preferred and Secondary sites. The vsan Witness Appliance must reside in a different location. Stretched Clusters and 2 Node configurations are not supported when the vsan Witness Appliance is running in the same location as either side of the compute/data nodes. There are latency requirements between the Witness Appliance and each data site, as well as throughput. Latency requirements are 200ms RTT for Stretched Clusters with up to 10 hosts per site 100ms RTT for Stretched Clusters with more than 10 hosts per site (up to the max of 15) 500ms RTT for 2 Node configurations. Bandwidth requirements are entirely dependent on the amount of components on vsan. These are covered in the Stretched Cluster Bandwidth Sizing Guidance documentation previously as well as the Stretched Cluster and 2 Node Guide. 4
1.3 Choosing vcloud Air for the Third Site There are some things to consider when choosing vcloud Air as a location for the location of the Witness Appliance. Running the vsan Witness Appliance in vcloud Air has a some distinct advantages: No third site to maintain No hardware or underlying infrastructure to maintain Run run the appliance No facilities to maintain Easy to grow as needed Included remote network capabilities VPN connectivity Easily manageable firewall rules Easily configurable networking 1.4 How can a vsan Witness be run in vcloud Air? The setup process isn t significantly difficult, but there are some distinct requirements. The vsan Witness Appliance must be uploaded to vcloud Air, networking must be in place between the vsan cluster and vcloud Air, as well as a few more tasks to make this work successfully. This document covers how to accomplish these tasks, showing how to run a vsan Witness Appliance in vcloud Air. 5
2. Uploading the vsan Witness Appliance to vcloud Air An overview of the process of uploading the vsan Witness Appliance OVA into vcloud Air. 6
2.1 The vsan Witness Appliance OVA & OVF The vsan Witness Appliance OVA This appliance is provided in the Open Virtualization Appliance (OVA) format from VMware. This OVA is a handy-dandy file that includes the vsan Witness Appliance, as well as some logic to allow an administrator to deploy the appliance in with different characteristics. It also provides the ability to enter the default password for the appliance, enhancing the ability to provide a password which meets an organization s security requirements. There are different versions of the Witness Appliance for different builds of vsan, but are mostly similar. This article will discuss using the vsan 6.5 Witness Appliance. The OVA format of the appliance allows for a single file download of the Witness Appliance. While this works very well with vsphere, the vcloud Air interface does not allow OVA appliances to be imported into vcloud Air. Because of this, the vsan Witness Appliance must be converted to an OVF format. The VMware OVF Tool can be used to convert an OVA to an OVF fairly simply. For starters, download the vsan Witness Appliance for your version of vsan, then download the OVF tool and install it. The syntax for converting an OVA to OVF in Linux/OS X is: 1 ovftool /path/vsan-witness-appliance.ova /path/vsan-witness-appliance.ovf The vsan Witness Appliance has now been converted to the OVF format. This consists of multiple vmdk (virtual disk) files, an.ovf file, a.mf (manifest), and a.cert file (to ensure the.ovf maintains an integrity check). The vsan Witness Appliance OVF Importing the OVF and OVA are the same when using vsphere, but remember that vcloud Air doesn t support importing an OVA. It is also important that some changes need to take place in the.ovf file, which contains the appliance s settings, before it is a good fit in vcloud Air. Using a preferred text editor, some changes will need to be made to the.ovf file. The.ovf includes more disks that it actually needs, to meet the needs of the different profiles that an admin could deploy. The appliance has a few sizes, including Tiny, Normal/Medium, and Large. The Tiny size is the only one that does not include any 350GB vmdks. The Medium includes one, and the Large includes three. 2.2 Appliance Size Considerations Appliance Size Considerations The appliance size is important to consider, especially since our appliance will be consuming space in vcloud Air. In the case of running the vsan Appliance for a 2 Node Direct Connected cluster, some of the larger disks could be dispensed with. Larger configurations using the normal or large size will require the larger vmdks. The space the appliance requires, regardless of which profile it defaults to, will be the entire space the OVF could consume. So the boot vmdk (12GB), the cache disk vmdk (10GB), the tiny data vmdk (15GB), the normal data vmdk (350GB), and the 2 additional large data vmdks (350GB & 350GB) are uploaded. A The tiny profile consumes 27GB, the medium profile consumes 372GB, and the large consumes 1,072GB. 7
Default Appliance Size The Witness Appliance profile that the OVF defaults to, is the normal or medium size. This is indicated by ovf:default= true. This will consume at least 372GB. To change the default deployment ovf:default="true" can be added to an alternative configuration, such as "tiny" below. The default "normal" will need to be changed to ovf:default="false". Choose which ever size is best for your environment as the default. Remember that all of the vmdks are uploaded, regardless of whether they are in the default profile or not. Removing some disks In the case of using tiny or normal, some drives can be removed to keep the capacity in vcloud Air smaller. To keep the capacity size smaller, because tiny is suitable for 2 Node needs, below is an example of removing the three 350GB disk entries in the References and DiskSection section of the.ovf file. 8
Before: With disk-1, disk-3, disk-4 & each capacity entry of 350 After: With disk-1, disk-3, disk-4 & each capacity entry of 350 removed 2.3 Uploading the Witness Appliance OVF to vcloud Air Uploading the Witness Appliance OVF to vcloud Air The next step is the actual process of uploading the Witness Appliance OVF to vcloud Air into a catalog that may be deployed from. The upload process can be performed using either the vcloud Air upload UI. The OVF tool is also an option. The OVF Tool makes things pretty easy once the syntax is correct. The syntax is as follows: 1 ovftool --sourcetype="ovf" "VMware-VirtualSAN-Witness.ovf" "vcloud://vcloudairurl? org=yourorg&vdc=vdcname&vapptemplate=templatename&catalog=catalogname" A couple switches have to be used to handle the upload process. This is because our OVF has a EULA that has to be accepted, as well as the fact that the manifest file (.mf) will not match up with the original after the modifications to profile and disk have been made. 1 ovftool --skipmanifestcheck "VMware-VirtualSAN-Witness.ovf" "vcloud://vcloudairurl? org=yourorg&vdc=vdcname&vapptemplate=templatename&catalog=catalogname" Uploading from a windows environment to a vcloud Air instance is easy as a result. 9
*Update & quick note: Do not use the allowextraconfig parameter. This will not allow the witnessswitch to be created. Logging into vcloud Air, and looking at the catalog, the appliance can be seen. 10
3. Networking in vcloud Air Required Configuring the networking for the vsan Witness Appliance in vcloud Air. 11
3.1 Networking in vcloud Air Networking in vcloud Air In vcloud Air, multiple independent networks can be created with visibility across different Virtual Applications (vapps), Virtual Data Centers (vdcs), and sites external to vcloud Air. Some of the networking setup can be accomplished through the basic interface, and other pieces must be configured through the vcloud Director interface that makes up the backend of vcloud Air. This post is going to focus on using basic networking in vcloud Air. Advanced networking in vcloud Air is similar, but not required. Creating Networks in vcloud Air The vsan Witness Appliance, includes two NICs (vmnics), which are each tied to an independent vsphere Standard Switch. The first NIC is attached to vswitch0 and the second NIC is attached to the witnessswitch. Attached to vswitch0, is a VMkernel interface (vmk0) that is used to manage the vsan Witness Appliance. The second switch (witnessswitch) has a second VMkernel interface (vmk1), that is provided to handle vsan Witness traffic. These two interfaces, vmk0 & vmk1, can be on the same segment, or they may be on different segments. One, and only one, of them must be tagged (vmk1 by default) for vsan traffic. This is traffic that is destined for communication with the vsan network in traditional vsan 2 Node & Stretched Cluster Configurations. In configurations using Witness Separated Traffic, vmk1 would need to be able to communicate with the VMkernel interfaces that have a traffic type of witness. To demonstrate independent networks for each traffic type, management vs. vsan, two networks can be created from within the vcloud Air interface. Simply chose Add One, and enter the relevant network settings. As can be seen in the illustration, one will be used for Management Traffic and the other will be used for Witness Traffic. Firewall Rules Firewall Rules are important to allow vsan Witness Appliances to be able to communicate across networks both internal and external to vcloud Air. The networks above are on the 192.168.109.0/24 and 192.168.110.0/24 ranges. 12
Under the Gateways tab, in the Firewall Rules section, new rules may be added to properly allow for traffic. Four rules are created below, two for each network. Management Network Manage the Witness as a Virtual Host Allow-ROBO-IN The source of traffic is the remote lab network of 192.168.1.0/24, destined for the vsan Witness Appliance s management interface (vmk0) Allow-VCA-OUT The source of traffic is from the Witness Appliance s management interface, destined for the vcenter server managing the Witness Appliance as a host. vsan Traffic Network Metadata updates Allow-ROBO-WIT-IN Allow traffic to the Witness Appliance vsan tagged interface from data nodes at a ROBO site. Allow-VCA-WIT-OUT Allow traffic to the Data Nodes interfaces for metadata traffic from the Witness Appliance. These could be normal vsan tagged interfaces or witness tagged interfaces when using Witness Traffic Separation. *Notice in the illustration, for the purpose of this post, all protocols/ports were allowed between networks to quickly deploy the network and ease configuration. In production, the minimum requirement for ports an protocols for vsphere and vsan need to be put in place. These ports can easily be found in KB Article 1012382. **Also notice that the Allow-VCA-OUT and Allow-VCA-WIT-OUT rules allow for any destination. While these could be locked down to only the ROBO lab site, they have been purposefully allowed access out to the Internet as well. That is because these rules are being used for some other testing that is unrelated. IPSec VPN Setup in vcloud Air The next portion of the configuration requires using the vcloud Director interface. Clicking Manage Advanced Gateway Settings will launch a different interface depending on whether the vcloud Air instance has basic or advanced networking as the default. Again, this post will cover basic networking. Additionally, in this post, only a single IPSec VPN will be configured for my 2 Node vsan instance. In a Stretched Cluster configuration that is located in 2 different sites, an IPSec VPN will be required to each individual site. 13
When using advanced networking, the interface will show the gateway services. Using basic networking, we ll need to go into Edge Gateway Services to perform some of the next steps. Configuring DHCP will make it easy to get our the vsan Witness Appliance upon initial boot up. The DHCP tab is where this is setup. Next, to configure one or more IPSec VPNs, select the VPN tab. 14
Setting up a VPN from within vcloud Air is pretty straight forward, provided all the addresses/settings are correct. A name is required, and it is a good practice to include a description. The Establish VPN to setting allows us to setup a VPN internally within vcloud Air, or to an external IPSEC VPN remote network. Because the ROBO site will not be in vcloud Air, a remote network is chosen. The two networks created earlier vsan Witness Management and vsan Witness Traffic are going to be included in this multi-tunnel IPSec VPN. In the example to the right, I setup a single peer network (192.168.1.0/24), because I wanted to keep the lab simple. Multiple networks can be configured on both sides of the VPN. If adding a second network to the peer network, be sure to include appropriate firewall rules. The Local ID is the external IP for the vcloud Air edge gateway (that s how we re exiting vcloud Air and getting to the ROBO site/lab. The Peer ID is the external IP for the ROBO site/lab, and the Peer IPcould possibly be different. The Peer IP happens to be the same as the Peer ID in this case because the ROBO site has a router that allows for IPSec VPN capability. The VPN is between vcloud Air and the ROBO site segment of 192.168.1.0/24. If another appliance or VPN appliance were behind the ROBO site router, the Peer ID would b the external IP, and Peer IP would be the IP of the appliance. Several different encryption algorithms are available, and I chose AES-256 in this case. Make sure to write down the Shared Key, and take note of the MTU setting. More detailed info on how to setup a VPN can be found in KB Article 2051370. 15
16
3.2 Data Site Configuration Data Site Configuration Setting up the IPSec VPN in vcloud Air is half of the task of getting networking properly configured. Different solutions are going to have different settings when it comes to setting up a VPN with vcloud Air. Because of this, I won t go into significant detail on the setup in this post. It is important though, to remember to configure firewall rules, any Network Address Translations (if necessary), and have all necessary ports open. While it isn t necessary to have a single segment on the remote side, it is possible. A single segment can make things easy and dispenses with the requirement of the use of things like a Virtual Tunnel Interface (vti) in the configuration. Here are a few external links that might provide some guidance on how to configure different solutions to connect with an IPSec VPN to vcloud Air: VMware KBTV More detail: How to connect your Datacentre to vcloud Air over IPSec VPN Not supported by VMware, but worthy of mention: Creating a VPN between a Cisco ASA and vcloud Air Creating a site-to-site VPN from Sonicwall to vshield Edge in vcloud Director 5.1 Also ensure the IPSec VPN configuration includes settings to handle dead peer detection or other mechanism to bring the VPN up in the event it were to go down. Connected Once the vcloud Air VPN is properly communicating with the remote site, everything should show green. When deploying 2 Node vsan clusters, a single VPN connection will likely be the only one required. When deploying Stretched Clusters across sites, it is necessary to create another VPN to the secondary site. While not addressed in this post, Static Routing may be required to ensure that traffic from the Witness Appliance to each site only uses the appropriate VPN. 17
More information on Stretched Cluster Witness routing can be found in the Stretched Cluster and 2 Node guide. Summary Now that a working connection is in place, the Witness Appliance can be deployed, configured. Once complete, it can be added to a vsan 2 Node or Stretched Cluster configuration. Proper networking is required to allow the vsan Witness Appliance to communicate with 2 Node or Stretched Cluster configurations. 18
4. Deploying the vsan Witness OVA in vcloud Air Deploying the vsan Witness Appliance from a template in vcloud Air. 19
4.1 What's Next? With the vsan Witness Appliance OVA converted and uploaded to vcloud Air, and connectivity in place to the networks that will be used, the next step is to deploy an appliance. This can be done from the normal vcloud Air interface, but we won t be able to power on our Witness Appliance due to a password missing. It will be easier to accomplish this using the vcloud Director interface of vcloud Air. From the vcloud Air Dashboard, select Manage Catalogs in vcloud Director. This will open up the vcloud Director interface in the Catalog section and select a catalog. The catalog the Witness Appliance was uploaded previously, was VSAN_WITNESS. A catalog can contain multiple vapps, which can have one or more virtual machines in it. After choosing the vsan_witness_6.5 and selecting Add to My Cloud wizard will start the process of deploying this vapp. Click to see topic media 4.2 Once the vsan Witness Appliance has been deployed Once the vsan Witness Appliance has been deployed 20
Now that the Witness Appliance is deployed, let s see if it is accessible. In the vcloud Director interface, select vapps and open the vapp that was just deployed. Inside the vapp, there is only a vsan Witness Appliance. Right clicking on the VM will present the option to open a console and see the IP address of the VM. 21
The management VMkernel interface (vmk0) successfully obtained an IP address from the vsan Witness Management network. If the VPN connection is properly configured, vmk0 should be pingable from the remote site. In this example, the vcenter Server is running on the remote site. It should be possible to add vsan Witness Appliance to vcenter. The Witness Appliance can be seen in the vsphere Web Client. Notice also that vmk1 has an IP address from the DHCP pool configured previously. 22
When using vsan 6.1 or 6.2, which are built into corresponding builds of ESXi 6.0, additional routing might be required if vmk1 is on a different segment than vmk0, the management VMkernel. This is because the default gateway used by vmk0 is also used by other VMkernel interfaces. In ESXi 6.5, different gateways can be configured for different VMkernel interfaces. If using DHCP in vcloud Air, provided the gateway is configured properly, ESXi 6.5 will use the gateway provided by the DHCP service and not the default vmk0 gateway. Now that the Witness Appliance has been deployed and added to vcenter, it is important to ensure connectivity between the vmk1 and the vsan ports at the remote site. In the remote site, vmk0 has been tagged to accommodate vsan Witness Traffic. Host1 192.168.1.21 Host2 192.168.1.22 Witness 192.168.110.102 From the Witness to Host 1 and Host 2 Going through the vsan Cluster Wizard, vsan should be setup properly. 23
Once configured, select the Monitor tab from the vsan Cluster to run the vsan Health Check. It should report a Healthy vsan Configuration. Success! Now the remote/lab 2 Node vsan cluster is using a Witness Appliance in vcloud Air! Summary This series has given a brief overview of how to run a vsan witness in vcloud Air. Uploading a Witness Appliance, configuring networking, and deploying the Witness Appliance are not a difficult task, but do have specific requirements. The Stretched Cluster and 2 Node guide covers the bulk of the requirements specific to the use of a Witness. Using vcloud Air as a tertiary location is a very viable solution when another physical site is either not possible or not cost effective 24