Configuration and Day 2 Operations First Published On: 05-12-2017 Last Updated On: 12-26-2017 1
Table of Contents 1. Configuration and Day 2 Operations 1.1.Top Day 2 Operations Knowledge Base Articles 2. vcenter Server High Availability 2.1.vCenter High Availability Overview 2.2.vCenter High Availability Technical Details 2.3.vCenter Server and Platform Services Controller 6.5 High Availability 2.4.vCenter High Availability Deployment - Basic workflow 2.5.vCenter High Availability Deployment - Advanced workflow 3. vcenter Backup and Restore 3.1.File-based Backup and Restore Overview 3.2.vCenter Server Appliance File-based Backup 3.3.vCenter Server File-based Restore 4. vsphere Certificate Management 4.1.Hybrid vsphere SSL Certificate Replacement 4.2.Hybrid Mode Certificate Replacement Walk-through 4.3.vSphere Certificate Management for Mere Mortals 2
1. Configuration and Day 2 Operations Learn about general day 2 operations such as configuring PSC HA with a load balancer, determining health of PSC replication agreements, and repointing vcenter Server to a different PSC. 3
1.1 Top Day 2 Operations Knowledge Base Articles Top Day 2 Operations Knowledge Base Articles Title URL Configuring Platform Service Controller HA in vsphere 6.5 (2147018) https://kb.vmware.com/kb/2147018 Determining replication agreements and status with the Platform Services Controller 6.X (2127057) https://kb.vmware.com/kb/2127057 How to repoint vcenter Server 6.x between External PSC within a site (2113917) https://kb.vmware.com/kb/2113917 4
2. vcenter Server High Availability Learn about how to ensure the uptime of vcenter Server, with information on High Availability and related topics. 5
2.1 vcenter High Availability Overview If you aren t familiar with vcenter High Availability, it is a new feature introduced in vsphere 6.5 and exclusively available for the vcenter Server Appliance (VCSA). When vcenter HA is enabled, a threenode vcenter Server cluster (Active, Passive, and Witness nodes) is deployed. vcenter HA provides an RTO of about 5 minutes for vcenter Server greatly reducing the impact of host, hardware, and application failures with automatic failover between the Active and Passive nodes. vcenter HA is included with the vcenter Server Standard license which means that no additional licensing is required. vcenter HA can also be enabled, disabled, or destroyed at any time allowing customers to easily take advantage of this new capability. There is also a maintenance mode that prevents planned maintenance from causing an unwanted failover. From an architecture perspective, vcenter HA supports both embedded and external Platform Services Controllers. An embedded Platform Services Controller instance can be used when there are no other vcenter Server or Platform Services Controller instances within the single sign-on domain. In other words, an external Platform Services Controller instance is required when there are multiple vcenter Server instances in an Enhanced Linked Mode configuration. When using vcenter HA with an external Platform Services Controller deployment, an external load balancer is required to provide high availability to the Platform Services Controller instances. There is little bene t to using vcenter HA without also providing high availability at the Platform Services Controller layer. Supported load balancers for Platform Services Controller instances in vsphere 6.5 include VMware NSX, F5 BIG-IP LTM, and Citrix NetScaler. 2.2 vcenter High Availability Technical Details 6
Learn more technical details on vcenter Server High Availability, first introduced in vsphere 6.5, with this innovative white board style presentation. Click to see topic media 2.3 vcenter Server and Platform Services Controller 6.5 High Availability Click to see the HTML page 2.4 vcenter High Availability Deployment - Basic workflow The basic workflow can be used in most scenarios in which all vcenter HA nodes run within the same cluster. As its name suggests, this workflow is very simple and automatically creates the passive and witness nodes. It also creates VMware vsphere Distributed Resource Scheduler (vsphere DRS) antiaffnity rules if vsphere DRS is enabled on the destination cluster and uses VMware vsphere Storage DRS for initial placement if enabled. Some flexibility is built into this workflow, so you can choose specific destination hosts, datastores, and networks for each node. This is a very simple, easy way to get a vcenter HA cluster up and running. Click to see topic media 2.5 vcenter High Availability Deployment - Advanced workflow The Advanced workflow is an alternative that can be used when the active, passive, and witness nodes are to be deployed to different clusters, vcenter Server instances, or even other data centers. This process requires the customer to manually clone the source vcenter Server instance for the passive and witness nodes and to then place those nodes in the chosen locations with the appropriate IP address settings. This is a more involved process, but it enables greater flexibility for those customers that require it. 7
Click to see topic media 8
3. vcenter Backup and Restore Learn about ways to Backup and Restore vcenter Server 9
3.1 File-based Backup and Restore Overview The vcenter Server Appliance (VCSA) 6.5 is full of new and exclusive features, one of which is the native file-based backup and restore. File-Based Backup This new out-of-the-box functionality supports the backup of the vcenter Server Appliance or Platform Services Controller (PSC). This includes both embedded or external deployments. To begin the backup workflow, log in the VMware vsphere Appliance Management Interface (VAMI) of the VCSA or PSC. Navigate to the summary tab and click the Backup button to launch the Backup Appliance wizard. The VCSA file-based backup feature requires no quiescing or downtime of the selected appliance. By default, the configuration and inventory data of a vcenter Server Appliance are backed up. There is also the option to backup the historical and performance data of the VCSA. The PSC will not have this option since all historical and performance data is kept in the VCSA database. Keep in mind that selecting this option could increase the backup time of the VCSA. The backup workflow will produce a set of files for the designated appliance. An option to encrypt the backup files using AES 256 is available by checking a box and entering a password. The encryption password is not stored and if lost, there is no way to recover those backup files. The backup files are then streamed to a backup target using one of the supported protocols: FTP(s), HTTP(s), and SCP. Once the backup workflow completes successfully, the files are visible at the backup target. 10
File-Based Restore In the case of a disaster and your VCSA or PSC is no longer available, it s time to put that backup to use. Make sure to have the original ISO used when deploying or upgrading your VCSA or PSC instance handy. During the restore workflow, enter the backup protocol and location used. If the option to encrypt your backup was selected, the same password used will be required to decrypt it. Without the encryption password, the restore of an appliance is not possible. As part of the restore workflow, a new VCSA or PSC is deployed retaining its UUID and system name (FQDN). There are opportunities to change a few of the appliance settings during a restore. The deployment and storage sizes can remain the same or increase, but can not decrease in size. The 11
appliance IP address, mask, gateway, and DNS can be changed. Two key points to remember when changing the appliance network settings. The first, if an appliance uses an IP address for its system name (FQDN), then it can not be changed. Second, in changing any appliance network settings remember to update DNS. The restore workflow will process to configure the appliance to the selected backup file. Once the restore workflow has completed, log in the vsphere Web Client to verify. 3.2 vcenter Server Appliance File-based Backup A new feature in vsphere 6.5 is vcenter Server Appliance File-Based Backup and Restore. This walkthrough demonstrates backup of the vcenter Server Appliance. Click to see topic media 3.3 vcenter Server File-based Restore This walkthrough demonstrates restore of the vcenter Server Appliance. Click to see topic media 12
4. vsphere Certificate Management Learn about SSL certificates in vsphere, including architecture and the use o the VMware Certificate Authority (VMCA) 13
4.1 Hybrid vsphere SSL Certificate Replacement VMCA Overview Over time, certificates within a vsphere environment have become much more important. Certificates ensure that communication between services, solutions, and users are secure and that systems are who we think they are. By default, VMCA acts as a root certificate authority. Certificates are issued that chain to VMCA where the root certificate of VMCA is self-signed as it is the end of the chain. These VMCA-signed certificates generate those thumbprint and browser security warnings you may be used to seeing because they are not trusted by the client computers by default. The VMCA acts as a central point in which certificates can be deployed to a vsphere environment without having to manually create Certificate Signing Requests (CSRs) or to manually install the certificates once they are minted. The VMCA, working in conjunction with its new purpose-built certificate store called the VMware Endpoint Certificate Store (VECS), has made managing certificates much easier than in prior vsphere releases. As shown in the graphic below, the VMCA operates within the Platform Services Controller (PSC). Depending on the topology of your installation, you can choose to deploy a vcenter Server with an embedded PSC or utilize separate external PSCs. The VMCA then issues certificates to any vcenter Servers and associated ESXi hosts that are registered to it. Many of the certificates issued by the VMCA are for internal service-to-service communication within vcenter Server. These services, also called Solution Users, use the certificates to authenticate to one another. As vsphere Users and Administrators, we do not interact directly with these services and therefore these certificates are less impactful to our overall certificate strategy. Note that a vcenter Server has four Solution Users while a PSC has one. A vcenter Server with an embedded PSC has four Solution Users as well. 14
In vsphere 6.0 we also added a reverse proxy to vcenter Server so than when we do need to communicate with vcenter Server services, that communication is all done via port 443 and secured by the Machine SSL certificate of the vcenter Server. The Machine SSL certificate becomes the primary way in which users secure communications with vcenter Server and the PSC. Remember those annoying web browser certificate warnings when accessing the vsphere Web Client? Those are caused by an untrusted (and perhaps self-signed) Machine SSL certificate. The real value of the VMCA is in the automation of replacing and renewing certificates without having to manually generate CSRs, mint certificates, then manually install those certificates. If you ve replaced certificates in a vsphere 5.x (or prior) environment then you know the challenges and time commitment involved in that process prior to the VMCA. The VMCA allows us to drastically reduce the overhead of the certificate lifecycle. I should note that use of the VMCA is not required. The VMCA can essentially be bypassed and custom certificates can be requested and installed for each of the different vsphere components, however, this comes with a higher operational cost. Additionally, it may introduce more opportunity for misconfiguration which could lead to a lower standard of security. Tread wisely. Next, let s take a look at some different operational models for the VMCA along with a recommendation on the best approach. The Subordinate CA Approach One of the operational models of the VMCA is to act as a Subordinate (or Intermediate) Certificate Authority. Initially, with the release of vsphere 6.0 and the VMCA, this was a rather attractive option for customers. As a sub CA to an already established Certificate Authority in an environment, the VMCA could issue certificates to vcenter Server and ESXi hosts that would be inherently trusted and easily get rid of those pesky self-signed certificate errors with ease. However, over time it became very apparent that the risk of this model has outweighed the benefit. From a security perspective, by having a Subordinate CA, a rogue administrator with full access to the PSC could mint fully trusted and valid certificates that are trusted all the way up to the organization s Root CA. In talking with our customers, many of them who operate in a highly security conscious manner, this type of risk is a deal breaker for the Security teams in those organizations. The Full Custom Approach The Subordinate CA approach sounded like a great win for operational simplicity but its downfall was the security risk. On the other end of the spectrum we have the Full Custom approach where every 15
certificate within the vsphere environment is replaced by a unique custom certificate minted by a Root CA. This approach is, in theory, the most secure but as previously mentioned, it introduces a lot more complexity and opportunity for misconfiguration, thereby impacting security negatively. It has a high operational cost in order to gain higher security which means generating a CSR for each vcenter Server and PSC VM, each Solution User, and each ESX host. This could be hundreds or thousands of CSRs to generate and certificates to manage. Once that s all done then you must worry about renewing all those certs or replacing revoked certificates. This is definitely a tradeoff in simplicity and time in order to gain more security. The Rise of the Hybrid Approach The question now becomes, How can we take advantage of the Certificate Lifecycle benefits of the VMCA (and VECS), mitigate the risk of a subordinate CA, and reduce the overall time and effort it takes to manage all of this? And thus, a hybrid model was born. A few short months after vsphere 6.0 was released, Mike Foley wrote about a new approach in a post titled, Custom certificate on the outside, VMware CA (VMCA) on the inside Replacing vcenter 6.0 s SSL Certificate. With this hybrid approach, custom certificates are used for the Machine SSL certificates of the Platform Services Controller and vcenter Server VMs and then the VMCA is left to manage the Solution Users and ESXi host certificates. This method of certificate lifecycle management does not use the VMCA as a subordinate CA. It lets the VMCA function as an independent CA and issue the internal Solution User and ESXi host certificates. Meanwhile, custom certificates from an external CA will adhere to the controls of the Enterprise PKI policies. Put these two pieces together and this hybrid approach reduces the work of certificate lifecycle management for Operations while increasing security with the custom certificates. This model even meets strict auditing standards such as with the IRS. Let s look at an example. Consider a vsphere 6.x environment that contains 4 Platform Services Controllers and 6 vcenter Servers across 2 sites with 50 hosts per vcenter Server. Let s look at replacing certificates in this environment while comparing and contrasting the Subordinate CA, Full Custom, and Hybrid approaches we discussed earlier. 16
First, if we were to use the Subordinate CA approach we would want each PSC in the SSO Domain to also be a Subordinate CA. While not a requirement, this ensures consistency across the environment and will make life easier if there is ever a need to repoint a vcenter Server from one PSC to another. Given that each PSC will be a Subordinate CA, we need to generate a CSR for each of those PSC Sub CAs and submit to the Root CA. Once that is completed and the VMCAs are fitted with their new signing certificates, the VMCAs can then issue Solution User, Machine SSL, and Host certificates. So, in this environment we only have to manually manage 4 CSRs to get 4 certificates. Not bad. But remember, most security teams will forbid this type of deployment because of the risks involved. Next, let s go to the Full Custom approach. Recall that this method uses custom certificates for everything. So, we need to generate CSRs for the Solution Users, Machine SSL, and Hosts. This adds up to 338 CSRs to generate the required certificates. Whoa, that s going to take some time. Not only that, but when it comes to renewal time you get to do this all over again not to mention that certificates could get revoked, hosts could be replaced, and other operations that would require a new certificate. You should be able to see that this causes the most management overhead but it is the most secure way of deploying certificates. There are some environments that may require this approach but for a normal production environment this should not be required. Last, the Hybrid approach mashes components of the previous two together to get the best of both worlds. We still need to manually generate a handful of CSRs for the Machine SSL certificates of each PSC and vcenter Server which gives us 10 certificates to install and manage over time. And by letting the VMCA do its thing, we gain operational benefits as we grow our datacenter. We don t have to mint a new CSR for every new ESXi host we add into a cluster. We just add it and let VMCA do it s thing. The same is true for Solution Users. Below is a table that captures the totals for each of the methods we ve discussed. Conclusion What we have found in talking to these customers that are embracing the hybrid approach is that security teams are most concerned with securing the control plane of the administrators with certificates issued by the security team via their enterprise PKI. The hybrid approach addresses that for securing access to vsphere by replacing the Machine SSL certificate. Per best practices, access to ESXi management should be limited in nature and only done on an isolated network. To address 17
administrative access to functions like the ESXi UI (introduced in 5.5 U3 and 6.0 U2), the VMCA CA certificate can be exported and added to the Trusted Root Certification Authorities container in an Active Directory group policy. As you can see, the Hybrid approach is the best of both worlds. It addresses the security needs of the Security Team by protecting access to vcenter Server while it also addresses the operational needs of the IT team. 4.2 Hybrid Mode Certificate Replacement Walk-through The VMware Certificate Authority (VMCA) was first introduced in vsphere 6.0 to improve the lifecycle management of SSL Certificates. This click-by-click walkthrough has been created to serve as a guide for planning a hybrid mode certificate deployment. Click to see topic media 4.3 vsphere Certificate Management for Mere Mortals 18
19
20
21
22
23
24
25
26
27
28
29
30