IBM Cloud for VMware Solutions Standard Reference Architecture Implementation Guide. Date: 01 Oct 2016 Version: 1.3

Transcription

1 IBM Cloud for VMware Solutions Standard Reference Architecture Implementation Guide Date: 01 Oct 2016 Version: 1.3

2 1 Document description 1.1 Purpose This document provides detailed steps on implementing the Standard Reference Architecture of the IBM Cloud for VMware Solutions offering. Because the implementation steps will change frequently as automation steps are introduced, always verify that you have the most current version of this document before starting any deployment. 1.2 Summary of changes This section records the history of significant changes to this document. Only the most significant changes are described here. Version Date Author Description of change May, May, June, Sept, 2016 Simon Kofkin-Hansen Bob Kellenberger Daniel Arrieta Alvarez Daniel de Araujo Frank Chodacki same same Same Initial distribution of document Updates in many areas including changes in SoftLayer ordering process and replacing sitespecific system and variable names. Updates to match Standard Reference Architecture v1.2 including changing to two private network VLANs. Added VSAN preparation and configuration steps using the storcli utility. Copyright 2016 IBM and VMware Page 2 of 143

3 Table of Contents 1 Document description Purpose Summary of changes Overview Standard architecture overview Deployment overview Prerequisites Conventions Acquire network infrastructure Create VLANs Acquire additional subnet Acquire infrastructure servers and external storage ESXi bare metal servers Utility server Acquire licenses for VMware components Merging licenses for VMware components Configure utility server Secure access to the environment Install utilities Install vsphere Client Add secondary network interface Trunk VLANs to each of the ESXi hosts ESXi hardware validation CPU, memory, and network hardware verification Change advanced power management settings on the ESXi hosts VLAN trunking and jumbo frames verification Verify SoftLayer trunking and SoftLayer jumbo frames configuration Configure an additional management network on all hosts Create new VMkernel connection Create new port group Copyright 2016 IBM and VMware Page 3 of 143

4 12 Configure Active Directory Domain Controller, CA, DNS, and NTP Deploy three Windows VMs Configure NTP client and time sync Install Active Directory and DNS server Configure DNS forwarders Configure Active Directory Configure DNS zones and transfer zones Install offline root certificate authority server Configure root certificate authority server Configure the root CA settings Configure subordinate certificate authority server Configure the subordinate CA settings Adjust vsphere host settings Add ESXi hosts to the active directory domain Deploy Platform Services Controller Install Platform Services Controller Join the Platform Services Controller to the Active Directory Deploy the vcenter Server Appliance Install the vcenter Server appliance Add new licenses for the vcenter Server Appliance Assign the newly added license to the vcenter Server Appliance Assign the vcenteradmins domain group to the vcenter Server administrator role Configure the SDDC cluster Create the cluster Add hosts to the cluster Create the resource pools on the cluster Create a Distributed Virtual Switch Create a Distributed Virtual Switch Enable jumbo frames on the vds-sddc and vds-sddc-ext distributed switches Create new port groups in the distributed switch Attach the ESXi hosts to the distributed switch Copyright 2016 IBM and VMware Page 4 of 143

5 18.1 Attach the ESXi hosts to the private distributed switch, configure the VMkernel network adapters, edit the existing, and add new adapters as needed Attach the ESXi hosts to the public distributed switch Migrate the Platform Services Controller, vcenter Server, and Domain Controller instances from the standard switch to the distributed switch Define Network I/O Control Shares values for the different traffic types Migrate the last physical adapter from the standard switch to the distributed switch Enable VSAN Verify VSAN network connectivity Configure VSAN vmkernel IP multicast Identify SSD drives Enable vsan on the cluster Rename the Virtual SAN datastore Migrate existing VMs to VSAN Set the virtual machines to the default Virtual SAN storage policy Enable vsphere HA for the Cluster Assign the newly added license to the vcenter Server Appliance NSX configuration Prepare permissions for NSX Deploy the NSX manager Connect the NSX Manager to the Management vcenter Server Add new licenses for the NSX Assign the newly added license to NSX Deploy the NSX Controllers Configure an IP pool for the NSX controller cluster Deploy the NSX Controller cluster Configure DRS affinity rules for the NSX Controllers Prepare the ESXi Hosts for NSX Install the NSX kernel modules on the management cluster ESXi hosts Configure the NSX logical network Configure the Segment ID allocation Configure the VXLAN networking Copyright 2016 IBM and VMware Page 5 of 143

6 26.3 Configure the VXLAN networking Configure the transport zone Deploy and configure gateway Deploy Network Exchange Logical Switch Deploy and configure NSX Edge Create edge Configure additional interfaces SNAT rules and static routes Create Source NAT rules allowing VMware solutions to access public network Create static route allowing VMware solutions to access SoftLayer private network Reconfigure hosts and utility server to use NSX Edge Replace certificates Create and add a Microsoft Certificate Authority template Obtain custom certificate and replace the Platform Services Controller certificate Obtain custom certificate and replace the vcenter Server machine certificate Obtain and replace the NSX Manager SSL certificate Configure lockdown mode on all ESXi hosts Appendix A VLANs and subnets Appendix B Bare metal summary Appendix C Host servers Appendix D Management servers Appendix E Userids Appendix F Edge IP addressing Appendix G Active Directory groups Appendix H Active Directory users Appendix I Acquire VLANs and subnets via ticket Copyright 2016 IBM and VMware Page 6 of 143

7 2 Overview IBM Cloud for VMware Solutions allows existing VMware virtualized datacenter clients to extend into the IBM Cloud. This permits uses like capacity expansion into the cloud (and contraction when not needed), migration to the cloud, disaster recovery to the cloud, backup into the cloud, and the ability to stand up a dedicated cloud environment for development, test, training, or lab. This is accomplished by exploiting the operational benefits and agility of IBM Cloud with the ability to quickly expand or contract the environment to meet business needs. Two initial architectures have been defined to ensure standardized deployments that have been tested and verified. The advanced architecture is for large scale deployments and has been certified to be compliant to the VMware validated design. The standard architecture is a smaller footprint deployment suitable for deployments of up to 10,000 average sized virtual machines while still providing high levels of availability and scalability. This document provides detailed, step-by-step instructions for provisioning this standard architecture. The entire reference architecture can be viewed in the IBM Cloud Architecture Center. 2.1 Standard architecture overview The standard architecture provides a VMware environment complete with the high availability, load balancing (VMware DRS), high performance storage, and flexible networking that seamlessly extend an on-premises VMware deployment into a hybrid cloud solution based on IBM Cloud. It also includes automation and orchestration for provisioning customer workloads into this cloud. This architecture specifies a configuration with a minimum of four physical servers sized to run over 120 average sized virtual guests (2 vcpu, 8 GB memory, 70 TB storage). Having a pre-defined server configuration ensures consistency across different implementations and provides a consistent platform for extending with additional services. Additional servers of the same configuration can be added, either initially or later, to provide additional capacity. These servers will be deployed with VMware vsphere (ESXi) 6.x licensed from IBM Cloud. Shared storage is provided using VMware VSAN, which clusters the local storage of the ESXi hosts and provides single tenant, highly available storage with very high IOPS. As additional hosts are added to meet additional capacity demands, the storage simultaneously grows. The reference architecture configuration of SSD and SATA disks and specified disk controller are officially supported by VMware. VMware NSX is also deployed to provide highly flexible networking, permitting the client to segregate application traffic and tiers as wanted with the ability to create essentially unlimited number of VLANs. Furthermore, NSX provides the edge gateway Copyright 2016 IBM and VMware Page 7 of 143

8 and firewall to connect the IBM Cloud for VMware Solutions to the client s on-premises datacenter. This includes allowing the client to use their own IP address space (BYOIP) on their deployed workloads in IBM Cloud. The client can optionally exploit NSX microsegmentation, load balancing, and other features as required. The inclusion of vrealize Automation and vrealize Orchestration gives the client administrator the ability to quickly deploy standard or template patterns or blueprints such as deploying a virtual server of a pre-defined size with the OS and application of their choice. Additional optional tooling can be added later, including VMware tooling such as vrealize Log Insights, IBM tooling such as Spectrum Protect, or any other tool from the VMware ecosystem. 2.2 Deployment overview The remainder of this document describes the step-by-step instructions. The first step is to understand the prerequisites and validate that they are all met. This includes sizing the expected workload and determining the number of servers needed initially. It also requires determining the appropriate IBM Cloud data center into which to deploy. Frequently this means selecting the closest data center to minimize network latency, although in other scenarios, geographic dispersion may be required. The data center selected must support dual 10 GB servers. If you require a large number of physical servers, you should contact IBM SoftLayer sales before proceeding to ensure that there is sufficient capacity in the selected data center. The actual installation starts by acquiring the basic infrastructure from SoftLayer. This includes acquiring the VLANs in the preferred data center, ordering the bare metal ESXi host servers, and some supporting steps. Once the infrastructure is in place, the VMware management stack can be deployed and the cluster created. This includes steps for creating and configuring the VSAN cluster, followed by deploying the NSX stack, and concludes with deploying vrealize Automation and vrealize Orchestration. 2.3 Prerequisites Before beginning the installation, ensure that the following prerequisites are met: Acquire all necessary software licenses and keys for all products used in this design. This includes licenses for management components, specifically, Microsoft licenses for the Windows 2012 virtual servers used by management stack components and one Microsoft SQL Server license required for vrealize Automation. Have an active SoftLayer account and a userid with authority to provision hardware and network and to submit tickets. Identify the preferred SoftLayer data center with the hardware availability and the ability to support dual 10 GB servers. Determine the number of vsphere hosts. Copyright 2016 IBM and VMware Page 8 of 143

9 Provide domain name and hostname prefixes for the vsphere hosts. Determine and establish required connectivity to provisioned guest workloads. a. Using NSX, this design provides a VPN endpoint and firewall. b. SoftLayer-provided VPN connectivity can be used for initial administrator access. Client is responsible for licensing of any software products in provisioned workloads. a. Windows or Linux licenses for workloads. b. Middleware and application licenses as necessary. 2.4 Conventions This implementation guide uses example values for domain names and host names. For example, it uses tornado and tornado.local as domain names, and uses sddc01esx01 as a host name. These values are listed in Appendices C and D. The user can, and should, replace these with names matching their requirements. Similarly, this guide uses example VLAN numbers and IP addresses that would be obtained from the SoftLayer portal. The user should replace those with the actual values provided when the resources are provisioned by SoftLayer. The example values are listed in Appendices C and D. However, IP addresses in the 192.x.x.x range do not need to be changed. These are values created within the NSX environment. With care, different values could be used, but it is acceptable and easier to use the values listed in this document. Sample values are italicized in this document as in this example: tornado.local 3 Acquire network infrastructure Building an IBM Cloud for VMware Solutions cluster begins with acquiring both the network infrastructure and the hardware for the ESXi servers. You should start by acquiring the network infrastructure, that is the VLANs and subnets required for the configuration. VMware recommends segregating different types of network traffic onto their own VLANs. However, in the public cloud such as on SoftLayer, the client does not have the unrestricted access to the networking infrastructure that they may have in their own data center. The reference architecture calls for three VLANs. Two are on the SoftLayer private network: one for management and the second for VSAN, NSX, vmotion, vsphere Replication, and NFS traffic. The third is on the SoftLayer public network to allow Internet connectivity. 3.1 Create VLANs Create three VLANs in this step Log in to SoftLayer portal at Select Network > IP Management > VLANs. Copyright 2016 IBM and VMware Page 9 of 143

10 3. Select Order VLAN If this option is not available in your account, you must submit a ticket to acquire the VLANs. Refer to Appendix I for instructions. Select Order by Router. Select the appropriate router from the Router list. The first three characters of the router name specify whether this is a back-end customer router (BCR) or a frontend customer router (FCR). The next two digits signify the pod number within the data center specified by the second portion of the router name. The public VLAN must be assigned to the FCR and the private VLANs are allocated to the BCR in the same pod of the desired data center. Select 16 IP Addresses as the Primary Subnet Size. 7. Include a VLAN name according to Appendix A Click Continue. Complete the form with the corresponding information. Indicate that 16 IPs will be used. For description, enter Will be used for servers within a VMware environment. 10. Select the two check boxes. Copyright 2016 IBM and VMware Page 10 of 143

11 11. Click Place Order and SoftLayer will provision the VLAN. 12. Repeat the process for the second private VLAN and the public VLAN. Do not start ordering servers until all of the VLANs have been provisioned. 13. Review the allocated VLANs in the SoftLayer portal by selecting Network > IP Management > VLANs. 14. After the VLANs have been provisioned, record the VLAN numbers, subnet range, and gateway. Use the worksheet in Appendix A to record the VLAN and the associated information. 3.2 Acquire additional subnet An additional portable private subnet must be allocated to the second VLAN to be used for the different VMware products that will be installed. The portable private subnet will be created in this step Log in to SoftLayer portal at Select Network > IP Management > Subnet. Copyright 2016 IBM and VMware Page 11 of 143

12 3. Select Order IP Addresses In the drop-down menu, select Portable Private. Select 64 Portable Private IP Addresses from the Select Option list Click Continue. Select the second private VLAN ID which will be used for VSAN, NSX, and vmotion. Click Continue. Indicate that 64 IPs will be used. Complete the form asking contact information in the same way as the previous VLAN request steps. 10. Select the two check boxes regarding contact information and the agreement. 11. Click Place Order and SoftLayer will provision the subnet. 12. Review the allocated subnet in the SoftLayer portal by selecting Network > IP Management > Subnets. After the VLANs and subnets have been provisioned, record the VLAN numbers, subnet range, and gateway. Use the worksheet in Appendix A to record the VLAN and the associated information. 4 Acquire infrastructure servers and external storage With the network infrastructure set up, the next step is to acquire the initial set of ESXi servers and a utility server that will be used to access the environment and perform subsequent configuration steps. IBM Cloud has the option of ordering both bare metal servers and virtual servers, either of which can be ordered on an hourly or a monthly basis. These different server types can be deployed in the same location and can be on the same network (VLAN). Copyright 2016 IBM and VMware Page 12 of 143

13 Before proceeding, gather the information on the acquired VLANs and the number of hosts required to provide sufficient capacity for the initial workload. 4.1 ESXi bare metal servers Acquire the vsphere (or ESXi) hosts. These servers must be bare metal servers running the ESXi hypervisor and will be used to host both the virtual machines running the client applications and the VMware management servers. A bare metal server is a physical machine that is dedicated to the client and provides the client complete access. You can order bare metal servers in pre-defined configurations on an hourly basis or order fully customizable servers on a monthly billing basis. However, only monthly servers are suitable for this implementation as VMware is only licensed for monthly servers, and monthly servers provide the necessary customized hardware configuration as specified in the reference architecture (see Appendix B). The appropriate number of bare metal servers depends upon the size of the expected workload. The reference architecture specifies a minimum of four servers, as VSAN requires a minimum of three servers and it is best practice to have a fourth (N+) server to provide resiliency in case of the loss of one server. If you require more capacity, you can order additional servers initially, or add additional servers later as demand increases. The hardware specifications in the reference architecture should be sufficient for over 120 average sized customer VMs. Review the reference architecture for specific sizing and how to determine the appropriate number of host servers. Follow these steps to order these bare metal servers. 1. Log in to 2. Select Account > Place an Order. Copyright 2016 IBM and VMware Page 13 of 143

14 3. Select Monthly in the Bare Metal Servers section. 4. In the pop-up window, select the data center in the drop-down menu. 5. Scroll to the Dual Processor section and select the E v3 server that supports up to 12 drives and up to 512 GB memory. If this specific processor is no longer available and has been replaced with a newer model, select the newer model but ensure that the server allows 12 drives and 512 GB memory. Copyright 2016 IBM and VMware Page 14 of 143

15 8. Enter the required quantity with a minimum of Scroll to the RAM tab and select 512 GB. 10. Scroll to Operating Systems and select VMware and vsphere Enterprise Plus 6.0 (2 Processor) 11. In the Disk Controller section, select disk 1 and 2. Click Assign Disks and select the 1 TB SATA. Select Create RAID Storage Group, set these two disks to RAID 1, and click Done. 12. Select only disk 3 and 8 (and deselect disks 1 and 2). Assign these disks as 1.2 TB SSD (10 DWPD). Select Create RAID Storage Group, set these two disks to JBOD, and click Done. 13. Select only the remaining disks 4,5,6,7,9,10,11,12 (deselecting 3 and 8) and assign these disks as 2.00 TB SATA. Select Create RAID Storage Group, set these eight disks to JBOD, and click Done. Copyright 2016 IBM and VMware Page 15 of 143

16 14. In the Network options section, select 10 Gbps Dual Public & Private Network Uplinks (Unbonded) in the Uplink Port Speeds drop-down menu. 15. In the System Addons - Advanced Monitoring options section, select Monitoring Package Basic. Copyright 2016 IBM and VMware Page 16 of 143

17 16. In the System Addons - Power Supply options section, select Redundant Power Supply. 17. In the Service Addons - Monitoring options section, select Host Ping and TCP Service Monitoring. 18. In the Service Addons - Response options section, select Automated Notification. 19. Click Add to Order. 20. On the resultant screen, review the summary of the order. Scroll to the Backend VLAN drop-down menu and select the first primary private VLAN previously acquired. 21. Scroll to the Subnet drop-down menu and select the first primary private subnet previously acquired. (Note: The selection of VLANs and subnet is very important so that this server is placed in the correct VLAN, subnet, and pod.) 22. In the Frontend VLAN drop-down menu, select the first primary public VLAN acquired. Scroll to the Subnet drop-down menu and select the first primary public subnet previously acquired. 23. Enter a hostname and domain name for the four servers. 24. Select the two check boxes and click Submit Order. Copyright 2016 IBM and VMware Page 17 of 143

18 25. An will be sent to the requestor when the order is confirmed and another will be sent to the master account userid when the server is provisioned. 26. Total provisioning time could take four hours or longer. You can monitor the progress of the provisioning in the portal by going to Devices > Device List. Update the worksheet in Appendix C with the IP addresses, hostname, and root password for each server. 4.2 Utility server A Windows-based server will be used during subsequent configuration steps. By provisioning this server with both public and private network access, it can be used as a jump box. Administrators can access the server over the public Internet via remote desktop and from there directly access the rest of the environment over the SoftLayer private network. It can also be used to store and transfer assets such as ISO images. A small virtual server is sufficient for these purposes. Whether it is private node or public node is not important, nor are hourly or monthly options. Acquire this virtual server with the following steps Log in to control.softlayer.com. Select Account > Place an Order. Under the Virtual Server (public node) heading, select the Monthly option. 4. On the resultant order screen, specify the following options: a. Select the appropriate data center (where the VLANs were created) b. Computing Instance: 2x2.0 GHz Cores c. RAM: 8 GB d. Operating System: Microsoft >Windows Server 2012 R2 Standard Edition (64-bit) (1-16 Core) e. First Disk: 100 GB (SAN) f. Uplink Port Speeds: 1 Gbps Public and Private Network Uplinks g. Hardware & Software Firewalls: Microsoft Windows Firewall h. Anti-Virus & Spyware Protection: McAfee Virus Scan Enterprise i. Intrusion Detection & Protection: McAfee Host Intrusion Protection w/reporting (this is optional but ensures that this system is protected from attack upon deployment) j. Advanced Monitoring: Monitoring Package Basic k. Monitoring: Host Ping and TCP Service Monitoring l. Response: Automated Notification Copyright 2016 IBM and VMware Page 18 of 143

19 Click Add to Order. Review the summary of the order. Scroll to the Backend VLAN drop-down menu and select the second private VLAN previously acquired. Scroll to the Subnet drop-down menu and select the second primary private subnet previously acquired. (Note: The selection of VLANs and subnet is very important so that this server is placed on the same VLAN and in the same pod as the ESXi hosts.) In the Frontend VLAN drop-down menu, select the first primary public VLAN acquired. Scroll to the Subnet drop-down menu and select the first primary public subnet previously acquired. 9. Enter a hostname and domain name for the server. 10. Select the two check boxes and click Submit Order. 11. An will be sent to the requestor when the order is confirmed and another will be sent to the master account userid when the server is provisioned. 12. Provisioning time can take 20 minutes or longer. You can monitor the progress of the provisioning in the portal by going to Devices > Device List. 13. Add the IP and login information to Appendix C. By default, this system is exposed to the Internet, so either lock it down as soon as possible as described in the Configure utility server section, or shut the system down until that action is completed. 5 Acquire licenses for VMware components The licenses for the additional VMware products used in this design must be acquired from the SoftLayer portal. The VMware vsphere (ESXi) licenses were already acquired with the servers but you must order the licenses for vcenter server, VSAN, NSX, and other VMware components such as vrealize Automation. The licenses for each VMware Copyright 2016 IBM and VMware Page 19 of 143

20 component must be ordered one at a time so you must repeat this process for each of the required components Log in to Select Devices > Manage > VMware licenses. Click Order VMware Licenses. Select the VMware component which requires licensing. On the initial pass, select vcenter Server Appliance 6.0. Click + Add License. Click Continue. Select the check box and click Place Order. Repeat this process for each of the components from the following list that will be used in this deployment. The ordering process only permits you to select one component for each iteration. The first two, VSAN and NSX, are required for all IBM Cloud for VMware Solutions deployments. The others can be ordered at this time as a convenience if you intend to install these products eventually. a. Virtual vsan Standard Tier II TB 6.x b. VMware NSX Enterprise 6.2 c. VMware vrealize Automation Enterprise 7.0 d. VMware vrealize Operations Enterprise Edition 6.0 e. VMware vrealize Log Insight 3.0 f. VMware Site Recovery Manager 6.1 VMware components are licensed per processor so you must order enough licenses based on the number of processors in the deployment. For example, if you are deploying the basic four server configuration with two CPUs, a total of eight CPU licenses will be required for each. At the time of this publication, the licenses for these components can be ordered in quantities of 1,2 or 4. You must place multiple orders to acquire the necessary number of licenses. For example, you need two sets of four licenses to obtain the required eight licenses for the basic four server configuration. These different sets of licenses will be merged into one set in the next step. 6 Merging licenses for VMware components As described previously, the ordering portal may require you to obtain multiple sets of licenses to cover all of the processors in the deployment. These different sets must be Copyright 2016 IBM and VMware Page 20 of 143

21 merged before they can be applied to the different VMware products. The licenses for the required products can be merged as follows Log in to the SoftLayer portal at Select Support > Add Ticket Select Sales Request in Subject drop-down menu. Enter the title Merge VMware licenses. Enter the following text into the details field. Include the respective license key for each product line. The list of keys can be obtained from Devices > Manage > VMware licenses. Please merge the licenses for each of the VMware products listed Virtual vsan Standard Tier II TB 6.x VMware NSX Enterprise 6.2 Submit the ticket and SoftLayer will work with VMware to merge the licenses. 7 Configure utility server At this stage the necessary infrastructure should be available. There are some additional tasks to perform before building the VMware cluster. You must set up the utility server or jump box by protecting it from attack and installing some tools that will be needed. 7.1 Secure access to the environment To provide a basic level of security and protection of the environment, direct Internet access to the environment should be tightly controlled. Even though the provisioned ESXi hosts are assigned public IP addresses, they are only connected to the SoftLayer private network and are therefore not connected to the external network. However, the utility server is directly accessible from the public Internet because it will be used as a jump box to access the servers on the SoftLayer internal network. Since it is on the Internet, it is open to attack. Consider reducing this risk by using Windows firewall settings to restrict access to this server. 1. Log in via RDP to the server using the administrator password found in the portal. 2. Consider creating a new userid, restricting RDP access only to this userid, and logging this in the list of client-created userids. 3. Add this userid and all future userids that are created to Appendix F. Copyright 2016 IBM and VMware Page 21 of 143

22 7.2 Install utilities Several other tools must be installed on this utility server for future steps Acquire and install Chrome from (At the time of this writing, there is a known bug with Firefox when using the Adobe Flash Plugin at vcenter Web Client.) Acquire and install Putty from Acquire and install WinSCP from Acquire and install Java from Acquire and install Remote Desktop Connection Manager Install vsphere Client VMware is transitioning to use a web-based client for vsphere administration and most new functions will only be available in the web client. However, some functions still need to performed in the traditional vsphere client Open Chrome, type the first ESXi SoftLayer Primary Private IP and click enter to connect to the first ESXi host. On the VMware ESXi Welcome page, click Download vsphere Client for Windows. Install the vsphere client. 7.4 Add secondary network interface This utility server will need to communicate with the vsphere hosts and the vcenter over a VXLAN network, which will be created in a later step. This requires adding a secondary IP address to the utility server from the SoftLayer IP portable segment at the second private VLAN that was previously requested. Add the IP and login information to Appendix C. 1. Open Control Panel\Network and Sharing Center. 2. Change adapter settings. 3. Open properties of PrivateNetwork-A. 4. Open properties of TCP/IP v4. 5. Open Advanced settings. 6. Add IP address of with a subnet mask of Click OK, OK, and Close. Copyright 2016 IBM and VMware Page 22 of 143

23 8 Trunk VLANs to each of the ESXi hosts In a previous step, two VLANs were acquired on the SoftLayer private network. Traffic on both of these two VLANs must be made available to each of the physical hosts. This is done by submitting a ticket requesting SoftLayer to trunk the two VLANs. By default, SoftLayer places ports on the back-end customer switches in access mode. As a result, the ports attached to the ESXi hosts need to be trunked so that the hosts can access storage and the VMs can communicate on the private network Log in to Determine which network interfaces for each host are on the private network. Select the ESXi host in the Device Details list. Record the first value in the Private column under the Network section. Usually this will be eth0, meaning the eth0 and eth2 pair are the private network adapters. A ticket is required to trunk the VLANs. Select Support > Add Ticket. Enter the following information: Subject: Private network question Title: Trunk VLANs on unbonded private NICs Details: Please trunk VLANs <Management VLAN> and <Converged VLAN> on eth0 and eth2 private NIC pair for the following hosts [list each ESXi host] Click Add Ticket and SoftLayer will reply to the ticket when complete. This is usually completed quickly but could take one hour or longer. Copyright 2016 IBM and VMware Page 23 of 143

24 9 ESXi hardware validation Now that the physical servers have been acquired and the network has been configured, you should validate certain aspects of the setup before proceeding with subsequent configuration steps. 9.1 CPU, memory, and network hardware verification Using the vsphere Host client, connect to each SoftLayer ESXi host Primary Private IP and compare CPU and memory listed in SoftLayer with the wanted hardware configurations. Table 1. Management Cluster Hosts SoftLayer Management IP Example Hostname Example FQDN Example SoftLayer Private Primary IP Example Subnet mask sddc01esx01 sddc01esx01.tornado.local sddc01esx02 sddc01esx02.tornado.local sddc01esx03 sddc01esx03.tornado.local sddc01esx04 sddc01esx04.tornado.local CPU and memory 1. Remote desktop to the utility server. 2. Open Chrome, type the first ESXi SoftLayer Primary Private IP, and click enter to connect to the first ESXi host. 3. Click Open the VMware Host Client and enter the ESXi credentials from the SoftLayer portal. 4. Select Host. 5. Expected results show a 2690 v3 with 2.6GHz and 512 GB of RAMM. Copyright 2016 IBM and VMware Page 24 of 143

25 Storage 6. Select Storage > Devices. 7. Expected results show 12 items (2 SSD, 1 RAID1 virtual disk, 1 Enclosure, and 8 SATA DRIVES). Network 8. Select Networking > Physical NICs. 9. Expected results are four network cards with 10 Gb. Repeat for each additional ESXi host Copyright 2016 IBM and VMware Page 25 of 143

26 10. Repeat the memory, CPU, storage, and network validation steps for each of the other hosts. If there are any discrepancies, open a support ticket or support chat with SoftLayer to determine if the server can be reconfigured. 9.2 Change advanced power management settings on the ESXi hosts At this time, you should also change the advanced power management settings on each ESXi from the SoftLayer default of Balanced to the recommended High Power. 1. Remote desktop to the utility server. 2. Open Chrome, type the first ESXi SoftLayer Primary Private IP , and click enter to connect to the first ESXi host. 3. Click Open the VMware Host Client and enter the ESXi credentials from the SoftLayer portal. 4. Select Host. 5. Select Manage. 6. Click Hardware tab. 7. Click Change policy. 8. Select High Performance. Click OK. 9. Repeat the steps for each of the other hosts. 10 VLAN trunking and jumbo frames verification Any issues in the network configuration could cause problems in the future that are difficult to diagnose. Although issues are unlikely, consider performing some verification now to avoid issues later Verify SoftLayer trunking and SoftLayer jumbo frames configuration By default, jumbo frames are enabled in SoftLayer switching, and you can quickly verify this setting before continuing. All that is required is to establish a temporary portgroup and vmkernel on each VLAN, then ping with a large package private network is used to temporarily set up the vmkernel IP address. The first vmkernel does not need to have a VLAN ID assigned because it uses the natively configured VLAN. The second vmkernel VLAN ID is set in this example as When running these Copyright 2016 IBM and VMware Page 26 of 143

27 commands, replace this value with the SoftLayer provided second private VLAN ID for this deployment SSH using Putty to the first ESXi host SoftLayer Primary Private IP Enter the following commands: esxcli network vswitch standard set -m v vswitch0 esxcli network vswitch standard portgroup add -p testpg_vlan1 -v vswitch0 esxcli network ip interface add --interface-name=vmk1 -- portgroup-name=testpg_vlan1 esxcli network ip interface set -i vmk1 -m 9000 esxcli network ip interface ipv4 set -i vmk1 -I N t static esxcli network vswitch standard portgroup add -p testpg_vlan2 -v vswitch0 esxcli network vswitch standard portgroup set -p testpg_vlan2 -v 1557 esxcli network ip interface add --interface-name=vmk2 -- portgroup-name=testpg_vlan2 esxcli network ip interface set -i vmk2 -m 9000 esxcli network ip interface ipv4 set -i vmk2 -I N t static 3. After the commands, the ESXi host will display the following network settings. 4. Repeat these configuration steps on the remaining servers, replacing the IP address with the specific IP address assigned to the respective hosts (such as for ESXi 02). # Setup for ESXi esxcli network vswitch standard set -m v vswitch0 esxcli network vswitch standard portgroup add -p testpg_vlan1 -v vswitch0 Copyright 2016 IBM and VMware Page 27 of 143

28 esxcli network ip interface add --interface-name=vmk1 --portgroupname=testpg_vlan1 esxcli network ip interface set -i vmk1 -m 9000 esxcli network ip interface ipv4 set -i vmk1 -I N t static esxcli network vswitch standard portgroup add -p testpg_vlan2 -v vswitch0 esxcli network vswitch standard portgroup set -p testpg_vlan2 -v 1557 esxcli network ip interface add --interface-name=vmk2 --portgroupname=testpg_vlan2 esxcli network ip interface set -i vmk2 -m 9000 esxcli network ip interface ipv4 set -i vmk2 -I N t static # Setup for ESXi esxcli network vswitch standard set -m v vswitch0 esxcli network vswitch standard portgroup add -p testpg_vlan1 -v vswitch0 esxcli network ip interface add --interface-name=vmk1 --portgroupname=testpg_vlan1 esxcli network ip interface set -i vmk1 -m 9000 esxcli network ip interface ipv4 set -i vmk1 -I N t static esxcli network vswitch standard portgroup add -p testpg_vlan2 -v vswitch0 esxcli network vswitch standard portgroup set -p testpg_vlan2 -v 1557 esxcli network ip interface add --interface-name=vmk2 --portgroupname=testpg_vlan2 esxcli network ip interface set -i vmk2 -m 9000 esxcli network ip interface ipv4 set -i vmk2 -I N t static # Setup for ESXi esxcli network vswitch standard set -m v vswitch0 Copyright 2016 IBM and VMware Page 28 of 143

29 esxcli network vswitch standard portgroup add -p testpg_vlan1 -v vswitch0 esxcli network ip interface add --interface-name=vmk1 --portgroupname=testpg_vlan1 esxcli network ip interface set -i vmk1 -m 9000 esxcli network ip interface ipv4 set -i vmk1 -I N t static esxcli network vswitch standard portgroup add -p testpg_vlan2 -v vswitch0 esxcli network vswitch standard portgroup set -p testpg_vlan2 -v 1557 esxcli network ip interface add --interface-name=vmk2 --portgroupname=testpg_vlan2 esxcli network ip interface set -i vmk2 -m 9000 esxcli network ip interface ipv4 set -i vmk2 -I N t static When the configuration of the second host and all subsequent hosts is complete, verify that the VLAN trunking and jumbo frame configuration works by pinging between two hosts with 8172 byte packet size. 5. Execute the vmkping command on the first host and ping the address of the other hosts in the environment. vmkping -I vmk1 -d -s vmkping -I vmk1 -d -s vmkping -I vmk1 -d -s vmkping -I vmk2 -d -s vmkping -I vmk2 -d -s vmkping -I vmk2 -d -s If successful, all the VLANs are correctly trunked and jumbo frames are properly configured. If unsuccessful, the following message will be displayed. 6. In this case, open a support ticket in SoftLayer portal requesting verification of the VLAN trunking, the jumbo frame availability, or both. After verifying that the vmkping was successful on each host, delete the portgroup on each host. esxcli network ip interface remove --interface-name=vmk1 Copyright 2016 IBM and VMware Page 29 of 143

30 esxcli network vswitch standard portgroup remove -p testpg_vlan1 -v vswitch0 esxcli network ip interface remove --interface-name=vmk2 esxcli network vswitch standard portgroup remove -p testpg_vlan2 -v vswitch0 11 Configure an additional management network on all hosts Some additional network configuration is required on each of the ESXi hosts. This requires configuring an additional vmkernel in the existing vsphere Standard Switch. This configuration provides connectivity and common network configuration for virtual machines that reside on each host Create new VMkernel connection Log in to the sddc01esx01.tornado.local vsphere Host client using the IP On the Home page, click Networking and click the VMkernel NICs tab. Click Add VMkernel NIC. In the Add Network NIC wizard, enter the following values. Port group: New port group New port group: Internal Management Network Virtual switch: vswitch0 VLAN ID: 1557 IP version: IPv4 only IPv4 settings: Static TCP/IP stack: Default TCP/IP stack Services -> Management: Checked In the IPv4 settings option, expand it and enter the following values depending on the host that you are working on. Remember that these values are reference examples for this document and must be replaced with the actual values for your deployment. Table 2. Management Cluster Hosts Internal Management IP Example Hostname Example FQDN Example SoftLayer Private Portable IP Example Subnet mask sddc01esx01 sddc01esx01.tornado.local Copyright 2016 IBM and VMware Page 30 of 143

31 sddc01esx02 sddc01esx02.tornado.local sddc01esx03 sddc01esx03.tornado.local sddc01esx04 sddc01esx04.tornado.local Click Create Create new port group From the utility server, open the vsphere Client and log in to the first ESXi hosts sddc01esx01 using the address for Internal Management vmkernel set up in the previous step and the root userid and password shown in the SoftLayer portal. On the Home page, click Inventory, click the Configuration tab, and click Networking. Click vsphere Standard Switch and click Properties next to the vswitch0. While still in the vswitch0 Properties window, click Add. In the Add Network wizard, on the Connection Type page, select Virtual Machine and click Next. In the Virtual Machines - Connection Settings page, enter the following settings and click Next. Network Label: VM Network VLAN ID: 1557 In the Ready to Complete page, click Finish. 12 Configure Active Directory Domain Controller, CA, DNS, and NTP With the physical infrastructure in place, you can start the deployment of the VMware environment. This is started by setting up some required key services such as an Active Directory (AD) Server, Certificate Authority (CA), DNS, and NTP time source. The standard reference architecture calls for two instances to provide AD/DNS/NTP and subordinate CA redundancy and high availability, and for a third server for offline root Certificate Authority. All servers will be Windows 2012 R2 Standard servers. You must have downloaded and saved a Windows Server ISO in the utility server before starting this deployment. In this section, you will establish the three guest VMs on one of the ESXi hosts and deploy the respective services. For now, the servers will be provisioned on local storage on that server and later will be migrated to VSAN shared storage. Copyright 2016 IBM and VMware Page 31 of 143

32 Deploy three Windows VMs From the utility server, open the vsphere Client and log in to the first ESXi hosts sddc01esx01 using the address for Internal Management vmkernel set up in the previous step and the root userid and password shown in the SoftLayer portal. Click Inventory. Select Configuration tab. Click Storage. Select datastore1. Right-click and select Browse Datastore. 3. In the datastore browser window, click the import icon and select Upload File. Select the Windows ISO file and click Open. 4. Using the vsphere client, press Ctrl+N to open the Create New Virtual Machine wizard and proceed through the wizard with the following steps. Select Custom. Click Next. Name it sddc01dc01. Click Next. Select datastore1 and click Next. Select Virtual Machine Version :11. Click Next. Select Windows and version Microsoft Windows Server 2012 (64-bit). Click Next. Select 2 virtual sockets and 2 cores per virtual socket. Click Next. Select 8 GB Memory Size. Click Next. Select 1 NIC, Network VM Network, and Adapter VMXNET 3. Click Next. Select VMware Paravirtual SCSI controller. Click Next. Select Create a new virtual disk. Click Next. Select 80 GB Disk Size, Thin Provision, and Store with the virtual machine. Click Next. Select SCSI (0:0). Click Next. Select the check box Edit the virtual machine settings before completion. Click Continue. Select New CD/DVD (adding), select Datastore ISO File, click Browse, double-click datastore1, select the Windows Server ISO file, and click OK. Check Connect at power on. Select New Floppy (adding), select Use existing floppy image in datastore, Click Browse, double-click vmimages, double-click floppies, select pvscsi- Copyright 2016 IBM and VMware Page 32 of 143

33 Windows2008.flp, click Open, and check Connect at power on. Click Finish. 5. Right-click the VM, select Open Console. Click the Power on button. 6. Install Windows Server 2012 R2 Standard Edition. Click Next. Click Install Now. Copyright 2016 IBM and VMware Page 33 of 143

34 Enter product key and click Next. Select Windows Server 2012 R2 Standard (Server with a GUI). Click Next. Copyright 2016 IBM and VMware Page 34 of 143

35 Check I accept the license terms. Click Next. Click Custom: Install Windows only (advanced). Click Load driver. Click Browse. Expand Floppy Disk Drive (A:). Select AMD64. Click OK. Copyright 2016 IBM and VMware Page 35 of 143

36 Select VMware PVSCSI Controller (A:\AMD64\PVSCSI.INF). Click Next. Copyright 2016 IBM and VMware Page 36 of 143

37 Select Drive 0 Unallocated Space. Click Next. Wait until the installation completes and reboots the VM automatically. Copyright 2016 IBM and VMware Page 37 of 143

38 Set administrator password. Click Finish. 7. Click VM menu, select Guest, and click Send Ctrl+Alt+Del. Enter administrator password and click enter. Click VM menu, select Guest, and click Install/Upgrade VMware Tools. Click OK. Click the pop-up window. Click Run setup64.exe. Click Next. Copyright 2016 IBM and VMware Page 38 of 143

39 Select Complete. Click Next. Click Install. Copyright 2016 IBM and VMware Page 39 of 143

40 Click Finish. Click Yes to restart the VM. Click the VM menu, select Guest, and click Send Ctrl+Alt+Del. Enter administrator password and click enter. Copyright 2016 IBM and VMware Page 40 of 143

41 8. 9. The VM is now built and the base OS is installed and is ready to be customized. Click the Start menu in the Windows OS. Click Control Panel. Click System and Security. Click See the name of this computer. Click Change Settings. Click Change. Type sddc01dc01. Click OK. Click OK. Click Remote tab. Select Allow remote connections to this computer. Click OK. Click Close. Click Restart now to apply the changes. Click the VM menu. Select Guest. Click Send Ctrl+Alt+Del. Enter administrator password and press enter. Right-click the network icon. Click Open Network and Sharing Center. Click Change adapter settings. Right-click Ethernet0 and click Properties. Select Internet Protocol Version 4 (TCP/IPv4) and click Properties. Set the static IP, subnet mask, and gateway for this VM according to Appendix D. Click OK. Click Close. 10. Return to the Open Network and Sharing Center window. Click Windows Firewall. Click Allow an app or feature through Windows Firewall. Enable Remote Desktop for Private and Public network. Click OK. Reboot the server. 11. Repeat the process to create the second domain controller, naming it sddc01dc02, using the same virtual hardware specifications. 12. Repeat the process to create the Master CA sddc01ca01 with the following virtual hardware specifications. vcpu RAM Disk Space OS Version Type Adapter SCSI Controller Type Floppy Drive Copyright 2016 IBM and VMware Page 41 of 143

42 2 4 GB 60 GB Microsoft Windows Server 2012 (64- bit) VMXNET 3 Paravirtual [] /vmimages/floppies/pvscsi- Windows2008.flp 12.2 Configure NTP client and time sync NTP client services must be configured for Domain Controllers and Certificate Authority server. 1. Open Remote Desktop Connection Manager Click Ctrl+N to create a new file. Save it on the desktop with the name SDDC. 3. Right-click SDDC and click Add server. 4. Enter the sddc01dc01 IP Click OK. 5. Repeat the process to create a connection for sddc01dc02 and sddc01ca Select the new connection , right-click and click Connect Server. Type the administrator password to log in to the VM. Configure NTP services on sddc01dc01 by executing the following commands in a command prompt window opened with administrator permissions. net start w32time w32tm.exe /config /manualpeerlist: servertime.service.softlayer.com /syncfromflags:manual /reliable:yes /update w32tm.exe /config /update net stop w32time net start w32time w32tm /resync w32tm /query /status w32tm /query /peers w32tm /query /configuration Repeat the process for sddc01dc02 and sddc01ca01. Copyright 2016 IBM and VMware Page 42 of 143

43 12.3 Install Active Directory and DNS server 1. Open the Server Manager from the task bar on sddc01dc From the Server Manager dashboard, select Add roles and features. 3. Click Next. 4. Select Role-based or features-based installation. Click Next. 5. Click Next to proceed to the Server Roles tab. 6. Select the Active Directory Domain Services check box. Click Add Features. 7. Select the DNS Server check box. Click Add Features. 8. Click Next. 9. Click Next. 10. Click Next. 11. Click Next. 12. Click Next. 13. Click Next. 14. Click Next. Copyright 2016 IBM and VMware Page 43 of 143

44 15. Click Install. 16. Click Close. No restart is required at this point. 17. Repeat the process for sddc01dc Configure DNS forwarders These two servers are configured as DNS servers but now they must be configured to use SoftLayer DNS forwarders. 1. Open the Server Manager from the task bar on sddc01dc Click Tools and open DNS Manager. 3. In the console tree, click the sddc01dc01 DNS server. 4. On the Action menu, click Properties. 5. On the Forwarders tab, click the Edit button. 6. Under the IP address list, type the IP address and click enter. Repeat the process to add , click OK, and click OK once more. 7. Repeat the process for sddc01dc Configure Active Directory Start by configuring the sddc01dc01 server for the new domain Open the Server Manager from the task bar on sddc01dc01. Open the Notifications Pane and click Promote this server to a domain controller. Copyright 2016 IBM and VMware Page 44 of 143

45 Select Add a new forest and enter the root domain name tornado.local. Click Next. Select a Windows Server 2012 as the Domain and Forest functional level. The functional level could be set as Windows Server 2008 in case this deployment will be connected with an existing customer on-premises Active Directory domain with an older domain version. Enter a new password for the Directory Services Restore Mode (DSRM). Ignore the prompt error. Click Next. 8. Enter TORNADO as the NetBIOS domain name and click Next. 9. Set the SYSVOL, Log files, and Database folders location as default. Click Next. 10. Click Next. 11. Ignore alerts and click Install. The server will be rebooted automatically after the installation be completed. With the first Domain Controller now established as an AD domain server, the second server can now also be configured as a domain server and joined to the first domain server. 12. On sddc01dc02, right-click the network icon. 13. Click Open Network and Sharing Center. 14. Click Change adapter settings. 15. Right-click Ethernet0 and click Properties. 16. Select Internet Protocol Version 4 (TCP/IPv4) and click Properties. 17. Set the DNS servers as and for this VM. 18. Click OK. 19. Click Close. 20. Open the Server Manager from the task bar on sddc01dc Open the Notifications Pane and click Promote this server to a domain controller. Copyright 2016 IBM and VMware Page 45 of 143

46 22. Select Add a domain controller to an existing domain and insert the root domain name and sddc01dc01 administrator credentials. 23. Click Next. 24. Select the Domain Name System (DNS) server and Global Catalog (GC) check box. 25. Enter a new password for the Directory Services Restore Mode (DSRM). 26. Click Next. 27. Ignore warning and click Next 28. Select sddc01dc01.tornado.local from the drop-down menu. 29. Click Next. 30. Set the SYSVOL, Log files, and Database folders location as default. 31. Click Next. 32. Click Next. 33. Ignore alerts and click Install. Copyright 2016 IBM and VMware Page 46 of 143

47 The server will be rebooted after the installation completes. 34. Open Active Directory Users and Computers console and create the accounts and groups required from Appendix G and Appendix H Configure DNS zones and transfer zones In the previous step, the Active Directory wizard created a domain forward zone and configured the transfer zone between sddc01dc01 and sddc01dc02. Now a DNS server reverse lookup zone must be configured on sddc01dc Open the Server Manager from the task bar on sddc01dc Click Tools and open DNS Manager. 3. In the console tree, right-click a sddc01dc01 server and click New Zone to open the New Zone wizard. 4. Select Primary Zone, click Next. 5. Select To all DNS servers running on domain controllers in this forest: tornado.local and click Next. 6. Select Reverse lookup zone and click Next. 7. Select IPv4 Reverse Lookup Zone Enter at Network ID field and click Next. Select Allow only secure dynamic updates (recommended for Active Directory) and click Next. 10. Click Finish. 11. Repeat the process to create the corresponding reverse lookup zone for each subnet that will be used in the environment. 12. Select in-addr.arpa, right-click and select Properties option. 13. In the opened domain s properties box, go to the Zone Transfers tab. 14. Ensure that the Allow zone transfers check box is selected. 15. Ensure that the Only to servers listed on the Name Servers tab radio button is selected. 16. Once verified, go to the Name Servers tab. 17. From the displayed interface, click Add. 18. On the opened New Name Server Record box, type the FQDN for the target DNS server in the Server fully qualified domain name (FQDN) field sddc01dc02.tornado.local. Note that BOTH of the name servers should be listed with their respective IP. 19. Click the Resolve button to resolve the IP address for the typed host name. 20. When the IP address is resolved, click OK. 21. On the domain s properties box, click OK to save the changes and to close the box. 22. On DNS Manager, right-click the server name. 23. From the displayed context menu, go to All Tasks and click Restart from the submenu that appears. Copyright 2016 IBM and VMware Page 47 of 143

48 24. Wait until the DNS service restarts and the DNS server starts using the modified settings. 25. Repeat steps 12 to 24 in sddc01dc02 to configure zone transfers in the second domain controller. 26. Open sddc01dc01 DNS console and create AAAA and PTR records for the VMs and ESXi from Appendix E. Note: Before continuing, the utility server DNS settings must be modified to add the new DNS servers and as primary and secondary in the private and public network interface DNS list. Keep and as third and fourth option respectively. The four ESXi hosts and the Certificate Authority server themselves must also be modified to use the and servers Install offline root certificate authority server A certificate authority is required for later steps in this deployment and this capability will be deployed on the first domain controller server Open Server Manager on sddc01dc01. Click Add roles and features. Click Next. Select Role-based or features-based installation. Click Next. Click Next to proceed to the Server Roles tab. Select the Active Directory Certificate Services check box. Click Add Features. Click Next. Click Next. Copyright 2016 IBM and VMware Page 48 of 143

49 9. Select the Certificate Authority and Certificate Authority Web Enrollment check box. 10. Click Next. 11. Click Install. 12. Repeat the process for sddc01dc02. For sddc01ca01, repeat the process but only install Certificate Authority feature Configure root certificate authority server Start by configuring the sddc01ca01 server as Root CA. Open the Server Manager from the task bar on sddc01ca01. Open the Notifications Pane and click Configure Active Directory Certificate Services on the destination server. 4. Click Next. 5. Select Certification Authority. 6. Click Next. 7. Select Standalone CA. 8. Click Next. 9. Select Root CA. 10. Click Next. 11. Select Create a new private key. 12. Click Next. 13. Select RSA#Microsoft Software Key Storage Provider, SHA256 and Click Next. 15. Set Common name SDDC01CA01-CA, Distinguished name suffix DC=tornado,DC=local, Preview CN=SDDC01CA01-CA,tornado.local. 16. Click Next. 17. Select 10 Years as the validity period. 18. Click Next. 19. Set database locations as default. 20. Click Next. 21. Click Configure. 22. Click Close. 23. Click No on Do you want to configure additional role services? Configure the root CA settings In Server Manager, click Tools and click Certification Authority. In the Certification Authority console tree, expand SDDC01CA01-CA. Copyright 2016 IBM and VMware Page 49 of 143

50 Right-click Revoked Certificates and click Properties. On the CRL Publishing Parameters tab, ensure that Publish Delta CRLs is cleared (not selected). Set CRL publication interval for 1 Years and click OK. The certificate revocation list (CRL) publication interval is very important because the certificate expires after that interval. You must bring the offline root CA server back online and republish the CRL to the sub-ca server. Setting the value to one year means that you can keep the root CA powered off for a year until it is required for the next publication. In the Certification Authority console tree, right-click SDDC01CA01-CA and click Properties. Click the Extensions tab. Ensure that Select extensions is set to CRL Distribution Point (CDP) and add a CDP pointing to the subordinate CAs that will be the ones actually distributing certificates. You must add one for each subordinate certificate authority. Select the check boxes Include in CRLs, Clients use this to find Delta CRL locations, and Include in the CDP extension of issued certificates for each entry. DeltaCRLAllowed>.crl DeltaCRLAllowed>.crl 7. Change Select extension to Authority Information Access (AIA) and add two new locations for AIA, one for each subordinate certificate authority. Select the check box for Include in the AIA extension of issued certificates for each entry. sddc01dc01.tornado.local/certdata/<serverdnsname><caname><certifi catename>.crt sddc01dc02.tornado.local/certdata/<serverdnsname><caname><certifi catename>.crt Click OK. If you are prompted to restart Active Directory Certificate Services, click Yes. 8. Right-click Revoked Certificates, click All Tasks, and click Publish. 9. Select New CRL and click OK. 10. Right-click SDDC01CA01-CA and click Properties. 11. In the SDDC01CA01-CA properties box, click View Certificate. 12. In the Certificate box, click the Details tab and click Copy to File. 13. In the Certificate Export wizard, in the Welcome box, click Next. 14. On the Export File Format box, select DER encoded binary X.509 (.CER) and click Next. 15. On the File to Export box, click Browse. In the File name text box, type \\sddc01dc01\c$ and click enter. 16. In the File name text box, type RootCA, click Save, and click Next. 17. Click Finish and click OK. 18. Browse to c:\windows\system32\certsrv\certenroll, copy all files, and paste to \\sddc01dc01\c$. Copyright 2016 IBM and VMware Page 50 of 143

51 19. From sddc01dc01.tornado.local, open Windows PowerShell, change to the C drive using the cd command, and run the following commands. certutil -dspublish -f sddc01ca01_sddc01ca01-ca.crt RootCA certutil -addstore -f root sddc01ca01_sddc01ca01-ca.crt certutil -addstore -f root SDDC01CA01-CA.crl 20. Repeat the process to copy and distribute the same files on the sddc01dc02 server Configure subordinate certificate authority server Start by configuring the sddc01dc01 server as subordinate CA. Open the Server Manager from the task bar on sddc01dc01. Open the Notifications Pane and click Configure Active Directory Certificate Services on the destination server. 4. Click Next. 5. Select Certification Authority and Certification Authority Web Enrollment. 6. Click Next. 7. Select Enterprise CA. 8. Click Next. 9. Select Subordinate CA. 10. Click Next. 11. Select Create a new private key. 12. Click Next. 13. Select RSA#Microsoft Software Key Storage Provider, SHA256 and Click Next. 15. Set Common name tornado-sddc01dc01-ca, Distinguished name suffix DC=tornado,DC=local, Preview CN=tornado-SDDC01DC01- CA,DC=tornado,DC=local. 16. Select Save a sddc01dc01.tornado.local_tornado-sddc01dc01-ca.req certificate request to file on C drive. 17. Click Next. 18. Set database locations as default. 19. Click Next. 20. Click Configure. 21. Click Close. 22. Click No on Do you want to configure additional role services? Copyright 2016 IBM and VMware Page 51 of 143

52 Configure the subordinate CA settings Repeat the process on the sddc01dc02 server. 1. Install the root CA certificate RootCA file (that you previously copied onto the C drive) in the local machine s Trusted Root CA certificate store. 2. Right-click the RootCA file and click Install Certificate. 3. In the Certificate Import wizard, click Local Machine and click Next. 4. On the Certificate Store box, click Place all certificates in the following store and click Browse. 5. Click Trusted Root Certification Authorities. 6. Click OK. 7. Click Next. 8. Click Finish. 9. Click OK in the Certificate Import wizard pop-up message. 10. Create c:\inetpub\wwwroot\certdata folder. Copy the SDDC01CA01-CA.crl and sddc01ca01_sddc01ca01-ca.crt files previously copied onto the C drive from the root CA server sddc01ca01 to c:\inetpub\wwwroot\certdata on the C drive. 11. Copy sddc01dc01.tornado.local_tornado-sddc01dc01-ca.req file to \\sddc01ca01\c$. 12. In the sddc01ca01.tornado.local Certificate Authority console, right-click SDDC01CA01-CA, click All Tasks, and click Submit new request. 13. In the Open Request File box, browse to the C drive, click file sddc01dc01.tornado.local_tornado-sddc01dc01-ca.req, and click Open. 14. In the right pane, select the Pending Requests folder, right-click the request, click All Tasks, and click Issue. 15. Click Issued Certificates container, double-click the certificate, click the Details tab, and click Copy to File. 16. In the Certificate Export wizard, on the Welcome page, click Next. 17. On the Export File Format box, click Cryptographic Message Syntax Standard PKCS #7 Certificates (.P7B), select the check box Include all certificates in the certification path if possible, and click Next. 18. On the File to Export box, click Browse. In the File name text box, type \\sddc01dc01.tornado.local\c$\subca and press enter. 19. Click Next. 20. Click Finish. 21. Click OK. 22. In the sddc01dc01.tornado.local Certification Authority console, right-click SDDC01DC01-CA, click All Tasks, and click Install CA Certificate. 23. Navigate to the C drive, click the SubCA.p7b file, and click Open. 24. Wait for a few seconds, right-click SDDC01DC01-CA, click All Tasks, and click Start Service. 25. Verify that the CA starts successfully. Copyright 2016 IBM and VMware Page 52 of 143

53 13 Adjust vsphere host settings Add ESXi hosts to the active directory domain Log in to the sddc01esx01.tornado.local vsphere Host client using the IP Click Manage, click Security & Users, and select Authentication. Click Join Domain. In the Join Domain dialog box, enter the following values and click OK. Setting Domain User name Password Value tornado.local administrator ad_admin_password 4. Repeat the previous step to add all remaining hosts to the domain. FQDN sddc01esx02.tornado.local sddc01esx03.tornado.local sddc01esx04.tornado.local Copyright 2016 IBM and VMware Page 53 of 143

54 14 Deploy Platform Services Controller With the physical infrastructure in place and the ESXi hosts initialized, the VMware management stack installation is ready to be started Install Platform Services Controller Setting up the VMware management stack starts by deploying a Platform Services Controller (PSC) instance from the vcenter Server appliance ISO file Access the utility server via RDP and log in as an administrator. Open a browser and download the appliance from Mount the VMware-VCSA-all vcenter Server Appliance ISO file by double-clicking the icon and opening it. Navigate to the vcsa directory. Start the VMware-ClientIntegrationPlugin-x.x.x.exe file. Follow the prompts and finish the installation. Start the VMware vcenter Server Appliance Deployment wizard. Browse to the vcenter Server Appliance ISO file. Open the vcsa-setup.html file in a web browser. Note: there may be a prompt to accept the Client Integration Plugin. Click Install to start the installation. Complete the VMware vcenter Server Appliance Deployment wizard. Copyright 2016 IBM and VMware Page 54 of 143

55 10. On the End User License Agreement page, select the I accept the terms of the license agreement check box and click Next. On the Connect to target server page, enter the following values and click Next. Setting FQDN or IP Address User name Password Value sddc01esx01.tornado.local root esxi_root_user_password 11. In the Certificate Warning dialog box, click Yes to accept the host certificate. 12. On the Set up virtual machine page, enter the following values and click Next. Setting Value Copyright 2016 IBM and VMware Page 55 of 143

56 Appliance name OS password Confirm OS password sddc01psc01 SDDCpsc_root_password SDDCpsc_root_password 13. On the Select deployment type page, under External Platform Services Controller, select the Install Platform Services Controller radio button and click Next. 14. On the Set up Single Sign-on (SSO) page, select the Create a new SSO Domain radio button, enter the following values, and click Next. Setting vcenter SSO Password Confirm password SSO Domain name SSO Site name Value vcenter_admin_password vcenter_admin_password vsphere.local SoftLayer data center site code Copyright 2016 IBM and VMware Page 56 of 143

57 15. On the Select appliance size page, click Next. 16. On the Select datastore page, select the local storage datastore to deploy the Platform Services Controller on, select the Enable Thin Disk Mode check box, and click Next. Copyright 2016 IBM and VMware Page 57 of 143

58 17. On the Network Settings page, enter the following values and click Next. Setting Choose a network IP address family Network type Value VM Network IPv4 Static Network address System name sddc01psc01.tornado.local Subnet mask Network gateway Network DNS servers (comma separated) Configure time sync Enable SSH , servertime.service.softlayer.com. Enabled (Select checkbox) Copyright 2016 IBM and VMware Page 58 of 143

59 18. On the Ready to complete page, review the configuration and click Finish to start the deployment Join the Platform Services Controller to the Active Directory When the Platform Services Controller instance has been successfully installed, the appliance must be added to the Active Directory domain. After that, add the Active Directory domain as an identity source to vcenter Single Sign-On. When this is done, users in the Active Directory domain are visible to vcenter Single Sign-On and can be assigned permissions to view or manage IBM Cloud for VMware Solutions components. Log in to the Platform Services Controller. In a browser, go to and log in with these credentials. Setting User name Password Value administrator@vsphere.local vcenter_admin_password 2. Add the management Platform Services Controller instance to the Active Directory domain. In the Navigator, click Appliance Settings, click the Manage tab, and click the Join button. Copyright 2016 IBM and VMware Page 59 of 143

60 3. In the Join Active Directory Domain dialog box, enter the following values and click OK. Setting Domain User name Password Value tornado.local ad_admin_password 4. Click the Appliance settings tab and click VMware Platform Services Appliance. Copyright 2016 IBM and VMware Page 60 of 143

61 5. Log in to the VMware vcenter Server Appliance administration interface with the following credentials. Setting User name Password Value root SDDCpsc_root_password 6. On the Summary page, click Reboot. 7. In the System Reboot dialog box, click Yes. Copyright 2016 IBM and VMware Page 61 of 143

62 8. Once the reboot process finishes, log in to again using the following credentials. Setting User name Password Value vcenter_admin_password 9. To verify that the Platform Services Controller successfully joined the domain, click Appliance Settings and click the Manage tab. Add the Active Directory as a vcenter Single Sign-On identity source. 10. In the Navigator, click Configuration and click the Identity Sources tab. 11. Click the Add icon to add a new identity source. 12. In the Add Identity Source dialog box, select the following values and click OK. Setting Value Copyright 2016 IBM and VMware Page 62 of 143

63 Identity source type Domain name Use machine account Active Directory (Integrated Windows Authentication) TORNADO.LOCAL selected 13. Under Identity Sources, select the tornado.local identity source and click Set as Default Domain. 14. In the confirmation dialog box, click Yes. 15. Log in to the Platform Services Controller console. In a browser, go to and log in with these values. Setting User name Password Value root PSC_password Copyright 2016 IBM and VMware Page 63 of 143

64 16. Go to Time, and modify the Time zone in order to be consistent with the rest of the environment time zone configuration. 15 Deploy the vcenter Server Appliance This design uses the VMware vcenter Server Appliance (VCSA) to manage the entire cluster that runs as a virtual server within the VMware cluster. The appliance version of the vcenter Server is VMware s statement of direction. Running as a virtual server in the VMware cluster enables it to take advantage of VMware reliability for mission critical servers such as HA, DRS, backup, and others, and its delivery as an appliance ensures consistent and optimal configuration Install the vcenter Server appliance Installation of the appliance is wizard driven and straightforward. Start the VMware vcenter Server Appliance deployment wizard Mount the VMware-VCSA-all vcenter Server Appliance ISO file by double-clicking the icon. Browse to the vcenter Server Appliance ISO file. Copyright 2016 IBM and VMware Page 64 of 143

65 3. 4. Open the vcsa-setup.html file in a browser. Click Install to start the installation. Complete the VMware vcenter Server Appliance deployment wizard On the End User License Agreement page, select the I accept the terms of the license agreement check box and click Next. On the Connect to target server page, enter the following values and click Next. Setting FQDN or IP Address User name Password Value sddc01esx01.tornado.local root esxi_root_user_password Copyright 2016 IBM and VMware Page 65 of 143

66 7. 8. In the Certificate Warning dialog box, click Yes to accept the host certificate. On the Set up virtual machine page, enter the following values and click Next. Setting Appliance name OS password Confirm OS password Value sddc01vc01.tornado.local SDDCvc_root_password SDDCvc_root_password 9. On the Select deployment type page, under External Platform Services Controller, select Install vcenter Server (Requires External Platform Services Controller) radio button and click Next. Copyright 2016 IBM and VMware Page 66 of 143

67 10. On the Configure Single Sign-On (SSO) page, enter the following values and click Next. Setting Platform Services Controller FQDN or IP address vcenter SSO password Value sddc01psc01.tornado.local vcenter_admin_password vcenter Single Sign-On HTTPS Port 443 Copyright 2016 IBM and VMware Page 67 of 143

68 11. On the Select appliance size page, select Large (up to 1000 hosts, 10,000 VMs) and click Next. 12. On the Select datastore page, select the local storage datastore, select the Enable Thin Disk Mode check box, and click Next. 13. On the Configure database page, select the Use an embedded database (PostgreSQL) radio button and click Next. Copyright 2016 IBM and VMware Page 68 of 143

69 14. On the Network Settings page, enter the following values and click Next. Setting Choose a network IP address family Network type Value VM Network IPv4 Static Network address System name sddc01vc01.tornado.local Subnet mask Network gateway Network DNS servers (comma separated) Configure time sync Enable SSH , servertime.service.softlayer.com. Enabled (Select check box) Copyright 2016 IBM and VMware Page 69 of 143

70 15. On the Ready to complete page, review the configuration and click Finish to start the deployment. 16. Log in to the vcenter console. In a browser, go to and log in with these values. Setting User name Password Value root vcenter_password Copyright 2016 IBM and VMware Page 70 of 143

71 17. Go to Time and modify the Time zone to be consistent with the rest of the environment time zone configuration Add new licenses for the vcenter Server Appliance In a browser go to and log in using the following credentials. Setting User name Password Value administrator@vsphere.local vcenter_admin_password 2. Click the Licensing icon in the bottom section. Copyright 2016 IBM and VMware Page 71 of 143

72 3. Click the Licenses tab. 4. Click the Create New Licenses icon to add license keys. Copyright 2016 IBM and VMware Page 72 of 143

73 5. On the Enter license keys page, enter the license key previously acquired for vcenter Server Appliance and click Next On the Edit license name page, enter a vcenter and click Next. On the Ready to complete page, review the entries and click Finish. Repeat the process to include the VSAN license Assign the newly added license to the vcenter Server Appliance 1. Click the Assets tab. Copyright 2016 IBM and VMware Page 73 of 143

74 2. Select the vcenter Server instance and click the Assign License icon. 3. Select the vcenter Server license entered in the previous step and click OK Assign the vcenteradmins domain group to the vcenter Server administrator role In the Navigator, click Home. Click Hosts and Clusters. Select the sddc01vc01.tornado.local tree. Click the Manage tab, click Permissions, and click the Add icon. Copyright 2016 IBM and VMware Page 74 of 143

75 In the sddc01vc01.tornado.local - Add Permission dialog box, click Add. In the Select Users/Groups dialog box, select TORNADO from the Domain drop-down menu. In the search box, enter vcenteradmins and click Enter. Select vcenteradmins and click Add. Copyright 2016 IBM and VMware Page 75 of 143

76 9. Click OK. 10. In the sddc01vc01.tornado.local - Add Permission dialog box, select Administrator as Assigned Role and select the Propagate to children check box. 11. Click OK. 16 Configure the SDDC cluster Now that the vcenter server has been deployed, you can build the VMware cluster with the ESXi hosts added to it and the resource pools defined to isolate management, edge, and client workload resources Create the cluster Log in to the Management vcenter Server by using the vsphere Web Client. In a browser, go to and log in with the following credentials. Copyright 2016 IBM and VMware Page 76 of 143

77 Setting User name Password Value vcenter_admin_password Create a new datacenter. 2. In the Navigator, click Hosts and Clusters. 3. Click Actions > New Datacenter. 4. In the New Datacenter dialog box, enter SDDC as the name and click OK. 5. Right-click the SDDC datacenter and click New Cluster. 6. In the New Cluster wizard, enter the following values and click OK. Setting Name DRS Value SDDC01DC01 Turn ON (select check box) Leave other DRS options with default values vsphere HA EVC Virtual SAN Do not turn ON (leave check box unselected) Leave off Leave off Copyright 2016 IBM and VMware Page 77 of 143

78 16.2 Add hosts to the cluster To add the first host to the cluster, right-click the SDDC01DC01 cluster and click Add Host. On the Name and location page, enter sddc01esx01.tornado.local in the Host name or IP address text box and click Next. On the Connection settings page, enter the following credentials and click Next. Setting User name Password Value root esxi_root_user_password In the Security Alert dialog box, click Yes. On the Host summary page, review the host information and click Next On the Assign license page, use the ESXi license key that is already assigned to the host. On the Lockdown mode page, leave the default value of disabled and click Next. On the Resource Pool page, leave the default and click Next. On the Ready to complete page, review the entries and click Finish. 10. Repeat the previous step for the three remaining hosts to add them to the converged cluster Create the resource pools on the cluster 1. In the Navigator, click Hosts and Clusters, expand the entire sddc01vc01.tornado.local tree, right-click the SDDC01DC01 cluster and select All vcenter Actions > New Resource Pool. Copyright 2016 IBM and VMware Page 78 of 143

79 2. 3. From the table below use the values in each column to create the Management ResPool. Type the name for the resource pool and choose its settings in the New Resource Pool wizard. Repeat the previous step for the two remaining resource pools. Option Managment Edge Pool Workload Name Management ResPool Edge ResPool Workload ResPool CPU Shares Normal High High CPU Reservation Accept default of 0 Accept default of 0 Accept default of 0 CPU Reservation Type Expandable Expandable Expandable CPU Limit Unlimited Unlimited Unlimited Memory Shares Normal High High Memory Reservation Memory Reservation Type Accept default of 0 Accept default of 0 Accept default of 0 Expandable Expandable Expandable Memory Limit Unlimited Unlimited Unlimited 4. Migrate the existing VMs to the new Management ResPool new resource pool. Copyright 2016 IBM and VMware Page 79 of 143

80 17 Create a Distributed Virtual Switch Now that vcenter server is deployed and the cluster has been created containing all of the ESXi hosts, the next step is to create a Distributed Virtual Switch and then migrate the Platform Services Controller and vcenter Server instances to this new distributed switch Create a Distributed Virtual Switch 1. In a browser, go to and log in using the following credentials. Setting User name Password Value administrator@vsphere.local vcenter_admin_password Right-click the SDDC data center and select Distributed Switch > New Distributed Switch to start the New Distributed Switch wizard. On the Name and location page, enter vds-sddc as the name and click Next. 4. On the Select version page, ensure that the Distributed switch version radio button is selected and click Next. 5. On the Edit settings page, enter the following values and click Next. Setting Value Number of uplinks 2 Network I/O Control Create a default port group Enabled No (deselect check box) On the Ready to complete page, review the entries and click Finish. Repeat the process to create a second Distributed switch version with the same values and vds-sddc-ext as the name Enable jumbo frames on the vds-sddc and vds-sddc-ext distributed switches Right-click the vds-sddc distributed switch and select Settings > Edit Settings. Click the Advanced tab. Enter 9000 as the MTU (Bytes) value and click OK. Repeat the process for vds-sddc-ext distributed switch. Copyright 2016 IBM and VMware Page 80 of 143

81 Create new port groups in the distributed switch. Right-click the vds-sddc distributed switch and select Distributed Port Group > New Distributed Port Group. Three port groups must be created, one at a time, with the following settings. Port Group Name Port Binding VLAN type VLAN ID vds-sddc-management Ephemeral - no binding None None vds-sddc-internal- Management Ephemeral - no binding VLAN 1557 vds-sddc-vmotion Static binding VLAN 1557 vds-sddc-vsan Static binding VLAN Enter the Port Group Name on the first screen and select the appropriate port binding and VLAN type on the second screen. During NSX Manager configuration you will create another VXLAN port group. 4. On the Ready to complete page, review the entries and click Finish. 5. Repeat this process for the remaining port groups from steps 1 to Right-click the vds-sddc-ext distributed switch and select Distributed Port Group > New Distributed Port Group. One port group must be created with the following settings. Port Group Name Port Binding VLAN type VLAN ID Copyright 2016 IBM and VMware Page 81 of 143

82 vds-sddc-ext- Management Static binding VLAN None 8. Enter the Port Group Name on the first screen and select the appropriate port binding and VLAN type on the second screen. 9. On the Ready to complete page, review the entries and click Finish. 18 Attach the ESXi hosts to the distributed switch 18.1 Attach the ESXi hosts to the private distributed switch, configure the VMkernel network adapters, edit the existing, and add new adapters as needed Right-click the vds-sddc distributed switch and click Add and Manage Hosts. On the Select task page, select Add hosts and click Next On the Select hosts page, click New hosts. In the Select new hosts dialog box, select all four hosts and click OK. Copyright 2016 IBM and VMware Page 82 of 143

83 On the Select hosts page, select Configure identical network settings...(template mode) check box and click Next. On the Select template host page, select the first host as a template host and click Next. On the Select network adapter tasks page, ensure that both Manage physical adapters (Template mode) and Manage VMkernel adapters (template mode) check boxes are selected, and click Next. On the Manage physical network adapters (template mode) page, click vmnic0 and click Assign uplink. In the Select an Uplink for vmnic0 dialog box, select Uplink 1 and click OK. 10. On the Manage physical network adapters (template mode) page, click Apply to all and click Next. 11. On the Manage VMkernel network adapters (template mode) page, click vmk0 and click Assign port group. Copyright 2016 IBM and VMware Page 83 of 143

84 vmnic Source Port Group Destination port group Port Properties vmk0 Management Network vds-sddc- Management Management traffic 12. In the Assign destination port groups dialog box, select vds-sddc- Management and click OK. 13. On the Manage VMkernel network adapters (template mode) page, click vmk1 and click Assign port group. vmnic Source Port Group Destination port group Port Properties vmk1 Internal Management Network vds-sddc-internal- Management Management traffic 14. In the Assign destination port groups dialog box, select vds-sddc-internal- Management and click OK. 15. On the Manage VMkernel network adapters (template mode) page, click On this switch and click New Adapter to add a new VM kernel adapter. 16. On the Select target device page, select vds-sddc-vmotion as the existing network and click Next. 17. On the Port properties page, select vmotion traffic and click Next. 18. Under IPv4 settings, select Use static IPv4 settings, enter as the IPv4 address, enter as subnet mask, and click Next. Adapter Existing network Service Static IPv4 Address Subnet mask MTU vmk2 vds- SDDCvMotion Virtual vmotion traffic On the Ready to complete page, click Finish. 20. Click Edit adapter to change the MTU setting for the vmk2 adapter. 21. In the vmk2 - Edit Settings wizard, click the NIC Settings page, enter 9000 as MTU value, and click OK. 22. Add more network adapters with the following values. Adapter Existing network Service Static IPv4 Address Subnet mask MTU Copyright 2016 IBM and VMware Page 84 of 143

85 vmk3 vds- SDDC- VSAN Virtual SAN traffic On the Manage VMkernel network adapters (template mode) page, click Apply to all. 24. In the sddc01esxi01...configuration to other hosts dialog box, enter the following IPv4 addresses, respective for each of the VMkernel adapters, and click OK. vmk IPv4 address vmk , , vmk #3 vmk #3 vmk #3 25. Click Next. 26. On the Analyze impact page, click Next. 27. On the Ready to complete page, review the entries and click Finish Attach the ESXi hosts to the public distributed switch Right-click the vds-sddc-ext distributed switch and click Add and Manage Hosts. Copyright 2016 IBM and VMware Page 85 of 143

86 2. On the Select task page, select Add hosts and click Next On the Select hosts page, click New hosts. In the Select new hosts dialog box, select all four hosts and click OK On the Select hosts page, select Configure identical network settings...(template mode) check box and click Next. On the Select template host page, select the first host as a template host and click Next. On the Select network adapter tasks page, ensure that only the Manage physical adapters (Template mode) check box is selected and click Next. On the Manage physical network adapters (template mode) page, click vmnic1 and click Assign uplink. In the Select an Uplink for vmnic1 dialog box, select Uplink 1 and click OK. 10. In the Select an Uplink for vmnic3 dialog box, select Uplink 2 and click OK. 11. On the Manage physical network adapters (template mode) page, click Apply to all and click Next. 12. On the Analyze impact page, click Next. 13. On the Ready to complete page, review the entries and click Finish. Copyright 2016 IBM and VMware Page 86 of 143

87 18.3 Migrate the Platform Services Controller, vcenter Server, and Domain Controller instances from the standard switch to the distributed switch Right-click the vds-sddc distributed switch and click Migrate VM to Another Network. On the Select source and destination networks page, browse the following networks and click Next. Setting Source network Destination network Value VM Network vds-sddc-internal-management On the Select VMs to migrate page, select sddc01dc01, sddc01dc02, sddc01ca01, sddc01psc01.tornado.local, and sddc01vc01.tornado.local, and click Next. On the Ready to complete page, review the entries and click Finish Define Network I/O Control Shares values for the different traffic types In the Navigator, click the Networking icon and click the SDDC data center. Click the vds-sddc distributed switch. Click the Manage tab and click Resource Allocation. Under System Traffic, edit each of the following traffic types with the following values. Traffic Type Physical Adapter Shares Virtual SAN Traffic High, 100 vmotion Traffic High, Migrate the last physical adapter from the standard switch to the distributed switch Right-click the vds-sddc distributed switch and select Add and Manage hosts. On the Select task page, select Manage host networking and click Next. On the Select hosts page, click Attached hosts. In the Select member hosts dialog box, select all four ESXi hosts and click OK. On the Select hosts page, click Next. On the Select network adapter tasks page, select Manage Physical adapters only and click Next. Copyright 2016 IBM and VMware Page 87 of 143

88 On the Manage physical network adapters page, under sddc01esx01.tornado.local, select vmnic2 and click Assign uplink. In the Select an Uplink dialog box, select Uplink2 and click OK. Assign uplinks for the three remaining hosts to reassign their vmnics and click Next. 10. On the Analyze Impact page, click Next. 11. On the Ready to complete page, click Finish. 19 Enable VSAN Virtual SAN or VSAN is VMware s storage component of a software defined data center. It pools disk space from the ESXi hosts and creates a high throughput, highly available distributed shared storage. The ESXi hosts in this design include two SSD drives which VSAN uses for caching and the majority of IO requests are serviced from the cache in most workloads. The eight SATA drives per host provide the persistent storage for the entire cluster. VSAN provides replication of data automatically to withstand the loss of any disk or even an entire server without loss of data or functionality Verify VSAN network connectivity Before proceeding with the VSAN deployment, verify that the VLAN allocated for VSAN is correctly established on the hosts. See Appendix A for the specific VLAN SSH into the first ESXi host. Run ping or vmkping and verify connectivity to each ESXi host in the VSAN cluster of the assigned VSAN vmkernel interface IP addresses. NOTE: Use the -I switch in the ping or vmkping command and specify the vmkernel adapter assigned to VSAN. vmkping -I vmk3 -d -s vmkping -I vmk3 -d -s vmkping -I vmk3 -d -s Configure VSAN vmkernel IP multicast VMware VSAN requires IP multicast to establish the VSAN cluster and exchange information between the hosts. SoftLayer uses the range to for multicast in each pod. Each ESXi host in a VSAN cluster is joined to the same two multicast IP addresses: Agent Group Multicast Address: Master Group Multicast Address: Copyright 2016 IBM and VMware Page 88 of 143

89 Log in to the sddc01esx01.tornado.local host over SSH using Putty with the root user name and esxi_root_user_password. Run the following command to configure the multicast IP membership for the vsan vmk. esxcli vsan network ipv4 set -i vmkx -d u where vmkx = 3, the vmk ID number of the vsan vmkernel interface. The output will be similar to the following. esxcli vsan network list Interface VmkNic Name: vmk2 IP Protocol: IPv4 Interface UUID: 2976c456-7c33-cb85-0e85-0cc47a6657f0 Agent Group Multicast Address: Agent Group Multicast Port: Master Group Multicast Address: Master Group Multicast Port: Host Unicast Channel Bound Port: Multicast TTL: 5 Run this same command, using the same multicast IP addresses, on each ESXi host participating in the same vsan cluster Install Avago storcli Next serveral steps should be taken to verify the disk related hardware and firmware are defined correctly for the VSAN deployment. The storcli command from Avago, the supplier of the disk controller is needed to query and configure the disk controller settings. This step outlines the process to install the Avago Storcli command on each ESXi host Download ZIP Using a browser on the utility server, navigate to and accept the download. Unzip StorCLI.zip file Transfer file Using the WinSCP application on the utility server, transfer the vib file that was downloaded in the previous step to each ESXi host. The VIB required is from the VMware-MN folder. Be sure to place the file in /tmp. Install Storcli Using Putty, SSH into each host and run the following command: esxcli software vib install -v=/tmp/ vmware-esx-storcli vib --no-sig-check Copyright 2016 IBM and VMware Page 89 of 143

90 5. Verify Storcli To verify the installation of Storcli succeeded, SSH into each host using Putty and execute the following command: esxcli software vib list grep -i storcli 19.4 Verify RAID Controller Verify that the correct RAID controller was installed via SoftLayer. To do this, login each ESXi host and execute the following command: /opt/lsi/storcli/storcli /c0 show grep -I avago The results should show AVAGO MegaRAID SAS i At the time of writing the VMware certified firmware version for the MegaRAID SAS i controller is version This should be included as part of the bare metal system deployment. This can be verified, and updated if necessary, using StorCLI /opt/lsi/storcli/storcli /c0 show all If the server does not include the certified version, it can be upgrade from SoftLayer portal. For each Device, Put ESXi host into maintenance mode Power off the host Go to device deteails Copyright 2016 IBM and VMware Page 90 of 143

91 4. Click Actions > Update Firmware. 6. To check if the configuration is successful you can check by running esxcli storage core adapter list or esxcli storage vib list grep scsi-megaraid-sas or esxcfg-module -i megaraid_sas 19.5 Driver Upgrade If for any reason, the previosus upgrade of the firmware fails, it can be done manually using the following steps Download the Avago (LSI) i driver from the VMware website MEGARAID-SAS OEM&productId=491. Unzip megaraid_sas zip file Transfer file Using the WinSCP application on the utility server, transfer the vib file that was downloaded in the previous step to each ESXi host. The VIB required is megaraid_sas zip. Be sure to place the file in /tmp. Install the driver Using Putty, SSH into each host and run the following command: Copyright 2016 IBM and VMware Page 91 of 143

92 esxcli software vib install -v=/tmp/scsi-megaraid-sas_ oem vib Next the the old lsi_mr3 driver must be disabled esxcfg-module -d lsi_mr3 One by one, put each host in maintenance mode, reboot, and then exit maintenance mode To check if the configuration is successful you can check by running esxcli storage core adapter list or esxcli storage vib list grep scsi-megaraid-sas or esxcfg-module -i megaraid_sas 19.6 Prepare disks for VSAN The Avago StorCLI is now used to verify and reconfigure the SoftLayer deployed disk layout. Thus, connect to each ESXi hosts via Putty. 1. vsan associated disks must configured as RAID0 with cache specified as: RWTD (read ahead always, write through, direct). Currently the SoftLayer provisioning process will set them up as RWBD (read ahead always, write back, direct) so this setting must be modified before proceeding. The example below is the output of a default ESXi host deployed in SL using the StorCli utility: /opt/lsi/storcli/storcli /c0 show all 2. Reconfiguring the disks is done using the StorCli command to first delete the virtual disks, and then re-create them. From the above example, this is done using the following commands: NOTE: Be sure to match the commands to the disk layout in your system, and adjust as necessary /opt/lsi/storcli/storcli /c0/v1 del /opt/lsi/storcli/storcli /c0/v2 del /opt/lsi/storcli/storcli /c0/v3 del /opt/lsi/storcli/storcli /c0/v4 del /opt/lsi/storcli/storcli /c0/v5 del Copyright 2016 IBM and VMware Page 92 of 143

93 /opt/lsi/storcli/storcli /c0/v6 del /opt/lsi/storcli/storcli /c0/v7 del /opt/lsi/storcli/storcli /c0/v8 del /opt/lsi/storcli/storcli /c0/v9 del /opt/lsi/storcli/storcli /c0/v10 del 3. /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan-ssd drive=8:2 nora wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:3 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:4 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:5 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:6 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan-ssd drive=8:7 nora wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:8 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:9 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:10 ra wt direct strip=256 /opt/lsi/storcli/storcli /c0 add vd type=raid0 name=vsan drive=8:11 ra wt direct strip=256 Once the disks are re-created, the output of the StorCli command now displays the proper RAID-0 and cache configuration: /opt/lsi/storcli/storcli /c0 show all Copyright 2016 IBM and VMware Page 93 of 143

94 19.7 Identify SSD drives The basic networking for the ESXi hosts is now configured and ready for vsan. The next steps are to prepare the ESXi storage array controller and disks associated with the vsan cluster. First, because the disks are configured in RAID0 mode, vsphere cannot determine which of the disks are SSD/Flash. Therefore, the SSD drives must to be manually marked as Flash or SSD. 1. In a browser go to and log in using the following credentials. Setting User name Password Value administrator@vsphere.local vcenter_admin_password Navigate to the first ESXi host. Mark the SSD drives as flash devices on the ESXi host by going to Configuration > Manage > Storage > Storage Devices tab as follows Select the two SSD drives. These will be the drives that have a value of 1.09 TB. After clicking the F (flash) button, verify that the appropriate SSD disk is selected and click Yes when presented with the following warning. Copyright 2016 IBM and VMware Page 94 of 143

95 6. Repeat for all hosts in the cluster. When all of the disk devices are prepared, the vsan cluster is ready to be created Set disks as local Now the Avago StorCLI utility is used to configure the recently configured disks layout as local. Thus, connect to each ESXi hosts via Putty. Using Putty, SSH into each host and run the following command: for sldisk in $(esxcli storage core device list grep "Is Local: false" -B 15 grep naa awk '{print $1}' grep naa) do esxcli storage nmp satp rule add --satp=vmw_satp_local --device $sldisk -- option "enable_local" esxcli storage core claiming unclaim --type=device --device $sldisk esxcli storage core claimrule load esxcli storage core claimrule run esxcli storage core claiming reclaim -d $sldisk done 19.9 Additional ESXi configuration There are two additional configuration required prior to enabling and using vsan. This step outlines the process to applied the fixes from VMware KB KC&externalId= and KC&externalId= for VSAN 6.x. 1. Connect to each ESXi hosts via Putty and SSH into each host and run the following command to configure the vsan IO timeout settings. These result in a persistent changes and remains configured even after the hosts are rebooted. esxcfg-advcfg -s /LSOM/diskIoTimeout esxcfg-advcfg -s 4 /LSOM/diskIoRetryFactor Copyright 2016 IBM and VMware Page 95 of 143

96 Verify that the command has been run successfully or whether it needs to be run. This should show that the value of diskiotimeout is and diskioretryfactor is 4 once the commands has been run. esxcfg-advcfg -g /LSOM/diskIoTimeout esxcfg-advcfg -g /LSOM/diskIoRetryFactor Place the first ESXi host into maintenance mode with the Ensure Accessibility option. Using Putty, SSH into first ESXi host and run the following command to disable the dedup scanner. These result in a persistent changes and remains configured even after the hosts are rebooted. esxcfg-advcfg s 0 /LSOM/lsomComponentDedupScanType Verify the command has been run successfully. This should show that the value of lsomcomponentdedupscantype is 0 when disabled esxcfg-advcfg g /LSOM/lsomComponentDedupScanType The first ESXi host should be rebooted. Wait until host reboot be completed, and put it out from Maintenance Mode. Steps3 to 6 should be repeated on each ESXi host one by one. When all of the ESXi are prepared, the vsan cluster is ready to be created Enable vsan on the cluster Select the SDDC01DC01 cluster. Click the Manage tab and click General under Virtual VSAN section. Click Configure. Copyright 2016 IBM and VMware Page 96 of 143

97 4. Select Manual on Add disks to storage option Click Next. Click Next. On Claim disks, select the disks that will be used by VSAN. Remember that each ESXi hosts will contain two disk groups. Each disk group includes one SSD drive and four SATA drives. Click Next. Click Finish Rename the Virtual SAN datastore The VSAN datastore is initially created with a generic name, which may be sufficient. However, typically you will change it to a client-selected name Select the SDDC01DC01 cluster. Click Related Objects and click Datastores. Copyright 2016 IBM and VMware Page 97 of 143

98 3. Select vsandatastore and click Actions > Rename. 4. In the Datastore - Rename dialog box, enter SDDC-VSAN01-SDDC01DC01 as the datastore name and click OK. 20 Migrate existing VMs to VSAN The VMs created so far were created on local storage on the first ESXi host. With the VSAN cluster created, you can migrate these VMs to the VSAN cluster In the Navigator, click the sddc01dc01 VM and select the Migrate option. Select the Change storage only option as the migration type. Select SDDC-VSAN01-SDDC01DC01 as the destination datastore. Review the settings and click Finish to start the migration. Repeat the step for sddc01dc02, sddc01ca01, sddc01psc01, and sddc01vc01 virtual machines. 21 Set the virtual machines to the default Virtual SAN storage policy In the Navigator, click Hosts and Clusters. Expand the entire SDDC01DC01 cluster tree. Select the sddcpsc01.tornado.local virtual machine. Click the Manage tab, click Policies, and click Edit VM Storage Policies. Copyright 2016 IBM and VMware Page 98 of 143

99 5. In the sddcpsc01.tornado.local:manage VM Storage Policies dialog box, from the VM storage policy drop-down menu, select Virtual SAN Default Storage Policy, and click Apply to all Click OK to apply the changes. Verify that the Compliance Status column shows a Compliant status for all items in the table. 8. Repeat the step to apply the Virtual SAN Default Storage Policy on the sddc01dc01, sddc01dc02, sddc01ca01, and sddc01vc01 virtual machines. 22 Enable vsphere HA for the Cluster In the Navigator, click Hosts and Clusters. Expand the entire sddc01dc01vc01.tornado.local tree and click the SDDC01DC01 cluster. Click the Manage tab, click Settings, click vsphere HA, and click the Edit button. In the Edit Cluster Settings dialog box, select the Turn on vsphere HA check box. Copyright 2016 IBM and VMware Page 99 of 143

100 In the Edit Cluster Settings dialog box, under Virtual Machine Monitoring, select VM Monitoring Only from the drop-down menu. Under Virtual Machine Monitoring, expand the Failure conditions and VM response setting. Select Shutdown and restart VMs from the Response for Host Isolation dropdown menu. Under Virtual Machine Monitoring, expand the Admission Control setting. Select Define failover capacity by reserving a percentage of the cluster resources, enter the following values, and click OK. Setting Value Reserved failover CPU capacity (% CPU) 25 Reserved failover Memory capacity (% Memory) Assign the newly added license to the vcenter Server Appliance Go to Home. Click the Licensing icon in the bottom section. Click the Licensing tab. Click the Assets tab. Select Clusters and click the Assign License icon. Select the VSAN license entered in the previous step and click OK. Copyright 2016 IBM and VMware Page 100 of 143

101 23 NSX configuration At this stage the VMware cluster has been created with the software defined compute and storage established. The next step is to deploy the software defined networking, which in this design is provided by VMware NSX. A VMware NSX deployment includes manager, controller, and edge components. Installation of NSX begins with deploying the manager component as covered in this step. The controller and edge deployments are covered in subsequent sections. Note that there is a one-to-one relationship between NSX managers and vcenter servers. Every instance of NSX Manager is connected to one, and only one, vcenter server. The NSX manager provides a UI interface and APIs to manage and configure the controller and edge components Prepare permissions for NSX The first step is to create an administrator userid for the NSX manager. 1. In a browser, go to and log in using the following credentials. Setting User name Password Value administrator@vsphere.local vcenter_admin_password Assign a service account the vcenter Server Administrator role. 2. In the Navigator, click Hosts and Clusters. 3. Select the sddc01vc01.tornado.local tree. Copyright 2016 IBM and VMware Page 101 of 143

102 4. Click the Manage tab, click Permissions, and click the Add icon In the sddc01vc01.tornado.local - Add Permission dialog box, click the Add button. In the Select Users/Groups dialog box, select TORNADO from the Domain drop-down menu. In the search box, enter svc-nsxmanager and press Enter. Select svc-nsxmanager and click Add. Click OK. 10. In the sddc01vc01.tornado.local - Add Permission dialog box, select Administrator as Assigned Role and select the Propagate to children check box. Copyright 2016 IBM and VMware Page 102 of 143

103 11. Click OK Deploy the NSX manager You can now deploy the NSX manager component Open a browser and download the appliance from In the Navigator, click Hosts and Clusters. Copyright 2016 IBM and VMware Page 103 of 143

104 Expand the entire sddc01dc01vc01.tornado.local tree. Right-click the SDDC01DC01 cluster and click Deploy OVF Template. On the Select source page, click the Browse button, select the VMware NSX Manager.ova file VMware-NSX-Manager ova, and click Next. On the Review details page, select the Accept extra configuration option check box and click Next. On the Accept License Agreements page, click Accept and click Next. On the Select name and folder page, enter the following values and click Next. Setting Name Folder or Datacenter Value sddc01nsxm01 Discovered virtual machines 9. On the Select storage page, enter the following values and click Next. Setting VM Storage Policy Datastore Value Virtual SAN Default Storage Policy SDDC-VSAN01-SDDC01DC01 Copyright 2016 IBM and VMware Page 104 of 143

105 10. On the Setup networks page, under Destination, select vds-sddc- Management and click Next. 11. On the Customize template page, expand all options, enter the following values, and click Next. Setting Value CLI "admin" User Password / enter mngnsx_admin_password CLI "admin" User Password / confirm CLI Privilege Mode Password / enter CLI Privilege Mode Password / confirm Hostname mngnsx_admin_password mngnsx_privilege_password mngnsx_privilege_password sddc01nsxm01.tornado.local Network 1 IPv4 Address Network 1 Netmask Copyright 2016 IBM and VMware Page 105 of 143

106 Setting Value Default IPv4 Gateway DNS Server List (separated by a space) Domain Search List NTP Server List Enable SSH tornado.local servertime.service.softlayer.com Yes (Select check box) 12. On the Ready to complete page, select the Power on after deployment check box and click Finish Connect the NSX Manager to the Management vcenter Server In a browser, go to Use the following credentials to log in. Setting User name Password Value admin mngnsx_admin_password Click Manage vcenter Registration. Under Lookup Service, click the Edit button. In the Lookup Service dialog box, enter the following values and click OK. Setting Lookup Service IP Value sddc01psc01.tornado.local Lookup Service Port 443 SSO Administrator User Name Password administrator@vsphere.local vcenter_admin_password In the Trust Certificate? dialog box, click Yes. Under vcenter Server, click the Edit button. In the vcenter Server dialog box, enter the following values and click OK. Copyright 2016 IBM and VMware Page 106 of 143

107 Setting vcenter Server vcenter User Name Password Value sddc01vc01 svc-nsxmanager_password 9. In the Trust Certificate? dialog box, click Yes. 10. Wait until the Status indicators for the Lookup Service and vcenter Server change to Connected. 11. Go to the Home menu, click on Manage Appliance Settings, click Edit on Time Settings, and modify the Time zone to be consistent with time zone configuration for the rest of the environment Add new licenses for the NSX 1. In a browser go to and log in using the following credentials. Setting User name Password Value svc-nsxmanager@tornado.local svc-nsxmanager_password Migrate the new NSX Manager to the Edge ResPool resource pool. Click the Licensing icon in the bottom section Click the Licenses tab. Click the Create New Licenses icon to add license keys. On the Enter license keys page, enter the license key previously acquired for NSX and click Next. On the Edit license name page, enter a NSX and click Next. On the Ready to complete page, review the entries and click Finish Assign the newly added license to NSX 1. Click the Assets tab. 2. Select Solutions and click the Assign License icon. 3. Select the NSX license entered in the previous step and click OK. Copyright 2016 IBM and VMware Page 107 of 143

108 24 Deploy the NSX Controllers With the NSX Manager successfully created and connected to the Management vcenter Server, the next step is to deploy the NSX Controller nodes that form the NSX Controller cluster. The controller nodes are the central control point for the logical switches and maintain the information of the virtualized network including the VXLANs, virtual servers, and hosts. The NSX architecture requires a minimum of three controller nodes, which is the number that will be used in this design. Each node must be deployed one at a time after the previous one is successfully deployed Configure an IP pool for the NSX controller cluster From a browser, log in to the Management vcenter Server by using the vsphere Web Client with these credentials. Setting User name Password Value svc-nsxmanager@tornado.local svc-nsxmanager_password 2. Under Inventories, click Networking & Security In the Navigator, click NSX Managers. Under NSX Managers, click the instance. Click the Manage tab, click Grouping Objects, click IP Pools, and click the Add New IP Pool icon. Copyright 2016 IBM and VMware Page 108 of 143

109 6. In the Add Static IP Pool dialog box, enter the following values and click OK. Setting Name Value Mgmt01-NSXC01 Gateway Prefix Length 26 Primary DNS Secondary DNS DNS Suffix tornado.local Static IP Pool Deploy the NSX Controller cluster In the Navigator, click Networking & Security and click Installation. Under NSX Controller nodes, click the Add icon. 3. In the Add Controller page, enter the following values and click OK. The password is only specified and configured during the deployment of the first controller. The other controllers use the same password. Setting Value NSX Manager Datacenter SDDC Copyright 2016 IBM and VMware Page 109 of 143

110 Setting Cluster/Resource Pool Datastore Folder Connected To IP Pool Password Confirm Password Value Edge ResPool SDDC-VSAN01-SDDC01 Discovered virtual machine vds-sddc-internal-management Mgmt01-NSXC01 mngnsx_controllers_password mngnsx_controllers_password 4. After the Status of the controller node changes to Connected, repeat the step and deploy the remaining two NSX Controller nodes with the same configuration to form the controller cluster Configure DRS affinity rules for the NSX Controllers Affinity rules are required to ensure that the three controllers run different physical hosts. This ensures that NSX can continue to run if a host is lost. 1. Return to the Home page. Copyright 2016 IBM and VMware Page 110 of 143

111 In the Navigator, click Hosts and Clusters and expand the sddc01vc01.tornado.local tree. Select the SDDC01 cluster and click the Manage tab. Under Configuration, click VM/Host Rules. Under VM/Host Rules, click Add. In the SDDC01DC01 - Create VM/Host Rule dialog box, enter the following values and click Add. Setting Name Enable rule Type Value SDDC_NSX_Controllers Yes (select check box) Separate Virtual Machine In the Add Rule Member dialog box, select the three NSX Controller VMs and click OK. In the SDDC01DC01 - Create VM/Host Rule dialog box, click OK. 25 Prepare the ESXi Hosts for NSX The underlying functionality of NSX such as VXLAN bridging, routing, and firewall is performed within the ESXi hypervisor kernel. This functionality is added to the ESXi hypervisor by installing NSX kernel modules packaged in VIB files Install the NSX kernel modules on the management cluster ESXi hosts. From a browser, log in to the Management vcenter Server by using the vsphere Web Client with these credentials. Setting User name Password Value svc-nsxmanager@tornado.local svc-manager_password 2. In the Navigator, click Networking & Security. Copyright 2016 IBM and VMware Page 111 of 143

112 3. In the Navigator, click Installation and click the Host Preparation tab. 4. Change the NSX Manager to Under Installation Status, click Install for SDDC01DC01 clusters. 6. Verify that the Installation Status column shows the NSX version for all hosts in the cluster to confirm that NSX kernel modules are successfully installed. Copyright 2016 IBM and VMware Page 112 of 143

113 26 Configure the NSX logical network Now you can create the components basic VXLAN structure Configure the Segment ID allocation From a browser, log in to the Management vcenter Server by using the vsphere Web Client with these credentials. Setting User name Password Value svc-manager@tornado.local svc-manager_password In the Navigator, click Networking & Security. Click Installation, click the Logical Network Preparation tab, and click Segment ID. 4. In the NSX Manager drop-down menu, select Click Edit, enter the following values, and click OK. Copyright 2016 IBM and VMware Page 113 of 143

114 Setting Value Segment ID pool Enable Multicast addressing No Configure the VXLAN networking In the Navigator, click NSX Managers. Under NSX Managers, click the instance. Click the Manage tab, click Grouping Objects, click IP Pools, and click the Add New IP Pool icon. In the Add Static IP Pool dialog box, enter the following values and click OK. Setting Name Value Mgmt01-NSXVTEP01 Gateway Prefix Length 24 Primary DNS Secondary DNS DNS Suffix tornado.local Static IP Pool Configure the VXLAN networking Click the Host Preparation tab. Under VXLAN, click Not Configured, enter the following values, and click OK. Setting Switch Value vds-sddc VLAN 0 MTU 1600 Copyright 2016 IBM and VMware Page 114 of 143

115 VMKNic IP Addressing VMKNic Teaming Policy Mgmt01-NSXVTEP01 Load Balance - SRCID VTEP Configure the transport zone With Installation still selected in the Navigator, click the Logical Network Preparation tab and click Transport Zones. Under NSX Managers, click the instance. 3. Click the Add New Transport zone icon, enter the following values, and click OK. Setting Name Replication mode Select clusters part of the Transport Zone Value SDDC Transport Zone Unicast SDDC01 Copyright 2016 IBM and VMware Page 115 of 143

116 27 Deploy and configure gateway The final components of an NSX implementation are the edge gateways. These provide the interface or connectivity between the virtualized network within the VMware environment and the physical, external network Deploy Network Exchange Logical Switch Deploy a Logical Switch to host the router exchange network. This network is used for the interconnection of management application gateways to transit traffic between networks and also exchange routing information in the form of OSPF. Six logical switches will be created 1. In the Navigator, click Networking & Security and click Logical Switches. 2. From the NSX Manager drop-down menu, select Click the New Logical Switch icon to create a new logical switch. 4. In the New Logical Switch dialog box, enter the following values and click OK. Setting Name Transport Zone Replication mode Enable IP Discovery Enable MAC Learning Value networkexchange-vxlan SDDC Transport Zone Unicast Selected Deselected 28 Deploy and configure NSX Edge The instructions in this section configure ESG SoftLayer NS. Copyright 2016 IBM and VMware Page 116 of 143

117 28.1 Create edge 1. In a browser, go to and log in using the following credentials. Setting User name Password Value svc-nsxmanager@tornado.local svc-nsxmanager_admin_password 2. In the Navigator, click Networking & Security and click NSX Edges. 3. From the NSX Manager drop-down menu, select Click the Add icon to deploy a new NSX Edge. 5. On the Name and description page, enter the following values and click Next. Setting Install Type Name Hostname Deploy NSX Edge Enable High Availability Value Edge Service Gateway softlayer01-edge softlayer01-edge Selected Selected Copyright 2016 IBM and VMware Page 117 of 143

118 6. On the Settings page, enter the following values and click Next. Setting User Name Password Enable SSH access Enable auto rule generation Edge Control Level logging Value admin sddc_edge_admin_password Selected Selected INFO Copyright 2016 IBM and VMware Page 118 of 143

119 7. On the Configure deployment page, enter the following values. Click the Add icon to configure two appliances with identical settings. Setting Datacenter Appliance Size Value SDDC Large 8. In the Add NSX Edge Appliance dialog box, enter the following values and click OK. Setting Cluster/Resource Pool Datastore Folder Value Edge ResPool SDDC-VSAN01-SDDC01DC01 Discovered virtual machine 9. On the Configure deployment page, click Next. 10. On the Configure Interfaces page, click the Add icon to configure a new interface. Copyright 2016 IBM and VMware Page 119 of 143

120 11. In the Add NSX Edge Interface dialog box, enter the following values. Setting Name Type Connected To Connectivity Status Value Internal-Management Uplink vds-sddc-internal-management (Distributed Portgroup) Connected MTU Click the Add icon. 13. Under Primary Address, enter Under Subnet Prefix length, enter 26 and click OK. 15. Add a second Primary Address, enter Under Subnet Prefix length, enter 26 and click OK Configure additional interfaces Configure one more interface with the following settings, and click Next. Name Type Connected To Connectivit y Status Primary Address Secondary Address Subne t Prefix Lengt h MTU Public Uplin k vds-sddc-ext- Management (Distributed Port Group) Connected networkexchang e Uplin k (Logical Switch) networkexchang e-vxlan Connected On the Default gateway settings page, use the values in the following table. For the others, leave the Configure Default Gateway check box unselected. Setting Configure Default Gateway Selected Copyright 2016 IBM and VMware Page 120 of 143

121 vnic Public Gateway IP MTU On the Firewall and HA page, enter the following values and click Next. Setting Configure Firewall default policy Default Traffic Policy Logging vnic Value Selected Accept Enable Any Declare Dead Time 15 Management IPs / /30 4. On the Ready to complete page, review the entries and click Finish. 29 SNAT rules and static routes Source NAT (SNAT) rules and static routes are required to enable VMware solutions to access the public network and the SoftLayer private network Create Source NAT rules allowing VMware solutions to access public network In a browser, go to and log in using the following credentials. Setting Value Copyright 2016 IBM and VMware Page 121 of 143

122 User name Password svc-nsxmanager_admin_password In the Navigator, click Networking & Security and click NSX Edges. From the NSX Manager drop-down menu, select and doubleclick the softlayer01-edge device to edit its settings. Click the Manage tab, click NAT, and click the Add > Add SNAT Rule icon to create a new Source NAT rule for the vsphere management network. 5. In the Add SNAT Rule dialog box, enter the following values and click OK. Setting Applied On Original Source IP/Range Translated Source IP/Range Enabled Enable logging Value Public / Selected Deselected 6. Under NAT, click Publish Changes for the new SNAT rule changes to take effect Create static route allowing VMware solutions to access SoftLayer private network In a browser, go to and log in using the following credentials. Copyright 2016 IBM and VMware Page 122 of 143

123 Setting User name Password Value svc-nsxmanager_admin_password In the Navigator, click Networking & Security and click NSX Edges. From the NSX Manager drop-down menu, select and doubleclick the softlayer01-edge device to edit its settings. Click the Manage tab, click Routing, and click the Add icon to create new static route. In the Add Static Route dialog box, enter the following values and click OK. Setting Value Network /8 Next Hop Interface Internal-Management MTU 1500 Admin Distance 1 6. Under Routing, click Publish Changes for the new SNAT rule changes to take effect Reconfigure hosts and utility server to use NSX Edge Reconfigure SDDC hosts and VMs to use the new NSX Edge Gateway as their default gateway as show in Appendix D. Update the static route configured in the utility server that was pointing to the SoftLayer BCR to point to the new NSX Edge Gateway. 30 Replace certificates At this stage all the necessary components have been deployed and the vsphere cluster has been created. However, the PSC and the vsphere management components have been deployed using default security certificates. These default certificates, signed by VMware Certificate Authority (VMCA), are not trusted by user devices and therefore a certificate Copyright 2016 IBM and VMware Page 123 of 143

124 warning results when a user connects to a vcenter Server system with the vsphere Web Client. To address this, in this design these user-facing certificates are replaced with certificates that are signed by a custom Microsoft Certificate Authority (CA). Certificates for machine-to-machine communication are not replaced; if necessary, those certificates can be trusted manually. Certificate replacement is required for the following VMware products: Platform Services Controller vcenter Server system VMware NSX Manager The overall process is: Generate a Certificate Signing Request (CSR) for the certificate to be replaced from the machine where the certificate lives. For vcenter Server and Platform Services Controller certificate replacement, use the vsphere Certificate Manager utility. Submit the CSR to the CA where the signed certificate is created. The certificate and the associated root certificate are copied to the machine where they will be used and will replace the existing certificates with the new certificates Create and add a Microsoft Certificate Authority template The first step is to add a custom Certificate Authority (CA) template to the CA server. This IBM Cloud for VMware Solutions design sets up an offline root CA and a redundant subordinate CA on the AD-CA servers sddc01dc01.tornado.local and sddc01dc02.tornado.local, which are running Microsoft Windows Server 2012 R2 and is the Active Directory server Use RDP to connect to the Certificate Authority server, sddc01dc01.tornado.local, as the AD administrator with the ad_admin_password password. Click Start > Run, enter certtmpl.msc, and click OK. In the Certificate Template Console, under Template Display Name, rightclick Web Server and click Duplicate Template. 4. In the Duplicate Template window, leave Windows Server 2003 Enterprise selected for backward compatibility and click OK In the Properties of New Template dialog, click the General tab. In the Template display name text box, enter VMware as the name of the new template. On the Extensions tab, select Application Policies and click Edit. Copyright 2016 IBM and VMware Page 124 of 143

125 8. 9. Select Server Authentication, click Remove, and click OK. Select Key Usage and click Edit. 10. Click the Signature is proof of origin (nonrepudiation) check box, leave the default for all other options, and click OK. 11. Click the Subject Name tab, ensure that the Supply in the request option is selected, and click OK to save the template. 12. To add the new template to the CA, click Start > Run, enter certsrv.msc, and click OK. 13. In the Certification Authority window, expand the left pane if it is collapsed. 14. Right-click Certificate Templates and select New > Certificate Template to Issue. 15. In the Enable Certificate Templates dialog, in the Name column, select the VMware certificate just created and click OK. 16. Repeat the process on sddc01dc02.tornado.local Obtain custom certificate and replace the Platform Services Controller certificate This section illustrates how to generate the signed certificate for the sddc01psc01.tornado.local Platform Services Controller instance and how to replace it Initiate an SSH connection to the Platform Services Controller sddc01psc01. Provide the root user name and password when prompted. Run this command to enable the Bash shell. shell.set --enable True Run this command to access the Bash shell. shell Create a folder for the certificates. mkdir /tmp/ssl Launch the vsphere 6.0 Certificate Manager. /usr/lib/vmware-vmca/bin/certificate-manager Select Option 1 (Replace Machine SSL certificate with Custom Certificate). Provide the administrator@vsphere.local password when prompted. Select Option 1 (Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate). 10. Enter the directory in which you want to save the certificate signing request and the private key. 11. Output directory path is /tmp/ssl. Copyright 2016 IBM and VMware Page 125 of 143

126 12. Enter the appropriate company information for the Country, Name, Organization, OrgUnit, State, Locality, and certificate information. Enter proper value for 'Country' [Default value : US] : Enter proper value for 'Name' [Default value : CA] : Enter proper value for 'Organization' [Default value : VMware] : Enter proper value for 'OrgUnit' [Default value : VMware] : Enter proper value for 'State' [Default value : California] : Enter proper value for 'Locality' [Default value : Palo Alto] : Enter proper value for 'IPAddress' [optional] : Enter proper value for ' ' [Default value : @acme.com] : Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : sddc01psc01.tornado.local 13. The files created will have the names vmca_issued_csr.csr and vmca_issued_key.key in /tmp/ssl. 14. Select Option 2 (Exit certificate-manager). 15. In the Bash shell, run this command to change the default shell to Bash. chsh -s /bin/bash root 16. Use WinSCP to download the certificate files vmca_issued_csr.csr and vmca_issued_key.key from the PSC folder /tmp/ssl. Save the certificate files in a new folder named C:\Users\%username%\Downloads\PSC on the utility server. 17. To return to the Appliance Shell, run this command. chsh -s /bin/appliancesh root 18. Log in to the Microsoft CA certificate authority web interface at Click Request a certificate. 20. Click advanced certificate request. 21. Open the certificate request vmca_issued_csr.csr in the Notepad text editor and copy from -----BEGIN CERTIFICATE REQUEST----- to -----END CERTIFICATE REQUEST----- into the Saved Request field. 22. Select the VMware Certificate Template. 23. Click Submit. 24. Click Base 64 encoded on the Certificate issued page. 25. Click Download Certificate. 26. File is saved at C:\Users\%username%\Downloads\ as certnew.cer. Rename certnew.cer to PSC.crt and move it to the C:\Users\%username%\Downloads\PSC\ directory. Copyright 2016 IBM and VMware Page 126 of 143

127 27. Return to the home page of the certificate server and click Download a CA certificate, certificate chain or CRL. 28. Select the Base 64 option. 29. Click Download CA Certificate chain. 30. The file will be saved in C:\Users\%username%\Downloads\ directory as certnew.p7b. 31. Double-click the certnew.p7b file to open it in the Certificate Manager. 32. Navigate to Certificates. 33. Right-click the first certificate listed (SDDC01CA01-CA), click All Tasks, and select Export. 34. Click Next. 35. Select Base-64 encoded X.509 (.CER) and click Next. 36. Complete the File name field. C:\Users\%username%\Downloads\PSC\SDDC01CA01-CA.cer 37. Click Next. 38. Click Finish. 39. Right-click the second certificate listed (tornado-sddc01dc01-ca), click All Tasks, and select Export. 40. Click Next. 41. Select Base-64 encoded X.509 (.CER) and click Next. 42. Complete the File name field. C:\Users\%username%\Downloads\PSC\ tornado-sddc01dc01-ca.cer 43. Click Next. 44. Click Finish. 45. After completion, use Notepad to concatenate the tornado-sddc01dc01-ca.cer and SDDC01CA01-CA.cer certificates into a single new file named cachain.cer. Example: -----BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----tornado-sddc01dc01-ca.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----SDDC01CA01-CA.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE----- Copyright 2016 IBM and VMware Page 127 of 143

128 46. Using Notepad, concatenate the certificates PSC.crt, tornado-sddc01dc01- CA.cer, and SDDC01CA01-CA.cer into a new single file named sddc01psc01_ssl.cer in the C:\Users\%username%\Downloads\PSC\ directory. Example: -----BEGIN CERTIFICATE----- MIIFxTCCBK2gAwIBAgIKYaLJSgAAAAAAITANBgkqhkiG9w0BAQUFADBGMRMwEQYK CZImiZPyLGQBGRYDbmV0MRYwFAYKCZImiZPyLGQBGRYGbW5uZXh0MRcwFQYDVQQD Ew5tbm5leHQtQUQtMS1DQTAeFw0xMzAyMDExNjAxMDNaFw0xNTAyMDExNjExMDNa <-----PSC.crt SMhYhbv3wr7XraAnsIaBYCeg+J7fKTFgjA8bTwC+dVTaOSXQuhnZfrOVxlfJ/Ydm NS7WBBBFd9V4FPyRDPER/QMVl+xyoaMGw0QKnslmq/JvID4FPd0/QD62RAsTntXI ATa+CS6MjloKFgRaGnKAAFPsrEeGjb2JgMOpIfbdx4KT3WkspsK3KPwFPoYza4ih 4eT2HwhcUs4wo7X/XQd+CZjttoLsSyCk5tCmOGU6xLaE1s08R6sz9mM= -----BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----tornado-sddc01dc01-ca.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----SDDC01CA01-CA.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE Use WinSCP to upload the new certificate files cachain.cer and sddc01psc01_ssl.cer to the PSC path /tmp/ssl. 48. Launch the vsphere 6.0 Certificate Manager. /usr/lib/vmware-vmca/bin/certificate-manager 49. Select Option 1 (Replace Machine SSL certificate with Custom Certificate). 50. Provide the administrator@vsphere.local password when prompted. 51. Select Option 1 (Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate). 52. Select Option 2 (Import custom certificate(s) and key(s) for Machine SSL certificate). 53. Provide a valid custom certificate for Machine SSL. File : /tmp/ssl/sddc01psc01_ssl.cer 54. Provide a valid custom key for Machine SSL. File : /tmp/ssl/vmca_issued_key.key 55. Provide the signing certificate of the Machine SSL certificate File : /tmp/ssl/cachain.cer 56. Answer Yes (Y) to the confirmation request to proceed. Get site namecompleted [Replacing Machine SSL Cert...] ams03 Lookup all services. Copyright 2016 IBM and VMware Page 128 of 143

129 Updated 9 service(s) Status : 100% Completed [All tasks completed successfully] 57. Initiate an SSH connection to the vcenter Server Appliance sddc01vc Provide the root user name and password when prompted. 59. Run this command to enable the Bash shell. shell.set --enable True 60. Run this command to access the Bash shell. shell 61. Restart all running vcenter services. service-control --stop --all service-control --start --all 30.3 Obtain custom certificate and replace the vcenter Server machine certificate After replacing the Platform Services Controller certificate, replace the vcenter Server machine SSL certificate. These step-by-step instructions explain how to replace the certificate for a vcenter Server system named sddc01vc Initiate an SSH connection to the vcenter Server machine sddc01vc01. Provide the root user name and password when prompted. Run this command to enable the Bash shell. shell.set --enable True Run this command to access the Bash shell. shell Create a folder for the certificates. mkdir /tmp/ssl Launch the vsphere 6.0 Certificate Manager. /usr/lib/vmware-vmca/bin/certificate-manager Select Option 1 (Replace Machine SSL certificate with Custom Certificate). Provide the administrator@vsphere.local password when prompted. 9. Provide the vcenter IP Select Option 1 (Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate). 11. Enter the directory in which you want to save the certificate signing request and the private key. 12. Output directory path is /tmp/ssl. Copyright 2016 IBM and VMware Page 129 of 143

130 13. Enter the appropriate company information for the Country, Name, Organization, OrgUnit, State, Locality, and certificate information. Enter proper value for 'Country' [Default value : US] : Enter proper value for 'Name' [Default value : CA] : Enter proper value for 'Organization' [Default value : VMware] : Enter proper value for 'OrgUnit' [Default value : VMware] : Enter proper value for 'State' [Default value : California] : Enter proper value for 'Locality' [Default value : Palo Alto] : Enter proper value for 'IPAddress' [optional] : Enter proper value for ' ' [Default value : @acme.com] : Enter proper value for 'Hostname' [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : sddc01vc01.tornado.local 14. The files created will have the names vmca_issued_csr.csr and vmca_issued_key.key in /tmp/ssl. 15. Select Option 2 (Exit certificate-manager). 16. In the Bash shell, run this command to change the default shell to Bash. chsh -s /bin/bash root 17. Use WinSCP to download the certificate files vmca_issued_csr.csr and vmca_issued_key.key from the PSC folder /tmp/ssl. Save the certificate files in a new folder named C:\Users\%username%\Downloads\VCENTER on the utility server. 18. To return to the Appliance shell, run this command. chsh -s /bin/appliancesh root 19. Log in to the Microsoft CA certificate authority web interface at Click Request a certificate. 21. Click advanced certificate request. 22. Open the certificate request vmca_issued_csr.csr in the Notepad text editor and copy from -----BEGIN CERTIFICATE REQUEST----- to -----END CERTIFICATE REQUEST----- into the Saved Request field. 23. Select the VMware Certificate Template. 24. Click Submit. 25. Click Base 64 encoded on the Certificate issued page. 26. Click Download Certificate. Copyright 2016 IBM and VMware Page 130 of 143

131 27. File is saved at C:\Users\%username%\Downloads\ as certnew.cer. Rename certnew.cer to VCENTER.crt and move it to the C:\Users\%username%\Downloads\VCENTER\ directory. 28. The following steps will reuse three files from the previous section. C:\Users\%username%\Downloads\PSC\tornado-SDDC01DC01-CA.cer C:\Users\%username%\Downloads\PSC\SDDC01CA01-CA.cer C:\Users\%username%\Downloads\PSC\cachain.cer. 29. Using Notepad, concatenate the certificates VCENTER.crt with the files created in the previous step, tornado-sddc01dc01-ca.cer and SDDC01CA01-CA.cer into a new single file named sddc01vc01_ssl.cer in the C:\Users\%username%\Downloads\VCENTER\ folder. Example: -----BEGIN CERTIFICATE----- MIIFxTCCBK2gAwIBAgIKYaLJSgAAAAAAITANBgkqhkiG9w0BAQUFADBGMRMwEQYK CZImiZPyLGQBGRYDbmV0MRYwFAYKCZImiZPyLGQBGRYGbW5uZXh0MRcwFQYDVQQD Ew5tbm5leHQtQUQtMS1DQTAeFw0xMzAyMDExNjAxMDNaFw0xNTAyMDExNjExMDNa <-----VCENTER.crt SMhYhbv3wr7XraAnsIaBYCeg+J7fKTFgjA8bTwC+dVTaOSXQuhnZfrOVxlfJ/Ydm NS7WBBBFd9V4FPyRDPER/QMVl+xyoaMGw0QKnslmq/JvID4FPd0/QD62RAsTntXI ATa+CS6MjloKFgRaGnKAAFPsrEeGjb2JgMOpIfbdx4KT3WkspsK3KPwFPoYza4ih 4eT2HwhcUs4wo7X/XQd+CZjttoLsSyCk5tCmOGU6xLaE1s08R6sz9mM= -----BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----tornado-sddc01dc01-ca.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----SDDC01CA01-CA.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE Use WinSCP to upload the existing certificate files cachain.cer and the new sddc01psc01_ssl.cer to the PSC path /tmp/ssl. 31. Launch the vsphere 6.0 Certificate Manager. /usr/lib/vmware-vmca/bin/certificate-manager 32. Select Option 1 (Replace Machine SSL certificate with Custom Certificate). 33. Provide the administrator@vsphere.local password when prompted. 34. Provide the vcenter IP Select Option 1 (Generate Certificate Signing Request(s) and Key(s) for Machine SSL certificate). 36. Select Option 2 (Import custom certificate(s) and key(s) for Machine SSL certificate). 37. Provide a valid custom certificate for Machine SSL. File : /tmp/ssl/sddc01vc01_ssl.cer Copyright 2016 IBM and VMware Page 131 of 143

132 38. Provide a valid custom key for Machine SSL. File : /tmp/ssl/vmca_issued_key.key 39. Provide the signing certificate of the Machine SSL certificate File : /tmp/ssl/cachain.cer 40. Answer Yes (Y) to the confirmation request to proceed. Get site namecompleted [Replacing Machine SSL Cert...] ams03 Lookup all services. Updated 9 service(s) Status : 100% Completed [All tasks completed successfully] 41. Restart all running vcenter services. service-control --stop --all service-control --start --all 30.4 Obtain and replace the NSX Manager SSL certificate After replacing the certificates of all Platform Services Controller instances and all vcenter Server instances, the certificates for the NSX Manager instances can be replaced Use RDP to connect to the utility server. Log in to the NSX Manager web interface at URL User Password admin nsx_password Click Manage Appliance Settings. In the Settings panel on the left, click SSL Certificates. Under SSL Certificates on the right, click Generate CSR. In the Generate Certificate Signing Request dialog, supply the following values and click OK. Country, Organization, OrgUnit, State and Locality certificate information must be completed with the respective company information. CSR Info Algorithm Value RSA Copyright 2016 IBM and VMware Page 132 of 143

133 Key size 2048 Common Name Organization Unit Organization Name Locality Name State Name Country Code sddc01nsxm01.tornado.local VMware VMware Palo Alto California US 7. Under SSL Certificates, click Download CSR. VMware NSX downloads a.csr file named NSX to the C:\Users\%username%\Downloads directory Rename the file to add the csr extension to the file name. Create a new directory named C:\Users\%username%\Downloads\NSX and move the NSX file to it. 10. Log in to the Microsoft CA certificate authority Web interface Click Request a certificate. 12. Click advanced certificate request. 13. Open the certificate request NSX.csr in the Notepad text editor and copy from BEGIN CERTIFICATE REQUEST----- to -----END CERTIFICATE REQUEST into the Saved Request field. 14. Select the VMware Certificate Template. 15. Click Submit to submit the request. 16. Click Base 64 encoded on the Certificate issued screen. 17. Click Download Certificate. Copyright 2016 IBM and VMware Page 133 of 143

134 18. File is saved at C:\Users\%username%\Downloads\ as certnew.cer. Rename certnew.cer to NSX.crt and move it to the C:\Users\%username%\Downloads\VCENTER\ directory. 19. The following steps will reuse three files from previous sections. C:\Users\%username%\Downloads\PSC\tornado-SDDC01DC01-CA.cer C:\Users\%username%\Downloads\PSC\SDDC01CA01-CA.cer 20. Using Notepad, concatenate the NSX.crt certificates with the files created previously, tornado-sddc01dc01-ca.cer and SDDC01CA01-CA.cer, into a new single file named sddc01nsxm01_ssl.cer in the C:\Users\%username%\Downloads\NSX\ folder. Example: -----BEGIN CERTIFICATE----- MIIFxTCCBK2gAwIBAgIKYaLJSgAAAAAAITANBgkqhkiG9w0BAQUFADBGMRMwEQYK CZImiZPyLGQBGRYDbmV0MRYwFAYKCZImiZPyLGQBGRYGbW5uZXh0MRcwFQYDVQQD Ew5tbm5leHQtQUQtMS1DQTAeFw0xMzAyMDExNjAxMDNaFw0xNTAyMDExNjExMDNa <-----NSX.crt SMhYhbv3wr7XraAnsIaBYCeg+J7fKTFgjA8bTwC+dVTaOSXQuhnZfrOVxlfJ/Ydm NS7WBBBFd9V4FPyRDPER/QMVl+xyoaMGw0QKnslmq/JvID4FPd0/QD62RAsTntXI ATa+CS6MjloKFgRaGnKAAFPsrEeGjb2JgMOpIfbdx4KT3WkspsK3KPwFPoYza4ih 4eT2HwhcUs4wo7X/XQd+CZjttoLsSyCk5tCmOGU6xLaE1s08R6sz9mM= -----BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----tornado-sddc01dc01-ca.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE BEGIN CERTIFICATE----- MIIDZzCCAk+gAwIBAgIQNO7aLfykR4pE94tcRe0vyDANBgkqhkiG9w0BAQUFADBG K73RIKZaDkBOuUlRSIfgfovUFJrdwGtMWo3m4dpN7csQAjK/uixfJDVRG0nXk9pq GXaS5/YCv5B4q4T+j5pa2f+a61ygjN1YQRoZf2CHLe7Zq89Xv90nhPM4foWdNNkr <-----SDDC01CA01-CA.cer /Esf1E6fnrItsXpIchQOmvQViis12YyUvwko2aidjVm9sML0ANiLJZSoQ9Zs/WGC TLqwbQm6tNyFB8c= -----END CERTIFICATE From the web browser that is connected to the NSX Manager interface, with the Manage tab and the SSL Certificate setting still selected on the left, click Import and provide the chained certificate file. Host sddc01nsxm01.tornado.local Filenames sddc01nsxm01_ssl.cer Copyright 2016 IBM and VMware Page 134 of 143

135 22. Reboot NSX Manager so that the custom certificate is used. In the right corner of the NSX Manager page, click the Settings icon. From the drop-down menu, choose Reboot Appliance. 23. Log in to the NSX Manager web interface URL User Password admin nsx_password 24. Click Manage vcenter Registration. 25. Under Lookup Service, click Edit. 26. In the Lookup Service dialog box, enter the password values and click OK. 27. In the Trust Certificate? dialog box, click Yes. 28. Under vcenter Server, click Edit. 29. In the vcenter Server dialog box, enter the password value and click OK. 30. In the Trust Certificate? dialog box, click Yes. 31. Wait until the Status indicators for the Lookup Service and vcenter Server change to Connected. 31 Configure lockdown mode on all ESXi hosts To increase security of the ESXi hosts, each host should be put into lockdown mode, so that administrative operations can be performed only from vcenter Server. vsphere supports an Exception User list, which is for service accounts that must log in to the host directly. Accounts with administrator privileges that are on the Exception Users list can log in to the ESXi shell. In addition, these users can log in to a host's DCUI in normal lockdown mode and can exit lockdown mode. 1. In a browser, go to and log in using the following credentials. Option Value Copyright 2016 IBM and VMware Page 135 of 143

136 User name Password vcenter_admin_password In the Navigator, click Hosts and Clusters, and expand the entire sddc01vc01.tornado.local tree. Select the sddc01esx01.tornado.local host. Click the Manage tab and click Settings. Under System, select Security Profile. In the Lockdown Mode panel, click Edit. In the Lockdown Mode dialog box, select the Normal radio button and click OK. 8. Repeat the previous step to enable normal lockdown mode for all remaining hosts in the data center. Object Management host 2 Management host 3 Management host 4 FQDN sddc01esx02 sddc01esx03 sddc01esx04 Copyright 2016 IBM and VMware Page 136 of 143

137 Copyright 2016 IBM and VMware Page 137 of 143