Dell EMC VxRack System SDDC Upgrade Guide Document revision 1.0 April 2017
Contents Revision history...3 Introduction...4 Upgrade prerequisites...5 Prerequisite skills... 5 General system prerequisites...5 Prerequisite tasks...5 Prerequisites for Lifecycle Manager... 6 Upgrade best practices and planning...7 Estimating the upgrade time...8 Upgrading Converged System releases...9 Updating the VMware Cloud Foundation LCM Management Domain...10 Updating the LCM Workload Domain...15 Upgrading Dell PowerEdge R630 servers... 16 Upgrading firmware and BIOS on Dell servers... 16 Updating PERC H730 firmware...22 Upgrading Cisco switches...23 Upgrading NIC firmware...24 Resolving BMC alerts...28 Contents 2
Revision history Date Document revision Description of changes April 2017 1.0 Initial version 3 Revision history
Introduction This document describes how to upgrade VxRack System SDDC to the current software releases. The current RCM release supports the upgrade from VMware Cloud Foundation 2.1 to 2.1.1. These upgrade instructions apply to the following VxRack System releases: VxRack System release Based On VxRack System SDDC 1.0 VMware Cloud Foundation 2.1 When to use this guide Use this guide to incrementally upgrade from one VxRack System RCM release to another. From this release To this release On these Converged Systems 3.2.1 3.2.2 VxRack System SDDC Refer to the Glossary for terms, definitions, and acronyms. Introduction 4
Upgrade prerequisites Prerequisite skills This topic describes the skills required for performing this upgrade. To perform this upgrade, you should have the following knowledge and skills: Converged System concepts and terminology Troubleshooting Converged Systems Experience in VMware Cloud Foundation system administration Understand virtual machine technology and datacenter operations Ensure that you understand the entire upgrade process before you begin the Converged System upgrade. Complete all of the planning and preparation requirements before you begin the upgrade. General system prerequisites This topic describes the general system prerequisites for performing an upgrade. Before starting the upgrade, you should have access to the following: Network access to the Converged Systems and the management systems Utilities for ping, traceroute, SSH, FTP server, TFTP server, and Telnet Prerequisite tasks Complete the following tasks before you continue upgrading your system. Review the release notes and complete any required actions. Ensure that the appropriate security policies are applied to the Converged Systems. Upgrades do not modify Converged System security. Plan for the upgrade. Baseline security standards for Converged Systems are available in Converged Systems security documents. Calculate the upgrade time. Schedule adequate downtime (if required) for system maintenance. 5 Upgrade prerequisites
Ensure that all health checks have been completed. Confirm that all components are in a ready state with no alarms. Prerequisites for Lifecycle Manager Ensure you meet the prerequisites listed here before starting a Lifecycle Manager (LCM) update. Before starting the LCM update, do the following: Ensure all the steps in the Configuring the LCM section of the VxRack System with SDDC Field Implementation Guide have been completed. Complete all workload domain related tasks (create/delete/expand). Ensure that the Support VM or the client device running the Cloud Foundation Manager is time synced with the SDDC Manager NTP server. To find the NTP server IP address: Log on to the Cloud Foundation Manager, go to Dashboard and then to Workload Domain. Click Management Domain for rack 1, go to MANAGEMENT INFO tab, scroll down and look for the NTP IP address. For the Horizon View VDI environment, the manual patch must be done before any updates to the VMware Cloud Foundation (VCF) environment. Refer to the standard Horizon View procedures for information on patching Horizon View VDI. Ensure you have a valid MyVMware account. Use an existing account or go to https:// my.vmware.com/ for information on creating a new account. In VCF 2.1. there is a known issue where updates fail to download. Follow the steps in the VMware KB to mitigate this issue. Upgrade prerequisites 6
Upgrade best practices and planning This topic describes how to plan for an upgrade and the best practices for performing the upgrade. Use the following guidelines to ensure minimum impact on the production traffic: Review this entire upgrade document and each referenced component upgrade document before you begin the upgrade. Perform the upgrade during scheduled maintenance windows or non-disruptively when virtual machines are running. The best practice is to perform operational verification after each component or logical grouping of components have been upgraded, this should be taken into consideration during assessment and planning. Before upgrading confirm the overall status and software version of each component. Verify that the component is in the ready state and is not generating any alarms or errors. Create a configuration backup file for each Converged System component before you begin the upgrade. As this process can take several hours to complete, upgrades should be scheduled outside of business hours. The Cloud Foundation Manager is available throughout the process, but it is recommended that the Management Domain update be scheduled when the Cloud Foundation Manager is expected to have least amount of traffic. Workload Domain updates should be done after the Management Domain updates are successfully completed. Update each Workload Domain separately to minimize issues. The Management Domain should be on the latest version. Workload Domains should be within three versions of the Management Domain. 7 Upgrade best practices and planning
Estimating the upgrade time This task allows you to estimate the amount of time you need to perform the upgrade. Plan your upgrade using the minimum time that it takes to upgrade the software on each component. The minimum time estimates do not include the time for preparation, configuration, VM migration, backup, and download. Review the list of components and determine the upgrade time for the components in the system. If a component is not in the system, do not estimate the upgrade time for that component. For cluster configurations, include the master and subordinate components. Order Component Upgrade time Notes 1. VMware Cloud Foundation Life Cycle Manager (LCM) LCM Management domain LCM Workload domain 2. Dell PowerEdge R630 node BIOS and Firmware 3. Cisco Nexus Switches (9000 series ToR and Spine) Eight hours per domain One hour for firmware One hour for BIOS Two hours 4. NIC firmware for Dell PowerEdge R630 One hour per server Estimating the upgrade time 8
Upgrading Converged System releases This topic describes the upgrade sequence for Converged System components. Upgrade the software and firmware on the Converged System in the order listed in the following table. Before upgrading, read the release notes and the release certification matrices for the Converged System release on your Converged System. Some Converged System components are optional or specific to a customer environment. If the Converged System that you are upgrading does not have a component listed in the following table, skip that component and proceed with the next component. Order Component 1. VMware Cloud Foundation Lifecycle Manager Management Domain Workload Domain 2. Dell PowerEdge R630 node Upgrade BIOS Upgrade Firmware 3. Cisco Nexus 9000 Series Switches Upgrade guides for Cisco Nexus 9000 Series Switches 4. NIC firmware for the Dell PowerEdge R630 server 9 Upgrading Converged System releases
Updating the VMware Cloud Foundation Lifecycle Manager Management Domain The Lifecycle Manager (LCM) provides for automated updates to Cloud Foundation components. Before you begin Before starting a LCM update, ensure all prerequisites (see page 6) are met. Before starting the upgrade process, get snapshots of all the Cloud Foundation VMs, including SDDC Manager VM (VRM), ISVMs, LCM VM, and the LCM-Backup-Repository VM. To access the VMware vcenter web client do the following: Log on to the SDDC Manager. Go to the Dashboard and click Workload Domains. Click Management Domain for VxRack 1. Under Cluster Info, click VCENTER. Click the Virtual Machines section to show all of the management VMs. During the upgrade process, LCM generates the following backups: PSC VM snapshot vcenter VM snapshot ESXi configuration backup. (The backup directory on the LCM-Backup-Repository VM is / backup/lcm/esx. Backup for each ESXi host is stored in its GUID (stored in VCF database). Each backup is in a date sub folder. An example ESXi backup file name is /backup/lcm/esx/ f32b407e-8fdb-4999-ab80-2082003586fb/02022017014114/backup/ firmwareconfig865759412960483871.tgz) NSX configuration backup. (The backup directory on LCM-Backup-Repository VM is /backup. File names have the prefix evo-nsx-domain_id_date. An example of a NSX backup file name is /backup/evo-nsx-888bcb7f-30d0-3aee-9291- b7777052a1ac-01_00_00_tue07march2017) To display Backup Locations for LCM and ESXi updates once the backups have been completed: On the Lifecycle Management page, click the INVENTORY tab. Click an LCM or ESXi resource. The Resource Details page appears. Click to the right of the component name and then click GET BACKUP LOCATION. The backup file name and location displays. Procedure 1 Log on to the SDDC Manager. When an update bundle is available, a notification displays on the Dashboard. Click on the notification to open the Lifecycle Management window. Optionally, click the Lifecycle tab on the left navigation pane. Updating the VMware Cloud Foundation Lifecycle Manager Management Domain 10
2 The Repository tab shows the number of available updates. 3 Expand the VMware Cloud Foundation drop-down and the VMware Software Update dropdown to see available updates for each section. It may only show VMware Cloud Foundation when there are no Software Updates. All available bundles appear, regardless of the version installed in the environment. Click MORE to see more details. Some items may be marked with Previous Download Required and some may display with a blue Download button. This signifies the order in which the updates will be downloaded. Download all the bundles which have the Download link enabled, even if it is already installed in the environment. 4 Click Download to download the bundle now, or click the blue icon to schedule a download. On the Review Download page, review the download schedule, bundle type and bundle size. Click Download. If the download is not scheduled, the status for the bundle changes to Download starts in less than an hour. After few minutes, click Refresh Status to see download status. 5 Large bundles may take a significant amount of time to download. Completed downloads are marked as Downloaded. Check both the VMware Cloud Foundation and the VMware Software Update sections to verify that everything has been downloaded. VCF 2.1 has a known issue where updates fail to download. Follow the steps in the VMware KB https://kb.vmware.com/kb/2148653 to mitigate the issue. 11 Updating the VMware Cloud Foundation Lifecycle Manager Management Domain
6 Click the Update tab and look at Available Updates to see the updates to be installed. These items may not match all the bundles downloaded in the Repository tab. When the bundles have been downloaded, an automatic system scan determines which updates need to be installed. If an update is dependent on another update, it displays PREVIOUS UPDATE REQUIRED. Once the dependency update is installed, the UPDATE button becomes available. 7 Ensure that the SDDC Manager and HMS are at the same version. In a multi rack scenario, the SDDC Manager and HMS versions must be the same on all racks. To confirm the versions are the same, do the following: a b c Click the LIFECYCLE tab and click INVENTORY. For the Management Domain of each rack, click Expand. Under the section VMware Cloud Foundation Software, check versions for VRM, HMS, and LCM and verify that they are the same. 8 Click UPDATE. The UPDATE button is enabled only for one update at a time. It is recommended to install/schedule only one update at a time. The system validates that update pre-requisites have been met. Updating the VMware Cloud Foundation Lifecycle Manager Management Domain 12
9 On the TARGET page, select the domains where the update is to be applied. Management Domain is compulsory and is automatically selected for update. Click EXPAND next to the domain to verify all the components to be updated. The Primary rack is updated before the other racks. It is recommended to perform the Workload Domain updates separately once the Management Domain updates complete successfully. 10 Click Next. The SCHEDULE page appears. Select the date and time for the update to be applied to the target domains. The default is two minutes. Click NEXT. 11 On the Review Update page, review the update bundle, targets, and schedule. Click SCHEDULE UPDATE. 13 Updating the VMware Cloud Foundation Lifecycle Manager Management Domain
12 The pending update appears in the SCHEDULED UPDATES section along with the scheduled time. Click VIEW UPDATE DETAILS to show the Update Status page with the update progress. If an update on a resource fails, a failure message displays on the Update Status page. The issue with the resources must be resolved before running the update again. 13 During the update the LCM page displays the following message. Do not cancel any update task while it is running. Also, do not reboot any device or any VM in the Management Domain. 14 Refresh the browser and log back on to the SDDC Manager and click the Updates tab under LIFECYCLE. Once the update process is finished the installed updates are displayed under COMPLETED UPDATES with the status of SUCCESS. 15 To download the log file, click next to SUCCESS and click DOWNLOAD UPDATE LOG. 16 To review the updated components click the INVENTORY tab and expand the various domains to check the components and the updated versions. 17 If there are no Workload Domains to be patched wait 24 hours before removing the snapshots created at the start of this procedure. If Workload Domain patches are required, go to Updating the LCM Workload Domain (see page 15). Updating the VMware Cloud Foundation Lifecycle Manager Management Domain 14
Updating the LCM Workload Domain Update the Lifecycle Manager (LCM) Workload Domain after completing the update to the Management Domain. The recommendation is to wait 24 hours after completing Management Domain updates to begin Workload Domain updates. Update Workload Domains one at a time. Procedure 1 Disable the anti-affinity rule that separates NSX controllers across hosts, as follows: a b c d e f g Log on to the SDDC Manager. Go to Dashboard > Workload Domains. Click the Workload Domain to be updated. Go to Cluster Info > VCENTER to open the vcenter web client. Right-click the cluster and click Settings. Go to Configuration and click VM/Host rules. Select the NSX-Controller Anti-Affinity rule. Click Edit. De-select Enable rule. 2 Go to the Updating the LCM Management Domain (see page 10) and perform Steps 8-16 for each Workload Domain. 3 Refer to Step 1 and enable the anti-affinity rule that separates NSX controllers across hosts. For Step 1g, select the Enable rule. 4 When all domains are successfully updated, wait 24 hours before removing the snapshots created in Step 1 of Updating the LCM Management Domain (see page 10). 15 Updating the LCM Workload Domain
Upgrading Dell PowerEdge R630 servers The following procedures provided information for upgrading Dell PowerEdge R630 servers. Upgrading firmware and BIOS on Dell servers Check firmware and BIOS versions and perform upgrades on Dell severs, as needed. Before you begin Verify that the Dell firmware version is at the correct RCM level. Verify that the Dell BIOS version is at the correct RCM level. Both the firmware and BIOS upgrades can be done at the same time by uploading both files and then doing the installation. A reboot is required after the BIOS upgrade. Procedure Upgrading Firmware 1 Log on to Dell idrac and make note of the BIOS version, firmware version, and Lifecycle Controller Firmware version. Upgrading Dell PowerEdge R630 servers 16
2 Select Quick Launch Tasks and click Update and Rollback. 3 Click Choose file, select the file and click Upload. 4 Expand the section and check the Current and Available firmware. Check the box and click Install. 5 Click Job Queue to monitor the firmware install. 17 Upgrading Dell PowerEdge R630 servers
6 Be aware that idrac restarts when the firmware update is complete. Continue to monitor the status of the upgrade. 7 When status shows Completed, wait a few minutes and log back on to idrac. 8 In idrac, verify the new firmware version. Upgrading Dell PowerEdge R630 servers 18
Upgrading BIOS After upgrading BIOS, a reboot is required. If you are using VMware vsphere, be sure the node is in maintenance mode. 1 Log on to Dell idrac and make note of the BIOS version, firmware version, and Lifecycle Controller Firmware version. 2 Select Quick Launch Tasks and click Update and Rollback. 19 Upgrading Dell PowerEdge R630 servers
3 Click Choose file and select the BIOS file and click Upload. 4 Expand the section and check the Current and Available BIOS. Check the box and click Install and Reboot. 5 Click Job Queue to monitor the BIOS upgrade. Upgrading Dell PowerEdge R630 servers 20
6 Take note of the jobs in the queue and monitor the progress of the node reboot. 7 When status shows as Completed, verify the BIOS version in idrac. 21 Upgrading Dell PowerEdge R630 servers
Updating PERC H730 firmware All Flash configurations of VxRack System SDDC require the 25.5.0.0018 version of the PERC H730 firmware on each host. Verify the firmware version and perform an update, if required. For hybrid systems, refer to the RCM for the compatible firmware version. Before you begin Ensure that RACADM is installed to verify the firmware version and perform updates. Procedure 1 Check the firmware version. Open RACADM and type sudo racadm -r $ip -u root -p calvin storage get controllers -o grep FirmwareVersion 2 For each host, update the firmware version by typing sudo racadm -r $ip -u root -p calvin update f SAS-RAID_Firmware_2H45F_WN32_25.5.0.0018_A08.EXE 3 Reboot the host after the firmware is installed by typing sudo racadm -r $ip -u root -p calvin serveraction powercycle. Upgrading Dell PowerEdge R630 servers 22
Upgrading Cisco switches Refer to Cisco documentation at Cisco Nexus 9000 Series Switches for information on upgrading the Cisco Nexus 9000 series switches. 23 Upgrading Cisco switches
Upgrading NIC firmware The firmware for the NIC adapters in the Dell PowerEdge R630 must match the version in the RCM. Use these procedure to upgrade the NIC firmware, if necessary. Before you begin Before starting the firmware upgrade, do the following to obtain the installation software: 1 Go to support.dell.com. Click Choose from all products > Servers, Storage, & Networking > Networking > Fixed Port Switches > PowerEdge > PowerEdge R630. Click Drivers & Downloads and click Network. Look for the firmware version listed in the RCM. 2 Download the file, saving it to your hard drive. The following procedures describe two methods for upgrading the NIC firmware: local or remote. Use the appropriate procedure. Procedure Installing firmware with Lifecycle Controller (local to server) 1 Copy the downloaded firmware to a removable USB storage. 2 Insert the removable USB storage in a USB port of the server to be updated. 3 Reboot the server. 4 Press F10 to access the Lifecycle Controller. 5 In the left navigation pane, click Firmware Update: Upgrading NIC firmware 24
6 Click Launch Firmware Update: 7 Select Local Drive (CD or DVD or USB). Click Next. 8 The Step 2 of 3: Enter Access Details page appears. Select the removable USB storage from the Local Drive drop-down list. 9 Type the path and filename of the firmware DUP in the File Path or Update Package Path text box. For example: \Network_Firmware_V6TPJ_WN64_17.5.10_A00.exe 10 Click Next. 11 On the Step 3 of 3: Select Updates page, select the checkbox next to one or more of the Available System Updates. 12 Click Apply to complete the installation. Installing firmware with Lifecycle Controller (remote) 1 Copy the downloaded firmware to your local drive. 25 Upgrading NIC firmware
2 Use a browser to access idrac on the server. Log on with username root and password calvin. 3 Select Update and Rollback: Upgrading NIC firmware 26
4 Browse to the downloaded file and select it: 5 Click Upload. 6 Select the firmware version you want to install. Click either Install and Reboot or Install Next Reboot to complete the installation. 27 Upgrading NIC firmware
Resolving BMC alerts Incorrect BMC alerts may appear on the dashboard due to out-of-band IP addresses. Modifythe /etc/ dhcp/dhcpd.conf file to resolve the issue. Procedure 1 Open the /etc/dhcp/dhcpd.conf file in a text editor. 2 Add a new line ping-check false;, as shown in the following sample: ddns-update-style none; option domain-name "vrack-mgmt.local"; default-lease-time 86400; max-lease-time 86400; log-facility local7; subnet 192.168.0.0 netmask 255.255.255.0 { option routers 192.168.0.1; range 192.168.0.50 192.168.0.100; ping-check false; 3 Save and close the modified /etc/dhcp/dhcpd.conf file. 4 Restart the DHCP service on the management switch by typing /etc/init.d/isc-dhcprelay restart. Resolving BMC alerts 28
The information in this publication is provided "as is." Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software described in this publication requires an applicable software license. Copyright 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners. Published in the USA in April 2017. Dell EMC believes the information in this document is accurate as of its publication date. The information is subject to change without notice. 29 Copyright