Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 3.5

Transcription

1 Cisco HyperFlex Systems Upgrade Guide for VMware ESXi, Release 3.5 First Published: Last Modified: Americas Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA USA Tel: NETS (6387) Fax:

2 THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental. All printed copies and duplicate soft copies of this document are considered uncontrolled. See the current online version for the latest version. Cisco has more than 200 offices worldwide. Addresses and phone numbers are listed on the Cisco website at Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: go trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R) 2018 Cisco Systems, Inc. All rights reserved.

3 CONTENTS CHAPTER 1 Overview 1 About This Guide 1 Upgrade Features 1 CHAPTER 2 Prerequisites and Guidelines 3 Overview 3 Prerequisites 3 Upgrade Guidelines 4 Software Requirements for VMware ESXi 6 Supported Upgrade Paths 9 Upgrade from Pre 3.5.x Releases 14 Upgrading SED Ready Systems 14 Cautions and Recommendations 14 CHAPTER 3 Pre-Upgrade Validation Checks 17 HyperFlex Node Upgrade Validations 17 Viewing HyperFlex Cluster Health 18 Verifying If DRS Is Enabled 19 Viewing ESX Agent Manager 19 Verify the Health of a HyperFlex Cluster In Cisco UCS Manager 19 Verify UCS Server Firmware (C-Bundle) Version 20 Configuring vmotion Interfaces 20 Configure Lenient Mode 21 CHAPTER 4 Detailed Pre-Upgrade Procedures 23 Important 23 iii

4 Contents Downloading Software 23 Test Upstream Network Connectivity 24 Graceful Shutdown of a HX Cluster 25 Modifying Host Firmware Package Using Cisco UCS Manager 25 Auto Bootstrap Upgrade Process from HX Connect UI 26 Manual Bootstrap Upgrade Process 29 Verify vmotion Configuration for HX Cluster 31 Entering Cisco HyperFlex Maintenance Mode 32 Exiting Cisco HyperFlex Maintenance Mode 33 CHAPTER 5 Upgrade Procedures 35 Recommended Upgrade Method 35 Upgrading Your HyperFlex Cluster Using the HX Connect UI 35 Online Upgrade Process Workflow 38 Offline Upgrade Process Workflow 39 Offline Upgrade Guidelines 42 Offline Upgrade Using CLI 42 Start Cluster and Power On VMs 43 CHAPTER 6 Upgrading Cisco HyperFlex Software Components 45 Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager 45 Upgrading from 2.5 or Later Releases 48 Upgrading Cisco HyperFlex Data Platform Using HX Connect 48 Upgrading Cisco UCS Server Firmware Using the HX Connect UI 49 Upgrading ESXi Using the HX Connect UI 51 Upgrading from 2.1 or Earlier Releases 52 Upgrading Cisco UCS Firmware From 2.1 and Earlier Releases 52 Upgrading Cisco HX Data Platform Using vsphere Web Client From 2.1 or Earlier Releases 54 Upgrading ESXi From 2.1 or Earlier Releases 55 Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware From 2.1 and Earlier Releases 57 Compute-only Node Upgrade Considerations 59 Upgrading Compute-only Nodes from Version 2.2(7c) to 3.1(2b) 59 Upgrading Cisco UCS Firmware on Compute-only Node from Version 1.7.x to 1.8.x or Later 59 iv

5 Contents CHAPTER 7 HyperFlex Edge Upgrade 61 Overview 61 Cisco HyperFlex Edge Firmware Recommended Versions 61 Upgrading HyperFlex Edge Using vsphere Web Client from 2.1 or Earlier Releases 62 Upgrading HyperFlex Edge Using HX Connect from 2.5(1a) or Later Releases 63 Server Firmware Upgrade Using the Cisco Host Upgrade Utility Tool 64 Server Firmware Upgrade Using Cisco Integrated Management Controller Supervisor 64 Post Upgrade Tasks for HyperFlex Edge 66 CHAPTER 8 Stretched Cluster Upgrade 69 Overview 69 Upgrade Guidelines for Stretched Cluster 69 Upgrading HyperFlex Stretched Cluster Using HX Connect 69 CHAPTER 9 Post Upgrade Tasks 73 Confirm That Upgrade Is Complete 73 Check Firmware Versions in UCSM 74 Verify If Cleaner Is Running 74 CHAPTER 10 Known Issues 77 Overview 77 ESXi Host Is In Lockdown Mode 77 Offline Upgrade Cluster Start Command Error: Node Not Available 78 A Node Fails to Upgrade due to vcenter Issues 78 v

6 Contents vi

7 CHAPTER 1 Overview About This Guide About This Guide, on page 1 Upgrade Features, on page 1 This document describes how to upgrade an existing installation of Cisco HX Data Platform. Cisco HyperFlex Systems has several components that may be upgraded depending on your environment. The core components in a HyperFlex system are: Cisco UCS server firmware (the UCS C-bundle, which consists of UCS server, BIOS, CIMC, NIC, and so on) Cisco HX Data Platform software VMware ESXi software Upgrade Features Table 1: Upgrade Features for 3.5(1a) and Later Feature Cisco customized VMware ESXi upgrade No maintenance mode Bootstrap Process Description You can upgrade VMware ESXi using the HX Connect UI. You are not required to place the HyperFlex nodes in maintenance mode before initiating upgrade. When you upload the upgrade package in the HX Connect UI, you will see an informational pop-up. Manual cluster bootstrap is required for upgrade from a pre-3.5 release to 3.5(1a). Auto bootstrap is supported for upgrade from 3.5(1a) to later releases. 1

8 Upgrade Features Overview Feature Pre-upgrade validations Description When you upload the upgrade package in HX Connect and click Upgrade, pre-upgrade validation checks are performed automatically. 2

9 CHAPTER 2 Prerequisites and Guidelines Overview, on page 3 Prerequisites, on page 3 Upgrade Guidelines, on page 4 Software Requirements for VMware ESXi, on page 6 Supported Upgrade Paths, on page 9 Upgrade from Pre 3.5.x Releases, on page 14 Upgrading SED Ready Systems, on page 14 Cautions and Recommendations, on page 14 Overview Before you upgrade the Cisco HX Data Platform and the Cisco UCS server firmware in your Cisco HyperFlex System, consider the guidelines, best practices, and recommendations listed in this chapter. Prerequisites 1. See Resolved Caveats and Open Caveats before upgrading and review the New Features for this release. Refer to the latest Cisco HX Data Platform Release Notes. 2. Review supported versions and system requirements. See Hardware and Software Interoperability for Cisco HyperFlex HX-Series for more details. Important Verify that you have the latest software bundle versions, review the software versions. Refer to the latest Cisco HX Data Platform Release Notes. Ensure that the operating systems on all servers have the right driver levels for the release of Cisco UCS to which you plan to upgrade. See Cisco UCS Driver Installation Guide for identifying the server hardware. 3. Back up the configuration into an All Configuration backup file. See Cisco UCS Manager Backing Up and Restoring the Configuration Guide for the detailed steps. 3

10 Upgrade Guidelines Prerequisites and Guidelines 4. Before you perform firmware updates, use the Cisco UCS Manager Firmware Management interface to download relevant images to the fabric interconnect. Images are stored in bootflash partitions in the fabric interconnect. See Downloading Software, on page 23 for more details. 5. An ESXi upgrade may be required when upgrading to newer Cisco HyperFlex HX Data Platform versions. See the supported versions for each HX Data Platform version in Software Requirements for VMware ESXi, on page Keep SSH enabled on all ESXi Hosts. 7. Enable vmotion so that the VMs can be moved automatically during the upgrade and MTUs are set as required in the environment. See Configuring vmotion Interfaces, on page 20 for details on adding VMkernel interface. 8. Verify that the HyperFlex cluster is healthy. See HyperFlex Node Upgrade Validations, on page 17 for more details. 9. Verify that the cluster is in lenient mode. If not, set the cluster to lenient mode, refer Configure Lenient Mode, on page 21. Upgrade Guidelines The list below is a highlight of critical criteria for performing an upgrade of your HyperFlex system. Required vcenter upgradefor enhanced security, HXDP release 3.5(1a) and later requires the use of TLS 1.2. Therefore, vcenter must be upgraded to 6.0 U3f or later prior to upgrading to HX 3.5. In addition, ESXi should be upgraded as required to meet HXDP compatibility requirements. Minimum HXDP version for upgradehxdp clusters running 2.1(1b) or later may upgrade directly to 3.5. HX Data Platform 1.7.x and 1.8.x clustersusers upgrading from 1.7.x or 1.8.x prior to 2.0(1a) must step through an intermediate version before upgrading to 3.0 or later releases. For more information, see the Cisco HyperFlex Systems Upgrade Guide. Cluster ReadinessEnsure that the cluster is properly bootstrapped and the updated plug-in loaded before proceeding. Manual cluster bootstrap is required for upgrade from a pre-3.5 release to 3.5. Initiating Upgrade Use the HX Connect UI or CLI stcli commands when upgrading from 2.5(1a) or later releases. Use either the CLI stcli commands or the HX Data Platform Plug-in to the vsphere Web Client when upgrading from a pre-2.5(1a) release. The vcenter plug-in should no longer be used for upgrades starting with the 2.5(1a) release. vsphere 5.5 UpgradesUsers on vsphere 5.5 must upgrade to 6.0 U3/6.5 U1 before starting HX Data Platform upgrade. vsphere 5.5 support was deprecated with HX Data Platform 2.5(1a) and upgrade fails if attempted. For HX220 users running 5.5, contact TAC for upgrade assistance. For HX240 users running 5.5, upgrade components in the following order. 1. Upgrade vcenter to 6.0 U3f or later. If upgrading to 6.5, you must upgrade your vcenter in place. Using a new vcenter 6.5 is not supported for users migrating from Upgrade ESXi to 6.0/6.5 using the offline zip bundle. 4

11 Prerequisites and Guidelines Upgrade Guidelines Note During upgrade, it might be necessary to reconnect ESXi host manually in vcenter after ESXi upgrade and host reboot. 3. Upgrade HX Data Platform (and optionally the UCS firmware). If Upgrading to vsphere 6.5: Certain cluster functions such as native and scheduled snapshots, ReadyClones, and Enter/Exit HX Maintenance Mode will not operate from the time the upgrade is started until the HX Data Platform upgrade to 3.5 or later is complete. After upgrading ESXi using the offline zip bundle, use the ESX Exit Maintenance Mode option. The HX Exit Maintenance Mode option does not operate in the vsphere Web Client until the HX Data Platform upgrade is complete. vsphere 6.0 UpgradesUsers on vsphere 6.0 migrating to 6.5, upgrade components in the following order: 1. Upgrade HX Data Platform and UCS firmware. 2. Upgrade HX Data Platform and ESXi. 3. Upgrade HX Data Platform only first, then upgrade ESXi and/or UCS firmware. M4 Server Firmware UpgradesServer firmware should be upgraded to ensure smooth operation and to correct known issues. Specifically, newer SAS HBA firmware is available in this release and is recommended for long-term stability. Note Users are encouraged to upgrade to 3.1(3c) C-bundle or later whenever possible. Users running C-bundle versions prior to 3.1(2f) must upgrade server firmware by performing a combined upgrade of UCS server firmware (C-bundle) to 3.1(3c) or later and HX Data Platform to 2.5. Do not split the upgrade into two separate operations. If the cluster is already on 3.1(2f) C-bundle or later, you may perform an HX Data Platform only or combined upgrade, as required. M5 Server Firmware UpgradesM5 generation servers must run firmware version 3.2(2d) or later. M4/M5 Mixed DomainsA mixed domain occurs when a new, separate M5 cluster is installed under the same UCS domain that contains existing M4 cluster(s). Under these conditions, orchestrated UCS server firmware upgrade will not operate until HX Data Platform 2.6 or later is installed on the M4 clusters. Therefore, it is best practice to first upgrade UCS server firmware to the latest 3.1(3) or 3.2(2) patch release prior to adding a new M5 cluster to the existing UCS domain. Additionally, any 1.7 HX Data Platform clusters must first be upgraded prior to adding any new M5 clusters to the same domain. 5

12 Software Requirements for VMware ESXi Prerequisites and Guidelines Maintenance WindowIf upgrading both HX Data Platform and UCS firmware, either a combined or split upgrade can be selected through the vsphere HX Data Platform Plug-in depending on the length of the maintenance window. Direct firmware upgrade using server firmware auto install through Cisco UCS Manager should not be attempted. Instead, use the UCS server upgrade orchestration framework provided by the HX Data Platform. HX Data Platform 2.1(1b) with SEDUpgrading SED-ready systems running 2.1 require UCS infrastructure and server firmware upgrades. Software Requirements for VMware ESXi The software requirements include verification that you are using compatible versions of Cisco HyperFlex Systems (HX) components and VMware vsphere components. HyperFlex Software Versions The HX componentscisco HX Data Platform Installer, Cisco HX Data Platform, and Cisco UCS firmwareare installed on different servers. Verify that each component on each server used with and within an HX Storage Cluster are compatible. Verify that the preconfigured HX servers have the same version of Cisco UCS server firmware installed. If the Cisco UCS Fabric Interconnects (FI) firmware versions are different, see the Cisco HyperFlex Systems Upgrade Guide for steps to align the firmware versions. M4: For NEW hybrid or All Flash (Cisco HyperFlex HX240c M4 or HX220c M4) deployments, verify that Cisco UCS Manager 3.1(3j) or later is installed. M5: For NEW hybrid or All Flash (Cisco HyperFlex HX240c M5 or HX220c M5) deployments, verify that Cisco UCS Manager 4.0(1a) or later is installed. For SED-based HyperFlex systems, ensure that the A (Infrastructure) and C (Rack server) bundles are at Cisco UCS Manager version 3.1(3h) or higher for M4 SED systems. Ensure that all bundles are at Cisco UCS Manager version 4.0(1a) or higher for M5 SED systems. To reinstall an HX server, download supported and compatible versions of the software. See the Cisco HyperFlex Systems Installation Guide for VMware ESXi for the requirements and steps. Table 2: HyperFlex Software Versions for M4 Servers HyperFlex Release 3.5(1a) 3.0(1i) 3.0(1h) 3.0(1e) 3.0(1d) 3.0(1c) M4 Recommended UCS FI Firmware 4.0(1b), 3.1(3j) 3.2(3h), 3.1(3j) 3.2(3h), 3.1(3j) 3.2(3h), 3.1(3j) 3.2(3g), 3.1(3j) 3.2(3g), 3.1(3h) 6

13 Prerequisites and Guidelines Software Requirements for VMware ESXi HyperFlex Release 3.0(1b) 3.0(1a) 2.6(1e) 2.6(1d) 2.6(1b) 2.6(1a) M4 Recommended UCS FI Firmware 3.2(3d), 3.1(3h) 3.2(3d), 3.1(3f) 3.2(3d), 3.1(3f) 3.2(3d), 3.1(3c) 3.2(2d), 3.1(3c) 3.2(2d), 3.1(3c) Table 3: HyperFlex Software Version for M5 Servers HyperFlex Release 3.5(1a) 3.0(1i) 3.0(1h) 3.0(1e) 3.0(1d) 3.0(1c) 3.0(1b) 3.0(1a) 2.6(1c) 2.6(1d) 2.6(1b) 2.6(1a) M5 Recommended UCS FI Firmware 4.0(1a) 3.2(3h) 3.2(3h) 3.2(3h) 3.2(3h) 3.2(3h) 3.2(3d) 3.2(3d) 3.2(3d) 3.2(3d) 3.2(2d) 3.2(2d) HyperFlex Licensing As of version 2.6(1a), HyperFlex supports VMware PAC licensing. Existing VMware embedded licenses will continue to be supported. As of version 2.5(1a), HyperFlex uses a smart licensing mechanism to apply your licenses. See the Cisco HyperFlex Systems Installation Guide for VMware ESXi for details and steps. Supported VMware vsphere Versions and Editions Each HyperFlex release is compatible with specific versions of vsphere, VMware vcenter, and VMware ESXi. 7

14 Software Requirements for VMware ESXi Prerequisites and Guidelines Verify that all HX servers have a compatible version of vsphere preinstalled. Verify that the vcenter version is the same or later than the ESXi version. Verify that the vcenter and ESXi versions are compatible by consulting the VMware Product Interoperability Matrix. Newer vcenter versions may be used with older ESXi versions, so long as both ESXi and vcenter are supported in the table below. Verify that you have a vcenter administrator account with root-level privileges and the associated password. HyperFlex Version vsphere Versions vcenter Versions vsphere Editions 3.5(1a) 6.0 U3, 6.5 U1, 6.5 U2 6.0U3, 6.5U1, 6.5U2, 6.7U1 1 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 3.0(1e) 6.0 U3, 6.5 U1, 6.5 U2 6.0U3, 6.5U1, 6.5U2 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 3.0(1d) 6.0 U3, 6.5 U1, 6.5 U2 6.0U3, 6.5U1, 6.5U2 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 3.0(1c) 6.0 U3, 6.5 U1, 6.5 U U3, 6.5U1, 6.5U2 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 3.0(1b) 6.0 U3, 6.5 U1, 6.5 U U3, 6.5U1, 6.5U2 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 3.0(1a) 6.0 U3, 6.5 U1 6.0 U3, 6.5 U1 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 2.6(1e) 6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0 U2 Patch 4, 6.0 U3, 6.5 U1 6.0U1/U2/U3, 6.5U1 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 2.6(1d) 6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0 U2 Patch 4, 6.0 U3, 6.5 U1 6.0U1/U2/U3, 6.5U1 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 2.6(1b) 6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0 U2 Patch 4, 6.0 U3, 6.5 U1 6.0U1/U2/U3, 6.5U1 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 2.6(1a) 6.0 U1b, 6.0 U2, 6.0 U2 Patch 3, 6.0 U2 Patch 4, 6.0 U3, 6.5 U1 6.0U1/U2/U3, 6.5U1 Enterprise, Enterprise Plus, Standard, Essentials Plus, ROBO 1 Use of vcenter 6.7U1 is supported only with the ESXi 6.0 and 6.5 versions listed. Support for ESXi 6.7U1 will come in a future release. 2 Cluster installation must first be performed with 6.0 U3 or 6.5U1. After cluster deployment, you may upgrade to 6.5 U2 using the Zip bundle available for download on Cisco.com. See the Upgrade Guide for detailed ESXi upgrade procedures. 8

15 Prerequisites and Guidelines Supported Upgrade Paths 3 Cluster installation must first be performed with 6.0 U3 or 6.5U1. After cluster deployment, you may upgrade to 6.5 U2 using the Zip bundle available for download on Cisco.com. See the Upgrade Guide for detailed ESXi upgrade procedures. VMware vsphere Licensing Requirements How you purchase your vsphere license determines how your license applies to your HyperFlex system. If you purchased your vsphere license with HyperFlex Each HyperFlex server either has the Enterprise or Enterprise Plus edition preinstalled at the factory. Note HX Nodes have OEM licenses preinstalled. If you delete or overwrite the content of the boot drives after receiving the HX servers, you also delete the factory-installed licenses. OEM license keys is a new VMware vcenter 6.0 U1b feature. Earlier versions do not support OEM licenses. All factory-installed HX nodes share the same OEM license key. With vsphere OEM keys, the Usage count can exceed the Capacity value. When you add an HX host to vcenter through the Add Host wizard, in the Assign license section, select the OEM license. We obfuscate the actual vsphere OEM license key; for example, 0N085-XXXXX-XXXXX-XXXXX-10LHH. Standard, Essentials Plus, and ROBO editions are not available preinstalled on HX servers. If you did NOT purchase your vsphere license with HyperFlex The HX nodes have a vsphere Foundation license preinstalled. After initial setup, the license can be applied to a supported version of vsphere. If you purchased a vsphere PAC license Follow the instructions in your PAC license letter from VMware to add the license to your MY VMware account, then follow the instructions to add your HX host to vcenter and assign the PAC license. Supported Upgrade Paths The following four table matrices map the supported upgrade paths for Cisco HyperFlex Systems. 9

16 Supported Upgrade Paths Prerequisites and Guidelines Cisco HX Data Platform Matrix Cisco HX Data Platform Current Version Supported Upgrade Paths Desired version 3.5(x) 3.0(1x) 2.6(1x) 2.5(1x) 2.1(1x) 2.0(1x) 1.8(1x) 3.0(1x) 2.6(1x) 2.5(1x) 2.1(1x) 2.0(1x) 1.8(1f) 1.8(1a) to 1.8(1e) x 5 4 Direct upgrade from 1.8.1a to 1.8.1e to any release beyond 2.6(1x) is not supported. See Upgrade from Pre 3.5.x Releases, on page 14 for more details. 5 Direct upgrade from 1.7.x to any release beyond 2.1(1x) is not supported. A direct upgrade to 2.1(1x) is supported. See Upgrade from Pre 3.5.x Releases, on page 14 for more details. Cisco UCS Manager Matrix Cisco UCS Manager Current Version Supported Upgrade Paths Desired Version 4.0(1a) 3.2(3g) 3.2(3d) 3.2(2d) 3.1(3j) 3.1(3h) 3.1(3f) 3.1(3c) 3.1(2g) 3.1(2f) 3.1(2b) 3.2(3g) 3.2(3d) 3.2(2d) 3.1(3j) 3.1(3h) 10

17 Prerequisites and Guidelines Supported Upgrade Paths Cisco UCS Manager Current Version Supported Upgrade Paths Desired Version 3.1(3f) 3.1(3c) 3.1(2g) 3.1(2f) 3.1(2b) VMware ESXi Matrix ESXi 5.5 support is deprecated with HXDP 2.5. If running ESXi 5.5 U3 on HX220, contact TAC for upgrade guidance. If running ESXi 5.5 U3 on HX240, see Upgrade Guidelines, on page 4 for further details. If you have the ESXi 6.0 U1 version, we recommend an ESXi upgrade. There is a known VMware issue where the node becomes unresponsive due to a PSOD and OS crash. See VMware KB article, VMware ESXi 6.0, Patch ESXi BG: Updates esx-base, vsanhealth, vsan VIBs ( ). Attention Upgrade to vcenter 6.0 U3f or later is required, due to TLS 1.2 support. Be sure to upgrade vcenter prior to upgrading the HX cluster. VMware ESXi Current Version Supported Upgrade Paths Desired Version 6.5 U1 6.0 U3 6.0 U2 (All Patch Levels) 6.0 U1b 6.0 U3 6.0 U2 (All Patch Levels) 6.0 U1b 5.5 U3 (Only on HX240c) 11

18 Supported Upgrade Paths Prerequisites and Guidelines Cisco HX Data Platform Versions Supported by Cisco UCS Manager Cisco UCS Manager Version Cisco HX Data Platform 3.5(x) 3.0(1x) 2.6(1x) 2.5(1x) 2.1(1x) 2.0(1x) 1.8(1x) 1.7.x 4.0(1a) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers 3.2(3g) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers 3.2(3g) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers 3.2(3d) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers 3.2(2d) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers hybrid, All Flash, M4 and M5 servers hybrid, All Flash and M4 servers hybrid, All Flash, and M4 servers hybrid, All Flash and M4 servers 12

19 Prerequisites and Guidelines Supported Upgrade Paths Cisco UCS Manager Version Cisco HX Data Platform 3.1(3j) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, and M4 servers hybrid and All Flash hybrid and All Flash hybrid and All Flash hybrid and All Flash 3.1(3h) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, and M4 servers hybrid and All Flash hybrid and All Flash hybrid and All Flash hybrid and All Flash 3.1(3f) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, and M4 servers hybrid and All Flash hybrid and All Flash hybrid and All Flash hybrid and All Flash 3.1(3c) hybrid, All Flash, M4 and M5 servers hybrid, All Flash, and M4 servers hybrid and All Flash hybrid and All Flash hybrid and All Flash hybrid and All Flash 3.1(2g) hybrid and All Flash hybrid and All Flash hybrid and All Flash hybrid and All Flash 3.1(2f) hybrid and All Flash 3.1(2b) hybrid 13

20 Upgrade from Pre 3.5.x Releases Prerequisites and Guidelines Upgrade from Pre 3.5.x Releases Clusters running HX Data Platform prior to 2.1(1x) must be first upgraded to an intermediate release. Refer to the following table to upgrade your existing HyperFlex cluster to 3.5(1x). From Version 1.7(1x) 1.8(1x) 2.0(1x) 2.1(1x) 2.5(1x) and Up To 3.5(x) 1. Upgrade to 2.1(1x) 2. Upgrade to 3.5(x) using the vcenter plug-in. 1. Cisco recommends upgrading to 2.5(1x). You may upgrade to either 2.1(1x), 2.5(1x), or 2.6(1x). 1. Cisco recommends upgrading to 2.6(1x). You may upgrade to either 2.5(1x), 2.6(1x), or 3.0(1x). Directly upgrade to 3.5(x) using the vcenter plug-in. Directly upgrade to 3.5(x) using HX Connect. 2. Upgrade to 3.5(x) using HX Connect. 2. Upgrade to 3.5(x) using HX Connect. Upgrading SED Ready Systems SED-ready systems are HyperFlex clusters running HXDP 2.1(1b) with self encrypting drives (SEDs) installed. UCS Manager must be upgraded to 3.1(3c) or later. In addition, UCS server firmware (C-bundle) must be upgraded to 3.1(3c) or later. Either combined or split upgrade may be used, so long as all the cluster nodes are upgraded prior to enabling key management in HX Connect. Cautions and Recommendations Before you begin upgrade of a Cisco HyperFlex System, consider the following cautions, guidelines, and limitations. To upgrade a cluster from 2.1.x or earlier releases, use the vcenter Plugin for all HX Data platform only, UCSM only, and combined upgrades. To upgrade a cluster from 2.1.x or earlier releases using split upgrade, upgrade the HX Data Platform first using the vcenter Plugin. After HX Data Platform upgrade is complete, upgrade the UCS Server firmware through the HX Connect UI. To upgrade a cluster from 2.5.x to 2.6(1b) and 3.0(1x), use the HX Connect UI for all HX Data platform only, UCSM only, and combined upgrades. If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI, on page 26). However, if you are running a HyperFlex release that is earlier than release 14

21 Prerequisites and Guidelines Cautions and Recommendations 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process, on page 29). Optimizations in Capacity TierBackend access is optimized to significantly reduce the magnitude and frequency of high latency spikes. This optimization is included automatically for all clusters upgraded to 2.5(1c). Upgrade VMware ESXi before starting the upgrade process. Important If you have to upgrade from VMware ESXi version 5.5 U3 on HX220, contact Cisco TAC for assistance. For HX240, see Upgrade Guidelines. If you have replication configured, put replication in pause mode prior to performing upgrade, expansion, or cluster maintenance. After the upgrade, expansion, or cluster maintenance is completed, resume replication. Perform the pause and resume on any cluster that has replication configured to or from this local cluster. Cisco recommends using GUI upgrade over CLI for ease of use and better reporting. When Upgrade is complete, for each browser interface you use, empty the cache and reload the browser to page to refresh the HX content. Ensure that all nodes (including compute nodes) are up and running and the cluster is healthy before starting an upgrade or other maintenance activities. The Cisco HX Data Platform and Cisco UCS firmware bundles must be compatible. Refer UCS Hardware and Software Compatibility Matrix for more details. For a split upgrade, Cisco HX Data Platform should be updated first before updating the Cisco UCS firmware. During online upgrade, as one node is being upgraded (put into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and Access Policy settings. All endpoints in a Cisco HyperFlex domain must be fully functional and all processes must be complete before you begin a firmware upgrade on those endpoints. For example, the firmware on a server that has not been discovered cannot be upgraded or downgraded. Each endpoint is a component in the Cisco HyperFlex domain that requires firmware to function. In a three node cluster, if you shut down one node or put into maintenance mode it makes the cluster unhealthy, but the cluster is still online. If you need to perform manual maintenance, put the hosts in maintenance mode one at a time and move to the next host only after the cluster is healthy. For HXDP and UCS server firmware upgrades, this process is automatic. Note You cannot remove a node from 3 node cluster by doing stcli node remove operation. To replace a node on a 3 node cluster, please contact Cisco TAC for assistance with the node replacement procedure. 15

22 Cautions and Recommendations Prerequisites and Guidelines Firefox browser is not supported due to an outdated version of flash that is bundled with the browser. Manual update of flash within Firefox is possible, but the recommendation is to use either Chrome or Internet Explorer with a modern version of flash. 16

23 CHAPTER 3 Pre-Upgrade Validation Checks HyperFlex Node Upgrade Validations, on page 17 Viewing HyperFlex Cluster Health, on page 18 Verifying If DRS Is Enabled, on page 19 Viewing ESX Agent Manager, on page 19 Verify the Health of a HyperFlex Cluster In Cisco UCS Manager, on page 19 Verify UCS Server Firmware (C-Bundle) Version, on page 20 Configuring vmotion Interfaces, on page 20 Configure Lenient Mode, on page 21 HyperFlex Node Upgrade Validations Perform the following validations on each HyperFlex node before moving on to upgrade the next node in the cluster. Verify that the HyperFlex cluster is healthy and online. Verify all HyperFlex cluster nodes are connected to the vcenter and are online. Verify that no major Alarms are reported for the HyperFlex cluster in HyperFlex Connect. Verify that DRS is enabled and set to fully automated. Verify that vsphere services are running and ESXi Agent Manager (EAM) health is normal. Verify the health of the cluster in Cisco UCS Manager. SSH into each controller VM in the HX cluster and run the command df -h to ensure that there is at least 50% free space in /var/stv. Output example root@springpathcontrollerg9es5wgwdg:~# df -h Filesystem Size Used Avail Use% Mounted on udev 24G 4.0K 24G 1% /dev tmpfs 4.8G 7.0M 4.8G 1% /run /dev/sda1 2.4G 1.5G 779M 67% / none 5.0M 0 5.0M 0% /run/lock none 24G 0 24G 0% /run/shm none 100M 0 100M 0% /run/user /dev/sdb1 158G 2.7G 147G 2% /var/stv /dev/sdb2 32G 50M 30G 1% /var/zookeeper none 4.0K 0 4.0K 0% /sys/fs/cgroup 17

24 Viewing HyperFlex Cluster Health Pre-Upgrade Validation Checks Important If there is not enough space (usage is over 50%), contact Cisco TAC for assistance. Viewing HyperFlex Cluster Health Using GUI From HyperFlex Connect, select System Information > Nodes page. Verify if the HyperFlex cluster is healthy and online. From the vsphere Web Client Navigator, select vcenter Global Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. View the cluster widget to verify if the HyperFlex cluster is healthy and online. From the vsphere Web Client Navigator, select vcenter Global Inventory Lists > Clusters > cluster > Summary. Verify if all HX Cluster nodes are connected to the vcenter and they are online. Using CLI Log in to any controller VM in the storage cluster. Run the command stcli cluster storage-summary -detail. address: name: HX-Cluster01 state: online uptime: 0 days 12 hours 16 minutes 44 seconds activenodes: 5 of 5 compressionsavings: deduplicationsavings: 0.0 freecapacity: 38.1T healinginfo: inprogress: False resiliencydetails: current ensemble size:5 # of ssd failures before cluster shuts down:3 minimum cache copies remaining:3 minimum data copies available for some user data:3 minimum metadata copies available for cluster metadata:3 # of unavailable nodes:0 # of nodes failure tolerable for cluster to be available:2 health state reason:storage cluster is healthy. # of node failures before cluster shuts down:3 # of node failures before cluster goes into readonly:3 # of hdd failures tolerable for cluster to be available:2 # of node failures before cluster goes to enospace warn trying to move the existing data:na # of hdd failures before cluster shuts down:3 # of hdd failures before cluster goes into readonly:3 # of ssd failures before cluster goes into readonly:na # of ssd failures tolerable for cluster to be available:2 resiliencyinfo: messages: Storage cluster is healthy. state: healthy 18

25 Pre-Upgrade Validation Checks Verifying If DRS Is Enabled hddfailurestolerable: 2 nodefailurestolerable: 1 ssdfailurestolerable: 2 spacestatus: normal totalcapacity: 38.5T totalsavings: usedcapacity: 373.3G clusteraccesspolicy: lenient datareplicationcompliance: compliant datareplicationfactor: 3 Sample response that indicates the HyperFlex storage cluster is online and healthy. Verifying If DRS Is Enabled Step 1 Step 2 From the vsphere Web Client Navigator, select vcenter Inventory Lists > Clusters > cluster > Configure tab. Verify that DRS is Enabled. Click the vsphere DRS tab. Check if Migration Automation Level is set to Fully Automated. Viewing ESX Agent Manager From the vsphere Web Client Navigator, select Administration > vcenter Server Extensions > vsphere ESX Agent Manager > Summary. Verify that ESX Agent Manager (EAM) health is normal. Verify the Health of a HyperFlex Cluster In Cisco UCS Manager Step 1 Step 2 Step 3 Step 4 Step 5 Verify if the high availability status of the fabric interconnects shows that both the fabric interconnects are up and running. See the Cisco UCS Manager System Monitoring Guide for more information. Verify that the data path is up and running. See the Cisco UCS Manager Firmware Management Guide for more information. Verify that the HyperFlex servers have no faults. Verify that vnic faults are cleared to ensure VMware ESXi vswitch uplinks are up and operational. Verify if all servers have been discovered. 19

26 Verify UCS Server Firmware (C-Bundle) Version Pre-Upgrade Validation Checks Verify UCS Server Firmware (C-Bundle) Version Using UCS Manager 1. Log in to UCS Manager. 2. Select the Server tab. 3. Select the Host Firmware Package policy by navigating to, Policies > Root > Sub-Organizations > <hx-cluster> > Host Firmware Packages > HyperFlex. Note Ensure that you select the desired cluster under the sub-org list. 4. Under properties, note the current Rack Package version. It is listed as X.Y(Z)C. For example, 3.1(2g)C. Using HX Connect 1. Log in to HX Connect. 2. In the Navigation pane, select Upgrade. 3. Select the UCS Firmware check box and click Discover. 4. Note the current C-bundle version displayed. Configuring vmotion Interfaces Complete the following steps to add the VMkernel interface necessary for vmotion to work: Before you begin You must pre-define vmotion networking by creating a vswitch and defining the vnics and VLANs in UCS Manager. Step 1 Step 2 Step 3 Step 4 Step 5 In the vsphere Web Client Navigator, click on Host > Inventory > Manage > Networking > VMkernel adapters. Click Add Host Networking. Select VMkernel Network Adapter. Select the existing vmotion vswitch by selecting browse. Provide a name, and refer to table below to enter the appropriate VLAN ID. Cluster Installation Version 1.7.x 1.8.x and later VLAN ID 0 (default) same as vmotion network 20

27 Pre-Upgrade Validation Checks Configure Lenient Mode Step 6 Step 7 Step 8 Provide a Static IP Address and complete the wizard. (Optional) To use jumbo frames, edit the vmk2 and set the MTU to Your upstream switch must be configured to pass jumbo frames on the vmotion VLAN. Repeat steps 1 to 6 for all hosts in the cluster. Configure Lenient Mode Cluster access policy is set by default to lenient mode. To manually set the cluster access policy to lenient, use the following procedure. SUMMARY STEPS 1. SSH to any one of the controller VMs and login as root. 2. Check if lenient mode is already configured. 3. If set to strict, change to lenient. If already set to lenient, no further action is required. 4. Confirm the change. DETAILED STEPS Command or Action Purpose Step 1 Step 2 Step 3 Step 4 SSH to any one of the controller VMs and login as root. Check if lenient mode is already configured. If set to strict, change to lenient. If already set to lenient, no further action is required. Confirm the change. #stcli cluster get-cluster-access-policy ~/#stcli cluster set-cluster-access-policy --name lenient stcli cluster info grep -i policy Example ~/#stcli cluster get-cluster-access-policy strict ~/#stcli cluster set-cluster-access-policy --name lenient stcli cluster info grep -i policy 21

28 Configure Lenient Mode Pre-Upgrade Validation Checks 22

29 CHAPTER 4 Detailed Pre-Upgrade Procedures Important, on page 23 Downloading Software, on page 23 Test Upstream Network Connectivity, on page 24 Graceful Shutdown of a HX Cluster, on page 25 Modifying Host Firmware Package Using Cisco UCS Manager, on page 25 Auto Bootstrap Upgrade Process from HX Connect UI, on page 26 Manual Bootstrap Upgrade Process, on page 29 Verify vmotion Configuration for HX Cluster, on page 31 Entering Cisco HyperFlex Maintenance Mode, on page 32 Exiting Cisco HyperFlex Maintenance Mode, on page 33 Important Warning This chapter contains a list of procedures required by various upgrade workflows. Follow only the procedures that are required for the specific upgrade workflow you intend to use. For step-by-step guidance on online and offline upgrades, refer to the Upgrade Procedures, on page 35 chapter. Downloading Software For a successful HyperFlex upgrade, the Cisco HyperFlex System component bundles can be downloaded from the Cisco HyperFlex Download website: 1. HX Data Platform upgrade bundle (.tgz file) 2. VMware ESXi Offline Zip bundle 3. Cisco UCS infrastructure bundle, blade firmware bundle, and rack firmware bundle. 23

30 Test Upstream Network Connectivity Detailed Pre-Upgrade Procedures After the Cisco UCS bundles and firmware are downloaded, they need to be copied to Cisco UCS Manager. To copy UCS software bundles to Cisco UCS Manager Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Log in to the Cisco UCS Manager GUI. Enter the username and password. In the Navigation pane, click the Equipment tab. On the Equipment tab, click the Equipment node. In the Work pane, click Firmware Management > Installed Firmware > Download Firmware. In the Download Firmware dialog box, click the Local File System radio button in the Location of the Image File field and fill in the required fields. In the Filename field, enter the full path and name of the image file. If you do not know the exact path to the folder where the firmware image file is located, click Browse and navigate to the file. Click OK. The Cisco UCS Manager GUI begins downloading the firmware bundle to the fabric interconnect. Monitor the status of the download on the Download Tasks tab. Note If Cisco UCS Manager reports that the bootflash is out of space, delete obsolete bundles on the Packages tab to free up space. To view the available space in bootflash, navigate to Equipment > Fabric Interconnect > Local Storage Information and look in the Work pane area under the General tab. Repeat this task until all the required firmware and bundles are downloaded to the fabric interconnect. Test Upstream Network Connectivity Ensure that the hx-storage-data and vmotion upstream switches are configured for Jumbo Frames. Skipping this step could lead to input/output interruption during Cisco UCS infrastructure upgrade. Step 1 Put a node in Cisco HX Maintenance mode (see Entering Cisco HyperFlex Maintenance Mode, on page 32). Step 2 SSH to the ESXi host in step 1. Step 3 Verify that the ping is working by pinging the corresponding vmk1 IP interface of another host. If using Jumbo Frames: vmkping -I vmk1 -d -s 8972 <data IP of address of another host> If not using Jumbo Frames: vmkping -I vmk1 -d -s 1472 <data IP of address of another host> Step 4 Swap the active interfaces in vswitch-hx-storage-data to force traffic upstream. esxcli network vswitch standard policy failover set -a vmnic2 -s vmnic3 -v vswitch-hx-storage-data Step 5 Again, verify that the ping is working by pinging the corresponding vmk1 IP interface of another host. If using Jumbo Frames: 24

31 Detailed Pre-Upgrade Procedures Graceful Shutdown of a HX Cluster vmkping -I vmk1 -d -s 8972 <data IP of address of another host> If the ping fails, try again with: vmkping -I vmk1 -d -s 1472 <data IP of address of another host> If not using Jumbo Frames: vmkping -I vmk1 -d -s 1472 <data IP of address of another host> Note If the ping fails, do not proceed to upgrade the Cisco UCS firmware. Investigate the network configuration including upstream switch to identify cause of failure. Step 6 Swap interface back to defaults even if ping fails. esxcli network vswitch standard policy failover set -a vmnic3 -s vmnic2 -v vswitch-hx-storage-data Step 7 Exit the node from Cisco HX Maintenance mode (see Exiting Cisco HyperFlex Maintenance Mode, on page 33). Graceful Shutdown of a HX Cluster Warning This chapter contains a list of procedures required by various upgrade workflows. Follow only the procedures that are required for the specific upgrade workflow you intend to use. For step-by-step guidance on online and offline upgrades, refer to the Upgrade Procedures, on page 35 chapter. Step 1 Step 2 Step 3 Step 4 SSH to any controller VM in the cluster. Check cluster health ~#stcli cluster info grep health. If healthy, shutdown the cluster ~#stcli cluster shutdown. Shutdown takes a few minutes, wait for the prompt to return. Modifying Host Firmware Package Using Cisco UCS Manager Host Firmware Package is set automatically, during Cisco UCS infrastructure upgrade. To manually stage the correct firmware version before starting the upgrade process use the following procedure. Step 1 Step 2 Step 3 Sign in to Cisco UCS Manager. In the Navigation pane, click Servers. Expand Servers > Policies > Sub-Organizations > hx-cluster. 25

32 Auto Bootstrap Upgrade Process from HX Connect UI Detailed Pre-Upgrade Procedures Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Expand Host Firmware Packages and choose the policy you want to update. In the Work pane, click the General tab. To modify the components in the host firmware package, click Modify Package Versions. Modify value for Blade Package and Rack Package, to the latest firmware version. See HyperFlex Software Versions for the list of recommended UCS FI firmware. In the Excluded Components area, check the boxes corresponding to the components that you want to exclude from this host firmware package. Click OK. Click yes for all the warnings. What to do next Verify pending activities. User acknowledgement of each server is performed automatically during upgrade. Optionally, you can manually acknowledge pending activities on the HyperFlex node. On the Cisco UCS Manager toolbar, click Pending Activities. The User Acknowledged Activities tab lists the HyperFlex nodes that require user acknowledgment within the selected sub-org. They are in Pending Reboot status. Note Do not acknowledge service profiles. Auto Bootstrap Upgrade Process from HX Connect UI Before you begin If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI. However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform as outlined in Manual Bootstrap Upgrade Process, on page 29. Step 1 Step 2 Log in to HX Connect. a) Enter the HX Storage Cluster management IP address in a browser. Navigate to b) Enter the administrative username and password. c) Click Login. In the Navigation pane, select Upgrade. The Select Upgrade Type page appears. 26

33 Detailed Pre-Upgrade Procedures Auto Bootstrap Upgrade Process from HX Connect UI Figure 1: Select Upgrade Type Page Step 3 Upload the new Cisco HX Data Platform upgrade package, and then click Upgrade. Figure 2: Uploading a Cisco HX Data Platform Upgrade File 27

34 Auto Bootstrap Upgrade Process from HX Connect UI Detailed Pre-Upgrade Procedures Step 4 Click Confirm in the Initiating Pre-Upgrade message box. The Pre-Upgrade process begins and will be performed on all nodes. Figure 3: Initiating the Pre-Upgrade Process Figure 4: Pre-Upgrade Progress Screen 28

35 Detailed Pre-Upgrade Procedures Manual Bootstrap Upgrade Process Step 5 When the Pre-Upgrade upgrade (management packages upgrade) process is completed, the HX Connect UI prompts you so with an onscreen message. The software will again ask you which component you want to upgrade. Click Upgrade again the complete the HX data platform part of the upgrade. Figure 5: Completing the Pre-Upgrade Process Manual Bootstrap Upgrade Process The manual bootstrap process enables you to upgrade the Cisco HX Data Platform and the Cisco HX Data Platform Plug-in. Note Perform this procedure on the node that has the Cluster Management IP address. Note If you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform as outlined in this procedure. However, if you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI as outlined in Auto Bootstrap Upgrade Process from HX Connect UI, on page

36 Manual Bootstrap Upgrade Process Detailed Pre-Upgrade Procedures Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 From the vsphere Web Client Navigator, select vcenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster. Navigate to Actions > Summary and note the Cluster Management IP address. SSH to the cluster management IP address with root privileges. Transfer the latest HX Data Platform upgrade bundle to the controller VM's /tmp directory. Depending on your operating system, use you can either use SCP directly or download third-party tools, such as WinSCP or MobaXterm. From the controller VM shell, change to the /tmp directory. Warning Do not use any folder other than /tmp and do not create any subfolders. Un-compress the package using tar zxvf <storfs package name>.tgz. Example: tar zxvf storfs-packages-3.5.1a tgz This un-compresses and extracts all files to the root of the /tmp folder. Invoke the cluster-bootstrap.sh script to bootstrap packages for upgrade. Execute the command ~#./cluster-bootstrap.sh Enter the vcenter FQDN or IP address and administrator level username and password. Example: root@ucs-stctlvm-097-1:/tmp/upgrade#./cluster-bootstrap.sh Current version of the cluster is "2.5.1c-26345" Current version is compatible with upgrade. Continuing... Preparing bootstrap process. Please wait, this can take a few seconds... Enter vcenter Host: pguo-vm3.eng.storvisor.com Enter vcenter User: administrator@vsphere.local Enter vcenter password: Installing legacy oss packages... remove asupcli.log* Installing HyperFlex Software Packages ====================================== Target OS: BUILD_ID: Release: 3.5.1a Tag: Date: Branch: 3.5 Install date: Mon Nov 19 20:29:52 UTC 2018 ========================================== Applying changes to HX controller cron stop/waiting cron start/running, process HX controller changes done Migrating install directory to front SSD Installing debian package cryptsetup-bin Selecting previously unselected package cryptsetup-bin. (Reading database files and directories currently installed.) Preparing to unpack.../cryptsetup-bin_2%3a ubuntu2_amd64.deb... Unpacking cryptsetup-bin (2: ubuntu2)... Setting up cryptsetup-bin (2: ubuntu2)

37 Detailed Pre-Upgrade Procedures Verify vmotion Configuration for HX Cluster.. reloading nginx * Reloading nginx configuration nginx...done. done Installing debian package storfs-mgmt-ui (Reading database files and directories currently installed.) Preparing to unpack.../storfs-mgmt-ui_3.5.1a-31462_x86_64.deb... Unpacking storfs-mgmt-ui (3.5.1a-31462) over (2.5.1c-26345)... Uninstalling storfs-mgmt-ui-2.5.1c done Setting up storfs-mgmt-ui (3.5.1a-31462)... Installing storfs-mgmt-ui-3.5.1a Restarting hxmanager... hxmanager start/running, process 6262 Setting up webapps for storfs-mgmt-ui-3.5.1a done Generating inventory json file Update cluster state in ZK Stoppping mgmt services stop: Unknown instance: Starting mgmt services Wait for system management service to restart... Wait for system management service to restart... All management services have started Expected Version: 3.5.1a wait_for_services succeeded after tries 1. Not running in daemon mode, skipping package links creation. Updating vcenter plugin... Executing MgmtAgentScript : clearmgmtconfig.sh Update sed configuration files... SED capability is: 0 Updating configuration with default SED values.. Disabling cluster wide rebalance using stcli Successfully disabled rebalance on the cluster Bootstrap completed. Please use HX Connect UI to upgrade cluster to "3.5(1a)" release. If you have an active HX Connect session on a browser, refresh the session manually or re-login into the session. Wait for the system management service to restart and the bootstrap process to complete. Verify if the HX Data Platform Plug-in is now updated. Verify vmotion Configuration for HX Cluster Before you perform maintenance operations on the HX cluster, verify all nodes in the HX cluster are configured for vmotion. Confirm the following from your vsphere Web Client: 1. Verify that the vmotion port group is configured with vmnic6 and vmnic7 in an active/standby configuration across all of the ESXi hosts in the cluster. 2. Verify that a port group is configured for vmotion, and that the naming convention is EXACTLY the same across all ESXi hosts in the cluster. 31

38 Entering Cisco HyperFlex Maintenance Mode Detailed Pre-Upgrade Procedures Note The name is case-sensitive. 3. Verify that you have assigned a static IP to each vmotion port group, and that the static IPs for each vmotion port group are in the same subnet. 4. Verify that the vmotion port group has the vmotion option checked in the properties, and that no other port groups (such as management) have this option checked, on each ESXi host in the cluster. 5. Verify in the settings that the vmotion port group is set to 9000 MTU, (if you are using jumbo frames), and the VLAN ID matches the network configuration for the vmotion subnet. 6. Verify you can ping from the vmotion port group on one ESXi host to the vmotion IP on the other host. Type vmkping -I vmk2 -d -s 8972 <vmotion IP address of neighboring server> Entering Cisco HyperFlex Maintenance Mode Using the HX Connect User Interface Note If you have replication configured, put replication in pause mode prior to performing upgrade or expansion or cluster maintenance. After the upgrade or expansion or cluster maintenance is completed, resume replication. Perform the pause and resume on any cluster that has replication configured to or from this local cluster. If the HX Storage Cluster is offline, use the VMware Maintenance Mode. Note Only supported on release 2.5(1a)/2.5(1b) and later releases. 1. Log in to HX Connect: management ip>. 2. In the menu, click System Information. 3. Click Nodes, and then click the row of the node you want to put in to maintenance mode. 4. Click Enter HX Maintenance Mode. 5. In the Confirm HX Maintenance Mode dialog box, click Enter HX Maintenance Mode. Note After you complete any maintenance tasks, you must manually exit HX maintenance mode. Using the vsphere Web Client 1. Log in to the vsphere web client. 32

39 Detailed Pre-Upgrade Procedures Exiting Cisco HyperFlex Maintenance Mode 2. Go to Home > Hosts and Clusters. 3. Expand the Datacenter that contains the HX Cluster. 4. Expand the HX Cluster and select the node. 5. Right-click the node and select Cisco HX Maintenance Mode > Enter HX Maintenance Mode. Using the Command-Line Interface 1. Log in to the storage controller cluster command line as a user with root privileges. 2. Move the node into HX Maintenance Mode. 1. Identify the node ID and IP address. # stcli node list --summary 2. Enter the node into HX Maintenance Mode. # stcli node maintenancemode (--id ID --ip IP Address) --mode enter (see also stcli node maintenancemode --help) 3. Log in to the ESXi command line of this node as a user with root privileges. 4. Verify that the node has entered HX Maintenance Mode. # esxcli system maintenancemode get You can monitor the progress of the Enter Maintenance Mode task in vsphere Web Client, under the Monitor > Tasks tab. If the operation fails, an error message displays. Try to fix the underlying problem and attempt to enter maintenance mode again. Exiting Cisco HyperFlex Maintenance Mode Using the HX Connect User Interface Note If you have replication configured, put replication in pause mode prior to performing upgrade or expansion or cluster maintenance. After the upgrade or expansion orcluster maintenance is completed, resume replication. Perform the pause and resume on any cluster that has replication configured to or from this local cluster. Note Only supported on release 2.5(1a)/2.5(1b) and later releases. 1. Log in to HX Connect: management ip>. 2. In the menu, click System Information. 33

40 Exiting Cisco HyperFlex Maintenance Mode Detailed Pre-Upgrade Procedures 3. Click Nodes, and then click the row of the node you want to remove from maintenance mode. 4. Click Exit HX Maintenance Mode. Using the vsphere Web Client 1. Log in to the vsphere web client. 2. Go to Home > Hosts and Clusters. 3. Expand the Datacenter that contains the HX Cluster. 4. Expand the HX Cluster and select the node. 5. Right-click the node and select Cisco HX Maintenance Mode > Exit HX Maintenance Mode. Using the Command-Line Interface 1. Log in to the storage controller cluster command line as a user with root privileges. 2. Exit the node out of HX Maintenance Mode. 1. Identify the node ID and IP address. # stcli node list --summary 2. Exit the node out of HX Maintenance Mode. # stcli node maintenancemode (--id ID --ip IP Address) --mode exit (see also stcli node maintenancemode --help) 3. Log in to the ESXi command line of this node as a user with root privileges. 4. Verify that the node has exited HX Maintenance Mode. # esxcli system maintenancemode get You can monitor the progress of the Exit Maintenance Mode task in vsphere Web Client, under the Monitor > Tasks tab. If the operation fails, an error message displays. Try to fix the underlying problem and attempt to exit maintenance mode again. 34

41 CHAPTER 5 Upgrade Procedures Recommended Upgrade Method, on page 35 Upgrading Your HyperFlex Cluster Using the HX Connect UI, on page 35 Online Upgrade Process Workflow, on page 38 Offline Upgrade Process Workflow, on page 39 Recommended Upgrade Method For both combined upgrade and split upgrade, Cisco recommends upgrading the HyperFlex components in the following order for optimizing the upgrade time: 1. Upgrade Cisco HX Data Platform 2. Upgrade Cisco customized VMware ESXi 3. Upgrade Cisco UCS firmware Upgrading Your HyperFlex Cluster Using the HX Connect UI Important Use the HX Connect UI when upgrading from a current HX Data Platform version of 2.5(1a) or later releases. Step 1 If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. See Downloading Software, on page 23 for more details. Step 2 Ensure that the hx-storage-data and vmotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. See Test Upstream Network Connectivity, on page 24 for more details. Step 3 Upgrade Cisco UCS Infrastructure as required. See Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager, on page 45 for more details. Step 4 Bootstrap to upgrade Cisco HX Data Platform. 35

42 Upgrading Your HyperFlex Cluster Using the HX Connect UI Upgrade Procedures Note If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI, on page 26). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process, on page 29). Step 5 Step 6 Step 7 Log in to HX Connect. a) Enter the HX Storage Cluster management IP address in a browser. Navigate to b) Enter the administrative username and password. c) Click Login. In the Navigation pane, select Upgrade. Choose the type of upgrade from the Select Upgrade Type page. Caution After manual bootstrap, validation will fail if you perform UCS only, ESXi only, or UCS and ESXi combined upgrade. For successful upgrade, Cisco recommends the following upgrade types: HX Data Platform only upgrade, followed by UCS Firmware and/or Hypervisor Software upgrade HX Data Platform and UCS Firmware HX Data Platform and Hypervisor Software HX Data Platform, UCS Firmware, and Hypervisor Software Step 8 Depending on the type of upgrade you want to perform, complete the following fields on the Enter Credentials tab. UCS Server Firmware Field Essential Information UCS Manager Connectivity UCS Manager FQDN/IP field User Name field Admin Password field Enter the Cisco UCS Manager FQDN or IP address. Example: Enter the Cisco UCS Manager <admin> username. Enter the Cisco UCS Manager <root> password. HX Server Firmware Discover button Click Discover to view the current UCS firmware package version, in the Current Version field. HX Data Platform UI Element Drag the HX file here or click to browse Essential Information Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform. Example: storfs-packages-2.5.1b tgz. 36

43 Upgrade Procedures Upgrading Your HyperFlex Cluster Using the HX Connect UI UI Element Current version Current cluster details Bundle version (Optional) Checksum field Essential Information Displays the current HyperFlex Data Platform version. Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state. Displays the HyperFlex Data Platform version of the uploaded bundle. The MD5 Checksum number is stored in a separate text file at the /tmp directory where the upgrade package was downloaded. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle. ESXi Note The ESXi upgrade option is supported in the HyperFlex Connect UI for HyperFlex release 3.5(1a) or later. UI Element Drag the ESXi file here or click to browse field Current version field Current hypervisor details field Bundle details field Essential Information Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform. Example: storfs-packages-2.5.1b tgz. Displays the current ESXi version. Lists the HyperFlex cluster details like the Hypervisor version and Cluster upgrade state. Displays the ESXi version of the uploaded bundle. vcenter Credentials UI Element User Name field Admin Password field Essential Information Enter the vcenter <admin> username. Enter the vcenter <admin> password. Step 9 Step 10 Click Upgrade to begin the cluster upgrade process. The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. When upgrade is in progress, you may see an error message, 'Websocket connection failed. Automatic refresh disabled'. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message. 37

44 Online Upgrade Process Workflow Upgrade Procedures Online Upgrade Process Workflow Attention If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI, on page 26). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process, on page 29). When using the online upgrade process workflow, consider the following: First upgrade Cisco UCS infrastructure to the latest version and then use the automated upgrade workflow for a combined upgrade of Cisco UCS firmware and Cisco HX Data Platform. Online upgrade uses host firmware packages to upgrade all server endpoints. During online upgrade, as one node is being upgraded (placed into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and the Access Policy settings. See Entering Cisco HyperFlex Maintenance Mode, on page 32 for the procedures on how to access Cisco HyperFlex Maintenance Mode. If upgrading both HXDP and UCS firmware, combined upgrade can be selected through HX Connect depending on the length of the maintenance window. Do not use Firefox browser. It is not supported due to an outdated version of flash that is bundled with the browser. Note Do not attempt to upgrade the server firmware directly through Cisco UCS Manager server firmware auto install. Use the UCS server upgrade orchestration framework provided by the HyperFlex Plug-in. The following table summarizes the online upgrade workflow: Step Description Reference If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. Ensure that the hx-storage-data and vmotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. Downloading Software, on page 23 Test Upstream Network Connectivity, on page Upgrade Cisco UCS Infrastructure as required. Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager, on page 45 38

45 Upgrade Procedures Offline Upgrade Process Workflow Step Description Bootstrap to upgrade Cisco HX Data Platform. Disable snapshot schedule, on the bootstrapped storage controller VM. Note It is enough to run this script on one of the controller nodes. Log in to HX Connect with administrator credentials. Start combined upgrade of: HX Data Platform and UCS Firmware HX Data Platform and Hypervisor Software Reference Manual Bootstrap Upgrade Process, on page 29 Run the command stcli snapshot-schedule --disable. Upgrading Your HyperFlex Cluster Using the HX Connect UI, on page 35 Attention To perform a split upgrade, you must upgrade HX Data Platform first. After HX Data Platform is upgraded to 3.5(1x), you can perform a split upgrade of UCSM only and/or ESXi only. When only UCS firmware is being upgraded, you may see the upgrade process pause at the validation screen after the fabric interconnect discovery. It may be a network connectivity failure issue, however, in most cases, it just requires waiting for the process to finish. Upgrading Cisco HyperFlex Data Platform Using HX Connect, on page 48 Upgrading Cisco UCS Server Firmware Using the HX Connect UI, on page 49 Upgrading ESXi Using the HX Connect UI, on page Confirm that the upgrade task is complete. On the same controller VM, enable snapshot schedule. Post Upgrade Tasks, on page 73 Run the command stcli snapshot-schedule - enable. Offline Upgrade Process Workflow The following table summarizes the offline upgrade workflow: Step 1. Description If UCSM (A-bundle) or UCS Server Firmware (C-bundle) upgrade is required, download Cisco UCS Infrastructure A, blade bundle B, and rack bundle C. Reference Downloading Software, on page 23 39

46 Offline Upgrade Process Workflow Upgrade Procedures Step Description Reference 2. Ensure that the hx-storage-data and vmotion upstream switches are configured for full network failover capability before proceeding forward. Otherwise the HyperFlex Cluster becomes offline and all datastores unmount from the ESXi hosts. Test Upstream Network Connectivity, on page Upgrade Cisco UCS Infrastructure as required. Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager, on page Launch the vsphere Web Client and power down all user VMs residing on HX servers and all user VMs running on HX datastores. This includes VMs running on compute-only nodes. After the VMs have been shut down, verify the health state of the cluster and perform a graceful shutdown. Important HyperFlex controller VMs (stctlvms) must remain powered on. Manually stage the correct firmware version before starting the upgrade process. Graceful Shutdown of a HX Cluster, on page 25 Modifying Host Firmware Package Using Cisco UCS Manager, on page Shutdown the HyperFlex Controller VMs (stctlvms). Once the Controller VMs are shutdown, place the ESXi hosts into Maintenance Mode. Acknowledge the pending reboot on the servers that comprise your HX cluster nodes, including both converged nodes and compute-only nodes connected to the cluster. Wait until all nodes are upgraded. Confirm that correct firmware packages have been installed before proceeding. In vcenter, right-click on each HX Controller VM (stctlvm) and select Power > Shut Down Guest OS. In vcenter, right-click on each ESXi host select Maintenance Mode > Enter Maintenance Mode. 40

47 Upgrade Procedures Offline Upgrade Process Workflow Step Description Once the ESXi hosts have booted, take them out of Maintenance Mode. Now the controller VM should come back online. Disable snapshot schedule, on the bootstrapped storage controller VM. Note It is enough to run this script on one of the controller nodes. Bootstrap to upgrade the Cisco HX Data Platform Plug-in. Important Be sure to copy the bootstrap file to the controller VM /tmp directory. Ensure that you confirm the version of the plug-in in the vcenter Administration > Client Plug-Ins page. Reference In vcenter, right-click on each ESXi host select Maintenance Mode > Exit Maintenance Mode. Run the command stcli snapshot-schedule --disable. Manual Bootstrap Upgrade Process, on page From the same controller VM, begin the upgrade. Confirm that upgrade is complete. After the upgrade is complete, start the cluster and power on VMs. On the same controller VM, enable snapshot schedule. Upgrading Your HyperFlex Cluster Using the HX Connect UI, on page 35 Offline Upgrade Using CLI, on page 42 Post Upgrade Tasks, on page 73 Start Cluster and Power On VMs, on page 43 Run the command stcli snapshot-schedule - enable. 41

48 Offline Upgrade Guidelines Upgrade Procedures Offline Upgrade Guidelines Important --ucsm-host and --ucsm-user parameters are required when you are upgrading from 1.7x to 1.8x. These parameters must not be used when moving up from 1.8(1a)/1.8(1b) to 2.0(1a) as we are not changing the Cisco UCS server firmware version. Cisco recommends using HX Connect UI to upgrade HyperFlex clusters over CLI for ease of use and better reporting. You can safely ignore Cluster Invalid State alerts in HX Connect. Before you proceed, with either combined or split upgrade consider the following guidelines: The package name must match the file that you uploaded to the controller VM. Enter passwords when prompted. Nodes are upgraded with the new version of the Cisco HX Data Platform software and rebooted one at a time. Offline Upgrade Using CLI Important If you need to perform a split upgrade, you must upgrade HX Data Platform first. After HX Data Platform is upgraded to 3.5(1x), you can perform a split upgrade of UCSM only and/or ESXi only. Combined Upgrade of Cisco HX Data Platform, ESXi and Cisco UCS Firmware M5 Servers # stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location/tmp/ <storfs package name,esxi package name> --ucsm-host <IP/FQDN of UCSM> --ucsm-user <UCSM User> --ucsm5-fw-version <UCSM Firmware Version> Example for M5 Servers: ~# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location /tmp/storfs-packages-3.5.1a tgz --ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucs5fw-version '3.1(2g)' M4 Servers # stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location/tmp/ <storfs package name, ESXi package name> --ucsm-host <IP/FQDN of UCSM> --ucsm-user <UCSM User> --ucsfw-version <UCSM Firmware Version> Example for M4 Servers: ~# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor --location /tmp/storfs-packages-3.5.1a tgz --ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucsfw-version '3.1(2g)' Combined Upgrade of Cisco HX Data Platform and ESXi M5 Servers 42

49 Upgrade Procedures Start Cluster and Power On VMs # stcli cluster upgrade --components hxdp, hypervisor --location/tmp/ <storfs package name,esxi package name> Example for M5 Servers: ~# stcli cluster upgrade --components hxdp, hypervisor --location /tmp/storfs-packages-3.5.1a tgz M4 Servers # stcli cluster upgrade --components hxdp, hypervisor --location/tmp/ <storfs package name, ESXi package name> Example for M4 Servers: ~# stcli cluster upgrade --components hxdp, hypervisor --location /tmp/storfs-packages-3.5.1a tgz Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware M5 Servers # stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/ <storfs package name> --vcenter-user <vcuser> --ucsm-host <IP/FQDN of UCSM> --ucsm-user <UCSM User> --ucsm5-fw-version <UCSM Firmware Version> M4 Servers # stcli cluster upgrade --components hxdp,ucs-fw --location/tmp/ <storfs package name> --vcenter-user <vcuser> --ucsm-host <IP/FQDN of UCSM> --ucsm-user <UCSM User> --ucsfw-version <UCSM Firmware Version> Example for M4 Servers: ~# stcli cluster upgrade --components hxdp,ucs-fw --location /tmp/storfs-packages-1.8.1c tgz --vcenter-user administrator@vsphere.local --ucsm-host eng-fi16.eng.storvisor.com --ucsm-user admin --ucsfw-version '3.1(2b)' Start Cluster and Power On VMs After the upgrade is complete and the cluster has been upgraded, log out and log back in to vcenter to see upgrade changes. Step 1 Step 2 After the upgrade is complete, start your cluster. Login to any controller VM through SSH. # stcli cluster start Example: HyperFlex StorageController 1.8(1c) Last login: Wed Sept 21 23:54: from pguo-dev.eng.storvisor.com root@ucs-stclivm ;~# stcli cluster upgrade-status Cluster upgrade succeeded. Cluster version: 1.8(1c) root@ucs-stctlvm-384;~# stcli cluster start waiting for Cluster to start on nodes: [ucs-383, ucs-384, ucs-385, ucs-386] This will start the cluster and mount the HX datastores. Wait for cluster to come online. You will see the prompt: Started cluster on nodes; [ucs-383, ucs-384, ucs-385, ucs-386] Cluster is online root@ucs-stctlvm-384-1;~# 43

50 Start Cluster and Power On VMs Upgrade Procedures Step 3 Wait for cluster to become healthy before starting the VMs. Run command: ~# stcli clustr info grep health Example: root@springpathcontrollerzrvf040451;~# stcli cluster info grep health healthstate: healthy state: healthy storage cluster is healthy Step 4 After the cluster is healthy, launch vsphere Web Client or Thick Client, navigate to Hosts and Cluster > Datacenter > Cluster >. Right click, select Power> Power On to start the VMs. 44

51 CHAPTER 6 Upgrading Cisco HyperFlex Software Components Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager, on page 45 Upgrading from 2.5 or Later Releases, on page 48 Upgrading Cisco HyperFlex Data Platform Using HX Connect, on page 48 Upgrading Cisco UCS Server Firmware Using the HX Connect UI, on page 49 Upgrading ESXi Using the HX Connect UI, on page 51 Upgrading from 2.1 or Earlier Releases, on page 52 Compute-only Node Upgrade Considerations, on page 59 Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager The UCS infrastructure includes UCS Manager, switch firmware, blade chassis IO module firmware and rack server FEX firmware. The UCS infrastructure firmware can be upgraded with no disruption to the HyperFlex workloads. This is achieved by doing a rolling upgrade of the UCS fabric interconnects. Note HX M4 servers are incompatible with 1227 VIC and UP fabric interconnects. The HX nodes are configured to failover all Ethernet traffic when a UCS fabric interconnect is rebooted to upgrade the firmware. Refer to the Cisco UCS Manager Firmware Management Guide for more details. Note Ensure that the hx-storage-data and vmotion upstream switches are configured for Jumbo Frames before proceeding forward, otherwise the HyperFlex Cluster will become offline and all datastore will unmount from the ESXi hosts. Step 1 Open the UCS Manager GUI. Step 2 Select Equipment > Firmware Management > Firmware auto-install. Step 3 Click Install Infrastructure Firmware. Step 4 Select the desired Service Pack. Select Upgrade Now box to immediately begin the firmware upgrade. Click Finish. 45

52 Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager Upgrading Cisco HyperFlex Software Components A pre-requisite dialog box will pop-up. You will see a list of warnings. Fix them first before proceeding further. 1. Cisco UCS Manager upgrade validation fails. In that case, stop the upgrade process. Determine the reason for upgrade validation failure. Take corrective action and restart the upgrade process. 2. Cisco UCS Manager GUI is disconnected. This is expected during UCS Manager upgrade as UCS Manager is stopped, then restarted at the new version. Wait until UCS Manager goes back online. Log back in to UCS Manager to complete the next steps. You may check Ignore All box if the warnings are not critical to user environment. Step 5 Step 6 Step 7 Check Upgrade Now box. Click Finish. UCS Manager software upgrades first, so UCS Manager may be inaccessible for sometime. Log back into UCS Manager. Wait for IOM to be upgraded (if the UCS blade server chassis is present). 1. Select Equipment > Installed Firmware, expand each chassis and check the Update Status of the IO Module. 2. During upgrade, the Update Status of the IO Modules will be Upgrading. 3. IOMs will be in a Pending Next Boot for Activate Status once the Update process completes. When IOM upgrade has completed, the Update Status of the IO Modules is set to Ready. Step 8 Wait for Subordinate FI to be activated. 1. Select Equipment > Installed Firmware > Fabric Interconnects. 2. Check the Activate Status of the kernel and switch images. During upgrade, the Activate Status is set to Activating. Step 9 Step 10 During FI reboot, all HX traffic will be forwarded to primary FI (based on ESXi vswitch failover policy). This will cause a brief traffic interruption. This will not cause storage IO failures. Verify subordinate FI has rebooted and joined the UCS cluster. 1. Select Equipment > Installed Firmware > Fabric Interconnects. 2. After activation, the Activate Status of the FI is set to Ready. 3. Check the Overall Status of the FI is operable. 4. Check the kernel and switch versions of the FI match the desired and updated version. 5. Check the FI has no fault. 6. Check the FI cluster membership is Subordinate. Step 11 If the UCS blade server chassis is present, wait for IOM activation to complete. Only the IOMs connected to the subordinate FI will enter Ready state, IOMs attached to the Primary FI will remain in Pending Next Boot Activate Status. 1. Select Equipment > Blade Chassis > IO Module. 2. Wait for the Activate Status of IO module to change to Ready. Step 12 Wait until HX traffic is re-pinned to both FIs. 46

53 Upgrading Cisco HyperFlex Software Components Upgrading Cisco UCS Infrastructure Using Cisco UCS Manager Wait for UCS Manager vnic faults to be cleared. The fault clearing indicates ESXi has loaded the ENIC driver and the interface is up. The traffic is not re-pinned immediately when the network interface goes up because ESXi has a fail back timer. But the Net.teampolicyupdelay timer is very low by default (100ms). Step 13 Verify the HX Cluster is online, and healthy before rebooting the primary fabric interconnect. Access summary tab from the vsphere Web Client Navigator. Select Home > vcenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. Step 14 In the UCS manager GUI, on the toolbar, click Pending Activities. Click on Fabric Interconnects tab that display the tasks requiring user acknowledgment before they can complete. 1. Click Reboot Now for each pending activity that you want to deploy immediately. 2. Click OK. Cisco UCS Manager immediately reboots the primary FI. This will cause the subordinate FI to become primary (FI failover). Step 15 Step 16 Step 17 During FI reboot, all HX traffic will be forwarded to the new primary FI. This will cause a brief traffic interruption. However, this will not cause storage IO failures. Wait for UCS Manager to be disconnected, then reconnected on the other FI. This is because a UCS Manager fail over occurs due to primary FI reboot. Check subordinate FI has become primary. Check that the FI cluster membership is Primary. Step 18 Wait for FI to be activated. 1. Select Equipment > Installed Firmware > Fabric Interconnects. 2. Wait for the Activate Status of the FI to be Ready. 3. Check the Overall Status of FI is operable. 4. Check the FI has no fault. Step 19 Verify FI has rebooted and joined the UCS cluster as subordinate. Check that the FI cluster membership is Subordinate. Step 20 Wait for IOM activation to complete a) Select Equipment > Blade Chassis > IO Module. b) Wait for the Activate Status of the IP module to be Ready. c) You can monitor the status on the FSM tab. Note You will lose connectivity to UCS Manager throughout the entire upgrade. This is a normal behavior. Step 21 Wait until HX traffic is re-pinned to both FIs. In the UCS manager GUI, wait until all server vnic faults have been cleared. Step 22 Verify the HX Cluster is online, and healthy after rebooting the FI. Access summary tab from the vsphere Web Client Navigator. Select Home > vcenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. 47

54 Upgrading from 2.5 or Later Releases Upgrading Cisco HyperFlex Software Components Upgrading from 2.5 or Later Releases Upgrading Cisco HyperFlex Data Platform Using HX Connect Before you begin Complete pre-upgrade validation checks. See Prerequisites for more details. Download the latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters from previous releases, from Software Download. Complete steps 1 to 6 in the Online Upgrade Process Work flow. See Online Upgrade Process Workflow, on page 38 for more details. Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform. Note If you are running HyperFlex release 3.5(1a) or later, you can upgrade the Cisco HX Data Platform by performing the auto-bootstrap process from the HX Connect UI (Auto Bootstrap Upgrade Process from HX Connect UI, on page 26). However, if you are running a HyperFlex release that is earlier than release 3.5(1a) you must run the manual bootstrap process to upgrade the Cisco HX Data Platform (Manual Bootstrap Upgrade Process, on page 29). Disable snapshot schedule, on the bootstrapped storage controller VM. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 Step 2 Step 3 Log in to HX Connect. a) Enter the HX Storage Cluster management IP address in a browser. Navigate to b) Enter the administrative username and password. c) Click Login. In the Navigation pane, select Upgrade. On the Select Upgrade Type page, select HX Data Platform and complete the following fields: 48

55 Upgrading Cisco HyperFlex Software Components Upgrading Cisco UCS Server Firmware Using the HX Connect UI UI Element Drag the HX file here or click to browse Current version Current cluster details Bundle version (Optional) Checksum field Essential Information Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform. Example: storfs-packages-2.5.1b tgz. Displays the current HyperFlex Data Platform version. Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state. Displays the HyperFlex Data Platform version of the uploaded bundle. The MD5 Checksum number is stored in a separate text file at the /tmp directory where the upgrade package was downloaded. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle. Step 4 Enter the vcenter credentials. UI Element User Name field Admin Password field Essential Information Enter the vcenter <admin> username. Enter the vcenter <admin> password. Step 5 Step 6 Click Upgrade to begin the cluster upgrade process. The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. When upgrade is in progress, you may see an error message, 'Websocket connection failed. Automatic refresh disabled'. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message. Upgrading Cisco UCS Server Firmware Using the HX Connect UI Caution Ensure you upgrade HX Data Platform first. After HX Data Platform is upgraded to 3.5(x), you can upgrade the UCS server firmware. Before you begin Complete pre-upgrade validation checks. See Prerequisites for more details. 49

56 Upgrading Cisco UCS Server Firmware Using the HX Connect UI Upgrading Cisco HyperFlex Software Components Download the latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters from previous releases, from Software Download. Complete steps 1 to 6 in the Online Upgrade Process Work flow. See Online Upgrade Process Workflow, on page 38 for more details. Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform. Disable snapshot schedule, on the bootstrapped storage controller VM. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 Step 2 Step 3 Log in to HX Connect. a) Enter the HX Storage Cluster management IP address in a browser. Navigate to b) Enter the administrative username and password. c) Click Login. In the Navigation pane, select Upgrade. On the Select Upgrade Type page, select UCS Server Firmware and complete the following fields: Field Essential Information UCS Manager Connectivity UCS Manager FQDN/IP field User Name field Admin Password field Enter the Cisco UCS Manager FQDN or IP address. Example: Enter the Cisco UCS Manager <admin> username. Enter the Cisco UCS Manager <root> password. HX Server Firmware Discover button Click Discover to view the current UCS firmware package version, in the Current Version field. Step 4 Step 5 Click Upgrade to begin the UCS firmware upgrade process. The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. 50

57 Upgrading Cisco HyperFlex Software Components Upgrading ESXi Using the HX Connect UI When upgrade is in progress, you may see an error message, 'Websocket connection failed. Automatic refresh disabled'. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message. Upgrading ESXi Using the HX Connect UI Caution Ensure that you upgrade HX Data Platform first. After HX Data Platform is upgraded to 3.5(x), you can upgrade the UCS server firmware. Before you begin Note The ESXi upgrade option is supported in the HyperFlex Connect UI for HyperFlex release 3.5(1a) or later. Complete pre-upgrade validation checks. See Prerequisites for more details. Download the latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters from previous releases, from Software Download. Complete steps 1 to 6 in the Online Upgrade Process Work flow. See Online Upgrade Process Workflow, on page 38 for more details. Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform. Disable snapshot schedule, on the bootstrapped storage controller VM. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 Step 2 Step 3 Log in to HX Connect. a) Enter the HX Storage Cluster management IP address in a browser. Navigate to b) Enter the administrative username and password. c) Click Login. In the Navigation pane, select Upgrade. On the Select Upgrade Type page, select ESXi and complete the following fields: 51

58 Upgrading from 2.1 or Earlier Releases Upgrading Cisco HyperFlex Software Components UI Element Drag the ESXi file here or click to browse field Current version field Current hypervisor details field Bundle details field Essential Information Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform. Example: storfs-packages-2.5.1b tgz. Displays the current ESXi version. Lists the HyperFlex cluster details like the Hypervisor version and Cluster upgrade state. Displays the ESXi version of the uploaded bundle. Step 4 Enter the vcenter credentials. UI Element User Name field Admin Password field Essential Information Enter the vcenter <admin> username. Enter the vcenter <admin> password. Step 5 Step 6 Click Upgrade to begin the VMware ESXi upgrade process. The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. When upgrade is in progress, you may see an error message, 'Websocket connection failed. Automatic refresh disabled'. You can either refresh the page or log out and log back in to clear the error message. You can safely ignore this error message. Upgrading from 2.1 or Earlier Releases Direct upgrade from 1.7.x to any release beyond 2.1(1x) is not supported. Direct upgrade from 1.8(1a) to 1.8(1e) to any release beyond 2.6(1x) is not supported. Direct upgrade from 2.0(1x) to any release beyond 2.6(1x) is not supported. See Upgrade from Pre 3.5.x Releases, on page 14 for more details. Upgrading Cisco UCS Firmware From 2.1 and Earlier Releases Important Do not manually acknowledge the pending activities in Cisco UCS Manager. Make sure that the HX Data Platform is already upgraded. When performing a split upgrade, ensure Cisco UCS Firmware is upgraded around the same time as the HX Data Platform upgrade. 52

59 Upgrading Cisco HyperFlex Software Components Upgrading Cisco UCS Firmware From 2.1 and Earlier Releases Before you begin Complete pre-upgrade validation checks. See Prerequisites for more details. Download the latest Cisco HX Data Platform upgrade bundle for upgrading existing clusters, from Download Software. Complete steps 1 to 6 in the Online Upgrade Process Work flow. Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform plug-in. Disable snapshot schedule, on the bootstrapped storage controller VM. Log in to the Cisco HX Data Platform plug-in with administrator credentials. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 Step 2 Step 3 Step 4 From the vsphere Web Client Navigator, select vcenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > HX-Cluster > Summary. Select Upgrade Cluster. Select only, UCS Firmware. Click Next. Enter administrator level UCS Manager credentials. Field UCS Manager Host Name User Name Password Data For example: eng-fi12.eng.cisco.com <admin> username <admin> password Step 5 Step 6 Step 7 Click Discover to view the current firmware package version. Type in the exact, latest version of Cisco UCS firmware in the Target version field. Click Upgrade. The Cisco UCS servers are now upgraded with the desired firmware packages. The pending activities will be automatically acknowledged in a rolling fashion. Note You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile. The Validation screen shows the progress of checks performed. Fix validation errors, if any. The upgrade process proceeds as follows: HyperFlex cluster readiness for upgrade is checked. One by one, the HX nodes enter maintenance mode. 53

60 Upgrading Cisco HX Data Platform Using vsphere Web Client From 2.1 or Earlier Releases Upgrading Cisco HyperFlex Software Components The HX Data Platform requests Cisco UCS Manager to begin firmware upgrade. This process can take up to 1 hour. Note You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile. HX node exits maintenance mode. Cluster begins to rebuild back to full health. Once the cluster is healthy, upgrade moves on to the next node in the HyperFlex cluster. What to do next Confirm that the upgrade is complete. See Post Upgrade Tasks for more details. Upgrading Cisco HX Data Platform Using vsphere Web Client From 2.1 or Earlier Releases Upgrading the HX Data Platform does not cause any disruption to the workloads running on the HyperFlex Cluster. Before you begin Complete pre-upgrade validation checks. Download the latest Cisco HX Data Platform upgrade bundle for upgrading existing clusters, from Software Download. For example: storfs-packages-3.0.1c tgz Complete steps 1 to 6 in the Online Upgrade Process Work flow. Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform plug-in. Disable snapshot schedule, on the bootstrapped storage controller VM. Log in to the Cisco HX Data Platform plug-in with administrator credentials. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 From the vsphere Web Client Navigator, select Global Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > HX-Cluster > Summary. 54

61 Upgrading Cisco HyperFlex Software Components Upgrading ESXi From 2.1 or Earlier Releases Step 2 Step 3 Step 4 Step 5 Step 6 Select Upgrade Cluster. Select HX Data Platform. Navigate to the.tgz package file on your local PC. Upload the Cisco HX Data Platform upgrade bundle. This is the same.tgz package file that was used to upgrade the HX Data Platform Plug-in. Enter administrator level vcenter credentials. Click Upgrade. The Validation screen shows the progress of checks performed. Fix validation errors, if any. The upgrade process proceeds as follows: HyperFlex cluster readiness for upgrade is checked. One by one, the HX nodes enter maintenance mode. HyperFlex vsphere Installation Bundles on Hypervisor are upgraded. Cisco HX Data Platform is upgraded. HX node exits maintenance mode. Cluster will begin to rebuild back to full health. Once the cluster is healthy, upgrade moves on to the next node in the HyperFlex cluster. What to do next Confirm that the upgrade is complete. See Post Upgrade Tasks for more details. Upgrading ESXi From 2.1 or Earlier Releases Caution Using VMware Update Manager (VUM) to upgrade ESXi is discouraged. If you are using VUM to upgrade ESXi do the following: Use VUM one host at a time. Make sure that the cluster is in healthy state before moving on to the next node. Do not use VUM to upgrade ESXi across a cluster, as there is no guarantee that the cluster will be healthy by the time VUM moves on to the next node. The ESXi hypervisor version can be upgraded with no disruption to the HyperFlex cluster workload. This is achieved by performing an online rolling upgrade of each node in the HX cluster. 55

62 Upgrading ESXi From 2.1 or Earlier Releases Upgrading Cisco HyperFlex Software Components Important ESXi upgrade requires a manual online upgrade. When upgrading VMware ESXi from 5.5 U3b through any version up to 6.0 U2, please contact Cisco TAC. Use the ESXi command line interface esxcli for upgrading or updating ESXi. Replace the build numbers provided in the examples below with the latest version. Before you begin Complete pre-upgrade validation checks. See upgrade prerequisites in Chapter 3. Ensure that you upgrade vcenter to a compatible version before beginning ESXi upgrades on the hosts. Step 1 Step 2 Step 3 Download ESXi upgrade package. When upgrading ESXi from 6.0 Ux to any newer version, use the offline zip file from Download Software. Example filename: HX-Vmware-ESXi-60U Cisco-Custom-Bundle zip Attention Do not use the HX ISO file or any other VMware ISO to attempt an ESXi upgrade. Select one of the hosts and put it in HX maintenance mode using the vsphere Web Client, refer Entering Cisco HyperFlex Maintenance Mode, on page 32. After the host enters maintenance mode, complete the following steps. Remote secure copy the ESXi upgrade bundle to an appropriate folder with sufficient space. To copy files using SCP, start the SSH service in the destination ESXi hosts as well. Note On HX240, you can use the local SpringpathDS datastore or a mounted HX datastore. On HX220, you can use either a mounted HX datastore or create a temporary RAM disk. scp local_filename user@server:/path/where/file/should/go Step 4 Log in to ESXi, and execute the following command to query the list of available image profiles and for profile name verification. esxcli software sources profile list -d <location_of_the_esxi_zip_bundle_on_the_datastore> Attention Full path must be used when running the esxcli software command. Example: [root@hx-esxi-01:/vmfs/volumes/1a bc1234] esxcli software profile update -d /vmfs/volumes/1a bc1234/hx-vmware-esxi cisco-custom bundle.zip -p Vmware-ESXi HX Custom-Cisco Name Vendor Acceptance Level Vmware-ESXi-60U Cisco-Custom-HX Cisco PartnerSupported Step 5 Run the following command to perform the upgrade. esxcli software profile update -d <path_to_profile_zip_file> -p < profile name> 56

63 Upgrading Cisco HyperFlex Software Components Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware From 2.1 and Earlier Releases Example: esxcli software profile update -d /vmfs/volumes/1a bc1234/hx-vmware-esxi-60u cisco-custom-bundle zip -p Vmware-ESXi-60U Cisco-Custom-HX Step 6 Once the upgrade completes, restart the ESXi host. esxcli system shutdown reboot -r Update -d 10 Step 7 After the ESXi host comes up, verify that the host has booted up with the correct version. Step 8 vmware -vl Wait for the ESXi host to auto reconnect to vcenter. In some upgrade scenarios it may be necessary to force ESXi to reconnect from vcenter. Right-click on the host and select Connection > Connect. Step 9 Exit maintenance mode using the vsphere Web Client, refer Exiting Cisco HyperFlex Maintenance Mode, on page 33. Step 10 Ensure that the cluster becomes healthy between each ESXi upgrade. stcli cluster storage-summary --detail Step 11 Repeat this process for all hosts in the cluster in a sequence. Note Make sure that the cluster becomes healthy between each ESXi upgrade. Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware From 2.1 and Earlier Releases Combined upgrade does not cause any disruption to the workloads running on the HyperFlex Cluster. Important Do not acknowledge the pending activities in Cisco UCS Manager. Before you begin Complete pre-upgrade validation checks. See Prerequisites for more details. Download the latest Cisco HX Data Platform upgrade bundle for upgrading existing clusters, from Download Software. Complete steps 1 to 6 in the Online Upgrade Process Work flow. Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform plug-in. Disable snapshot schedule, on the bootstrapped storage controller VM. Log in to the Cisco HX Data Platform plug-in with administrator credentials. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. 57

64 Combined Upgrade of Cisco HX Data Platform and Cisco UCS Firmware From 2.1 and Earlier Releases Upgrading Cisco HyperFlex Software Components Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 From the vsphere Web Client Navigator, select vcenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > HX-Cluster > Summary. Select Upgrade Cluster. Select both, HX Data Platform and UCS Firmware. Click Next. Navigate to the.tgz package file on your local PC. Upload the Cisco HX Data Platform upgrade bundle. This is the same.tgz package file that was used to upgrade the HX Data Platform Plug-in. Enter administrator level vcenter credentials. (Optional) Enter the MD5 Checksum # information under Advanced Options. The file checksum can be found on the Cisco.com download page by clicking on the download title to reveal the md5 checksum. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle. Enter administrator level UCS Manager credentials. Click Discover to view the current firmware package version. Type in the exact, latest version of Cisco UCS firmware in the Target version field. Click Upgrade. The Cisco UCS servers are now upgraded with the desired firmware packages. The pending activities will be automatically acknowledged in a rolling fashion. Note You can monitor the progress in the Cisco UCS Manager GUI, under the FSM tab for the service profile. The Validation screen shows the progress of checks performed. Fix validation errors, if any. The upgrade process proceeds as follows: HyperFlex cluster readiness for upgrade is checked. One by one, the HX nodes enter maintenance mode. HyperFlex vsphere Installation Bundles (VIBs) on Hypervisor are upgraded. Cisco HX Data Platform is upgraded followed by a reboot of the host server. The HX Data Platform requests Cisco UCS Manager to begin firmware upgrade. This process can take up to 1 hour. HX node exits maintenance mode. Cluster will begin to rebuild back to full health. Once the cluster is healthy, upgrade moves on to the next node in the HyperFlex cluster. 58

65 Upgrading Cisco HyperFlex Software Components Compute-only Node Upgrade Considerations What to do next Confirm that the upgrade is complete. See Post Upgrade Tasks for more details. Compute-only Node Upgrade Considerations Upgrading Compute-only Nodes from Version 2.2(7c) to 3.1(2b) The upgrade deployment for compute-only nodes can be done either in online of offline mode. If you are upgrading your compute-only node from 2.2(7c)/1.7.x to 3.1(2b)/1.8(1c), follow the non-disruptive online rolling upgrade process as described in the Online Upgrade Process Workflow, on page 38. The online upgrade workflow will automatically upgrade both compute-only node firmware and HX Data Platform components as part of the full upgrade. For offline upgrade, refer to the Offline Upgrade Process Workflow, on page 39. Upgrading Cisco UCS Firmware on Compute-only Node from Version 1.7.x to 1.8.x or Later If you are upgrading your compute-only node from 1.7.x to 1.8.x or later releases, the new HX Data Platform Upgrade workflow will automatically upgrade the HX Data Platform components. However, after the HX Data Platform upgrade completes, you need to manually complete the upgrade of the compute-only node server firmware. Step 1 Step 2 Step 3 Step 4 Step 5 Ensure Distributed Resource Scheduler (DRS) is enabled and set to automatic mode. This allows the HX node to be vacated automatically. Otherwise, use vmotion to manually migrate all VMs off the host. Migration with vmotion allows you to move a virtual machine to a new host without any interruption in the availability of the virtual machine. For detailed steps, see VMware KB article. Place each compute-only node into maintenance mode. In the UCS manager GUI, Servers > Service Profiles, click Pending Activities to display the tasks requiring user acknowledgment before they can complete. Manually Acknowledge the pending reboot for each of the compute-only server. This will trigger a reboot and subsequent installation of new version of UCS firmware. Wait for the compute-only node to upgrade and fully reboot. After the compute-only node is up, verify the correct UCS firmware is present in UCS Manager. See Check Firmware Versions in UCSM, on page 74. Repeat this process for all compute-only nodes in the cluster. 59

66 Upgrading Cisco UCS Firmware on Compute-only Node from Version 1.7.x to 1.8.x or Later Upgrading Cisco HyperFlex Software Components 60

67 CHAPTER 7 HyperFlex Edge Upgrade Overview, on page 61 Cisco HyperFlex Edge Firmware Recommended Versions, on page 61 Upgrading HyperFlex Edge Using vsphere Web Client from 2.1 or Earlier Releases, on page 62 Upgrading HyperFlex Edge Using HX Connect from 2.5(1a) or Later Releases, on page 63 Server Firmware Upgrade Using the Cisco Host Upgrade Utility Tool, on page 64 Server Firmware Upgrade Using Cisco Integrated Management Controller Supervisor, on page 64 Post Upgrade Tasks for HyperFlex Edge, on page 66 Overview This section provides information related to upgrading a Cisco HyperFlex Edge system. Important For upgrading a HyperFlex Edge system, use split upgrade only. Do not use combined upgrade. When upgrading a HyperFlex Edge system, only HyperFlex Data Platform can be upgraded from the HX Connect UI. Do not select the UCS Server firmware option. Instead, perform the firmware upgrade separately using the Host Upgrade Utility (HUU) tool or the Integrated Management Controller (IMC) Supervisor. Cisco HyperFlex Edge Firmware Recommended Versions Use the recommended CIMC firmware version for each HXDP release as listed in the following table: Table 4: Firmware Recommended Versions for M4 Edge Servers CIMC 4.0(1b) 3.0(4j) HX Data Platform 3.5.x 3.0.1x Host Upgrade Utility Download Link Release 4.0(1b) Release 3.0(4j) 61

68 Upgrading HyperFlex Edge Using vsphere Web Client from 2.1 or Earlier Releases HyperFlex Edge Upgrade CIMC 3.0(3e) 3.0(3a) 2.0(13i) 2.0(13i) HX Data Platform 2.6.1x 2.5.1x 2.1.1x 2.0.1x Host Upgrade Utility Download Link Release 3.0(3e) Release 3.0(3a) Release 2.0(13i) Release 2.0(13i) Table 5: Firmware Recommended Versions for M5 Edge Servers CIMC 4.0(1b) 3.1(3h) 3.1(3c) HX Data Platform 3.5.x 3.0.1x 2.6.1x Host Upgrade Utility Download Link Release 4.0(1b) Release 3.1(1h) Release 3.1(3c) Upgrading HyperFlex Edge Using vsphere Web Client from 2.1 or Earlier Releases Follow these steps when upgrading from a HyperFlex Data Platform version prior to 2.5(1a): Step 1 Bootstrap to upgrade Cisco HX Data Platform Plug-in. See Manual Bootstrap Upgrade Process, on page 29. Important Be sure to copy the bootstrap file to the controller VM /tmp directory. Ensure that you confirm the version of the plug-in in the vcenter Administration > Client Plug-Ins page. Step 2 Disable snapshot schedule, on the bootstrapped storage controller VM. Run the command stcli snapshot-schedule --disable. It is enough to run this script on one of the controller nodes. Step 3 Step 4 Step 5 Step 6 Log in to the vsphere Web Client Plug-in with administrator credentials. Perform a split upgrade of the HX Data Platform only. Confirm that upgrade is complete. See Post Upgrade Tasks for HyperFlex Edge, on page 66 for more details. On the same controller VM, to enable snapshot schedule, run the command stcli snapshot-schedule - enable. 62

69 HyperFlex Edge Upgrade Upgrading HyperFlex Edge Using HX Connect from 2.5(1a) or Later Releases Upgrading HyperFlex Edge Using HX Connect from 2.5(1a) or Later Releases Follow these steps when upgrading a HyperFlex Edge system from a current HX Data Platform version of 2.5(1a) or later releases. Step 1 Bootstrap to upgrade Cisco HX Data Platform Plug-in. See Manual Bootstrap Upgrade Process, on page 29 for more details. Important Be sure to copy the bootstrap file to the controller VM /tmp directory. Step 2 Step 3 Step 4 Step 5 Log in to HX Connect. In the Navigation pane, select Upgrade. On the Select Upgrade Type page, select HX Data Platform only. Click Continue. Complete the following fields on the Enter Credentials page. Upgrade HX Data Platform UI Element Drag the HX file here or click to browse Essential Information Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform. Example: storfs-packages-2.5.1b tgz. Current version Current cluster details Bundle version (Optional) Checksum field Displays the current HyperFlex Data Platform version. Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state. Displays the HyperFlex Data Platform version of the uploaded bundle. The MD5 Checksum number is stored in a separate text file at the /tmp directory where the upgrade package was downloaded. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle. vcenter Credentials UI Element User Name field Admin Password field Essential Information Enter the vcenter <admin> username. Enter the vcenter <admin> password. Step 6 Click Upgrade. 63

70 Server Firmware Upgrade Using the Cisco Host Upgrade Utility Tool HyperFlex Edge Upgrade Step 7 The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. Server Firmware Upgrade Using the Cisco Host Upgrade Utility Tool The following table summarizes the server firmware upgrade workflow on Cisco HX Servers: Step Description Place a node in HX maintenance mode. Note Upgrade one node at a time, for the cluster to stay online during upgrade. Upgrade server firmware using the Host Upgrade Utility tool. Reboot the node back into ESXi. Exit HX maintenance mode. Wait until the cluster becomes fully healthy. Repeat steps 1-4 on the remaining HX nodes in a rolling fashion. Note Ensure that you check the health state before entering maintenance mode on the next host in the cluster. Reference Verify vmotion Configuration for HX Cluster, on page 31 Entering Cisco HyperFlex Maintenance Mode, on page 32 See Updating the Firmware on Cisco UCS C-Series Servers in the Cisco Host Upgrade Utility User Guide. Exiting Cisco HyperFlex Maintenance Mode, on page 33 Viewing HyperFlex Cluster Health, on page 18 You can find current and previous releases of the Cisco Host Upgrade Utility User Guide at this location: products-user-guide-list.html. Server Firmware Upgrade Using Cisco Integrated Management Controller Supervisor The following table summarizes the server firmware upgrade workflow on Cisco HX Servers: 64

71 HyperFlex Edge Upgrade Server Firmware Upgrade Using Cisco Integrated Management Controller Supervisor Step Description Place a node in HX maintenance mode. Note Upgrade one node at a time, for the cluster to stay online during upgrade. Create a rack group. Add servers to the IMC Supervisor inventory. Configure auto-discovery profile. Run inventory on the rack group. Create firmware profile. Reference Verify vmotion Configuration for HX Cluster, on page 31 Entering Cisco HyperFlex Maintenance Mode, on page 32 See Managing Server Discovery, Rack Groups, and Rack Accounts in the Cisco IMC Supervisor Rack-Mount Servers Management Guide. See Configuring Auto Discovery Profile in the Cisco IMC Supervisor Rack-Mount Servers Management Guide. See Collecting Inventory for Rack Accounts or Rack Groups in the Cisco IMC Supervisor Rack-Mount Servers Management Guide. Refer to the following tasks in the Cisco IMC Supervisor Rack-Mount Servers Management Guide: Adding Images to a Local Server Uploading Images from a Local File System Adding Images from a Network Sever Upgrade firmware using IMC Supervisor, on the node put in maintenance mode. Reboot the node back into ESXi. Exit HX maintenance mode. Wait until the cluster becomes fully healthy. See Upgrading Firmware in the Cisco IMC Supervisor Rack-Mount Servers Management Guide. Exiting Cisco HyperFlex Maintenance Mode, on page 33 Viewing HyperFlex Cluster Health, on page 18 65

72 Post Upgrade Tasks for HyperFlex Edge HyperFlex Edge Upgrade Step 9. Description Repeat step 6 on the remaining HX nodes in a rolling fashion. Note Ensure that you check the health state before entering maintenance mode on the next host in the cluster. Reference You can find current and previous releases of the Cisco IMC Supervisor Rack-Mount Servers Management Guide here: integrated-management-controller-imc-supervisor/products-installation-and-configuration-guides-list.html. Post Upgrade Tasks for HyperFlex Edge After the upgrade is complete and the HyperFlex Edge cluster has been upgraded, log out and log back in to vcenter to see the upgrade changes. Step 1 Step 2 Confirm that the HX nodes match the expected firmware version. Check the firmware version in the IMC supervisor GUI to verify for the correct firmware version. To view the firmware version, in the IMC Supervisor GUI, navigate to the Systems > Firmware Management tab. See Upgrading Firmware using IMC Supervisor for more details. Log in to any controller VM through SSH. # ssh root@controller_vm_ip Step 3 Confirm the HyperFlex Data Platform version. # stcli cluster version Cluster version: 2.5(1c) Node HX02 version: 2.5(1c) Node HX01 version: 2.5(1c) Node HX03 version: 2.5(1c) Step 4 Verify that the HX storage cluster is online and healthy. # stcli cluster info grep -i health Sample output: healthstate : healthy state: healthy storage cluster is healthy Step 5 Verify that the datastores are up and are mounted properly on the ESXi host. From HX controller VMs run the command: # stcli datastore list From the ESXi host run: 66

73 HyperFlex Edge Upgrade Post Upgrade Tasks for HyperFlex Edge # esxcfg-nas -l Step 6 For each browser interface you use, empty the cache and reload the browser page to refresh the HX Connect content. 67

74 Post Upgrade Tasks for HyperFlex Edge HyperFlex Edge Upgrade 68

75 CHAPTER 8 Stretched Cluster Upgrade Overview, on page 69 Upgrade Guidelines for Stretched Cluster, on page 69 Upgrading HyperFlex Stretched Cluster Using HX Connect, on page 69 Overview This section provides information related to upgrading a Cisco HyperFlex Stretched Cluster. The procedure for performing a Stretched Cluster upgrade is similar to the regular HyperFlex cluster upgrade procedure. Upgrade Guidelines for Stretched Cluster Split upgrade of the HX Data Platform only is supported. Upgrading ESXi and UCS firmware upgrade is not supported. Manual cluster bootstrap is required for upgrade from a pre-3.5 release to 3.5(1a). Auto bootstrap is supported for upgrade from 3.5(1a) to later releases. HyperFlex Witness node version is supported from 3.5(1a) or later releases. An upgrade of the HyperFlex Witness node is not required when upgrading stretched clusters to 3.5(1a) or later releases. Upgrading HyperFlex Stretched Cluster Using HX Connect Follow these steps when upgrading a HyperFlex Stretched Cluster from a current HX Data Platform version of 3.0(1x) or later releases. Before you begin Complete pre-upgrade validation checks. See Prerequisites for more details. Download the latest Cisco HX Data Platform Upgrade Bundle for upgrading existing clusters from previous releases, from Software Download. Complete steps 1 to 6 in the Online Upgrade Process Work flow. See Online Upgrade Process Workflow, on page 38 for more details. 69

76 Upgrading HyperFlex Stretched Cluster Using HX Connect Stretched Cluster Upgrade Upgrade Cisco UCS Infrastructure. Bootstrap to upgrade Cisco HX Data Platform plug-in. Disable snapshot schedule, on the bootstrapped storage controller VM. If DRS is Enabled, the VMs are automatically vmotioned to other hosts. Note If DRS is Disabled, vmotion the VMs manually to continue the upgrade process. For more information, see VMware Documentation for Migration with vmotion. Step 1 Step 2 Step 3 Log in to HX Connect. a) Enter the HX Storage Cluster management IP address in a browser. Navigate to b) Enter the administrative username and password. c) Click Login. In the Navigation pane, select Upgrade. On the Select Upgrade Type page, select HX Data Platform and complete the following fields: UI Element Drag the HX file here or click to browse Essential Information Upload the latest Cisco HyperFlex Data Platform Upgrade Bundle for upgrading existing clusters with previous release.tgz package file from Download Software - HyperFlex HX Data Platform. Example: storfs-packages-2.5.1b tgz. Current version Current cluster details Bundle version (Optional) Checksum field Displays the current HyperFlex Data Platform version. Lists the HyperFlex cluster details like the HyperFlex version and Cluster upgrade state. Displays the HyperFlex Data Platform version of the uploaded bundle. The MD5 Checksum number is stored in a separate text file at the /tmp directory where the upgrade package was downloaded. This is an optional step that helps you verify the integrity of the uploaded upgrade package bundle. Step 4 Enter the vcenter credentials. UI Element User Name field Admin Password field Essential Information Enter the vcenter <admin> username. Enter the vcenter <admin> password. Step 5 Click Upgrade to begin the cluster upgrade process. 70

77 Stretched Cluster Upgrade Upgrading HyperFlex Stretched Cluster Using HX Connect Step 6 The Validation Screen on the Upgrade Progress page displays the progress of the checks performed. Fix validation errors, if any. Confirm that the upgrade is complete. 71

78 Upgrading HyperFlex Stretched Cluster Using HX Connect Stretched Cluster Upgrade 72

79 CHAPTER 9 Post Upgrade Tasks Confirm That Upgrade Is Complete, on page 73 Check Firmware Versions in UCSM, on page 74 Verify If Cleaner Is Running, on page 74 Confirm That Upgrade Is Complete Step 1 Log in to Cisco UCS Manager to ensure that the HX nodes have no pending server activities. From Servers tab > Servers > Pending Activities tab check for all server activities. Step 2 Confirm that the HX nodes match the expected firmware version. In Cisco UCS Manager, from Equipment > Firmware Management > Installed Firmware tab, verify for the correct firmware version. Step 3 Log in to any controller VM through SSH. # ssh root@controller_vm_ip Step 4 Confirm the HyperFlex Data Platform version. # stcli cluster version Cluster version: 2.5(1c) Node HX02 version: 2.5(1c) Node HX05 version: 2.5(1c) Node HX01 version: 2.5(1c) Node HX03 version: 2.5(1c) Node HX04 version: 2.5(1c) Step 5 Verify that the HX storage cluster is online and healthy. # stcli cluster info grep -i health Sample output: healthstate : healthy state: healthy storage cluster is healthy Step 6 Verify that the datastores are up and are mounted properly on the ESXi host. 73

80 Check Firmware Versions in UCSM Post Upgrade Tasks From the HX controller VMs: # stcli datastore list From the ESXi host: # esxcfg-nas -l Step 7 Verify that the upgrade is complete and is successful. stcli cluster upgrade-status Nodes up to date: [HX-Cluster, HX-Node-1( ), HX-Node-2( ), HX-Node-3( )] Cluster upgrade succeeded. Step 8 For each browser interface you use, empty the cache and reload the browser page to refresh the HX Connect content. Check Firmware Versions in UCSM In Cisco UCS Manager, from Equipment > Firmware Management > Installed Firmware tab, verify for the correct firmware version. For a complete list of hardware and software inter-dependencies, refer to respective UCSM release version using the UCS Hardware and Software Compatibility tool. Verify If Cleaner Is Running If Upgrade Fails If upgrade fails, run cleaner. This is required even if you do not want to continue with an upgrade. To run cleaner manually, restart the storage cluster cleaner using the following command. stcli cleaner start [-h] [--id ID --ip NAME] Syntax Description Option Required or Optional Description --id ID Optional. ID of storage cluster node. The ID is listed in the stcli cluster info command. --ip NAME Optional. IP address of storage cluster node. The IP is listed in the stcli cluster info command. If Upgrade Completes If upgrade completes, verify if cleaner is running. To obtain information about the storage cluster cleaner for the specified node, use the following command. stcli cleaner info [-h] [--id ID --ip NAME] 74

81 Post Upgrade Tasks Verify If Cleaner Is Running Syntax Description Option Required or Optional Description --id ID Optional. ID of storage cluster node. The ID is listed in the stcli cluster info command. --ip NAME Optional. IP address of storage cluster node. The IP is listed in the stcli cluster info command. 75

82 Verify If Cleaner Is Running Post Upgrade Tasks 76

83 CHAPTER 10 Known Issues Overview, on page 77 ESXi Host Is In Lockdown Mode, on page 77 Offline Upgrade Cluster Start Command Error: Node Not Available, on page 78 A Node Fails to Upgrade due to vcenter Issues, on page 78 Overview This chapter provides information to help you troubleshoot common problems that may occur during the Cisco HyperFlex upgrade process. ESXi Host Is In Lockdown Mode Description If the ESXi host is in lockdown mode, pre-upgrade validation will fail with the error message auth cancel. Action: Disable Lockdown mode on the ESXi host and enable it after the upgrade is successful. Enable/Disable Lockdown Mode Using HX Connect 1. Log in to HX Connect. 2. On the Navigation pane, select System Overview. 3. On the System Overview tab, from the Actions drop-down list, you can enable or disable access to the controller VM using SSH as an administrator. Using vsphere Web Client 1. Log in to vsphere Web Client. 2. Browse to the host in the vsphere Web Client inventory. 3. Click the Manage tab and click Settings. 4. Under System, select Security Profile. 77