StorageGRID Webscale 11.0 Upgrade Guide

Transcription

1 StorageGRID Webscale 11.0 Upgrade Guide January _B0

2

3 Table of Contents 3 Contents What's new in StorageGRID Webscale Removed or deprecated features... 6 Changes in the Grid Management API... 7 Upgrade planning and preparation... 9 How your system is affected during the upgrade... 9 Impact of an upgrade on groups and user accounts Impact of an upgrade on tenant accounts Verifying the installed version of StorageGRID Webscale Required materials for software upgrades Web browser requirements Downloading and extracting the StorageGRID Webscale upgrade files Checking the system's condition before upgrading software Performing the upgrade Verifying the completion of your upgrade Managing legacy ILM rules Backing up the Recovery Package Troubleshooting upgrade issues Cannot shut down server because a service fails to stop User interface issues Docker image availability check error messages Copyright information Trademark information How to send comments about documentation and receive update notifications Index... 30

4 4 What's new in StorageGRID Webscale 11.0 StorageGRID Webscale 11.0 introduces platform services for S3 tenant accounts, adds support for new Linux operating systems, enhances ILM functionality, simplifies log collection, adds support for a new StorageGRID Webscale appliance, and adds or enhances other features and capabilities. Platform services for S3 tenant accounts Platform services are now available for S3 tenant accounts, if enabled by a grid administrator. The following platform services give tenants the ability to use external services with the objects in S3 buckets: CloudMirror replication: This service enables tenants to automatically replicate objects from a StorageGRID Webscale S3 bucket to an external location, such as another StorageGRID Webscale deployment or Amazon Web Services. Notifications: This service enables tenants to configure event notifications. Notifications can be delivered to any external endpoint that supports the Simple Notification Service (SNS). Search integration: The search integration service is used to automatically send object metadata and tags to an external search index, where tenants can use Elasticsearch to perform searches and sophisticated data analysis. Attention: StorageGRID Webscale 11.0 includes the initial release of platform services. CloudMirror replication, notifications, and search integration are currently appropriate only for specific situations and workloads. You must contact your NetApp representative if you want to use the initial release of these services. Enhancements to tenant accounts The following changes have been made to tenant accounts: A tenant can now use up to 1000 buckets per S3 tenant account or 1000 containers per Swift tenant account. An S3 tenant account can now change the consistency level used for operations performed on objects. The setting is made on a per-bucket basis. An S3 tenant account can now enable Last Access Time updates for objects. Enabling Last Access Time allows tenants to take advantage of time-based ILM rules, such as rules that keep recently-viewed objects on local storage for faster access. Last Access Time is enabled on a perbucket basis. Support for Ubuntu, Debian, and CentOS Linux You can now deploy StorageGRID Webscale on Ubuntu, Debian, and CentOS platforms. For information, see the following publications: StorageGRID Webscale Installation Guide for Ubuntu and Debian Deployments StorageGRID Webscale Installation Guide for Red Hat Enterprise Linux and CentOS Deployments StorageGRID Webscale Expansion Guide StorageGRID Webscale Recovery and Maintenance Guide

5 What's new in StorageGRID Webscale Enhancements to information lifecycle management (ILM) ILM rules now support Location Constraints and can filter objects based on the regions where S3 buckets are created. This allows you to store objects from different regions in different storage locations. If you want to use S3 bucket regions as advanced filters in ILM rules, you must first create those regions using the new ILM > Regions option in the Grid Management Interface. You can also create and list regions using the new regions section of the Grid Management API. The ILM section of the Grid Management Interface has been enhanced with updated pages for creating Erasure Coding profiles and managing policies. S3 object tags are now supported, allowing you to create ILM rules that include advanced filters that filter on objects with specific tags. To make it easier to verify object information, for example, after updating the active ILM policy, the user interface for viewing object metadata has been enhanced. You can use the new ILM > Object Metadata Lookup option to view object metadata. You can also access this information using the new objects section of the Grid Management API. For information, see the StorageGRID Webscale S3 Implementation Guide and the StorageGRID Webscale Administrator Guide. SG5700 StorageGRID Webscale appliance A new SG5700 appliance is available that includes two models: The SG5712, which includes an E2800 controller and an E5700SG controller in a 2U, 12-drive enclosure The SG5760, which includes an E2800 controller and an E5700SG controller in a 4U, 60-drive enclosure For information, see the Hardware Installation and Maintenance Guide for StorageGRID Webscale SG5700 Series Appliances. New StorageGRID Webscale Appliance Installer A new StorageGRID Webscale Appliance Installer makes it easier to deploy appliance Storage Nodes during installation, expansion, and recovery operations. You are no longer required to connect to the Admin Network when deploying appliance Storage Nodes. As StorageGRID Webscale is being installed on the appliance, a fully interactive console appears below the list of operations, allowing you to log into directly the Storage Node. You no longer need to log into the node from a remote console or use SSH for troubleshooting or recovery. Erasure coding changes After you upgrade to version 11.0, the following changes to the erasure coding feature take effect: When you run foreground verification, erasure coded data is now checked. You can now begin node recovery for Storage Nodes that contain erasure coded data without all nodes in the grid being available. Recovery can also proceed even when grid nodes are temporarily unavailable.

6 6 StorageGRID Webscale 11.0 Upgrade Guide Improvements to grid node decommissioning The process of grid node decommissioning has been improved. Decommissioning now completes more quickly and reliably, and progress is displayed in the Grid Management Interface. See the Recovery and Maintenance Guide for more information. Simplified Lumberjack log collection You can now collect log files from grid nodes using the Grid Management Interface. See the Troubleshooting Guide for details. API support for Prometheus metrics In StorageGRID Webscale 11.0, you can use the Grid Management API to perform custom Prometheus queries on metrics. Both instant metric queries at a single point of time and range metric queries over a range of time are supported. Note: The Grid Management API uses the Prometheus systems monitoring tool as the backend data source. For information about constructing Prometheus queries, see the Prometheus web site. Support for Cassandra version 2.1 Version 11.0 of StorageGRID Webscale uses version 2.1 of the Cassandra database. The Cassandra version is automatically updated when you upgrade to StorageGRID Webscale version Documentation improvements The StorageGRID Webscale documentation has been simplified. There is now a single Expansion Guide and a single Recovery and Maintenance Guide. These guides include information on expanding or maintaining StorageGRID Webscale deployments on all supported platforms. Related concepts Changes in the Grid Management API on page 7 Related information Ubuntu or Debian installation Red Hat Enterprise Linux or CentOS installation Administering tenant accounts Administering StorageGRID Webscale S3 client application implementation Swift client application implementation Recovery and maintenance Expanding a StorageGRID Webscale grid SG5700 appliance installation and maintenance Prometheus: Query basics Removed or deprecated features Certain features have been removed or deprecated in StorageGRID Webscale The following feature has been removed: All support for the Cloud Data Management Interface (CDMI) has been removed from the StorageGRID Webscale 11.0 product.

7 What's new in StorageGRID Webscale The following features are deprecated and are not supported for new StorageGRID Webscale 11.0 installations: Connections that use Transport Layer Security (TLS) 1.0 are deprecated. S3, Swift, or management API clients that use TLS 1.0 connections should plan to use a newer version of TLS. Support for TLS 1.0 will be removed in StorageGRID Webscale NetApp-provided virtual machine disk files and scripts for OpenStack have been deprecated for new installations or expansions of StorageGRID Webscale. To install or expand an OpentStack deployment, download the files for your Linux operating system. Then follow the instructions in the appropriate StorageGRID Webscale 11.0 Installation Guide or the Expansion Guide. Note: NetApp-provided virtual machine disk files and scripts for OpenStack continue to be available for recovery. ILM rules that were created before StorageGRID Webscale 10.3 were deprecated in version 10.4; however, active policies with legacy rules will continue to work as expected after you upgrade to version You should recreate or delete these legacy rules after upgrading. See Managing legacy ILM rules for more information. Version 1 of the Grid Management API was deprecated in 10.3, but continues to be supported for 11.0 upgrades as follows: By default, new installs of StorageGRID Webscale 11.0 only allow version 2 API requests. Upgraded systems that were originally installed using StorageGRID Webscale 10.4 default to allowing only version 2 API requests. Upgraded systems that were originally installed using StorageGRID Webscale 10.3 or earlier default to allowing both version 1 and version 2 API requests. Support for version 1 API requests can be manually enabled (or disabled) for any grid if required. If a version was manually enabled or disabled in StorageGRID Webscale 10.4, upgrading to StorageGRID Webscale 11.0 does not change the allowed API versions. See Changes in the Grid Management API for more information. Related concepts Changes in the Grid Management API on page 7 Related tasks Managing legacy ILM rules on page 21 Related information Red Hat Enterprise Linux or CentOS installation Ubuntu or Debian installation Expanding a StorageGRID Webscale grid Changes in the Grid Management API StorageGRID Webscale 11.0 supports version 2 of the Grid Management API. Version 2 of the Grid Management API deprecates version 1 of the management API. Important: You can continue to use version 1 of the Grid Management API with StorageGRID Webscale 11.0; however, support for this version of the API will be removed in a future release of StorageGRID Webscale. If you are still using version 1 of the API, you should update all management API clients to use version 2 after you upgrade to StorageGRID Webscale Then,

8 8 StorageGRID Webscale 11.0 Upgrade Guide deactivate version 1 support. See information about the Grid Management API in the StorageGRID Webscale Administrator Guide. API error data can be returned in HTML Previously, error data was always returned in JSON format, regardless of what format was specified as preferred in the Accept header for the API request. Now, error data will be returned in HTML format, if HTML is specified as the preferred format. Changes to the Auth API to protect against Cross-Site Request Forgery (CSRF) The Authorize API accepts a new optional parameter, csrftoken. If true, CSRF tokens are required when signing in. The csrftoken is set to true when you sign in using the Grid Management Interface or Tenant Management Interface. Applications that share cookies with one of these StorageGRID Webscale Management user interfaces might be affected by this change. See How to protect against Cross-Site Request Forgery in the Administrator Guide for more information. Related information Administering StorageGRID Webscale

9 9 Upgrade planning and preparation You must plan the upgrade of your StorageGRID Webscale system to ensure that the system is ready for the upgrade, and that the upgrade can be completed with minimal disruption. Before beginning the upgrade from StorageGRID Webscale 10.4 to StorageGRID Webscale 11.0.x, or from version 11.0.x to version 11.0.y, you must read this guide in full and become familiar with the requirements and steps needed to upgrade your StorageGRID Webscale system. Before you begin, be aware of the following: You must understand how your system will be affected during the upgrade to ensure you do not lose configuration changes, logs, or locally applied network changes. See How your system is affected during the upgrade for more information. You must understand the impact of the upgrade, so that you can update user accounts, groups, and tenant accounts appropriately after the upgrade is complete. You must have all materials required for the upgrade, including the StorageGRID Webscale upgrade files. You must be at version 10.4 or later to upgrade to version 11.0.x. If you have an earlier version, you must upgrade to version 10.4 before proceeding with the steps described in this guide. You must complete the upgrade on the same hypervisor platform you started with. You must upgrade all grid nodes for all data center sites from the primary Admin Node, using the Grid Management Interface. You must open port 9999 on all nodes. In StorageGRID Webscale 11.0, Admin Nodes use port 9999 to communicate with all nodes for Prometheus metrics. Some features will not work if port 9999 is blocked. Related concepts How your system is affected during the upgrade on page 9 How your system is affected during the upgrade During the upgrade, your StorageGRID Webscale system remains available for object ingests and retrievals. In some cases, there might be periods within which client applications cannot connect to a grid node. Alarms might also be triggered during the upgrade; these should clear as the upgrade process completes. Client applications might experience short-term disruptions The StorageGRID Webscale system can ingest and retrieve data from client applications throughout the upgrade process except for a short period of time when services are restarting and client connections to individual API Gateway Nodes or Storage Nodes might be disrupted. Connectivity will be restored after the upgrade finishes and services resume on the individual nodes. Every node in your StorageGRID Webscale system will be brought down one at a time for several minutes each during the upgrade. You might need to schedule downtime for the upgrade if loss of connectivity for a short period is not permitted. Attention: If an API Gateway Node is connected to a NAS Bridge instance, you must manually connect the NAS Bridge to an alternate API Gateway Node before you upgrade the primary API Gateway Node. Otherwise, NFS and SMB clients will temporarily lose access to their data.

10 10 StorageGRID Webscale 11.0 Upgrade Guide Alarms might be triggered StorageGRID Webscale alarms might be triggered when the StorageGRID Webscale system is operating in a mixed-version environment (some grid nodes still running version 10.4, while others have been upgraded to version 11.0). In general, these alarms clear when the upgrade completes. Many s are generated When you upgrade grid nodes, notifications are generated when the node is stopped and restarted. To avoid excessive s, you can disable notifications before upgrading the first node and re-enable notifications after the upgrade is completed. Configuration changes are restricted During the upgrade to release StorageGRID Webscale 11.0, the following restrictions apply: Do not make any grid configuration changes until the upgrade is finished. Do not enable or disable any new features until the upgrade completes. Because of ILM policy configuration changes in StorageGRID Webscale, do not update the ILM configuration until you have upgraded all grid nodes. Otherwise, you might experience inconsistent and unexpected ILM behavior. Errors might occur when creating S3 buckets that use the LocationConstraint request element (regions) Previous releases of StorageGRID Webscale did not support using the LocationConstraint request element for S3 PUT Bucket API requests. Any regions that you might have added to S3 buckets were ignored. StorageGRID Webscale version 11.0 supports regions. Before upgrading your grid, be aware of the following behavior: After the upgrade, any buckets that do not have a region specified are considered to belong to the us-east-1 region, which is the default. After the upgrade, an error occurs if a PUT Bucket request specifies a region that has not been defined in StorageGRID Webscale. If a current application creates buckets with the LocationConstraint request element set to anything other than us-east-1, you must do either of the following to avoid these errors: Before upgrading to version 11.0, change the application's bucket-creation settings to use the useast-1 region. After upgrading the primary Admin Node, create these other regions in StorageGRID Webscale. You can use the Grid Management Interface (ILM > Regions) or the Grid Management API to create StorageGRID Webscale regions. Installed Storage Capacity might appear to decrease After you upgrade to StorageGRID Webscale version 11.0, you might notice a decrease in the value of the Installed Storage Capacity (XISC) attribute. This attribute indicates the total storage capacity of the installed object stores. Previously, the value for this attribute was being calculated incorrectly. The calculation has been corrected, resulting in an apparent reduction in the total installed storage capacity. The decrease is only because of the corrected calculation in StorageGRID Webscale version 11.0; the actual installed storage capacity remains the same as before.

11 Upgrade planning and preparation 11 Impact of an upgrade on groups and user accounts Before upgrading to a new version of StorageGRID Webscale, you should understand how groups and users might be affected. Changes to permissions The following management permissions have been changed in StorageGRID Webscale To enable users to perform these tasks, a user with Root Access permission must update permissions for the applicable groups. Permission ILM Description Provides access to the following items in the ILM menu: Rules Policies Erasure Coding Regions Metrics Query Object Metadata Lookup Provides access to custom Prometheus metrics queries using the Metrics section of the Grid Management API. Provides access to the following item in the ILM menu: Object Metadata Lookup Related information Administering StorageGRID Webscale Impact of an upgrade on tenant accounts Before upgrading to a new version of StorageGRID Webscale, you should understand how tenant accounts might be affected. Enabling platform services for tenant accounts Wait until the upgrade to StorageGRID Webscale 11.0 completes before enabling or using platform services. Platform services are not fully functional until the upgrade is complete. Platform services are not enabled for tenant accounts that existed before the upgrade. To allow these tenants to use CloudMirror replication, event notifications, or the search integration service for S3 buckets, you must edit each S3 tenant account to enable platform services. See the Administrator Guide for information on editing tenant accounts using the Grid Management Interface. See the Tenant Administrator Guide for more information on platform services. Changes to permissions for tenant users and groups The following management permissions have been changed for tenant users in StorageGRID Webscale To enable tenant users to perform these tasks, a tenant user with Root Access permission must update permissions for the applicable groups.

12 12 StorageGRID Webscale 11.0 Upgrade Guide Permission Manage all containers Description S3 tenants: Allows access to the S3 > Buckets menu. Permits a user to manage bucket settings for all S3 buckets in the tenant account using the Tenant Management Interface or the Tenant Management API. Bucket settings include Last Access Time, consistency level, and the settings required to enable each platform service. Swift tenants: Allows Swift users to control the consistency level for Swift containers using the Tenant Management API. The manage all containers permission can be granted to Swift groups using the Tenant Management API. Manage endpoints S3 tenants only. Allows users to use the Tenant Management Interface or the Tenant Management API to create or edit endpoints, which are used as the destination for StorageGRID Webscale platform services. Users who do not have this permission do not see the S3 > Endpoints menu option. Related information Administering StorageGRID Webscale Administering tenant accounts Verifying the installed version of StorageGRID Webscale Before starting the upgrade, you must verify which version of the StorageGRID Webscale is currently installed. Before you begin To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Sign in to the Grid Management Interface using a supported browser. 2. Select Maintenance > Software Upgrade. 3. Verify that the Current Version is or later. Important: You must be at version or later to upgrade to version 11.0.x. If you have an earlier version, you must upgrade to version before proceeding with the steps described in this guide. Related information Administering StorageGRID Webscale

13 Upgrade planning and preparation 13 Required materials for software upgrades Before you begin the software upgrade, you must gather the required materials so you can complete the upgrade successfully. What you need The latest release of the StorageGRID Webscale software Description You must download the StorageGRID Webscale upgrade file on your service laptop: NetApp_StorageGRID_version_Software_uniqueID.upgrade If you are upgrading a StorageGRID Webscale deployment running on Red Hat Enterprise Linux hosts, you must also download one of the StorageGRID Webscale installation packages: StorageGRID-Webscale-version-RPM-uniqueID.tgz StorageGRID-Webscale-version-RPM-uniqueID.zip Service laptop The service laptop must have: Network port SSH client (for example, PuTTY) Supported web browser Recovery Package (.zip) file You must download the most recent Recovery Package file from the StorageGRID Webscale system. Select Maintenance > Recovery Package. sgws-recovery-package-id-revision.zip The.zip file includes the SAID package. Use the latest revision of the SAID package. Passwords.txt file Provisioning passphrase Related documentation This file is included in the SAID package, which is part of the Recovery Package.zip file. The passphrase is created and documented when the StorageGRID Webscale system is first installed. The provisioning passphrase is not listed in the Passwords.txt file. Release Notes. Be sure to read these carefully before starting the upgrade. Administrator Guide If you are upgrading a Red Hat Enterprise Linux deployment, the Installation Guide for Red Hat Enterprise Linux or CentOS Deployments Other StorageGRID Webscale documents, as required. Related tasks Downloading and extracting the StorageGRID Webscale upgrade files on page 14 Related references Web browser requirements on page 14

14 14 StorageGRID Webscale 11.0 Upgrade Guide Related information Administering StorageGRID Webscale Red Hat Enterprise Linux or CentOS installation NetApp Documentation: StorageGRID Webscale Web browser requirements You must use a supported web browser. Web browser Google Chrome 54 Microsoft Internet Explorer Mozilla Firefox 50 Minimum supported version 11 (Native Mode) You should set the browser window to a recommended width. Browser width Pixels Minimum 1024 Optimum 1280 Downloading and extracting the StorageGRID Webscale upgrade files Before you can upgrade your StorageGRID Webscale system, you must download the software. About this task For all environments, you must download the StorageGRID Webscale upgrade package to your service laptop: NetApp_StorageGRID_version_Software_uniqueID.upgrade If your StorageGRID Webscale system is deployed on Red Hat Enterprise Linux hosts, you must also download either of the StorageGRID Webscale installation files: StorageGRID-Webscale-version-RPM-uniqueID.tgz StorageGRID-Webscale-version-RPM-uniqueID.zip Use the.zip file if you are running Windows on the service laptop Steps 1. Go to the Software Download page on the NetApp Support Site. NetApp Downloads: Software 2. Sign in using the username and password for your NetApp account. 3. Scroll to StorageGRID Webscale, select All Platforms, and click Go. Note: Be sure to select StorageGRID Webscale, not StorageGRID. 4. Select the StorageGRID Webscale release, and click View & Download.

15 Upgrade planning and preparation From the Software Download section of the page, click CONTINUE, and accept the End User License Agreement. 6. Download the appropriate archive. 7. If your StorageGRID Webscale system is deployed on Red Hat Enterprise Linux, you must also complete these steps: a. Download one of the StorageGRID Webscale installation files for Red Hat Enterprise Linux: StorageGRID-Webscale-version-RPM-uniqueID.tgz StorageGRID-Webscale-version-RPM-uniqueID.zip b. Extract the RPM packages from the installation file. c. Install the RPM packages on all Red Hat Enterprise Linux hosts as described in Installing StorageGRID Webscale host services in the Installation Guide. Note: You must complete these steps before continuing with the upgrade procedure. Related information Red Hat Enterprise Linux or CentOS installation Checking the system's condition before upgrading software Before upgrading a StorageGRID Webscale system, you must verify the system is in a state that is ready to accommodate the upgrade. You must ensure that the system is running normally and that all grid nodes are operational. Steps 1. Sign in to the Grid Management Interface using a supported browser. 2. Check for and resolve any active alarms. For more information on specific alarms, see the Troubleshooting Guide. 3. Select Grid > Site > primary Admin Node > CMN > Grid Tasks > Configuration, and confirm that no conflicting grid tasks are active or pending. The only exceptions are information lifecycle management evaluation (ILME) tasks. These grid tasks can run concurrently with the software upgrade. If any other grid tasks are active or pending, wait for them to finish or release their lock. For more information, see the Administrator Guide. Note: Contact technical support if a task does not finish or release its lock. 4. Optionally, recreate or delete any legacy ILM rules that are no longer part of the active policy. Legacy ILM rules are rules that you created before version You are not required to recreate these rules before upgrading to StorageGRID Webscale Active policies that contain legacy rules will continue to work after the upgrade. See Managing legacy ILM rules for information. Note: It is best to wait until after the upgrade has completed to recreate or delete any legacy rules. When you recreate a rule in version 11.0 you can use the ILM Simulation feature to confirm that the new rule is behaving as expected. 5. Ensure that your enterprise firewall rules allow traffic on port 9999 between all nodes.

16 16 StorageGRID Webscale 11.0 Upgrade Guide In StorageGRID Webscale 11.0, Admin Nodes use port 9999 to communicate with all nodes for Prometheus metrics. Some features will not work if port 9999 is blocked. Related tasks Managing legacy ILM rules on page 21 Related information Administering StorageGRID Webscale Expanding a StorageGRID Webscale grid Recovery and maintenance

17 17 Performing the upgrade You must upgrade all grid nodes in your StorageGRID Webscale system to complete the upgrade, but you can update your grid in stages to control the timing of the upgrade or to ensure the continuous availability of your StorageGRID Webscale system. Before you begin To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. About this task You must complete the upgrade of your StorageGRID Webscale system on the primary Admin Node. The Software Upgrade page guides you through the process of uploading the required files, upgrading all of the grid nodes in your StorageGRID Webscale system (starting with the primary Admin Node), and then upgrading the database. You can customize the upgrade sequence by selecting groups of grid nodes to upgrade. After the selected group of grid nodes has been upgraded, you can repeat the process of selecting grid nodes to upgrade as many times as necessary, until all grid nodes at all sites are upgraded. After all grid nodes are upgraded, you must upgrade the database to complete the process. API Gateway Nodes are deselected by default. You must decide when to upgrade API Gateway Nodes based on your grid's configuration. If your StorageGRID Webscale system has multiple API Gateway Nodes, you must sequence the upgrade so that client applications are always directed to an available API Gateway Node. If your StorageGRID Webscale system has only one API Gateway Node, you must plan a downtime for the upgrade because client applications will not be able to access the system while the API Gateway Node is being upgraded. Attention: If an API Gateway Node is connected to a NAS Bridge instance, you must manually connect the NAS Bridge to an alternate API Gateway Node before you upgrade the primary API Gateway Node. Otherwise, NFS and SMB clients will temporarily lose access to their data. See Defining the StorageGRID Webscale Gateway Node in the NAS Bridge Installation and Setup Guide. Because of ILM policy configuration changes in StorageGRID Webscale, do not update the ILM configuration until you have upgraded all grid nodes. Otherwise, you might experience inconsistent and unexpected ILM behavior. Steps 1. Sign in to the Grid Management Interface using a supported browser. 2. Optionally, disable notifications during the upgrade. a. Select Configuration > Display Options. b. Optionally, select the Notification Suppress All check box to avoid receiving excessive notifications about node outages and upgrade processes. When this check box is selected, all notifications are suppressed, included those unrelated to the upgrade.

18 18 StorageGRID Webscale 11.0 Upgrade Guide c. Click Apply Changes. 3. Select Maintenance > Software Upgrade. 4. In the Provisioning Passphrase text box, enter your provisioning passphrase, and click Verify. 5. Select the Upgrade file. a. Click Browse to the right of Upgrade file. b. Locate and select the StorageGRID Webscale software upgrade file in the File Upload dialog box:netapp_storagegrid_version_software_uniqueid.upgrade c. Click Open. The selected software upgrade file is uploaded and then validated. 6. When the upgrade file has been validated, click Start Upgrade. When you start the upgrade, pre-upgrade validations are performed, which might take awhile to complete. If any pre-upgrade validation errors are reported, resolve them and click Start Upgrade again. 7. If you are not already on the Software Upgrade Status page, select Maintenance > Software Upgrade > Status. 8. Click Start next to the Load Provisioning Software action. 9. Click Start next to the Run Grid Task action. Note: The Run Grid Task action applies only to major version upgrades, such as from version 10.4 to version It does not apply to minor upgrades, such as from version 11.0.x to version 11.0.y.

19 Performing the upgrade Click Start next to the Upgrade Primary Admin Node action. Users will not be able to access the Grid Management Interface on the primary Admin Node while the primary Admin Node is being upgraded. 11. Click OK in the confirmation dialog box. The Grid Management Interface and audit logs are unavailable for several minutes during the upgrade while the services are stopped and the software is upgraded and restarted. 12. Periodically, refresh the browser until the Grid Management Interface becomes available. You can follow the progress of the upgrade in the virtual machine console. 13. After the primary Admin Node is upgraded and the Grid Management Interface is available again, clear your web browser's cache and reload the page. You must clear the web browser's cache to remove outdated resources used by the previous version of the StorageGRID Webscale system. For instructions, see the documentation for your web browser. 14. If you are not already on the Software Upgrade Status page, select Maintenance > Software Upgrade > Status. 15. Select or deselect individual grid nodes or data center sites in the Update Queue section to specify which grid nodes to upgrade. 16. Scroll to the bottom of the Upgrade Stages section, and click Upgrade. As each grid node is upgraded, it goes offline as the services are restarted, and the system status displays an error state. The Upgrade Queue section displays the status of the grid nodes being upgraded. The Upgraded Grid Nodes section lists grid nodes when the upgrade on each grid node is complete. 17. After you have upgraded all grid nodes, return to the Software Upgrade Status page by selecting Maintenance > Software Upgrade > Status. 18. Scroll to the bottom of the Upgrade Stages section, and click Start to upgrade the database. Upgrading the database can take an extended amount of time (from days to weeks). The grid operates normally while the upgrade is in progress. You can monitor progress under Database Upgrade in the Upgrade Actions section. The database upgrade is complete when the status is listed as complete on all Storage Nodes. 19. Because the database upgrade can take a long time to finish, you should verify grid operations before the upgrade completes: a. Check that the services are operating normally and that there are no alarms. b. Review all custom alarms to verify that they are still required and usable. c. Confirm that client connections to the StorageGRID Webscale system are operating as expected. 20. Reenable notifications if you suppressed them for the upgrade. The upgrade processes that can trigger excessive alarms have now completed. a. Select Configuration > Display Options. b. Unselect the Notification Suppress All check box. c. Click Apply Changes.

20 20 StorageGRID Webscale 11.0 Upgrade Guide Related information Administering StorageGRID Webscale NAS Bridge installation and setup Verifying the completion of your upgrade After the database upgrade completes, you must verify that the upgrade to StorageGRID Webscale 11.0 completed successfully and make any required configuration changes to ensure that your grid is operating optimally. About this task The upgrade process is complete when each grid node has been upgraded, the database has been upgraded, and the StorageGRID Webscale system configuration finishes. After the status of the Database Upgrade entry in the Upgrade Actions section is listed as complete, you may need to refresh the page or browse to another Maintenance page to trigger the Status for the upgrade as a whole to be listed as complete. Make sure to refresh the page. If the upgrade to StorageGRID Webscale 11.0 is not marked as complete, the collection of Prometheus metrics will be delayed on some nodes. This procedure asks you to re-verify some items that you checked while the database upgrade was still in progress. This is necessary to ensure that the last stages of upgrade completed successfully. Steps 1. Sign in to the Grid Management Interface using a supported browser. 2. Confirm that the upgrade completed successfully. a. Select Maintenance > Software Upgrade. b. Select the Status tab, and confirm that Current Version shows the expected version and that Status is Complete. 3. Check that the services are operating normally and that there are no alarms. 4. Review all custom alarms to verify that they are still required and usable. 5. Confirm that client connections to the StorageGRID Webscale system are operating as expected. 6. Review your active ILM policy, and confirm the status of ILM rules. After the upgrade completes, warning icons indicate any legacy ILM rules (ILM rules that were created before StorageGRID Webscale 10.3). Legacy rules in active policies will continue to work as expected after you upgrade to version However, these rules are now deprecated and cannot be edited. See Managing legacy ILM rules to learn how to recreate or delete legacy ILM rules.

21 Performing the upgrade 21 Managing legacy ILM rules ILM rules that were created using StorageGRID Webscale 10.2 or earlier were deprecated in StorageGRID Webscale If your StorageGRID Webscale system still includes these legacy ILM rules, you must understand how to manage them after the upgrade to version Before you begin You must be signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. You have the latest version of the Administrator Guide. About this task The following conditions apply to legacy ILM rules (rules created before version 10.3): When a legacy rule is displayed, a warning icon indicates that it is a legacy rule. You can continue to use a legacy rule in the active policy, but you cannot edit the rule. You can add a legacy rule to a new policy as long as that rule already exists in the active policy. You cannot add a legacy rule to a new policy unless that rule is already part of the active policy. This is true even if the legacy rule is part of a historical policy. If you remove a legacy rule from a policy and then activate that policy, you cannot subsequently add the legacy rule to that policy. The Grid Management Interface displays version numbers for legacy rules. Note: Rules created in version 10.3 or later all have version 1.0 internally, and the version is not shown. A policy might contain different versions of a legacy ILM rule (for example, a policy might include both v1.0 and v1.1 of Rule A). After upgrading to StorageGRID Webscale 11.0, follow these steps to recreate or delete any legacy rules. Steps 1. Determine which legacy ILM rules are currently in use, and note all settings for those ILM rules. 2. Use the Create New ILM Rules wizard to create new ILM rules with equivalent settings. See the Configuring ILM rules section of the Administrator Guide for instructions. Note: You might need to replace a single legacy rule with multiple new rules. When ILM rules changed in version 10.3, certain rule features also changed. For example, you can no longer have multiple APIs in one ILM rule. 3. If the legacy rules are being used in the active policy, create a new proposed ILM policy. a. Clone the active policy. b. Select the new ILM rules and unselect the original legacy rules. c. Save the proposed policy.

22 22 StorageGRID Webscale 11.0 Upgrade Guide d. Follow the instructions in the Administrator Guide to simulate the proposed policy using ingested objects that are appropriate for testing the new ILM rules. e. Activate the proposed policy. 4. Recreate or delete any unused legacy rules. You cannot delete active ILM rules or ILM rules currently selected for the proposed ILM policy. Related information Administering StorageGRID Webscale

23 23 Backing up the Recovery Package You must download an updated copy of the Recovery Package file any time you make grid topology changes to the StorageGRID Webscale system or whenever you upgrade the software version. The Recovery Package file allows you to restore the system if a failure occurs. Before you begin You must be signed in to the Grid Management Interface using a supported browser. You must have the provisioning passphrase. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Maintenance > Recovery Package. 2. Enter the provisioning passphrase, and click Start Download. The download starts immediately. 3. When the download completes, open the.zip file and confirm it includes a gpt-backup directory and a second.zip file. Then, extract this inner.zip file and confirm you can open the passwords.txt file. 4. Copy the downloaded Recovery Package file (.zip) to two safe, secure, and separate locations. Important: The Recovery Package file must be secured because it contains encryption keys and passwords that can be used to obtain data from the StorageGRID Webscale system. Related information Administering StorageGRID Webscale

24 24 Troubleshooting upgrade issues If the upgrade does not complete successfully, you might able to resolve the issue yourself. If you cannot resolve an issue, you should gather the required information before contacting technical support. The following sections describe how to recover from situations where the upgrade has partially failed. Contact technical support if you cannot resolve an upgrade issue. Provisioning failures If the provisioning process fails, contact technical support. Grid node crashes or fails to start If a grid node crashes during the upgrade process or fails to start successfully after the upgrade finishes, contact technical support to investigate and to correct any underlying issues. Grid task pauses with error If a grid task pauses with error, contact technical support. Ingest or data retrieval is interrupted If data ingest or retrieval is unexpectedly interrupted when you are not upgrading a grid node, contact technical support. Database upgrade errors If the database upgrade fails with an error, retry the upgrade. If it fails again, contact technical support. Cannot shut down server because a service fails to stop If the upgrade does not complete successfully because a service fails to stop, you can attempt to correct the error by forcing the service to shut down, so that the upgrade process can resume. About this task The GDU Upgrade Software task cannot complete if the server is prevented from shutting down because a service fails to stop. When this occurs, the GDU Upgrade Software task stops at the Stopping services stage and manual intervention is required to stop the service. Allow 30 minutes to elapse for the service to complete processing before concluding that the service is hung, and the following troubleshooting steps are required. Steps 1. From the service laptop, log in to the grid node: a. Enter the following command: ssh admin@grid_node_ip b. Enter the password listed in the Passwords.txt file. c. Enter the following command to switch to root: su - d. Enter the password listed in the Passwords.txt file.

25 Troubleshooting upgrade issues 25 When you are logged in as root, the prompt changes from $ to #. 2. Display the status of services on the server: /usr/local/servermanager/reader.rb 3. Examine the output to determine if any services are stuck in the Stopping... state. 4. For any service that is hung, shut down the service forcibly: /etc/init.d/service_name force-stop The service may appear as unknown (blue) in the StorageGRID Webscale system until it is restarted. User interface issues You might see issues with the Grid Management Interface or Tenant Management Interface after upgrade. Web interface does not respond as expected The Grid Management Interface or Tenant Management Interface might not respond as expected after StorageGRID Webscale software is upgraded. For example, after you use the Grid Management Interface to acknowledge an alarm and click Apply Changes, the change might not be saved. If you experience issues with the web interface, clear your web browser cache. This removes outdated resources used by the previous version of StorageGRID Webscale software, and permits the user interface to operate correctly again. For instructions, see the documentation for your web browser. Docker image availability check error messages When attempting to start the upgrade process, you might receive an error message that states The following issues were identified by the Docker image availability check validation suite. All issues must be resolved before you can complete the upgrade. Contact technical support if you are unsure of the changes required to resolve the identified issues. Message Cause Solution Unable to determine upgrade version. Upgrade version info file {file_path} did not match the expected format. Upgrade version info file {file_path} was not found. Unable to determine upgrade version. Unable to determine currently installed release version on {node_name}. Connection error while attempting to list versions on {node_name} The upgrade package is corrupt. The upgrade package is corrupt. A critical file on the node is corrupt. The node is offline or the connection was interrupted. Re-upload the upgrade package, and try again. If the problem persists, contact technical support. Re-upload the upgrade package, and try again. If the problem persists, contact technical support. Contact technical support. Check to make sure that all nodes are online and reachable from the primary Admin Node, and try again.

26 26 StorageGRID Webscale 11.0 Upgrade Guide Message Cause Solution The host for node {node_name} does not have StorageGRID {upgrade_version} image loaded. Images and services must be installed on the host before the upgrade can proceed. Error while checking node {node_name} Uncaught error while running prechecks. {error_string} The RPMs for the upgrade have not been installed on the host where the node is running, or the images are still in the process of being imported. Note: This error only applies to nodes that are running as containers on Red Hat Enterprise Linux. An unexpected error occurred. An unexpected error occurred. Check to make sure that the correct versions of both the services and images RPM packages have been installed on all hosts where nodes are running. Wait a few minutes, and try again. Wait a few minutes, and try again. Wait a few minutes, and try again.

27 27 Copyright information Copyright 2018 NetApp, Inc. All rights reserved. Printed in the U.S. No part of this document covered by copyright may be reproduced in any form or by any means graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval system without prior written permission of the copyright owner. Software derived from copyrighted NetApp material is subject to the following license and disclaimer: THIS SOFTWARE IS PROVIDED BY NETAPP "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, WHICH ARE HEREBY DISCLAIMED. IN NO EVENT SHALL NETAPP BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp. The product described in this manual may be protected by one or more U.S. patents, foreign patents, or pending applications. RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS (October 1988) and FAR (June 1987).

28 28 Trademark information NETAPP, the NETAPP logo, and the marks listed on the NetApp Trademarks page are trademarks of NetApp, Inc. Other company and product names may be trademarks of their respective owners.

29 29 How to send comments about documentation and receive update notifications You can help us to improve the quality of our documentation by sending us your feedback. You can receive automatic notification when production-level (GA/FCS) documentation is initially released or important changes are made to existing production-level documents. If you have suggestions for improving this document, send us your comments by . To help us direct your comments to the correct division, include in the subject line the product name, version, and operating system. If you want to be notified automatically when production-level documentation is released or important changes are made to existing production-level documents, follow Twitter You can also contact us in the following ways: NetApp, Inc., 495 East Java Drive, Sunnyvale, CA U.S. Telephone: +1 (408) Fax: +1 (408) Support telephone: +1 (888)