StorageGRID Webscale 10.0

Transcription

1 StorageGRID Webscale 10.0 Expansion Guide NetApp, Inc. 495 East Java Drive Sunnyvale, CA U.S. Telephone: +1 (408) Fax: +1 (408) Support telephone: +1 (888) Web: Feedback: Part number: _A0 September 2014

2

3 Contents 1 Overview Introduction Who Should Read this Guide Grid Node Installation Options Expansion Overview Required Materials Checklist Limitations of the Expansion Procedure Data Center Sites Admin Nodes Storage Nodes Archive Nodes and Tivoli Storage Manager Prepare for Expansion Verify Hardware and Networking Configure Archive Node Hardware Check the StorageGRID Webscale Software Version Copy the Provisioned Grid Specification File Prepare the Expansion Grid Specification File Provision the StorageGRID Webscale System Expand your StorageGRID Webscale system Introduction Multiple Data Center Sites Deploy the StorageGRID Webscale Installer Deploy the Expansion Grid Nodes Run the Initial Expansion Grid Task Run the Add Server Grid Task Enable Services Apply Hotfixes and Maintenance Releases Clean Up Unused Cassandra Keys Configure the Expanded System Introduction Storage Node Configuration Verify ILM Storage Pool Configuration Verify that the Storage Node is Active Admin Node Configuration Copy the NMS Database Copy the Audit Logs StorageGRID Webscale _A0

4 Expansion Guide 4 Update Client Audit Share Integration Optional Configuration Archive Node Configuration Update the ILM Policy Recommended Custom Alarms for the Archive Node Update NTP Sources on Grid Nodes Troubleshooting Expansion Errors StorageGRID Webscale Software Installation Fails Grid Deployment Utility Contacting Support A Manually Install the Expansion Grid Nodes Introduction Expansion Overview Generate the Grid Deployment Files Deploy the OVF Template Install SUSE Linux Enterprise Server Install VMware Tools Install StorageGRID Webscale Software B How to Use the Grid Deployment Utility About the Grid Deployment Utility Start the Grid Deployment Utility How to Use the Grid Deployment Utility Entering Commands Quit the Grid Deployment Utility Troubleshooting the Grid Deployment Utility Display Problems Problems with Server Status Log Files Missing Tasks Troubleshooting with screen in Multi Display Mode About load_cds.py Copy ISO Files in a Multi-Site Environment C Provisioning Introduction Export a Provisioned Grid Specification File Provision the StorageGRID Webscale System Change the Provisioning Passphrase Provisioning Command Reference Preserving Copies of the Provisioning Data Troubleshooting Provisioning Provision Command Fails _A0 StorageGRID Webscale 10.0

5 5 Errors in Grid Specification File Initial Installation Expansion and Maintenance Procedures About the Software Activation and Integration Data Package Viewing Grid Configuration Files Support for Chrome About Grid Specification Files Naming Convention View a Copy of the Provisioned Grid Specification File D Connectivity Browser Settings Verify Internet Explorer Settings Enable Pop-ups NMS MI Connection Procedure Security Certificate Sign In Sign Out Command Shell Access Procedures Log In Log Out Accessing a Remote Grid Node Connect Using the Remote Server Password Connect Using the ssh Private Key Password Connect Without Using a Password Using GNU screen Copyright Information Trademark Information How to Send Comments Glossary Index StorageGRID Webscale _A0

6 Expansion Guide _A0 StorageGRID Webscale 10.0

7 Overview 1 A summary of the expansion process Introduction This guide provides all of the steps required to successfully expand a StorageGRID Webscale system without interrupting system operations. Read this guide in full before you begin an expansion, and become familiar with all of the steps and requirements needed to expand a StorageGRID Webscale system. This guide assumes that you are familiar with the StorageGRID Webscale system as described in the Grid Primer and Administrator Guide. This guide assumes that the expansion has been planned in advance and that all required hardware is installed, connected, and configured to specifications. NOTE This guide only covers the software expansion and configuration processes. Hardware specific instructions are not included. Who Should Read this Guide This guide is intended for technical personnel tasked with expanding a deployed StorageGRID Webscale system. An advanced level of computer literacy is required, including knowledge of Linux/Unix command shells, networking, VMware virtualization, and server hardware setup and configuration. This guide assumes that you are familiar with the StorageGRID Webscale system's general design, configuration, and options. For information on the limits placed on the types of expansions you can complete, see Limitations of the Expansion Procedure on page _A0 StorageGRID Webscale 10.0

8 8 Expansion Guide Grid Node Installation Options The standard expansion installation procedure automatically deploys grid nodes using the StorageGRID Webscale Installer. It is recommended that you follow this standard expansion installation procedure unless advised otherwise by NetApp. For information on the standard expansion procedure, see Chapter 3: Expand your StorageGRID Webscale system. If you cannot use the StorageGRID Webscale Installer to complete your expansion for any reason, an alternate manual expansion installation procedure is also supported. The manual expansion procedure requires the use of the command line to deploy grid nodes. For the manual expansion procedure, see Appendix A: Manually Install the Expansion Grid Nodes. Expansion Overview Figure 1 provides a detailed, high-level overview of the tasks you need to complete to expand your system using the StorageGRID Webscale Installer. The tasks are categorized into three sections which correspond to chapters in this guide. You must complete the expansion process in the order listed: 1. Preparation Edit the existing, deployed Grid Specification File and provision the StorageGRID Webscale system. For information on the preparation steps, see Chapter 2: Prepare for Expansion on page Expansion Deploy the virtual machine for each expansion grid node and complete the expansion. For information on the expansion steps, see Chapter 3: Expand your StorageGRID Webscale system on page 19. If you have been instructed by NetApp to use the manual installation option, see Appendix A: Manually Install the Expansion Grid Nodes on page Configuration Perform the additional configuration tasks that you may need to complete for each expansion grid node type. For information on the configuration steps, see Chapter 4: Configure the Expanded System on page 35. StorageGRID Webscale _A0

9 Overview 9 Figure 1: StorageGRID Webscale Expansion Process _A0 StorageGRID Webscale 10.0

10 10 Expansion Guide Required Materials Checklist Gather the materials listed in Table 1 to prepare for expansion. Table 1: Required Materials Checklist Item Notes Grid specification file SUSE Linux Enterprise Server (SLES) ISO Only use the latest version of the provisioned grid specification file copied from the primary Admin Node. For more information, see the Grid Designer User Guide. Use only the supported version of SLES for the StorageGRID Webscale system. For supported versions, see the Interoperability Matrix Tool (IMT) at mysupport.netapp.com/matrix. NOTE Use of any version of SLES other than the correct version will result in an installation failure. StorageGRID Webscale Installer OVA StorageGRID Webscale Software ISO Tivoli Storage Manager (TSM) Client packages ISO Hotfix and/ or Maintenance Release ISO VMware software and documentation USB flash drives (two) Open Virtualization Appliance (OVA) file used to deploy the StorageGRID Webscale Installer. Use only the latest version of the StorageGRID Webscale software ISO. The StorageGRID Webscale software includes the latest service pack release. Used to customize the Linux operating system, and install StorageGRID Webscale software on all virtual machines. To acquire the latest version of StorageGRID Webscale software, see the NetApp Support site at mysupport.netapp.com. Required if the expansion adds an Archive Node that uses TSM middleware to write to archival media. For supported versions, see the Interoperability Matrix Tool (IMT) at mysupport.netapp.com/matrix. Determine whether or not a hotfix and/or maintenance release has been applied to the grid node type that is being added. The expansion grid node must be updated to the same hotfix or maintenance release as the other installed grid nodes of the same type. (See the storage-grid-release version number listed on the <grid_node> SSM Services Main page.) To acquire hotfixes and maintenance releases, contact Support. For supported versions, see the IMT at mysupport.netapp.com/matrix. Used to safely store provisioning data. StorageGRID Webscale _A0

11 Overview 11 Table 1: Required Materials Checklist (cont.) Item Notes Provisioning passphrase Service laptop Documentation The passphrase is created and documented when the StorageGRID Webscale system is first installed. The provisioning passphrase is not in the Passwords.txt file. Laptop must have: Microsoft Windows operating system Network port Supported browser For information about supported browsers, see the IMT at mysupport.netapp.com/matrix. Telnet/SSH client (for example, PuTTY) SCP tool (for example, WinSCP) to transfer files to and from the primary Admin Node. Administrator Guide Grid Designer User Guide Release Notes Limitations of the Expansion Procedure Data Center Sites You cannot add a new data center site to a deployed StorageGRID Webscale system. The required data center sites must be identified and configured during the initial grid planning and implementation phases. Admin Nodes You can add an Admin Node to a data center site by expanding your StorageGRID Webscale system, if the site was deployed without an Admin Node. You cannot add multiple Admin Nodes to a single data center site. The primary Admin Node is always deployed during the initial implementation, and cannot be added using the expansion procedure _A0 StorageGRID Webscale 10.0

12 12 Expansion Guide Storage Nodes You can add new Storage Nodes to your StorageGRID Webscale system to increase the storage capacity of the system. While you are adding Storage Nodes during the expansion, you cannot decommission a Storage Node, or change the IP address of a previously installed Storage Node. Storage expansion requirements depend on the StorageGRID Webscale system s information lifecycle management (ILM) rules and the StorageGRID Webscale system s topology. You must always add enough storage to take copies into account. For example, if the ILM rules make one copy of each object ingested at data center site 1 (DC1) and a second copy at data center site 2 (DC2), you must add an equivalent amount of storage at each site to increase the overall capacity of the StorageGRID Webscale system. Most ILM policies create two copies of data, so most storage expansions must add two Storage Nodes. Archive Nodes and Tivoli Storage Manager If you are adding Archive Nodes that will be integrated with a Tivoli Storage Manager (TSM) server, you are required to upload the IBM Tivoli Storage Manager Backup-Archive Client installation files. Acquire these packages from IBM and make them available to the StorageGRID Webscale Installer. These packages are included in a.tar file available at: ftp://service.software.ibm.com/storage/tivoli-storage-management/ maintenance/client/ For information about supported versions, see the Interoperability Matrix Tool (IMT) at mysupport.netapp.com/matrix. You can integrate an Archive Node with either a new or existing TSM server. For information about integrating a TSM server with your StorageGRID Webscale system, see the Installation Guide. For more information about Archive Nodes, see the Administrator Guide. StorageGRID Webscale _A0

13 Overview 13 Information Lifecycle Management Policy When you add an Archive Node, you must update the information lifecycle management (ILM) policy so that the StorageGRID Webscale system uses the expansion Archive Node as a storage location. Depending upon system requirements and the site design, existing content might also be re-evaluated and directed to storage on an Archive Node. For more information about updating the StorageGRID Webscale system s ILM policy, see Update the ILM Policy on page _A0 StorageGRID Webscale 10.0

14 14 Expansion Guide StorageGRID Webscale _A0

15 Prepare for Expansion 2 Configure the StorageGRID Webscale environment for expansion and update the grid specification file Verify Hardware and Networking Before you begin the expansion of your StorageGRID Webscale system, ensure that you have installed and configured the necessary hardware to support the StorageGRID Webscale system. For information about supported servers, see the Interoperability Matrix Tool at mysupport.netapp.com/matrix. You must also verify network connectivity between servers at the data center site, and confirm that the primary Admin Node can communicate with all expansion servers that are intended to host the StorageGRID Webscale system. Configure Archive Node Hardware You can integrate an Archive Node with a new or an existing Tivoli Storage Manager (TSM) server. For more information about installing and configuring TSM middleware for use with an Archive Node, see Archive Nodes and Tivoli Storage Manager on page 12. Check the StorageGRID Webscale Software Version At the end of the expansion process, all grid services of the same type must be running the same version of the StorageGRID Webscale 10.0 software. This includes applied hotfixes and maintenance releases. Before performing an expansion, note the current installed version number of the StorageGRID Webscale software running on the grid node type that is to be added. Use this number as a reference to determine if hotfixes and/or maintenance releases must be applied when performing the expansion procedures _A0 StorageGRID Webscale 10.0

16 16 Expansion Guide Procedure 1. In the NMS management interface (MI), go to <grid_node> SSM Services Main. 2. Scroll to the bottom of the page and the Packages section. The Version attribute for the storage-grid-release package indicates the installed version number. Copy the Provisioned Grid Specification File Required Materials Passwords.txt file A SCP tool, such as WinSCP (available at download.php), to transfer files to and from the primary Admin Node Procedure 1. At the primary Admin Node server, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Create a directory to hold the provisioned grid specification file. Enter: mkdir -p /var/local/gspec 3. Copy the provisioned grid specification file from the StorageGRID Webscale system to the directory. Enter: copy-grid-spec /var/local/gspec 4. Enter the provisioning passphrase. 5. Use an SCP tool, such as WinSCP, to transfer the GID<Grid_ID>_REV<revision_number>_GSPEC.xml file from the directory created in step 2 to your service laptop. 6. Log out of the primary Admin Node. Enter: exit StorageGRID Webscale _A0

17 Prepare for Expansion 17 Prepare the Expansion Grid Specification File Use Grid Designer to update the provisioned grid specification file and generate the grid specification file used to expand your Storage- GRID Webscale system. For information about Grid Designer, see the Grid Designer User Guide. Provision the StorageGRID Webscale System Provisioning is the process of generating the collection of files needed to expand your StorageGRID Webscale system, including the SAID (Software Activation and Integration Data) package. The provisioning process also generates the grid tasks, which are used to add grid nodes to the StorageGRID Webscale system. For more information about provisioning, see Appendix C: Provisioning on page 67. Prerequisites and Required Materials You have a grid specification file that has been updated for the expansion. See Prepare the Expansion Grid Specification File on page 17. An SCP tool, such as WinSCP (available at download.php), to transfer files to and from the Admin Node Service laptop Two USB flash drives Procedure 1. At the primary Admin Node server, access a command shell and log in as root. When prompted for a password, press <Enter>. 2. Copy the expansion grid specification file to /var/local/provision. If the directory does not exist, you need to create it. Enter: mkdir -p /var/local/provision Use an SCP tool, such as WinSCP, to copy the edited grid specification file from your service laptop to the primary Admin Node. 3. Remove any old grid specification files from /var/local/provision. Ensure that there is only one file named GID<grid_ID>_REV<revision_number>_GSPEC.xml _A0 StorageGRID Webscale 10.0

18 18 Expansion Guide NOTE The /var/local/provision directory must contain only one grid specification file. Otherwise, provisioning will fail. 4. Provision the StorageGRID Webscale system: a. Enter: provision /var/local/provision b. When prompted, enter the provisioning passphrase. When the process is complete, Provisioning complete is displayed. If provisioning ends with an error message, see Troubleshooting Provisioning on page Use an SCP tool to copy the GID<grid_ID>_REV<revision_number>_ SAID.zip file from the /var/local/provision directory of the Admin Node to your service laptop. 6. Unzip the GID<grid_ID>_REV<revision_number>_SAID.zip file on your service laptop, and review the contents of the docs/index.html file to confirm that the StorageGRID Webscale system is configured correctly. For more information, see Viewing Grid Configuration Files on page Back up the provisioning data in the /var/local/provision directory to two USB flash drives. These copies can be used to restore the StorageGRID Webscale system in the case of an emergency or during an expansion or maintenance procedure. Copy the contents of the /var/local/provision directory to the service laptop (using an SCP tool, such as WinSCP) and then on to USB flash drives (at least two) for backup. Store your USB flash drives in separate and secure locations. For more information, see Preserving Copies of the Provisioning Data on page 72. WARNING Store copies of the /var/local/provision directory in at least two separate and secure locations. The /var/ local/provision directory contains a copy of encryption keys and passwords that can be used to obtain data from your StorageGRID Webscale system. The Provisioning directory is also required to recover from a primary Admin Node failure. StorageGRID Webscale _A0

19 Expand your StorageGRID Webscale system 3 How to add grid nodes and complete the expansion Introduction The following sections cover the tasks you need to complete to install the expansion grid nodes defined in the grid specification file using the StorageGRID Webscale Installer and complete your expansion. Before you can complete the installation tasks in this chapter, you must complete all of the prerequisite tasks in Chapter 2: Prepare for Expansion. Follow the standard installation procedure outlined in this chapter using the StorageGRID Webscale Installer, unless you are advised otherwise by NetApp. The manual install procedure is covered in Appendix A: Manually Install the Expansion Grid Nodes. You can expand a StorageGRID Webscale system by adding the following node types: Admin Nodes (add a single Admin Node to existing sites only) Storage Nodes Archive Nodes API Gateway Nodes StorageGRID Webscale _A0

20 20 Expansion Guide Multiple Data Center Sites You can expand a StorageGRID Webscale system that is distributed across multiple data center sites, spanning multiple networks. This includes data center sites that are geographically distributed in different regions or countries. The procedure for expanding the distributed StorageGRID Webscale system depends on the speed of the WAN link connecting the data center sites. The StorageGRID Webscale installation on each virtual machine involves approximately 3GB of network data transfer between the StorageGRID Webscale Installer virtual machine and the expansion grid node virtual machine. If the WAN link is relatively fast, you can continue to use the deployed StorageGRID Webscale installer at the first data center to deploy grid nodes at the other data centers. If the WAN link is relatively slow, you can deploy the StorageGRID Webscale Installer at each data center site and use this local instance to deploy the grid nodes at that site using the local network. Using a local instance of the StorageGRID Webscale Installer can significantly speed up the installation process. You must complete the following tasks to expand a distributed StorageGRID Webscale system: 1. Deploy the StorageGRID Webscale Installer at the data center site where the Primary Admin Node is located. For more information, see Deploy the StorageGRID Webscale Installer on page Follow all of the procedures in this guide to expand the Storage- GRID Webscale system at the data center site where the primary Admin Node node is located. Provision the system, and complete the installation and configuration of all of the expansion grid nodes in the first data center site by deploying the.ovf templates for the site. NOTE Provisioning for the StorageGRID Webscale system is completed once on the primary Admin Node for each revision of the grid specification file, and does not need to be repeated for each additional data center site _A0 StorageGRID Webscale 10.0

21 Expand your StorageGRID Webscale system Complete the appropriate option depending on the speed of the WAN link you are completing the deployment across: If the WAN link is relatively fast, navigate to the StorageGRID Webscale Installer, and use the links for the.ovf files to deploy the expansion grid nodes at the secondary data center sites. If the WAN link is relatively slow, use the StorageGRID Webscale Installer deployed at the secondary data center site to deploy the grid nodes for that site. For more information, see Deploy the StorageGRID Webscale Installer on page 21. If you use this option, you need to copy the SAID package from the primary Admin Node to the service laptop, or another location, where it is available for upload to the data center site you are completing the installation for. NOTE When you generate the.ovf files, all grid nodes defined in the Grid Specification File will be created. You only deploy the expansion grid nodes for the data center site where you are completing the installation. Deploy the StorageGRID Webscale Installer The standard expansion procedure for the StorageGRID Webscale system requires the use of the StorageGRID Webscale Installer, which is deployed as a virtual machine in VMware vsphere. Depending on your organization s policies, you may need to redeploy the Storage- GRID Webscale installer for expansion, or you may be able to power on the existing virtual machine from the initial installation. In either case, you need to ensure that the correct version of the StorageGRID Webscale Installer is deployed. Use your web browser to navigate to the StorageGRID Webscale Installer, and click the About link to view the version number. If the version of the StorageGRID Webscale Installer is older than the current version of the StorageGRID Webscale software you are going to use in the expansion you need to deploy the matching version of the StorageGRID Webscale Installer. The installer must be deployed on the same network as, or a network that is accessible to, the grid nodes being deployed for the Storage- GRID Webscale system at each site. Additionally, a unique IP address StorageGRID Webscale _A0

22 22 Expansion Guide is required for the installer, in addition to the IP addresses assigned to grid nodes in the grid specification file. Once deployed, the StorageGRID Webscale Installer is accessed through a web browser and is used to deploy all of the grid nodes for the StorageGRID Webscale system. The IP address is configured during this deployment procedure. Before starting this procedure, confirm that you have the network configuration information (IP address, network mask, and default gateway) for the StorageGRID Webscale Installer. Prerequisites and Required Materials VMware software is installed and correctly configured StorageGRID Webscale Installer OVA file Network configuration information for the StorageGRID Webscale Installer (IP address, network mask, and default gateway) Procedure 1. Open the VMware vsphere Client and log in. 2. Select File Deploy OVF Template. 3. In the Deploy OVF Template wizard, follow prompts on screen and update the default settings, as required. a. On the Source page, click Browse and select the StorageGRID Webscale Installer OVA file. b. On the Network Mapping page, select DestinationNetworks <grid_network> where <grid_network> is the grid network configured with Grid Designer. The StorageGRID Webscale Installer must be on the same network as the StorageGRID Webscale system s grid nodes _A0 StorageGRID Webscale 10.0

23 Expand your StorageGRID Webscale system 23 Figure 2: Network Mapping c. On the Properties page, enter network configuration information for the StorageGRID Webscale Installer. This network is used to access the installer through a web browser. StorageGRID Webscale _A0

24 24 Expansion Guide Figure 3: IP Addresses for the StorageGRID Webscale Installer d. Click Finish to deploy the StorageGRID Webscale Installer. 4. After the StorageGRID Webscale Installer is successfully deployed, it must be powered on. If it did not power on automatically, rightclick the virtual machine the StorageGRID Webscale Installer is deployed on and select Power Power On. The StorageGRID Webscale Installer takes a minute or two to start up, and then it can be accessed through your web browser using the IP address configured in step 3. Deploy the Expansion Grid Nodes This procedure creates one virtual machine for each expansion grid node defined in your grid specification file. The deployment of each grid node includes the deployment of the virtual machine in VMware vsphere and the installation of SUSE Linux Enterprise Server (SLES), VMware Tools, and StorageGRID Webscale software on the virtual machine. The software installation completes automatically when you power on the virtual machine _A0 StorageGRID Webscale 10.0

25 Expand your StorageGRID Webscale system 25 Prerequisites The StorageGRID Webscale Installer is deployed and configured. The SLES installation ISO must be copied to a VMware datastore that can be accessed by the StorageGRID Webscale Installer. The VMware Tools installation ISO must be available in a VMware datastore that can be accessed by the StorageGRID Webscale Installer. Procedure 1. In a supported web browser, navigate the StorageGRID Webscale Installer using the IP address configured when deploying the StorageGRID Webscale Installer. For information about supported browsers, see the Interoperability Matrix Tool (IMT) at support.netapp.com/matrix. 2. On the Welcome page, select Modify an existing StorageGRID Webscale System. 3. On the Initial Setup page complete the following steps: a. Click Upload for NetApp StorageGRID Webscale Software, locate and select the.iso file on the service laptop, and click Open. b. Click Upload for the Software Activation and Integration Data (SAID) package, locate and select the SAID package for your installation, and click Open. You must specify the location of the compressed SAID package (.zip) file on the service laptop. c. If your grid expansion adds an Archive Node, click Upload next to IBM Tivoli Storage Manager (TSM) client package, locate and select the IBM TSM client package, and click Open. d. Open the VMware vsphere Client and select the StorageGRID Webscale Installer virtual machine. The default name is NetApp-SGI. e. Connect the VMware Tools ISO from the VMware datastore to the virtual machine. Click the Connect/Disconnect the CD/DVD devices of the virtual machine icon, and select CD/DVD Drive <n> Connect to ISO image on a datastore. f. In the Browse Datastores dialog box, navigate to the /vmimages/ tools-isoimages subdirectory, select linux.iso and click Open. g. Connect the SLES ISO from the VMware datastore to the virtual machine. Click the Connect/Disconnect the CD/DVD StorageGRID Webscale _A0

26 26 Expansion Guide devices of the virtual machine icon, and select CD/DVD Drive <n> Connect to ISO image on a datastore. h. In the Browse Datastores dialog box, locate and select the SLES.iso image, and click Open. i. Return to the StorageGRID Webscale Installer in your web browser and click Verify Availability for Novell SuSE Linux Enterprise Server and then for VMware Tools to verify that these resources are available. It may take several minutes to validate the SLES installation ISO. j. When all of the required files are successfully loaded and the availability of resources is confirmed, click Next. 4. Deploy each expansion grid node defined in your grid specification by completing the following steps: a. On the Deploy Grid Nodes page, select and copy the.ovf file location for the expansion grid node you are deploying in the URL column. b. Open VMware vsphere Client, and select File Deploy OVF Template. c. Paste the URL for the expansion grid node.ovf file in the Deploy from a file or URL text box and click Next. d. Complete the remaining steps in the Deploy OVF Template wizard. 5. If you are deploying a Storage Node, add one or more hard disks to the virtual machine for use as RangeDBs. Open the VMWare vsphere Client and edit the virtual machine settings to add one or more additional hard disks. The hard disks are typically configured as Raw Device Mappings (RDM) in physical compatibility mode that map to the storage array s Logical Unit Number (LUN) used for storing objects. For more information about adding hard disks to virtual machines, see the VMware vsphere documentation. 6. Repeat steps 4 and 5 until all expansion grid nodes are deployed. 7. In vsphere Client, power on each of virtual machines, which automatically starts the installation of SLES and the StorageGRID Webscale software _A0 StorageGRID Webscale 10.0

27 Expand your StorageGRID Webscale system 27 NOTE Do not start the virtual machines until LUNs have been assigned to all Storage Nodes. Up to 20 virtual machines can be concurrently started. 8. When all virtual machines are started, return to the StorageGRID Webscale Installer and click Next. 9. Wait until the installation is complete on all of the grid node virtual machines and then click Finish. You are returned to the Welcome page and all files are deleted from the StorageGRID Webscale Installer. Run the Initial Expansion Grid Task Running the initial expansion grid task (Grid Expansion: Initial) adds configuration information to the StorageGRID Webscale system for all expansion grid nodes at all sites. The initial configuration grid task is run once per expansion. If you are deploying the expansion grid nodes using the manual installation procedure, you should complete the installation of StorageGRID Webscale software on all expansion grid nodes before running the initial grid task. Completing the installation on all expansion grid nodes helps to ensure that the grid specification file is correct. It is easier to recover from an error that requires you to reprovision the StorageGRID Webscale system if you reprovision the system before you run the Grid Expansion: Initial grid task. Prerequisites StorageGRID Webscale software installation has completed on all expansion grid nodes. Only the following grid tasks can be run concurrently: Clean Up Unused Cassandra Keys (CCLE) ILM Evaluation (Volume Lost) (ILME) ILM Re-evaluation (User Triggered) (ILME) If any other grid tasks are running, wait for them to complete or release their lock, or abort them as appropriate. The CCLE grid task can be aborted if your expansion includes a Storage Node. The expansion will generate a new CCLE grid task that you will need StorageGRID Webscale _A0

28 28 Expansion Guide to run during the expansion process. For more information about grid tasks and resource locking, see the Administrator Guide. Procedure 1. In the NMS MI, go to CMN Grid Tasks Configuration Main. Figure 4: Grid Tasks 2. In the Pending table, locate the Grid Expansion: Initial grid task, and select Start from the Actions drop-down list. NOTE If the Pending table includes grid tasks from multiple provisioning revisions, always run grid tasks from the earliest revision (lowest revision number) first. 3. Click Apply Changes. The grid task moves from the Pending table to the Active table. You must wait for the NMS MI page to auto-refresh before the change is visible. Do not submit the change again. The grid task continues to execute until it completes or fails. When the grid task completes successfully, it moves to the Historical table with a Status of Successful. If the grid task fails, it moves to the Historical table with a description of the error under Status _A0 StorageGRID Webscale 10.0

29 Expand your StorageGRID Webscale system 29 NOTE Grid tasks that are Paused can be resumed. If you Abort a grid task you cannot restart it while it is in the Historical table. For more information about grid tasks, see the Administrator Guide. If the grid task fails, contact Support. NOTE Wait until the grid task completes successfully before continuing. Run the Add Server Grid Task Before you customize or enable services for a grid node, you must run the Add Server expansion grid task (Grid Expansion: Add Server) for the grid node. This prepares the StorageGRID Webscale system for the new grid node. The Add Server grid task needs to be run once for each grid node added in the expansion. When adding more than one type of grid node in a single expansion, you should complete the configuration of the expansion grid nodes in the following order: 1. Admin Nodes 2. Storage Nodes and Archive Nodes 3. API Gateway Nodes Enabling services for grid nodes in this order prevents queue buildups, particularly in a larger expansion where the length of time to complete the process can be longer. Prerequisites Grid Expansion: Initial grid task has been run If you are adding more than one type of grid node, you have determined the correct order to run the grid tasks in Only the Clean Up Unused Cassandra Keys (CCLE) grid task can be run concurrently with the Add Server grid task. If any other grid tasks are running, wait for them to complete, release their lock, or abort them as appropriate. For more information about grid tasks and resource locking, see the Administrator Guide. StorageGRID Webscale _A0

30 30 Expansion Guide NOTE If the expansion requires you to run more than one grid task, allow each grid task to complete before you start the next grid task. Procedure 1. Go to CMN Grid Tasks Configuration Main. 2. In the Pending table, locate the Grid Expansion: Add Server <hostname> task that you want to start, and select Start from the actions drop-down list. NOTE If the Pending table includes grid tasks from multiple provisioning revisions, always run grid tasks from the earliest revision (lowest revision number) first. 3. Click Apply Changes. The grid task moves from the Pending table to the Active table. When a grid task completes (or fails), it moves to the Historical table with a Status description. NOTE Note that you can resume grid tasks that are Paused. If you Abort a grid task, you cannot restart it while it is in the Historical table. For more information about grid tasks, see the Administrator Guide. If the grid task fails, contact Support. NOTE Unless directed otherwise, before you enable services for an expansion grid node, wait at least two minutes after the grid task completes successfully. This allows for the propagation of authentication certificates throughout the system _A0 StorageGRID Webscale 10.0

31 Expand your StorageGRID Webscale system 31 Enable Services You need to enable services for each grid node added in the expansion. Prerequisites If you are adding more than one type of grid node, you have determined the correct order to enable services. You must use the same order as you used for Run the Add Server Grid Task on page 29. VMware Tools is installed You have run the Grid Expansion: Add Server grid task for grid nodes on which you will enable services. WARNING Do not enable services for a grid node until you complete all preparatory steps. Procedure 1. At the primary Admin Node server, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Type the following command: ssh-add 3. Enter the SSH Access Password listed in Passwords.txt. 4. Type the following command to start the Grid Deployment Utility (GDU): gdu-console 5. Type the provisioning passphrase, press <Tab> to select OK, and press <Enter>. 6. Complete the following steps for each additional grid node you are configuring: a. Ensure that the grid node you are configuring is selected and that the status is Available. b. Press <Tab>, use the down-arrow key to highlight Enable Services, and press the <Spacebar> to select it. c. Press <Tab> to move through the options and select the Start Task action. d. Press <Enter> to run the task. StorageGRID Webscale _A0

32 32 Expansion Guide 7. When the task completes for the last expansion grid node you are enabling services for, press the right-arrow key to move to the Quit action and press <Enter>. As each node joins the StorageGRID Webscale system, you can monitor it through the NMS MI. Alarms are normal until the services establish connectivity within the system. The alarms clear automatically as connections are established. NOTE Archive Nodes show a major alarm when they are first started. This is normal. You must configure the TSM server and the Archive Node to clear these alarms. For more information, see Archive Node Configuration on page 39. Apply Hotfixes and Maintenance Releases After enabling services, confirm that the expansion grid node is running the same version of the StorageGRID Webscale software as other grid nodes of the same type. If it is not, apply any necessary hotfixes or maintenance releases needed to update the version of the software the expansion grid node is running to that of other grid nodes of the same type. Prerequisites Services have been enabled for the expansion grid node. Procedure 1. Determine the current version of the StorageGRID Webscale software. Go to <grid_node_of same type> SSM Services Main and under Packages note the storage-grid-release number. 2. Determine the version of the StorageGRID Webscale software of the expansion grid node. Go to <expansion_grid_node> SSM Services Main and under Packages note the storage-grid-release number. Compare the two versions and if they differ apply hot fixes or maintenance releases as necessary to update the expansion grid node to the same software version as the rest of the grid nodes of the same type. For more information about available hotfixes and maintenance releases, contact Support _A0 StorageGRID Webscale 10.0

33 Expand your StorageGRID Webscale system 33 Clean Up Unused Cassandra Keys When you add one or more Storage Node during an expansion, you must run the Clean Up Unused Cassandra Keys grid task generated by the system. You must run the grid task generated for each data center site. NOTE A maximum of three Clean Up Unused Cassandra Keys grid tasks can be run at the same time. The order you run them in does not matter. Prerequisites All Storage Nodes are listed in the normal (green) state, meaning there are no alarms. Procedure 1. Go to CMN Grid Tasks Configuration Main. 2. In the Pending table, locate the Clean Up Unused Cassandra Keys, Site: <site_name> grid task, and select Start from the Actions dropdown list. NOTE If the Pending table includes Clean Up Unused Cassandra Keys grid tasks from previous provisioning revisions, abort all earlier versions of the Clean Up Unused Cassandra Keys grid task before starting the latest version. 3. Click Apply Changes. The grid task moves from the Pending table to the Active table. The grid task continues to execute until it completes or fails. When this grid task completes (or fails), it moves to the Historical table with a Status description. The stage Running Nodetool Cleanup may take several minutes to complete on large systems with many objects. NOTE You can resume grid tasks that are Paused. If you Abort a grid task, you cannot restart it while it is in the Historical table. For more information about grid tasks, see the Administrator Guide. StorageGRID Webscale _A0

34 34 Expansion Guide _A0 StorageGRID Webscale 10.0

35 4 Configure the Expanded System Complete the required configuration to expand your StorageGRID Webscale system Introduction The following sections cover the post-expansion configuration tasks for each grid node type. You only need to complete the tasks that apply to the grid nodes you are adding in your expansion, and some tasks may be optional depending on the options selected when installing and administering your system, and how you want to configure the grid nodes added during the expansion. Storage Node Configuration Prerequisites and Required Materials You cannot start the configuration steps in this section until the required expansion steps in Chapter 3: Expand your Storage- GRID Webscale system on page 19 have been completed for the Storage Node. Passwords.txt file Verify ILM Storage Pool Configuration You must verify that the expansion Storage Nodes are included in a storage pool used by the active ILM policy. Otherwise, the new storage will not be used by the StorageGRID Webscale system. For more information about ILM policies, see the Administrator Guide. StorageGRID Webscale _A0

36 36 Expansion Guide Verify that the Storage Node is Active After you enable services for a Storage Node, the StorageGRID Webscale system automatically starts using the new Storage Node. Use the NMS MI to verify that the StorageGRID Webscale system is using the new Storage Node. Procedure 1. Go to <Expansion_Storage_Node> LDR Storage Overview Main. 2. Click the immediate report button for the Objects Committed attribute. The StorageGRID Webscale system goes to a graph charting the data objects being stored to the Storage Node. 3. Verify that the value of the Objects Committed attribute is increasing. Admin Node Configuration Prerequisites and Required Materials You cannot start the configuration steps in this section until the required expansion steps in Chapter 3: Expand your Storage- GRID Webscale system on page 19 have been completed for the Admin Node. Passwords.txt file Copy the NMS Database When adding Admin Nodes, you can optionally copy the existing NMS database from the primary Admin Node to the expansion Admin Node. The StorageGRID Webscale software activation process creates an empty database for the NMS service on the expansion Admin Node. The expansion Admin Node starts recording attribute and audit information as if the installation were new. The new NMS service records attribute information for servers and services that are currently part of the system or added later. Historical NMS data is not copied unless _A0 StorageGRID Webscale 10.0

37 Configure the Expanded System 37 you manually copy the existing NMS database from the primary Admin Node to the expansion Admin Node. Procedure 1. Stop the MI service on both the primary Admin Node and the expansion Admin Node. For each Admin Node, complete the following steps: a. Access the command shell and log in using the password listed in the Password.txt file. b. Stop the MI service. Enter: /etc/init.d/mi stop 2. On the expansion Admin Node: a. Enter: /usr/local/mi/bin/mi-clone-db.sh <primary_admin_node_ip> where <primary_admin_node_ip> is the IP address of the primary Admin Node, which you are copying the NMS database from. b. When prompted, enter the root password for the primary Admin Node, as found in the Passwords.txt file. c. When prompted, press <Enter> to confirm that you want to overwrite the MI database on the expansion Admin Node. d. When prompted, enter the root password for the primary Admin Node, as found in the Passwords.txt file. The NMS database and its historical data is copied to the expansion Admin Node. When it is done, the script starts the expansion Admin Node. NOTE Copying the NMS database can take several hours. 3. Restart Server Manager on both the primary Admin Node and the expansion Admin Node. Enter: /etc/init.d/servermanager restart Copy the Audit Logs Each Admin Node AMS service generates an audit log file that records events and actions performed by all grid nodes and services, both before and after an expansion. When you add an expansion Admin Node, the AMS service will record information about all events and actions that occur after it joins the system. To make the his- StorageGRID Webscale _A0

38 38 Expansion Guide torical audit messages from other Admin Nodes available on the expansion Admin Node, you need to copy the audit log files manually from the primary Admin Node to the expansion Admin Node. Procedure 1. At the primary Admin Node server, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Stop the AMS service to prevent it from creating a new file. Enter: /etc/init.d/ams stop 3. Rename the audit.log file, so it does not overwrite the file on the expansion Admin Node you are copying it to. Enter: cd /var/local/audit/export ls -l mv audit.log <new_name>.txt 4. Copy all audit log files to the expansion Admin Node. Enter: scp -p * <IP_address>:/var/local/audit/export 5. When prompted, enter the password for the expansion grid node as listed in the Passwords.txt file. 6. Restore the original audit.log file. Enter: mv <new_name>.txt audit.log 7. Restart the AMS service stopped in step 2. Enter: /etc/init.d/ams start 8. Log out of the command shell. Enter: exit 9. At the expansion Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. 10. Update the user and group settings of the audit log files. Enter: cd /var/local/audit/export chown ams-user:bycast * 11. Log out of the command shell. Enter: exit Update Client Audit Share Integration When you add an expansion Admin Node, you can configure client access to audit shares. This configuration provides access to the audit logs on the Admin Node. You can complete the configuration for a Common Internet File System (CIFS) share, a Network Files System (NFS) share, or both. For information on configuring CIFS, see Add a User or Group to a CIFS Audit Share in the Administrator Guide. For information on configuring NFS, see Add an NFS Audit Client to an Audit Share in the Administrator Guide _A0 StorageGRID Webscale 10.0

39 Configure the Expanded System 39 Optional Configuration Depending on your required StorageGRID Webscale system configuration, you may need to complete some, or all, of the following tasks: If the IP address allocated to the Admin Node on the network changed since you planned the expansion (and generated the SAID package), update the IP address of the Admin Node. For information on updating IP addresses, see Change IP Addresses in the Administrator Guide. If you need to configure Domain Name Service (DNS) separately for each individual grid node, you need to specify the DNS information for the expansion Admin Node. For more information, see Modify the DNS Configuration for a Single Node in the Administrator Guide. If you want to change the preferred sender, you can update your configuration to make the expansion Admin Node the preferred sender. Otherwise, the existing Admin Node configured as the preferred sender continues to send notifications and AutoSupport messages. For information on configuring the preferred sender, see Select a Preferred Sender in the Administrator Guide. Archive Node Configuration Prerequisites and Required Materials You cannot start the configuration steps in this section until the required expansion steps in Chapter 3: Expand your Storage- GRID Webscale system on page 19 have been completed for the Archive Node. Passwords.txt file Procedure Before the Archive Node can communicate with the Tivoli Storage Manager (TSM) middleware running on the TSM server, you must configure a number of settings in the NMS that permit the Archive Node to recognize it. Until these settings are configured, the ARC service remains in a Major alarm state (as it is unable to communicate with the TSM). StorageGRID Webscale _A0

40 40 Expansion Guide 1. Open Internet Explorer, navigate to the NMS MI, and sign in. 2. In the Grid Topology tree, locate the Archive Node you want to configure and select ARC Middleware. 3. Click the Configuration tab. 4. Update the setting on the middleware page. Table 2 provides details on the required values. At a minimum, you need to specify the Server IP, Node Name, User Name, Password, and, depending on the situation, Server Port and Management Class. The Number of Sessions, Maximum Retrieval Sessions, and Maximum Archive Sessions specify values for a tape library with two drives allocated to the Archive Node. 5. Click Apply Changes. The Archive Node can now connect to the middleware, and the alarm on Middleware Connectivity clears within a few minutes. Table 2: ARC Middleware Component Configuration Settings Prompt Type Description Middleware Account Maximum Retrieve Sessions Text The maximum number of concurrent sessions that the ARC can open to the middleware server to retrieve objects. In most cases, set Maximum Retrieve Sessions to the Number of Sessions Maximum Store Sessions. For example, if the Number of Sessions is 5, and Maximum Store Sessions has its default value of 1, then Maximum Retrieve Sessions should be 4. To share one tape drive for both storage and retrieval, set the Maximum Retrieve Sessions equal to the Number of Sessions. When there are multiple tape drives, making this setting optimizes performance when objects on a Storage Node are being restored from copies on the Archive Node, or if the archive is full and operating in a read-only mode _A0 StorageGRID Webscale 10.0

41 Configure the Expanded System 41 Table 2: ARC Middleware Component Configuration Settings (cont.) Prompt Type Description Management Class Maximum Store Sessions Text Text The name of the default TSM management class (default) assigned to objects by the ARC service when it saves them to the TSM server. Middleware management classes outline how the middleware's backup and archive operations function, and can be used to specify business rules that are applied by the middleware server. (Such business rules operate independently of the StorageGRID Webscale system s business rules, and must be consistent with the system s requirement that objects are stored permanently and are always available for retrieval by the Archive Node.) The default management class is used if a management class is not specified for an object when it is saved to the StorageGRID Webscale system, or if the management class that is specified is not defined on the TSM middleware server. If the default management class you specify here does not exist on the TSM server, then an object saved using this management class is not stored to the middleware archive. Instead the StorageGRID Webscale system retains the object in a queue, and increments the value of CMS Content Overview Objects with ILM Evaluation Pending. The name of a TSM management class can include no more than 30 characters, is not case-sensitive, and can include only the following characters: alphabetic characters: A Z numerals: 0 9 the following special characters:. (period), - (hyphen), + (plus sign), & (ampersand), _ (underscore) The maximum number of concurrent sessions that the ARC service can open to the middleware server to store objects. Concurrent store sessions are not supported by the Archive Node. Set Maximum Store Sessions to 1 when the Archive Node is able to store or retrieve objects. Set the value to 0 when the archive managed by the middleware server is full, and the Archive Node can only retrieve objects. Node Name Text Sets the name of the Archive Node, as it appears to the middleware. The name that you enter here must be the same as the node name (arc-user) that you registered on the TSM server. StorageGRID Webscale _A0

42 42 Expansion Guide Table 2: ARC Middleware Component Configuration Settings (cont.) Prompt Type Description Number of Sessions Text The number of tape drives on the middleware server that are dedicated to the Archive Node. Set this value to be the same as the value for MAXNUMMP (maximum number of mount points) set when the TSM node was registered or updated. (In the register command, the default value of MAXNUMMP used is 1, if no value is set.) The value of MAXSESSIONS for the TSM server must be set to be at least as large as Number of Sessions set here for the ARC service. (The default value of MAXSESSIONS on the TSM server is 25). The Archive Node concurrently creates at most one session per mount point plus a small number (<5) of additional sessions. Password Text The value of the password used by the ARC service to log in to the TSM middleware. This is the password for the node (arc-user) that you registered for the client (arc-password), or the password of the administrative user you selected for the node. You are prompted to confirm the password after you enter it. Server IP Text Sets the IP address of the middleware server used by the ARC service. The default value is Server Port Text Sets the value of the port number on the middleware server that the ARC service uses for communications. The default port used by clients (such as the Archive Node) to communicate with the TSM server is User Name Text The user name of the account that the ARC service uses to log in to the middleware. By default, when you register a node, the TSM server creates an administrative user ID with client owner authority. If you chose to use this default administrative user, enter the client node name (arc-user) as the user name. If you chose to define (or use) a different administrative user for the node, enter its name here. Update the ILM Policy You must include the new Archive Node in a storage pool and activate an ILM policy that uses this storage pool before the StorageGRID Webscale system can use the new storage. For more information, see the Administrator Guide _A0 StorageGRID Webscale 10.0

43 Configure the Expanded System 43 Recommended Custom Alarms for the Archive Node You should establish custom alarms for the following attributes that are used to monitor the speed and efficiency of data retrieval from the Archive Node: ARQL Average Queue Length. The average time, in microseconds, that an object is queued for retrieval from the middleware. ARRL Average Request Latency. The average time, in microseconds, needed by the Archive Node to retrieve objects through the middleware. The normal values for these attributes (located at ARC Retrieve Overview Main) depend heavily on how the middleware and storage is configured and used by the customer. For the Archive Node, the values set in the middleware for request timeouts and the number of sessions made available for retrieve requests are particularly influential. After integration is complete, monitor retrieval from the Archive Node to establish values for normal retrieval times and queue lengths. Then create custom alarms for ARQL and ARRL that will notify an administrator of abnormal operating conditions that might require investigation. For the procedure to create custom alarms, see the Administrator Guide. Update NTP Sources on Grid Nodes If any of the grid nodes added in the expansion have been configured to act as a primary Network Time Protocol (NTP) time source for the StorageGRID Webscale system, you must update the list of NTP sources on all other grid nodes. Follow the procedure below to update all pre-expansion grid nodes to include the new primary NTP time sources in their list of NTP sources. Admin Nodes are commonly designated to be primary NTP time sources. Prerequisites All other expansion steps are complete. StorageGRID Webscale _A0

44 44 Expansion Guide Procedure 1. Check which grid nodes are currently configured to have the role of a primary NTP time source. This information is available in the SAID package. Go to the NTP tab of the Doc/index.html file. 2. If any expansion grid nodes are NTP primaries, note their grid IP addresses. 3. On each server that was in the StorageGRID Webscale system prior to the expansion, add the expansion grid nodes as NTP sources. For the detailed procedure, see the Administrator Guide. After you complete any necessary updates to NTP sources, expansion is complete. Troubleshooting Expansion Errors If you run into errors during the grid expansion process that you are unable to resolve, or if a grid task fails, contact Support. StorageGRID Webscale Software Installation Fails If installation fails because of a provisioning error, you can complete the following steps to attempt to resolve the issue before calling Support: Delete the grid tasks created by provisioning (Grid Expansion: Initial and all Grid Expansion: Add Server grid tasks). Correct the provisioning issue and generate a corrected grid specification file for the expansion, if required. Reprovision the system (including removing the failed revision), as described in Errors in Grid Specification File on page 74 Reinstall all expansion servers. This is required because reprovisioning the grid regenerates node IDs and certificates for all expansion servers _A0 StorageGRID Webscale 10.0

45 Configure the Expanded System 45 Grid Deployment Utility If you run into issues working with the Grid Deployment Utility (GDU), such as the GDU displaying incorrectly, the server status not updating correctly, or tasks are missing, see Troubleshooting the Grid Deployment Utility on page 61. Contacting Support When you contact support, be prepared to supply the following installation log files to assist Support in troubleshooting the errors you are encountering: /var/local/log/install.log (located on the grid node being added) /var/local/log/gdu-console.log (located on the primary Admin Node) StorageGRID Webscale _A0

46 46 Expansion Guide _A0 StorageGRID Webscale 10.0

47 Manually Install the Expansion Grid Nodes A Step-by-step instructions for manually installing the grid nodes for your StorageGRID Webscale system expansion Introduction These manual installation instructions for expansion grid nodes are included in the event that the automated installation is not applicable to your environment. Follow the standard installation procedure, as described in Chapter 3: Expand your StorageGRID Webscale system, unless you are advised otherwise by NetApp. In order to successfully complete your grid expansion, you must follow the instructions in Chapter 2: Prepare for Expansion before beginning the expansion tasks in this chapter. Expansion Overview The first procedure you need to complete to manually install the expansion grid nodes is to generate the required grid deployment files for each expansion grid node. For more information, see Generate the Grid Deployment Files on page _A0 StorageGRID Webscale 10.0

48 48 Expansion Guide You need to complete all of the following procedures, in the order listed, for each expansion grid node: 1. Deploy the OVF Template on page Install SUSE Linux Enterprise Server on page Install VMware Tools on page Install StorageGRID Webscale Software on page 53 TIP You may find it more efficient to proceed through the required deployment and installation tasks by completing each step on all nodes before moving on to the next step. For example, if you are installing four additional nodes, you can first deploy all four OVF templates, and then install SLES on all four virtual machines, and so on. Generate the Grid Deployment Files After you have provisioned the StorageGRID Webscale system, you can generate the virtual machine template files (.ovf) and boot image files (.flp) that are required to deploy the expansion grid nodes defined in your grid specification file. Prerequisites and Required Materials The expansion Grid Specification File has been provisioned. For more information, see Provision the StorageGRID Webscale System on page 17. Download and install an SCP tool, such as WinSCP (available at winscp.net/eng/download.php), on the service laptop to transfer files to and from the primary Admin Node. Service laptop Procedure 1. At the primary Admin Node server, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. The following commands create a directory and output the generated files in that directory. Enter: mkdir /var/local/deploy-files generate-grid-deployment-files.rb -o /var/local/deploy-files 3. Enter the provisioning passphrase at the prompt. StorageGRID Webscale _A0

49 Manually Install the Expansion Grid Nodes From the service laptop, use an SCP tool, such as WinSCP, to transfer the contents of the /var/local/deploy-files directory to the service laptop. Connect to the IP address for the Admin Node, and specify the root username and the generated password for the Admin Node in the Passwords.txt file. The generated files include a virtual machine template file (.ovf) and a boot image file (.flp) for each node in your grid specification file. You will use the.ovf and.flp files generated for each expansion grid node to deploy the grid node. For example, if you are expanding your StorageGRID Webscale system by adding a storage node named DC1-S4 in the grid specification, the generated files you will need to use to create the virtual machine and install the operating system are named DC1-S4.ovf and DC1-S4.flp. Deploy the OVF Template You need to deploy the OVF template for each expansion grid node in vsphere Client to create the virtual machine that will host the grid node. Procedure 1. Open VMware vsphere Client and log in. 2. Select File Deploy OVF Template. 3. In the Source page, click Browse. 4. In the Open dialog box, locate and select the.ovf file for the grid node you are installing, and click Open. The virtual machine template files (.ovf) for the expansion grid nodes are stored in the deploy-files subdirectory on the service laptop where you transferred them using an SCP tool, such as WinSCP. 5. In the OVF Template Details page, click Next. 6. In the Name and Location page, verify the name for the grid node virtual machine, select the Inventory Location, and click Next. 7. In the Host/Cluster page, select the host or cluster to run the deployed template on, and click Next. 8. In the Resource Pool page, select the resource pool to deploy the template into, and click Next _A0 StorageGRID Webscale 10.0

50 50 Expansion Guide 9. In the Storage page, select the storage hardware where the virtual machine files will be stored, and click Next. 10. In the Disk Format page, click Next. 11. In the Network Mapping page, verify the network selected for the virtual machine. You can select a different network by clicking the entry in the DestinationNetworks column, and selecting the correct network from the drop-down list. Click Next when the correct network is selected. 12. In the Ready to Complete page, review the deployment settings, and click Finish. CAUTION When deploying Storage Nodes, you must manually assign LUNs to each Storage Node before starting the virtual machine. Do not select the Power on after deployment checkbox. For more information about how to assign LUNs, see the VMware vsphere documentation. 13. If the virtual machine is created without any issues a Deployment Completed Successfully dialog box is displayed. Click Close. 14. Power on the grid node Virtual Machine. Install SUSE Linux Enterprise Server Use this procedure to install SUSE Linux Enterprise Server (SLES) on each virtual machine that will host a StorageGRID Webscale grid node. The installation process completely erases the server drives and installs the SLES operating system (OS), applications, and support files customized for StorageGRID Webscale. Prerequisites Verify that the virtual machine for the grid node is powered on. Download the required SLES ISO image for the StorageGRID Webscale release. For supported versions, see the Interoperability Matrix Tool (IMT) at mysupport.netapp.com/matrix. Locate the boot image file (.flp) for each grid node you need to complete the installation for. The boot image files (.flp) for the additional grid nodes are stored in the deploy-files subdirectory where you transfered the provisioning data to the service laptop using an SCP tool, such as WinSCP. StorageGRID Webscale _A0

51 Manually Install the Expansion Grid Nodes 51 Procedure 1. Open VMware vsphere Client and log in. 2. In the navigation tree, select the virtual machine. 3. Connect the SLES ISO from the VMware datastore to the virtual machine. Click the Connect/Disconnect the CD/DVD devices of the virtual machine icon, and select CD/DVD Drive <n> Connect to ISO image on a datastore. 4. In the Browse Datastores dialog box, locate and select the SLES.iso image, and click OK. 5. Click the Connect/Disconnect the floppy devices of the virtual machine icon and then select Floppy Drive 1 Connect to Floppy Image on local disk. 6. In the Open dialog box, locate and select the boot image file (.flp) that contains the activation file for this server and click Open. 7. Click the Console tab. 8. Click anywhere inside the Console pane to enter the Console pane. Your cursor disappears and you are locked into the Console pane. 9. Press <Ctrl>+<Alt>+<Insert> to restart the virtual machine. The server performs the following steps while rebooting: The BIOS runs a hardware verification. By default, the system boots from the ISO image connected on the CD/DVD drive, and loads the SLES boot screen in the VMware vsphere Client Console pane. Figure 5: SLES Boot Screen _A0 StorageGRID Webscale 10.0

52 52 Expansion Guide 10. When the SLES Boot Screen is displayed, press the down arrow key to select the Installation option. (Do not press <Enter>.) NOTE You must move the cursor to the Installation option within eight seconds. If you do not, SLES will automatically attempt to install from the hard drive and the installation process will fail. If this happens, you must restart the virtual machine and select the Installation option within the required time. 11. Press <Tab>, and at the Boot Options prompt, type the following command and press <Enter>: autoyast=device://fd0/autoinst.xml NOTE You must always specify the autoyast information at the Boot Options prompt. If you do not enter this information, AutoYaST does not complete the required custom installation for the server. If you enter an incorrect value, and are prompted to re-enter the device name and path, verify the floppy device name. When the SLES installation is complete, the server completes its configuration and starts the operating system. Installation is complete when the login prompt appears. Install VMware Tools You need to install VMware Tools on each virtual machine that will host a grid node. The required version is made available to virtual machines through the VMware vsphere Client. Prerequisites SLES is installed on the virtual machine Procedure 1. Open VMware vsphere Client and log in. 2. In the navigation tree, select the virtual machine to complete the installation on. Power on the virtual machine if it is not started. StorageGRID Webscale _A0

53 Manually Install the Expansion Grid Nodes Click the Connect/Disconnect the CD/DVD devices of the virtual machine icon, and select CD/DVD Drive <n> Connect to ISO image on a datastore. 4. In the Browse Datastores dialog box, navigate to the /vmimages/toolsisoimages subdirectory, select linux.iso and click Open. 5. Click in the Console tab. 6. Click anywhere in the Console pane and log in as the root user. Specify the generated password for the grid node stored in the Passwords.txt file. 7. Mount the VMware Tools installer. Enter: mount /cdrom 8. Copy the gzip installation package to a temporary directory on the virtual machine, and unpack it. Enter: mkdir /tmp/vmtools cd /tmp/vmtools tar -zxvf /cdrom/vmwaretools-*.tar.gz 9. Install VMware Tools with the default installation options. Enter: cd /tmp/vmtools/vmware-tools-distrib/./vmware-install.pl --default 10. When the installation is complete, verify that VMware Tools is running. Enter: /etc/init.d/vmware-tools status If the installation was successful you will see the following status message: vmtoolsd is running. 11. Remove the installation files from the virtual machine. Enter: cd /tmp rm -rf vmtools 12. Reboot the system to ensure that the changes take effect. Enter: reboot Install StorageGRID Webscale Software The Grid Deployment Utility (GDU) is a console application used to install StorageGRID Webscale software, enable services, and execute other tasks on individual grid nodes. The GDU is accessed through a telnet application, such as PuTTY. For general information on connecting to and working with the GDU, see Appendix B _A0 StorageGRID Webscale 10.0

54 54 Expansion Guide Prerequisites Ensure that a telnet application, such as PuTTY, is available on the service laptop. For more information about PuTTY, see: Procedure 1. At the primary Admin Node server, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Type the following command: ssh-add 3. Enter the SSH Access Password listed in Passwords.txt. 4. Type the following command to start the Grid Deployment Utility (GDU): gdu-console 5. Type the provisioning passphrase, press <Tab> to select OK, and press <Enter>. 6. Complete the following steps for each additional grid node you are configuring: a. Ensure that the grid node you are configuring is selected and that the status is Available. b. Press <Tab>, use the down-arrow key to highlight Install Software, and press the <Spacebar> to select it. c. Press <Tab> to move through the options and select the Start Task action. d. Press <Enter> to run the task. e. The status of the primary Admin Node changes to Busy. It typically takes a few minutes for the task to complete. f. The status of the primary Admin Node returns to Available when the task completes. Press the right-arrow key to move to the Quit action and press <Enter>. 7. You can use the same SSH session for the procedure to Enable Services on page 31. If you do not plan to continue on and complete that procedure, type the following command to end the SSH session: ssh-add -D StorageGRID Webscale _A0

55 Manually Install the Expansion Grid Nodes 55 WARNING Do not Enable Services after you install the Storage- GRID Webscale software. You need to run the expansion grid tasks before you Enable Services. You must perform the following tasks to complete the expansion: 1. Run the Initial Expansion Grid Task on page Run the Add Server Grid Task on page Enable Services on page Apply Hotfixes and Maintenance Releases on page 32 When you have completed all of the expansion steps, you need to continue with tasks in Chapter 4: Configure the Expanded System _A0 StorageGRID Webscale 10.0

56 56 Expansion Guide StorageGRID Webscale _A0

57 How to Use the Grid Deployment Utility B About the Grid Deployment Utility The Grid Deployment Utility (GDU) is used to facilitate the installation of software on all grid nodes and is installed and available on the primary Admin Node. Start the Grid Deployment Utility 1. At the primary Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. If you are using PuTTY as your Telnet/ssh client, in PuTTY, select Window Translation Remote character set UTF-8. Figure 6: PuTTY Settings for the Grid Deployment Utility _A0 StorageGRID Webscale 10.0

58 58 Expansion Guide 2. Enter: ssh-add You need to run ssh-add, which adds the ssh private key to the ssh agent, each time you start a new shell session. For more information about ssh access points, see the Administrator Guide. 3. If prompted, enter the SSH Access Password listed in the Passwords.txt file. 4. If using PuTTY, start screen. For example, enter: screen -S GDU NOTE Do not use screen if running GDU locally because the GDU console characters will not display properly. The name of the session (for example GDU in the command above) is optional, but recommended since it is useful for managing screen sessions. The screen program allows you to manage multiple shell instances concurrently, connect to the same session from different locations, detach from a session without stopping the program running within the session, and resume a session that was previously detached. To detach from a screen, press <Ctrl>+<A> and then <Ctrl>+<D>. To reattach to a screen, enter: screen -r 5. Start GDU. Enter: gdu-console 6. When prompted, enter the provisioning passphrase. Type the passphrase, press <Tab> to select OK, and then press <Enter>. Figure 7: Entering the Provisioning Passphrase If you encounter display problems, see Display Problems on page 61. StorageGRID Webscale _A0

59 How to Use the Grid Deployment Utility 59 How to Use the Grid Deployment Utility Figure 8: The Grid Deployment Utility Console The GDU console consists of five panels: Servers Displays the servers for the StorageGRID Webscale system. Tasks Displays the procedures that can be performed on the server selected in the Servers panel. Only the tasks applicable to the current situation are displayed. It is possible to run GDU tasks in parallel on different servers. The list of tasks includes: Task Enable Services Install Software Load Configuration Reboot Server Remount Storage Start Services Stop Services Update IP Configuration Update Status Select To Enable services for a grid node. Install StorageGRID Webscale software onto the virtual machine. Load NMS configuration settings. Reboot the server and start the services. Check for preserved storage volumes and remount them. Used for maintenance procedures on Storage Nodes. Start Server Manager and all services. Stop Server Manager and all services. Change IP addresses. Display the current server status _A0 StorageGRID Webscale 10.0

60 60 Expansion Guide These tasks are described in detail in the procedures where they are used. Server Info Displays the state of the server selected in the Servers panel. The status can be one of: Current State Available Busy Error Pingable Reachable Notes The server is available for the tasks listed in the Tasks panel. A GDU task is running on this server. A GDU task has failed. The server is pingable, but cannot be reached because there is a problem with the hostname. The server can be reached but is not available because the ssh host keys do not match. Log Messages Displays the output of the GDU task executed for the server selected in the Servers panel. If you are running multiple GDU tasks in parallel, you can display the output of each task by selecting the appropriate server in the Servers panel. Actions The actions are: Action Start Task ISO List Quit Select to Start the procedure selected in the Tasks panel. List the ISO images available in the /var/local/install directory of the primary Admin Node. Quit GDU. Entering Commands Use the keyboard to enter commands: To Go from panel to panel Go back from panel to panel Do Press <Tab>. Press <Shift> <Tab>. StorageGRID Webscale _A0

61 How to Use the Grid Deployment Utility 61 To Go up and down within a panel Go right and left within a panel Select a task Activate a command Do Press <Up Arrow> and <Down Arrow> Press <Page Up> and <Page Down> Press <Home> and <End> Press <Left Arrow> and <Right Arrow>. Press the space bar. X appears next to the selected task. Press <Enter>. Quit the Grid Deployment Utility If you quit GDU while a task is in progress, GDU pauses until the task completes, and then closes. Some tasks, such as formatting storage volumes on a new Storage Node, can take hours to complete. Avoid quitting GDU while long-running tasks are in progress. Continue working in another terminal window. 1. Quit GDU. Select Quit in the Actions panel and then press <Enter>. When prompted, confirm that you want to quit GDU. 2. Remove the ability to access servers without a server password. Enter: ssh-add -D 3. Close the screen session. Enter: exit Troubleshooting the Grid Deployment Utility Display Problems Under certain circumstances, the GDU console might not display properly. For an example, see Figure 9 below. Figure 9: Display Problems _A0 StorageGRID Webscale 10.0

62 62 Expansion Guide If using PuTTY, change the Window Translation setting to Use font encoding. If running GDU locally, do not use screen. Problems with Server Status When starting GDU, the status update of all servers might hang, or take a long time to complete. After server status is updated, many appear as Unknown or Pingable when it is known that the servers are Available. This typically only occurs occasionally in large systems with many servers. To correct the problem, quit GDU, and restart with the -k option. Enter: gdu-console -k Start GDU with the -k option bypasses its initial status update on startup. The state of all servers remains Unknown in GDU until you manually update them using the Update Status task. Log Files The GDU logs are located on the primary Admin Node in /var/local/log/ gdu-console.log. Missing Tasks If you installed your StorageGRID Webscale system using the manual installation procedure, and a GDU task that you must execute is missing from the Tasks panel, check the GDU log to determine the reason. For instance, it is possible that a required ISO image is missing. To list the ISO images currently available in the /var/local/install directory, use the ISO List GDU action. For a sample output, see Figure 10 below. Figure 10: ISO Images in /var/local/install The label Missing, required means that the ISO image of the CD required for the installation is not in the /var/local/install directory. StorageGRID Webscale _A0

63 How to Use the Grid Deployment Utility 63 The label Not present means that an ISO image that GDU expected to find is not in the /var/local/install directory, but GDU does not know whether this ISO image is actually required. Troubleshooting with screen in Multi Display Mode The screen program is useful when two or more people need to interact with a shell session simultaneously for troubleshooting purposes. Below is an example of two users connecting to GDU at the same time. User 1 creates a named screen session and starts GDU. # screen -S GDU # gdu-console User 2 lists the screen sessions and connects without detaching User 1. # screen -ls There is a screen on: 5361.GDU (Attached) 1 Socket in /var/run/uscreens/s-root. # screen -r -x GDU Now both users are viewing GDU and inputs can come from either user. About load_cds.py The load_cds.py command accepts two different inputs: physical installation CDs or ISO images of the installation CDs stored in a directory on the primary Admin Node. You can run the load_cds.py script as many times as you need. If you insert the same CD twice, no new ISO is created. The existing ISO will not be overwritten. If the load_cds.py script fails because you inserted a CD unrecognized by the script, eject the CD and continue with the correct CD (you do not have to start over from the first CD you loaded) _A0 StorageGRID Webscale 10.0

64 64 Expansion Guide Copy ISO Files in a Multi-Site Environment In a multi-site environment, copy ISO files to the servers in the remote location prior to installing or upgrading the software with GDU. This is an optional, but recommended, step to reduce the number of large files that would otherwise be transferred over a slow WAN link. Data Center Remote Site Server 1 Primary Admin Node Servers 2...n Figure 11: Copying Files to Remote Site Prerequisites and Required Materials ISO image of the StorageGRID Webscale Software Installation CD has been copied to the primary Admin Node using the load_cds.py command ssh access between the primary Admin Node and the servers at the remote location Procedure 1. At the primary Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Copy the ISO image of the StorageGRID Webscale Software Installation CD to a server at the remote location. Enter (on one line): scp /var/local/install/<storagegrid_webscale_iso> <destination>:/var/local/tmp where: <StorageGRID_Webscale_iso> is the name of the StorageGRID Webscale Software.iso file. <destination> is the hostname or IP address of the first server at the remote location. StorageGRID Webscale _A0

65 How to Use the Grid Deployment Utility Log in to the server at the remote site where you copied the ISO files. Enter: ssh <destination> When prompted, enter the password for the remote server listed in the Passwords.txt file. 4. Change to the /var/local/tmp directory. Enter: cd /var/local/tmp 5. Load the ISOs using the load_cds.py script. Enter: load_cds.py <StorageGRID_Webscale_iso> where <StorageGRID_Webscale_iso> is the name of the Storage- GRID Webscale Software.iso file. 6. Empty the temporary directory. Enter: rm -r /var/local/tmp/* 7. Copy ISOs from the first server at the remote location to the remaining servers at the remote location. For each remaining server: a. Copy the required ISO files to the server. Enter: scp /var/local/install/* <next_server>:/var/local/tmp where <next_server> is the hostname or IP address of the next server at the remote site. b. Log in to the next server at the remote site where you copied the ISO files. Enter: ssh <next_server> When prompted, enter the password for the remote server listed in the Passwords.txt file. c. Change to the /var/local/tmp directory. Enter: cd /var/local/tmp d. Load the ISOs using the load_cds.py script. Enter: load_cds.py <StorageGRID_Webscale_iso> where <StorageGRID_Webscale_iso> is the name of the Storage- GRID Webscale Software.iso file. e. Empty the temporary directory. Enter: rm -r /var/local/tmp/* f. End the ssh session. Enter: exit g. Repeat from step a for each server at the remote site. 8. End the ssh session on the remote server. Enter: exit 9. Log out of the primary Admin Node. Enter: exit _A0 StorageGRID Webscale 10.0

66 66 Expansion Guide StorageGRID Webscale _A0

67 Provisioning C Introduction Provisioning is the process of generating the set of configuration files required to install or expand your StorageGRID Webscale system. This collection of files is known as the GPT (Grid Provisioning Tool) repository, which includes the Software Activation and Integration Data (SAID) package. Export a Provisioned Grid Specification File If you want to make changes to the StorageGRID Webscale system, you need to export the current grid specification file from the system. The following procedure steps you through the process of copying the current grid specification file locally on the primary Admin Node, and then transferring it to the service laptop where it can be opened and edited in the StorageGRID Webscale Grid Designer. For information on viewing the grid specification file in the NMS MI, see View a Copy of the Provisioned Grid Specification File on page 79. Prerequisites and Required Materials Passwords.txt file An SCP tool, such as WinSCP (available at winscp.net/eng/download.php), to transfer files to and from the primary Admin Node Service laptop Procedure 1. At the primary Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Create a directory to hold the provisioned grid specification file. Enter: mkdir -p /var/local/provision _A0 StorageGRID Webscale 10.0

68 68 Expansion Guide 3. Copy the provisioned grid specification file to the directory. Enter: copy-grid-spec /var/local/provision You are asked for the password listed in the Passwords.txt file. Enter the password. The provisioned grid specification file is copied to the /var/local/ provision directory created in step Use an SCP tool, such as WinSCP, to copy the GID<Grid_ID>_REV<revision_number>_ GSPEC.xml file from the Admin Node to your service laptop. 5. Log out. Enter: exit Provision the StorageGRID Webscale System Use this procedure to implement the changes made to the provisioned grid specification file. The provisioning script imports the updated grid specification file into the StorageGRID Webscale system and generates any grid tasks required to complete the implementation of the changes. Prerequisites and Required Materials Grid specification file. See Export a Provisioned Grid Specification File on page 67. Passwords.txt file Provisioning passphrase An SCP tool, such as WinSCP, on the service laptop to transfer files to and from the primary Admin Node. Service laptop USB flash drives Procedure 1. Use an SCP tool, such as WinSCP, to transfer the grid specification file you want to provision from the service laptop to the /var/local/ provision folder on the primary Admin Node. 2. At the primary Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. 3. Verify that the grid specification file you want to provision is the only grid specification file located in the /var/local/provision directory. Delete any old grid specification files in the directory. StorageGRID Webscale _A0

69 Provisioning 69 NOTE The /var/local/provision directory must only contain the grid specification file being provisioned. Otherwise, provisioning will fail. 4. Provision the StorageGRID Webscale system. Enter: provision /var/local/provision 5. When prompted, enter the provisioning passphrase. When the process completes successfully, a Provisioning complete is displayed. If provisioning ends with an error message, see Troubleshooting Provisioning on page Back up the provisioning data in the /var/local/provision directory to two USB flash drives. These copies can be used to restore the StorageGRID Webscale system in the case of an emergency or during an expansion or maintenance procedure. Copy the contents of the /var/local/provision directory to the service laptop (using an SCP tool, such as WinSCP) and then onto USB flash drives (at least two) for backup. Store your USB flash drives in separate and secure locations. For more information, see Preserving Copies of the Provisioning Data on page 72. WARNING Store copies of the /var/local/provision directory in at least two separate and secure locations. The /var/ local/provision directory contains a copy of encryption keys and passwords that can be used to obtain data from your StorageGRID Webscale system. The Provisioning directory is also required to recover from a primary Admin Node failure. The contents of the Provisioning directory are used during expansion and maintenance of the StorageGRID Webscale system when a new SAID package must be generated. Change the Provisioning Passphrase Use this procedure to update the provisioning passphrase. The provisioning passphrase is used to encrypt the GPT repository. It is created when the StorageGRID Webscale system is first installed and is required for expansions and maintenance procedures _A0 StorageGRID Webscale 10.0

70 70 Expansion Guide WARNING The provisioning passphrase is required for many installation and maintenance procedures. The provisioning passphrase is not listed in the Passwords.txt file. Make sure that it is documented and kept in a safe and secure location. Prerequisites and Required Materials Passwords.txt file Current provisioning passphrase New provisioning passphrase An SCP tool, such as WinSCP, on the service laptop to transfer files to and from the primary Admin Node. Service laptop Procedure 1. At the primary Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. 2. Change the passphrase and write the updated GPT repository to the specified directory. Enter: change-repository-password <path> where <path> is the path to the directory on the server where you want to store a copy of the updated GPT repository. 3. When prompted, enter the current passphrase. 4. When prompted, enter the new passphrase. It must be at least six characters. 5. When prompted, enter the new passphrase again. The passphrase of the GPT repository is changed to the new value, and an updated copy of the repository that uses this password is saved in the <path> directory. WARNING Store the provisioning passphrase in a secure location. It is required for most installation, expansion, and maintenance procedures. Back up the provisioning files to another directory on the primary Admin Node. Create the backup directory and then back up the provisioning files to the new directory. Enter: StorageGRID Webscale _A0

71 Provisioning 71 mkdir -p /var/local/backup backup-repository /var/local/backup This backup copy can be used to restore the StorageGRID Webscale system the event of a hardware failure, or to preform expansion and maintenance procedures. When prompted, enter the new provisioning passphrase. 6. Log out. Enter: exit 7. Use an SCP tool, such as WinSCP, to transfer the provisioning files from the /var/local/backup folder on the primary Admin Node to the service laptop. 8. Store the contents of the Provisioning directory (found at </var/ local/gpt-data/>) and the Backup Provisioning directories (/var/local/ backup) separately in a safe place. For more information, see Preserving Copies of the Provisioning Data on page 72. WARNING Store copies of the backup directory in two separate and secure locations. The directory contains encryption keys and passwords that can be used to obtain data from the system, and is required to recover from a primary Admin Node failure. Provisioning Command Reference The following commands are run using a Telnet/SSH application, such as PuTTY, to connect to the primary Admin Node to either update or copy grid provisioning data: provision <path> change-repository-password <path> copy-grid-spec <path> backup-repository <path> restore-repository <path> The procedures in this appendix specify /var/local/provisioning as the <path> value. This is the recommended location, but you can specify a different path on the Admin Node and then transfer the generated files to the required location. Be aware of the following when you run provisioning commands: The size of the GPT repository increases every time the provision command is run because a new revision is created and is added to the GPT repository _A0 StorageGRID Webscale 10.0

72 72 Expansion Guide Preserving copies of the GPT repository is critical to the continued operation of the StorageGRID Webscale system, as described in Preserving Copies of the Provisioning Data on page 72. Preserving Copies of the Provisioning Data Preserving the GPT repository is critical to the continued operation of StorageGRID Webscale software. The contents of the GPT repository are required when running expansion or maintenance procedures. The GPT repository is also required to restore the primary Admin Node, should it fail and require replacement. Each time you run one of the commands that update or copy grid pro visioning data, you must back up the provisioning data to two secure locations, preferably in two distinct physical locations. For example, copy the provisioning data to two USB flash drives and store them separately in two geographically distinct secure locations such as a locked cabinet or safe. The USB flash drives contain encryption keys and passwords that can be used to obtain data from the StorageGRID Webscale system. WARNING Always store two copies of the GPT repository in two separate and secure locations. The GPT repository is essential to the continued operation of the Storage- GRID Webscale system, and to recover from a failed primary Admin Node. It is possible to back up provisioning data directly to the StorageGRID Webscale system. If you store a copy to the StorageGRID Webscale system, you must always store a second copy in another location outside of the StorageGRID Webscale system. The SAID package includes a two-part encryption key that permits you to recover data from grid nodes in the event of a catastrophic system failure. If the only copies of these keys are in the StorageGRID Webscale system, it is not possible to recover data after such a failure. StorageGRID Webscale _A0

73 Provisioning 73 Troubleshooting Provisioning In case of provisioning errors, follow the guidelines below. provision-fail.log is the only file Fix the grid specification file (keep same rev number). Run the provision command again. There is no need to run the command remove-provision. Provision command fails Look for log file on provisioning media Provision-crash- <grid_info>.log created Contact Support and supply the log files found on provisioning media. Provision command completes normally, but there are configuration errors in the SAID package Get revision number from grid spec file Rev number of grid spec file is 1 (new installation All other revisions Fix grid specification file (keep rev number at 1). Start installation over, i.e reinstall Linux, load provisioning software, run provisioning command. Run remove-revision, fix grid spec file (keep same rev number), run provision command again. Figure 12: Troubleshooting Provisioning Errors Provision Command Fails If provisioning fails because the grid specification file is incorrect, the file provision-fail.log is created in the /var/local/provision directory. This file contains the error message that the provisioning software displayed before terminating. If the provisioning program terminates abnormally (crash), two identical log files are saved to the /var/local/provision directory on the primary Admin Node. provision-fail.log provision-crash-<grid_info>.log where <grid_info> includes the grid ID, the grid revision being created and a timestamp. If the provision-fail.log file is the only file created, fix the grid specification file by updating it with Grid Designer and then run provisioning again. If the provision-crash-<grid_info>.log file is created, contact Support. If provisioning ends with an error, no information is saved and the remove-revision command does not need to be run _A0 StorageGRID Webscale 10.0

74 74 Expansion Guide Errors in Grid Specification File If the provision command completes normally, but you discover an error in the provisioning data after examining the configuration pages in the SAID package, fix the grid specification file by updating it with Grid Designer and then reprovision the StorageGRID Webscale system. Initial Installation NOTE Follow this procedure if the revision number of the grid specification file is 1. For example, GID234567_REV1_SAID.zip. If during the initial installation you discover errors in the SAID package, you must fix the grid specification file and reinstall the primary Admin Node from the beginning, that is, you must reinstall SLES, load provisioning software, and provision the StorageGRID Webscale system. Expansion and Maintenance Procedures NOTE Follow this procedure if the revision number of the grid specification file is greater than 1. For example, GID234567_REV3_SAID.zip. This procedure cannot be used for a new installation. Prerequisites and Required Materials Passwords.txt file Current provisioning passphrase Service laptop No scripts or grid tasks generated by provisioning are running. Procedure 1. Confirm that no scripts or grid tasks generated by provisioning have been started. 2. At the primary Admin Node, access a command shell and log in as root using the password listed in the Passwords.txt file. 3. Remove the provisioning data from the StorageGRID Webscale system. Enter: remove-revision StorageGRID Webscale _A0

75 Provisioning 75 The remove-revision command does not remove grid tasks generated by provisioning, or roll back grid tasks that have already been run. WARNING Do not use the remove-revision command if you have started any scripts or grid tasks that were generated by provisioning. Contact Support for assistance. 4. When prompted, enter the provisioning passphrase. 5. Cancel any pending grid tasks created by the provisioning. 6. Edit the /etc/ssh/ssh_known_hosts file to reverse the changes specified in the removed revision, by completing one of the following steps: Remove expansion servers: Remove each expansion server entry by deleting the line containing the corresponding host key. Restore IP Addresses: Revert each modified IP address by editing the line containing the corresponding host key and specify the original IP address. 7. Edit the /etc/hosts file to reverse the changes specified in the removed revision, by completing one of the following steps: Remove servers: Remove each expansion server entry by and deleting line containing the host IP address and the Hostname of the expansion server. Restore IP addresses: Revert each modified IP address by editing the line containing the corresponding IP address and specify the original IP address. 8. Fix the grid specification file with Grid Designer and save it to the root directory of the /var/local/provision directory. Do not change the REV<revision_number> value. NOTE The /var/local/provision directory must contain only one grid specification file at the root level. Otherwise, provisioning will fail. 9. Run provisioning again and generate a new SAID package. For more information, see Provision the StorageGRID Webscale System on page 68. The old SAID package is overwritten and a new one is generated that uses the same naming convention. 10. Review the contents of the SAID package to confirm that the provisioning information is correct _A0 StorageGRID Webscale 10.0

76 76 Expansion Guide About the Software Activation and Integration Data Package The Software Activation and Integration Data (SAID) package contains site-specific files for the StorageGRID Webscale system. It is generated during the provisioning process as a zip file and is named using the following naming convention: GID<grid_ID>_REV<revision_number>_SAID.zip The SAID package contains the following items: Table 3: SAID Package Contents Item Doc Escrow_Keys Grid_Activation Grid_Tasks Configuration.txt Grid specification file Passwords.txt Description Directory containing HTML files used to confirm provisioning. Directory containing Encryption keys used by the Data Recovery Tool. Directory containing activation files, one for each server. Activation files are named <servername>-autoinst.xml. Activation files are keyed to work with the hardware used for the StorageGRID Webscale system and the version of StorageGRID Webscale software. Directory containing files created by some types of changes to the grid specification file, such as expansion. Grid tasks are used to trigger various actions that implement changes to the StorageGRID Webscale system. File listing system-wide configuration and integration data generated during the provisioning process. XML file that encapsulates the topological design of the StorageGRID Webscale system. File name is GID<grid_ID>_REV<revision_number>_GSPEC.xml File listing passwords used to access the Storage- GRID Webscale system. StorageGRID Webscale _A0

77 Provisioning 77 Viewing Grid Configuration Files The Doc directory of the SAID package contains HTML files documenting the configuration of the StorageGRID Webscale system. Open the Doc/Index.html file in supported web browser and review its contents to confirm that the StorageGRID Webscale system is configured correctly. Figure 13: Index.html File For a list of supported browsers, see Interoperability Matrix Tool (IMT) mysupport.netapp.com/matrix. Support for Chrome By default, the Chrome web browser is configured with security features, which render it incapable of opening the Doc/Index.html file. In order to open the Doc/Index.html file in Chrome, configuration changes must be made _A0 StorageGRID Webscale 10.0

78 78 Expansion Guide Configuring Chrome to Open the Doc/Index.html File The path to the Run dialog box may be different depending on the version of Windows used. 1. In Windows, go to Start All Programs Accessories Run. The Run dialog box opens. 2. In the Run dialog box, click Browse. The Browse dialog box opens. 3. In the Browse dialog box, navigate to the Google directory, select the Chrome application and then click Open. The Run dialog box updates to include the path to Chrome. Figure 14: Run Dialog Box with Path to Chrome 4. In the Run dialog box, add the following to the end of the path to Chrome (after the quotation mark): --allow-file-access-from-files or --disable-web-security 5. Click OK. Chrome starts and is now able to open the Doc/Index.html file. Changes are not saved when you close Chrome. About Grid Specification Files The grid specification file is an XML file that encapsulates the configuration information needed to install, expand, and maintain a StorageGRID Webscale system. The file includes topology, servers, options, and networking details for the StorageGRID Webscale system. Grid specification files are generated by Grid Designer and then used to provision the StorageGRID Webscale system. Provisioning the StorageGRID Webscale system changes the grid specification file such that any subsequent changes made to the StorageGRID Webscale system (maintenance procedure or expansion) require the use of the provi- StorageGRID Webscale _A0

79 Provisioning 79 sioned grid specification file. See Export a Provisioned Grid Specification File on page 67. For more information about Grid Designer, see the Grid Designer User Guide. Naming Convention Grid specification files use the naming convention GID<grid_ID>_REV<revision_number>_GSPEC.xml, where <grid_id> refers to the StorageGRID Webscale system s unique identifier and <revision_number> refers to the revision number of the grid specification file, for example, GID1234_REV1_GSPEC.xml. The revision number is increased by 1 each time the grid specification file is modified; for example to add grid nodes or change IP addresses. For the initial installation of the StorageGRID Webscale system, the revision number must be 1 (REV1). Any other revision number will cause provisioning to fail. View a Copy of the Provisioned Grid Specification File You can view the contents of the provisioned grid specification file through the NMS MI. You cannot, however, open and update this version in Grid Designer. To obtain a copy of the provisioned grid specification file that you can update with Grid Designer, you must export the provisioned grid specification file using the procedure Export a Provisioned Grid Specification File on page Go to Grid Management Grid Configuration Configuration. 2. In the Main page, click Export at the bottom of the page, to the right of the Grid Specification File label. A new browser window opens, showing the grid specification file in raw XML _A0 StorageGRID Webscale 10.0

80 80 Expansion Guide View Grid Specification File Figure 15: View the Provisioned Grid Specification File StorageGRID Webscale _A0

81 Connectivity D Connection requirements for the NMS management interface and the server command shell Browser Settings Verify Internet Explorer Settings If you are using Internet Explorer, verify that the settings for temporary internet files, security and privacy are correct. 1. Go to Tools Internet Options General 2. In the Browsing history box, click Settings. 3. For Check for newer versions of stored pages, select Automatically. Figure 16: Temporary Files Setting _A0 StorageGRID Webscale 10.0

82 82 Expansion Guide 4. Go to Tools Internet Options Security Custom Level and ensure that the Active Scripting setting is Enable. Figure 17: Active Scripting Setting 5. Go to Tools Internet Options Privacy and ensure that the privacy setting is Medium or lower (cookies must be enabled). Enable Pop-ups To make any changes to passwords, you must ensure that your browser allows pop-up windows. For more information about allowing pop-up windows, see your browser s documentation. NMS MI Connection Procedure Access to the NMS MI is through a supported web browser. For a list of supported browsers, see Interoperability Matrix Tool (IMT) at mysupport.netapp.com/matrix. Connecting to the NMS MI at the site requires access to the customer network. In a multi-site StorageGRID Webscale deployment each site may include an Admin Node. You can connect to any Admin Node and each Admin Node displays a similar view of the StorageGRID Webscale system; however, alarm acknowledgments made at one Admin Node are not copied to the other Admin Node. It is therefore possible that the Grid Topology tree will not look the same for each Admin Node. 1. Work with the system administrator to establish the physical network connection to the service laptop. 2. From the Configuration.txt file, note the IP address of the Admin Node. This is needed to access the NMS MI. StorageGRID Webscale _A0

83 Connectivity From the Passwords.txt file, note the NMS MI password for the Vendor account or the Admin account. 4. Launch the web browser. 5. Open the address where <IP_address> is the address of the Admin Node on the customer network specified in the Configuration.txt file. Security Certificate Depending on your version of Windows and web browser, you might be warned of a problem with the security certificate when you access the NMS MI URL. Figure 18: Example of a Security Alert Window If this appears, you can either: Proceed with this session. The alert will appear again the next time you access this URL. Install the certificate. Follow the instructions of your browser. The NMS MI uses a self-signed certificate. For information about importing this certificate into a browser see the browser's documentation. Note that the self-signed certificate used by the NMS MI is based on the StorageGRID Webscale system s IP address. The expected URL to the interface is this IP address and not a domain name. In cases where the domain name is used, browsers might not be able to match the self-signed certificate to the identity of the NMS server. For more information see the browser's documentation _A0 StorageGRID Webscale 10.0

84 84 Expansion Guide Sign In 1. Enter the username Vendor for full access to the NMS MI. If you are not making system-wide configuration changes, you can also use the Admin account. 2. Enter the password for the NMS MI as specified in the Passwords.txt file. Figure 19: NMS MI Sign In Window Sign Out When you finish your NMS MI session, sign out to keep the system secure. 1. Click Sign Out located at the top-right corner of the screen. The logging out message appears. 2. You can safely close the browser or use other applications. NOTE Failure to sign out can give unauthorized users access to your NMS session. Simply closing your browser is not sufficient to sign out of the session. Command Shell Access Procedures Log In At the grid node, access a command shell and log in as root using the password listed in the Passwords.txt file. StorageGRID Webscale _A0