StorageGRID Webscale 10.3 Troubleshooting Guide

Transcription

1 StorageGRID Webscale 10.3 Troubleshooting Guide September _A0

2

3 Table of Contents 3 Contents Overview of problem determination... 4 Defining the problem... 4 Assessing the risk and impact on the system... 5 Reviewing the alerts history... 5 Data collection... 6 Analyzing data Escalation information checklist Troubleshooting Admin Nodes Connection to NMS unavailable New installations Error occurs unexpectedly while the system is running Troubleshooting Storage Nodes Object store (storage volume) failures Troubleshooting SAVP Total Usable Space (Percent) alarm Troubleshooting SSTS Alarms Lost and missing object data Determining the number of lost objects Tracking lost erasure coded object data Determining if lost object data can be recovered Checking unavailable resources for lost objects Resetting lost and missing object counts Troubleshooting SVST (Services: Status - Cassandra) alarm Rebuilding the distributed key value store Troubleshooting replication queues Other StorageGRID Webscale troubleshooting tips Time synchronization Alarms reference Log files StorageGRID Webscale software logs Deployment logs Third-party logs About the bycast.log Log file rotation Message description Severity level Error codes Copyright information Trademark information How to send comments about documentation and receive update notifications Index... 84

4 4 Overview of problem determination This overview provides the recommended problem determination methodology for the StorageGRID Webscale system. Follow the process outlined in this figure to determine the problem with the StorageGRID Webscale system. Defining the problem The first step to solving a problem is to define the problem clearly. This table provides an example of the type of information that you would collect to define the problem:

5 Overview of problem determination 5 Question What is the StorageGRID Webscale system doing or not doing? What are its symptoms? When did the problem start? Sample response Client applications are reporting that they cannot store objects to the StorageGRID Webscale system. Many alarms have been triggered. Object creation was denied at about 14:50 this afternoon. How did you first notice the problem? Notified by client application. Also received e mail notifications. Is the problem reproducible? What is the frequency of the problem? Problem is ongoing. This is the first time this has happened. Assessing the risk and impact on the system After you have defined the problem, assess its risk and impact on the StorageGRID Webscale system. For example, the presence of critical alarms does not necessarily mean that the system is not delivering core services. This table summarizes the impact the example problem is having on system operations: Question Can the StorageGRID Webscale system ingest content? Can client applications retrieve content? Is data at risk? Is the ability to conduct business severely affected? Sample response No. Some objects can be retrieved and others cannot. No. Yes, because client applications cannot store objects to the StorageGRID Webscale system and data cannot be retrieved consistently. Reviewing the alerts history To troubleshoot the example problem, you need to establish time lines and compare the current situation to baseline system operations. In order to do this, look at the alerts history and plot trends. Steps 1. Sign in to the Grid Management Interface using a supported browser. 2. Select Alarms. 3. From the Alarms menu, select History. 4. Select an attribute, and then choose one of the following: Run a Quick Query by selecting one of the quick filtering options: Last 5 Minutes Last Hour Last Day Last Week

6 6 StorageGRID Webscale 10.3 Troubleshooting Guide Last Month Run a Custom Query by entering a specific Start Date and End Date. 5. Review the History tab, which shows the alarms that have been triggered: Data collection After the problem has been defined and its risk and impact assessed, collect data for analysis. You can use the following steps to collect data, depending on the nature of the problem: General data collection steps: Perform ingest and retrieval tests. List recent changes. Check hardware monitoring. Grid Management Interface-related data collection steps: Check connectivity status. Check the status of services. Review alarms. Plot trends. Establish baselines. Monitor SSM events. Create custom syslog events. Trigger an AutoSupport message. Optional Grid Management Interface-related data collection steps: Run foreground verification: Not a source of data collection. Confirm data object locations. Non-Grid Management Interface data collection steps: Gather audit and application log files, and system data. Checking connectivity status Confirm that servers are online and connected to each other. In the Grid Topology tree, look for grid nodes whose state is unknown (blue) or are administratively down, that is, that have been purposely stopped (gray). If all grid nodes are green, the primary Admin Node is connected to all other grid nodes. If grid nodes are blue, there is likely a connectivity issue.

7 Overview of problem determination 7 Reviewing alarms An alarm is triggered when the value of an attribute reaches a set alarm threshold value. Check the Grid Topology tree for any icons that are not green, indicating that an alarm has been triggered. About this task For information about alarms, see the Grid Primer and the Administrator Guide. For information about how to check and review alarms, see the following table. To Get a list of all current alarms for the StorageGRID Webscale system Get a list of all alarms triggered over a period of time Do this Select Alarms. The alarms are sorted by severity. Note: The alarms indicator immediately tells you the most serious status (state or alarm) of the system. 1. Select Alarms. 2. From the Alarms menu, select History. 3. Do one of the following: Click one of the time periods. Enter a custom range and click Custom Query. Find out how often alarms have been triggered for a particular attribute 1. Select Grid. 2. Select grid node > service or component > Alarms > History. 3. Select the attribute from the list. 4. Do one of the following: Click one of the time periods. Enter a custom range and click Custom Query. The alarms are listed in reverse chronological order. 5. To return to the alarms history request form, click History. Determine whether an alarm has been disabled globally If a default alarm is disabled globally, an asterisk appears beside the alarm on the page. Related references Alarms reference on page 47 Related information StorageGRID Webscale 10.3 Grid Primer StorageGRID Webscale 10.3 Troubleshooting Guide

8 8 StorageGRID Webscale 10.3 Troubleshooting Guide Monitoring SSM events The SSM > Events component relays logged events from hardware drivers. Interpretation of these numbers depends on the hardware and drivers used for the server. Treat this data as a general indicator of server problems. Review SSM events The Last Event attribute (see Grid > site > grid node > SSM > Events) captures the last event detected by the grid node. You can perform a custom query to generate a list of the event messages generated by the server over time. These messages can contain useful troubleshooting information and can be used to help determine the source of a problem. View last SSM event To view the last SSM event, select Grid > site > grid node > SSM > Events > Overview > Main. Reviewing multiple SSM events You can view multiple logged events, which can help isolate issues. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Grid. 2. Select site > grid node > SSM > Events > Reports > Text. 3. Change Attribute to Last Event. 4. Optionally, select a time period for Quick Query. 5. Click Update.

9 Overview of problem determination 9 Related information StorageGRID Webscale 10.3 Administrator Guide Creating custom syslog events Custom events allow you to track all kernel, daemon, error and critical level user events logged to the syslog. A custom event can be useful for monitoring the occurrence of system log messages (and thus network security events and hardware faults) not captured by default by the SSM > Events component. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. About this task Consider creating custom events to monitor recurring problems. The following considerations apply to custom events. After a custom event is created, every occurrence of it is monitored. The result is an increase to the Count value of Custom Events on the SSM > Events > Overview page of the server being monitored. To create a custom event based on keywords in the /var/log/messages or /var/log/syslog files, the logs in those files must be: Generated by the kernel Generated by daemon or user program at the error or critical level Note: Not all entries in the /var/log/messages or /var/log/syslog files will be matched unless they satisfy the requirements stated above. Steps 1. Select Configuration > Events. 2. Click Edit (or Insert if this is not the first event). 3. Enter a custom event string, for example, shutdown 4. Click Apply Changes.

10 10 StorageGRID Webscale 10.3 Troubleshooting Guide 5. Select Grid. 6. Select grid node > SSM > Events > Overview > Main. 7. Under System Events, monitor the Count of Custom Events. If the count increases, a custom event you are monitoring is being triggered on that grid node. Related information StorageGRID Webscale 10.3 Administrator Guide Resetting the count of custom events to zero You can reset the count of custom events to zero. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Grid. 2. Select grid node > SSM > Events > Configuration > Main. 3. Select the Reset check box for Custom Events. 4. Click Apply Changes. Related information StorageGRID Webscale 10.3 Administrator Guide

11 Overview of problem determination 11 Triggering an AutoSupport message You can manually trigger the sending of an AutoSupport message to technical support. This message includes system and site level attribute information, all alarms raised in the last seven days, the current status of all grid tasks (including historical data), events information as listed on the Grid > site > SSM > Events > Overview page, Admin Node database usage, the number of lost or missing objects (zero or more), and the system s ILM policy. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. About this task This information can assist technical support in troubleshooting problems with the StorageGRID Webscale system. For more information about AutoSupport, see the Administrator Guide. Steps 1. Select Configuration > AutoSupport. 2. Select User-Triggered. 3. Click Send. Note: To send an AutoSupport message, the StorageGRID Webscale system s server must be correctly configured. For more information, see the Administrator Guide. Related information StorageGRID Webscale 10.3 Administrator Guide Plotting trends Reports (both chart and text) are an invaluable tool when troubleshooting. The fastest way to create a chart is to click the Chart button on the Overview tab of a component or service. This is known as an immediate report. You can also create charts from the Reports tab. For detailed information on how to create charts, see the Grid Primer. Related information StorageGRID Webscale 10.3 Grid Primer

12 12 StorageGRID Webscale 10.3 Troubleshooting Guide Establishing baselines Baseline information is operational data during normal system operations that provides clues that can help you solve problems. You can gather useful baseline information using the StorageGRID Webscale system: Property Value How to obtain Average size of ingested objects Average size of retrieved objects Average storage consumption Rate of S3/ Swift/CDMI Operations Failed S3/Swift/ CDMI Operations MB MB / consumed/day GB / consumed/day % Opera tions/s Divide the average number of bytes stored daily by the average number of objects stored daily. Divide the average number of bytes retrieved daily by the average number of objects retrieved daily. Go to the Grid Management Interface Dashboard, and click in the Available Storage chart panel. Find a period where the operation is fairly stable and estimate the daily storage consumption rate in bytes and in percentage. You can collect this information for the entire system or a subset of it. Select Grid Topology > Overview tab > Main. Under API Operations, view the S3 Operations - Rate field. Select Grid Topology > Overview tab > Main. Under API Operations, view the S3 Operations - Failed field. Average ILM Evaluation Time us Select Node > CMS > Content > Overview tab > Main. Under Information Lifecycle Management, view the Average ILM Evaluation Time field. ILM Evaluation Rate Objec ts/s Select Node > CMS > Content > Overview tab > Main. Under Information Lifecycle Management, view the ILM Evaluation Rate field. Average Query Latency ms Select Node > DDS > Data Store > Overview tab > Main. Under Queries, view the Average Query Latency field. Failed Queries Select Node > DDS > Data Store > Overview tab > Main. Under Queries, view the Total Queries - Failed (Timed-Out) field. Failed Queries for Consistency Level Unmet Select Node > DDS > Data Store > Overview tab > Main. Under Queries, view the Total Queries - Failed (Consistency Level Unmet) field. The following sample screenshot shows the Dashboard:

13 Overview of problem determination 13 The following sample screenshot shows how, from under each service, you can select each attribute to view baseline values available on the Overview tab for that attribute: List of recent changes made to the StorageGRID Webscale system It is important that you make a list of any recent changes made to the StorageGRID Webscale system or its environment. List of recent changes Was the StorageGRID Webscale system recently installed, expanded, or recovered? Has any hardware been repaired or changed recently? Have any object storage applications or users been added or removed? Have there been any changes to the network infrastructure? For example, VLANS, routers, or DNS.

14 14 StorageGRID Webscale 10.3 Troubleshooting Guide List of recent changes Have any long term grid tasks been recently triggered? Is data migration taking place? Have any changes been made to user authentication, such as adding a tenant or changing LDAP configuration? Have any changes been made to the ILM policy? Have any changes been made to storage compression and/or encryption? Have any changes been made to NTP sources? Have any changes been made to the Grid, Admin, and/or Client network interfaces? Have any configuration changes been made to the Archive Node? Have any other changes been made to the StorageGRID Webscale system or its environment? Checking the status of services Checking the status of all services to make sure they are running helps you troubleshoot issues. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Grid. 2. Select site > grid node > SSM > Services > Overview > Main, and check the status of all services on that grid node.

15 Overview of problem determination 15 Related information StorageGRID Webscale 10.3 Administrator Guide Gather application log files and system data To help troubleshoot a problem, you can retrieve log files and system data (including configuration data) for your StorageGRID Webscale system. This information can be retrieved manually or with Lumberjack, which is installed on every grid node as part of a standard StorageGRID Webscale deployment. About this task Lumberjack is a collection tool that gathers and organizes log files and system and configuration data on a per grid node basis. Retrievals must be set with a specific start time and are archived in a.tar file to a directory of your choice or by default to /var/local/tmp. System data is saved to the system_command file and configuration data in the form of various bundles files. Because application log files can be quite large, the destination directory must have at least 1 GB of free space. Steps 1. Determine the log files you need to gather and the time range within which to gather those files. 2. At the grid node hosting the log files, access a command shell and log in as root using the password listed in the password.txt file. 3. Run Lumberjack by entering: lumberjack.rb -s start_time options start_time is the start time within which to gather logs. Format is: YEAR-MM-DDTHH:MM:SS For example: lumberjack.rb -s T12:01:01 Options include the following: Option Description s start_time Start time within which to gather logs (mandatory). Format is YEAR- MM-DDTHH:MM:SS e end_time Ending time within which to gather logs. Format is YEAR-MM- DDTHH:MM:SS v, verbose o, output directory h, help -d --duration INTERVAL Displays details of the actions Lumberjack is performing as it performs them. Specify an output directory for logs other than the default. Default is /var/local/tmp. Help menu. Specify the length of the collection period as a string with a number and some units; for example, 2.5 hours.

16 16 StorageGRID Webscale 10.3 Troubleshooting Guide Option Description -w --working WORKDIR Specify the working directory for this tool. Default is /var/local/ tmp. By default, files are saved to /var/local/tmp and stored in a.tar file with the following naming convention:grid_id-hostname-start_date_time-end_date_time.tar.gz For example: tar.gz Note: For a detailed list of available commands, enter: lumberjack.rb -h Related concepts Log files on page 71 Performing ingest and retrieval tests To troubleshoot ingest and retrieval performance issues, you can perform a simple test that uses a workstation in the place of the actual client application and analyze the store/retrieve performance during the test compared to the performance normally seen with the client application. The audit logs can be a valuable source of information to quantify the time a grid takes to execute an operation. For example, to determine the time a grid takes to process an S3 retrieval operation, see the audit message SGET > TIME attribute value. The corresponding audit log TIME attribute can be found for the following operations: S3 GET, DELETE, HEAD, metadata update Swift DELETE, GET, HEAD, PUT CDMI GET, PUT/POST, DELETE Archive retrieve or store to cloud tier For more information, see StorageGRID Webscale 10.3 Audit Message Reference. Running foreground verification Foreground verification enables you to verify the existence of replicated object data on a Storage Node. This foreground verification process can help you to determine if there are integrity issues with a storage device. Missing objects might indicate an issue with the underlying storage, which the LDR service uses. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Ensure that the following grid tasks are not running: Grid Expansion: Add Server (GEXP), when adding a Storage Node Storage Node Decommissioning (LDCM) on the same Storage Node If these grid tasks are running, wait for them to complete or release their lock, or abort them as appropriate. Storage must be online.

17 Overview of problem determination 17 About this task Foreground verification only checks for missing replicated object data and does not check for missing fragments of erasure coded object data. You can configure foreground verification to check all of a Storage Node's object stores or only specific object stores. If foreground verification determines that a copy of replicated object data is missing, the count for the Missing Objects Detected (OMIS) attribute (see Grid > site > Storage Node > LDR > Verification > Overview > Main) goes up by one. A replacement copy is automatically created by the system and stored to a location that satisfies the active ILM policy. The replacement copy is not necessarily stored on the Storage Node from which it originally went missing. If a replacement copy cannot be made, the LOST (Lost Object) alarm might be triggered. Foreground verification generates an LDR Foreground Verification grid task that, depending on the number of objects stored on a Storage Node, can take days or weeks to complete. It is possible to select multiple Storage Nodes at the same time; however, these grid tasks are not run simultaneously, but rather queued and run one after the other until completion. When foreground verification is in progress on a Storage Node, you cannot start another foreground verification task on that same Storage Node even though the option to verify additional volumes might appear to be available for the Storage Node. If a Storage Node other than the one where foreground verification is being run goes offline, the grid task continues to run until the % Complete attribute reaches percent. The % Complete attribute then falls back to 50 percent and waits for the Storage Node to return to online status. When the Storage Node's state returns to online, the LDR Foreground Verification grid task continues until it completes. Steps 1. Select Grid. 2. Select Storage Node > LDR > Verification. 3. Click Configuration > Main. 4. Under Foreground Verification, select ID for the storage volume or volumes to verify.

18 18 StorageGRID Webscale 10.3 Troubleshooting Guide 5. Click Apply Changes. Wait until the page auto-refreshes and reloads before you leave the page. Once refreshed, object stores become unavailable for selection on that Storage Node. An LDR Foreground Verification grid task is generated and runs until it completes or is aborted. To view its progress, go to Grid > site > Admin Node > CMN > Grid Task > Overview > Main. If object data is found to be missing, the missing object data is automatically replicated. 6. Monitor missing objects: a. Select Storage Node > LDR > Verification. b. From the Grid Options menu, click Overview. c. Under Verification Results note the value of Missing Objects Detected. If the count for the attribute Missing Objects Detected is large (if there are a hundreds of missing objects), there is likely an issue with the Storage Node's storage. In this case, cancel foreground verification by aborting the Foreground Verification grid task, resolve the storage issue, and then rerun foreground verification for the Storage Node. If foreground does not detect a significant number of replicated objects are missing, then the storage is operating normally. After you finish If foreground verification finds no (or few) missing objects, and you still have concerns about data integrity, it is recommended that you verify the integrity of the stored objects on the LDR by increasing the priority of the background verification process. Related information StorageGRID Webscale 10.3 Cloud Data Management Interface Implementation Guide StorageGRID Webscale 10.3 Swift Implementation Guide Confirming object data locations Depending on the problem, you might want to confirm where object data is being stored. For example, you might want to verify that the ILM policy is performing as expected and object data is being stored where intended. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. You must have an object identifier, one of: CBID: An object s CBID can be obtained from latest audit log, which is located on the Admin Node at /var/local/audit/export. S3 bucket and key: When an object is ingested through the S3 interface, the client application uses a bucket and key combination to store and identify the object. For details, see the S3 Implementation Guide. CDMI data object ID: When an object is ingested through the CDMI interface, the StorageGRID Webscale system returns a CDMI data object ID to the client application. For details, see the Cloud Data Management Interface Implementation Guide.

19 Overview of problem determination 19 Swift container and object: When an object is ingested through the Swift interface, the client application uses a container and object combination to store and identify the object. For details, see the Swift Implementation Guide. Steps 1. Select Grid. 2. Select primary Admin Node > CMN > Object Lookup. 3. Click Configuration > Main. 4. Enter the object s identifier (CBID, S3 bucket/key, Swift container/object, or CDMI data object ID). 5. From the Grid Options menu, click Overview. The Object Lookup page displays the current location of the object and any metadata associated with the object.

20 20 StorageGRID Webscale 10.3 Troubleshooting Guide Related information StorageGRID Webscale 10.3 Administrator Guide Analyzing data Use the information that you collect to determine the cause of the problem and potential solutions. The analysis is problem dependent, but in general: Locate points of failure and bottlenecks using the alarms. Reconstruct the problem history using the alarm history and charts. Use charts to find anomalies and compare the problem situation with normal operation.

21 Overview of problem determination 21 Escalation information checklist If you cannot resolve the problem on your own, contact technical support. Before contacting technical support, gather the information listed in the following table to facilitate problem resolution. Item Notes Problem statement Impact assessment Grid ID Software version Customization Grid specification file copy Baseline information Recent changes What are the problem symptoms? What is the history of the problem? What is the severity of the problem? The Grid ID is displayed under Configuration > Grid Options > Overview. Check the value of storage-grid-release in the Grid Topology under Grid > Topology > site > grid node > SSM > Services. Summarize how the StorageGRID Webscale system is configured. For example, list whether it uses storage compression, storage encryption, content protection options, archiving target, metadata dual commit, content dual commit, and so on. Create a copy of the current version of the grid specification file by copying the latest version from the StorageGRID Webscale system. See Configuration > Grid Options > Configuration. For more information, see the Administrator Guide. Note that this is only a copy of the grid specification file and not the actual grid specification file. This copied version is only used for troubleshooting purposes. It cannot be used to provision the StorageGRID Webscale system. Collect baseline information regarding ingest operations, retrieval operations, and storage consumption. Summarize any recent changes made to the system or its environment. Related concepts Establishing baselines on page 12 Related references Defining the problem on page 4 Assessing the risk and impact on the system on page 5 List of recent changes made to the StorageGRID Webscale system on page 13

22 22 StorageGRID Webscale 10.3 Troubleshooting Guide Related information StorageGRID Webscale 10.3 Administrator Guide

23 23 Troubleshooting Admin Nodes There are several tasks you can follow to help determine the source of Admin Node related problems. Connection to NMS unavailable You might see the Connection to NMS Unavailable screen when attempting to sign in to the StorageGRID Webscale system on a new installation for the first time or unexpectedly, while using the system. About this task When StorageGRID Webscale software is first installed and the software is not yet ready to receive attribute data, the Connection to NMS Unavailable page is displayed. This is normal system behavior. When installation of the software completes, and the system begins to retrieve attribute data, you are redirected to the login window. If the page persists for more than a few minutes, there can be connectivity issues with the Admin Node. If this error occurs unexpectedly while the StorageGRID Webscale system is running, there might be a connectivity problem with the system, or issues with services running on the Admin Node. Check network connections between the Admin Node and the rest of the StorageGRID Webscale system. If there are no network connectivity issues, restart the MI service. Steps 1. From the service laptop, log in to the Admin Node as root using the password listed in the Passwords.txt file. 2. Enter: /etc/init.d/mi restart 3. Log out. Enter: exit After you finish If the StorageGRID Webscale login window does not reappear after the MI service has been restarted, contact technical support.

24 24 StorageGRID Webscale 10.3 Troubleshooting Guide New installations When StorageGRID Webscale software is first installed and the software is not yet ready to receive attribute data, the Connection to NMS Unavailable page is displayed. This is normal system behavior. When installation of the software completes and the system begins to retrieve attribute data, you are redirected to the login window. If the screen persists for more than a few minutes, there can be connectivity issues with the Admin Node. Error occurs unexpectedly while the system is running If this error occurs unexpectedly while the StorageGRID Webscale system is running, there might be a connection problem with the system, or there might be an issue with services running on the Admin Node. Checking NMS connection through other Admin Node If the StorageGRID Webscale system is deployed with multiple Admin Nodes, you can use another Admin Node to check the status of the unavailable Admin Node. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Grid. 2. Check an available Admin Node by selecting Site > Admin Node > SSM > Services > Overview > Main. 3. Select Site > unavailable Admin Node > SSM > Services > Overview > Main. A service that is experiencing a problem has a status of Not Running and might also be displayed in blue. Determine if alarms have been triggered and take appropriate action.

25 Troubleshooting Admin Nodes 25 Related information StorageGRID Webscale 10.3 Administrator Guide

26 26 Troubleshooting Storage Nodes There are several tasks you can follow to help determine the source of Storage Node related problems. Object store (storage volume) failures The underlying storage on a Storage Node is divided into object stores. These object stores (also referred to as storage volumes) are physical partitions that act as mount points for StorageGRID Webscale system s storage. Object stores are also known as storage volumes. You can view object store information for each Storage Node at Grid > site > Storage Node > LDR > Storage > Overview > Main. Depending on the nature of the failure, faults with a storage volume might be reflected in an alarm on the storage status or on the health of an object store. If a storage volume fails, you should repair the failed storage volume to restore the Storage Node to full functionality as soon as possible. If necessary, you can place the Storage Node in a read only state so that the StorageGRID Webscale system can use it for data retrieval while you prepare for a full recovery of the server. For recovery procedures, see the Maintenance Guide. Related information StorageGRID Webscale 10.3 Maintenance Guide for VMware Deployments

27 Troubleshooting Storage Nodes 27 Troubleshooting SAVP Total Usable Space (Percent) alarm If the SAVP Total Usable Space alarm appears, you can investigate the cause. Before you begin You must have signed in to the Grid Management Interface using a supported browser. Steps 1. Select Alarms. 2. Notice the SAVP alarm. 3. Click the attribute name to display more information as shown in the following figure. 4. Look up the troubleshooting information for the SAVP alarm. The recommended actions are: Add storage Migrate data to archive media 5. Determine when this LDR service s storage is likely to reach capacity: a. Select Grid. b. Select the LDR service s Storage component that triggered the alarm. Click the Service link in the System Status table. c. Select LDR > Storage > Alarms > Main.

28 28 StorageGRID Webscale 10.3 Troubleshooting Guide d. Notice the alarm for Total Usable Space (Percent). Click to chart usable space available over the last hour. Total space available has been steadily decreasing. e. Review the trend over the last day to estimate how fast storage capacity is decreasing. Total available space for this LDR service decreased about 15% in the last day.

29 Troubleshooting Storage Nodes 29 If there is no available storage for the StorageGRID Webscale system s other Storage Nodes, add storage to the system by adding Storage Node s. For more information about Storage Nodes, see the Administrator Guide. Related references Alarms reference on page 47 Related information StorageGRID Webscale 10.3 Administrator Guide Troubleshooting SSTS Alarms If a Storage Status (SSTS) alarm appears, you can investigate the cause. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. About this task In the following example, the SSTS (Storage Status) alarm at the Notice level of severity has been triggered. Steps 1. Select Grid. 2. Select Site > Storage Node > LDR > Storage > Alarms > Main to display the alarms on the Storage Node. 3. Select LDR > Storage > Overview > Main and review the Total Usable Space (STAS) attribute to determine the amount of storage available for the LDR service.

30 30 StorageGRID Webscale 10.3 Troubleshooting Guide In this example, there is only 10 GB of usable storage remaining. For this StorageGRID Webscale deployment, 10 GB is the default value of the Storage Volume Read-Only Watermark (see Configuration > Storage Options > Overview). 4. Plot storage consumption over the last few hours. In this example, Total Usable Space dropped from roughly 55 GB at 14:00 to 10 GB at 18:00, which corresponds to the time at which the SSTS alarm was triggered. 5. Verify the storage watermarks for the StorageGRID Webscale system.

31 Troubleshooting Storage Nodes 31 Select Configuration > Storage Options > Overview. When the amount of usable disk space (Total Usable Space (STAS) dropped below the Storage Volume Read-Only Watermark, Storage Status Current (SSCR) changed to Read-only. As Storage State Desired was set to Online, Storage Status (SSTS) changed to Insufficient Usable Space and a Notice alarm was triggered. In most situations, an alarm for Total Usable Space (Percent) SAVP is also triggered; however, this depends on the relationship between the watermark setting in gigabytes (GB) and the alarm settings in percent. In this example, the Storage Volume Read-Only Watermark threshold (10 GB) was crossed before for the SAVP alarm threshold (10%).

32 32 StorageGRID Webscale 10.3 Troubleshooting Guide 6. Consult the troubleshooting information for the SSTS alarm. This alarm is triggered because there is no more available storage for the Storage Node. Add storage capacity by expanding the StorageGRID Webscale system and adding Storage Nodes. For procedures on how to manage a full Storage Node, see the Administrator Guide. Related information StorageGRID Webscale 10.3 Administrator Guide Lost and missing object data Retrieval attempts are made for several reasons, including read requests from a client application, background verifications of replicated object data, ILM re-evaluations, and the restoration of object data during the recovery of a Storage Node. The StorageGRID Webscale system uses object location information listed in an object s metadata to determine the location from which it retrieves an object. If the object is not found at the expected location, the system attempts to retrieve another copy of the object from elsewhere in the system, provided an ILM policy that made two or more copies of an object is configured. If this retrieval is successful, the StorageGRID Webscale system works to replace the missing copy of the object. Whether or not an alarm is triggered depends on the type of missing copy (erasure coded or replicated) and whether or not the system is able to retrieve another copy and replace the missing one. If another copy cannot be retrieved, the object is considered lost and a LOST (Lost Objects) alarm is triggered. All LOST (Lost Objects) alarms should be investigated immediately to determine the root cause of the loss and to determine if the object might still exist in an offline or otherwise currently unavailable Storage Node or Archive Node. For erasure coded copies, if an object cannot be retrieved from the expected location an ECOR (Missing Copies Detected) alarm is triggered on the Storage Node from which the retrieval was attempted and an attempt is made to retrieve another copy. If no other copies are found, the LOST (Lost Objects) alarm is also triggered. If a LOST (Lost Objects) alarm is not triggered after an ECOR (Missing Copies Detected) alarm, a copy has been located elsewhere in the system and the missing copy replaced according to the active ILM policy. In this case, clear the ECOR (Missing Copies Detected) alarm. In the case where object data without copies is lost, there is no recovery solution. The only action to take is to reset the Lost Object attribute for the DDS service and clear the LOST (Lost Objects) alarm. Clearing the alarm will prevent known LOST alarm instances from masking any new LOST alarm instances. Related tasks Resetting lost and missing object counts on page 39 Determining the number of lost objects If a LOST (Lost Objects) alarm is triggered, the number of lost objects is tracked by the Lost Objects attribute on the Grid > site > Storage Node > DDS > Data Store > Overview > Main page. Both replicated and erasure coded object data are accounted for. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups.

33 Troubleshooting Storage Nodes 33 Steps 1. Select Grid. 2. Select site > Storage Node > DDS > Data Store > Overview > Main. 3. Review the Lost Objects attribute. 4. Investigate the LOST (Lost Objects) alarm immediately and determine the reason why there are lost objects. 5. After completing your investigation, reset the Lost Objects attribute to zero. Related tasks Determining if lost object data can be recovered on page 37 Related information StorageGRID Webscale 10.3 Administrator Guide Running foreground verification Foreground verification enables you to verify the existence of replicated object data on a Storage Node. This foreground verification process can help you to determine if there are integrity issues with a storage device. Missing objects might indicate an issue with the underlying storage, which the LDR service uses. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Ensure that the following grid tasks are not running:

34 34 StorageGRID Webscale 10.3 Troubleshooting Guide Grid Expansion: Add Server (GEXP), when adding a Storage Node Storage Node Decommissioning (LDCM) on the same Storage Node If these grid tasks are running, wait for them to complete or release their lock, or abort them as appropriate. Storage must be online. About this task Foreground verification only checks for missing replicated object data and does not check for missing fragments of erasure coded object data. You can configure foreground verification to check all of a Storage Node's object stores or only specific object stores. If foreground verification determines that a copy of replicated object data is missing, the count for the Missing Objects Detected (OMIS) attribute (see Grid > site > Storage Node > LDR > Verification > Overview > Main) goes up by one. A replacement copy is automatically created by the system and stored to a location that satisfies the active ILM policy. The replacement copy is not necessarily stored on the Storage Node from which it originally went missing. If a replacement copy cannot be made, the LOST (Lost Object) alarm might be triggered. Foreground verification generates an LDR Foreground Verification grid task that, depending on the number of objects stored on a Storage Node, can take days or weeks to complete. It is possible to select multiple Storage Nodes at the same time; however, these grid tasks are not run simultaneously, but rather queued and run one after the other until completion. When foreground verification is in progress on a Storage Node, you cannot start another foreground verification task on that same Storage Node even though the option to verify additional volumes might appear to be available for the Storage Node. If a Storage Node other than the one where foreground verification is being run goes offline, the grid task continues to run until the % Complete attribute reaches percent. The % Complete attribute then falls back to 50 percent and waits for the Storage Node to return to online status. When the Storage Node's state returns to online, the LDR Foreground Verification grid task continues until it completes. Steps 1. Select Grid. 2. Select Storage Node > LDR > Verification. 3. Click Configuration > Main. 4. Under Foreground Verification, select ID for the storage volume or volumes to verify.

35 Troubleshooting Storage Nodes Click Apply Changes. Wait until the page auto-refreshes and reloads before you leave the page. Once refreshed, object stores become unavailable for selection on that Storage Node. An LDR Foreground Verification grid task is generated and runs until it completes or is aborted. To view its progress, go to Grid > site > Admin Node > CMN > Grid Task > Overview > Main. If object data is found to be missing, the missing object data is automatically replicated. 6. Monitor missing objects: a. Select Storage Node > LDR > Verification. b. From the Grid Options menu, click Overview. c. Under Verification Results note the value of Missing Objects Detected. If the count for the attribute Missing Objects Detected is large (if there are a hundreds of missing objects), there is likely an issue with the Storage Node's storage. In this case, cancel foreground verification by aborting the Foreground Verification grid task, resolve the storage issue, and then rerun foreground verification for the Storage Node. If foreground does not detect a significant number of replicated objects are missing, then the storage is operating normally. After you finish If foreground verification finds no (or few) missing objects, and you still have concerns about data integrity, it is recommended that you verify the integrity of the stored objects on the LDR by increasing the priority of the background verification process. Related information StorageGRID Webscale 10.3 Cloud Data Management Interface Implementation Guide StorageGRID Webscale 10.3 Swift Implementation Guide

36 36 StorageGRID Webscale 10.3 Troubleshooting Guide Tracking lost erasure coded object data If an erasure coded object cannot be retrieved from the expected location, an ECOR (Missing Copies Detected) alarm is triggered on the Storage Node from which the retrieval was attempted and an attempt is made to retrieve another copy. If no other copes are found, the (Lost Objects) alarm will be triggered. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. About this task If an (Lost Objects) alarm is not triggered after an ECOR (Missing Copies Detected) alarm, a copy has been located elsewhere in the system and the missing copy replaced according to the active ILM policy. In this case, clear the ECOR (Missing Copies Detected) alarm. In rare cases, the retrieval of a deleted erasure coded object may result in the triggering of an ECOR (Missing Copies Detected) alarm. In this case, a LOST (Lost Objects) alarm will never be triggered. Confirm that the missing object has been deleted and then clear the ECOR (Missing Copies Detected) alarm. Note: The first time a retrieval attempt fails, the Reads-Failed attribute goes up; however, it does not go up on subsequent retrieval retries against the same lost object. Steps 1. Select Grid. 2. Select Site > Storage Node > LDR > Erasure Coding > Overview > Main for the Storage Node that has raised the (ECOR) Missing Copies Detected alarm. 3. Review the Missing Copies Detected attribute. 4. Determine whether a LOST (Lost Objects) alarm has also been triggered and if so, the reason why there are lost objects. A LOST (Lost Objects) alarm is not necessarily triggered when an ECOR (Missing Copies Detected) alarm triggers. The StorageGRID Webscale system first attempts to local another copy and replace the missing copy. Only when a missing copy cannot be replaced does the LOST (Lost

37 Troubleshooting Storage Nodes 37 Objects) alarm trigger. On a large system, it may take some time before the StorageGRID Webscale system completes its task of attempting to restore the missing copy. 5. After completing your investigation, reset the Missing Copies Detected attribute to zero. Related tasks Resetting lost and missing object counts on page 39 Related information StorageGRID Webscale 10.3 Administrator Guide Determining if lost object data can be recovered When a LOST (Lost Objects) alarm is triggered, determine if the lost object data can be recovered. Before you begin The LOST (Lost Objects) alarm must be triggered Steps 1. Use the audit log to determine the identifier (CBID) of the object that triggered the LOST (Lost Objects) alarm: a. From the service laptop, log in to the Admin Node as root using the password listed in the Passwords.txt file. b. Change to the directory where the audit logs are located. Enter: cd /var/local/audit/ export/ c. Use grep to extract the Object Lost (OLST) audit messages. Enter: grep OLST audit_file_name d. Note the CBID value included in the message. For example: Admin: # grep OLST audit.log T11:03: [AUDT:[CBID(UI64):0x498D8A1F681F05B3] [UUID(CSTR):"6213A021-91FC-49C0-AF44-EC6BF377D264"] [NOID(UI32): ] [VOLI(UI64):2][RSLT(FC32):NONE][AVER(UI32):10] [ATYP(FC32):OLST][ATIM(UI64): ] [ATID(UI64): ][ANID(UI32): ] [AMID(FC32):BCMS][ASQN(UI64):62] [ASES(UI64): ]] 2. Use the ObjectByCBID command to find the object by its identifier (CBID), and then determine if data is at risk. a. Telnet to localhost 1411 to access the DDS console. b. Enter: /proc/obrp/objectbycbid -h hexadecimal_cbid_value Metadata No UUID metadata Conclusion If there is no UUID metadata, it is safe to ignore the alarm. The absence of a UUID indicates that the object was intentionally deleted.

38 38 StorageGRID Webscale 10.3 Troubleshooting Guide Metadata UUID is present Locations = 0 Conclusion If there is a UUID and the number of locations is zero, the object is potentially missing. Check if the object is available on an offline or unavailable storage resource. If you do not find a copy of the object on an offline or unavailable storage resource and the ILM policy does not include an ILM rule with only one active content placement instruction, contact technical support. Support might ask you to determine if, for replicated object data, there is a storage recovery procedure in progress. That is, has a Volume_Lost command been issued on any Storage Node and is the recovery still in progress. Related tasks Checking unavailable resources for lost objects on page 38 Checking unavailable resources for lost objects When the LOST (Lost Objects) alarm is triggered, you might want to check Storage Nodes and Archive Nodes for services that might be offline or otherwise unavailable. It may be that an object is not lost, but rather that the system could not query a location because it is currently unavailable. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Grid. 2. To check if any LDR or ARC services are offline or otherwise unavailable, select Site > grid node > LDR > Overview > Main and check the LDR State, or ARC > Overview > Main and check the ARC State. 3. If an LDR or ARC service is offline, disconnected, or otherwise unavailable: a. Restore the LDR or ARC service s state to Online. b. When the state of the LDR or ARC service is restored to Online, run the manual recovery procedure to determine if the object has a location on the previously unavailable resource, and, if possible, recover the object. 4. If you cannot restore the LDR service s state, but its storage file system is intact, perform a manual search for the CBID in the /var/local/rangedb directory. Enter: find /var/ local/rangedb/?/p -type f -name CBIDp where p indicates that this is persistent content. If the object is not found in persistent storage, contact technical support. 5. If you cannot restore the ARC service s state: a. If your Archive Node targets is a Tivoli Storage Manager (TSM) middleware server, query the TSM database on the TSM server for the object. Enter: /usr/local/arc/tsmtool q -L CBID Warning: Use the tsmtool utility carefully as it permits low-level operations that can place content at risk. Use only the commands and options described below.

39 Troubleshooting Storage Nodes 39 For example: DC-ARC1: # /usr/local/arc/tsmtool q -L BCC6984C770 DC-ARC1: # 1, ,// ,/ ,/ BCC6984C770,F,66103, , In this example, the object data is stored on the TSM tape library and the command returns information about it, including the count (the count is the first parameter in the output) and the CBID. If the object data is not found, contact technical support. b. If your Archive Node targets the cloud through the S3 API, check the destination bucket on the external archival storage system. Related tasks Resetting lost and missing object counts on page 39 Related information StorageGRID Webscale 10.3 Administrator Guide Resetting lost and missing object counts After investigating the StorageGRID Webscale system and verifying that all recorded lost objects are permanently lost or that it is a false alarm, you can reset the value of the Lost Objects attribute to zero. For erasure coded object data, you can also reset the values of Copies Lost and Reads - Failed to zero. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. Steps 1. Select Grid. 2. Select Site > Storage Node > DDS > Data Store > Configuration for the Storage Node that has raised the (LOST) Lost Objects alarm. 3. Select Reset Lost Objects Count. 4. Click Apply Changes.

40 40 StorageGRID Webscale 10.3 Troubleshooting Guide The Lost Objects attribute is reset to 0 and the LOST (Lost Objects) alarm clears. It can take a few moments for the Lost Objects attribute to reset and the alarm to clear after you click Apply Changes. 5. For erasure coded object data, select Site > Storage Node > LDR > Erasure Coding > Configuration for the Storage Node that has raised the (ECOR) Missing Copies Detected alarm. 6. Select Reset Read Failure Count and Reset Missing Copies Detected Count. 7. Click Apply Changes. The Reads - Failed and Copies Lost attributes reset to 0 and the ECOR (Copies Lost) alarm clears. It can take a few moments for the attributes to reset and the alarm to clear after you click Apply Changes. Related information StorageGRID Webscale 10.3 Administrator Guide Troubleshooting SVST (Services: Status - Cassandra) alarm A SVST alarm may be an indication that you need to rebuild the DDS service s distributed key value store (Cassandra database) for a Storage Node. Before you begin You must have signed in to the Grid Management Interface using a supported browser. To perform this task, you need specific access permissions. For details, see information about controlling system access with administration user accounts and groups. About this task If Cassandra is stopped for more than 15 days (for example, the Storage Node is powered off), when it is brought back online, Cassandra will not start. You must rebuild the Cassandra database for the affected DDS service. Attention: If two or more of the Cassandra database services are down for more than 15 days, contact technical support, and do not proceed with the steps below.

41 Troubleshooting Storage Nodes 41 Steps 1. Select Grid. 2. Select site > Storage Node > SSM > Services > Alarms > Main to display alarms. The SVST alarm was triggered. Cassandra is not running, which is indicated on the Grid > Site > Storage Node > SSM > Services > Alarms > Main page. 3. Try restarting Cassandra: a. At the Storage Node, access a command shell and log in as root using the password listed in the password.txt file. b. Enter: /etc/init.d/cassandra status c. If Cassandra is not running, restart it: /etc/init.d/cassandra restart 4. If Cassandra does not restart, determine how long Cassandra has been down. If Cassandra has been down for longer than 15 days, you must rebuild the Cassandra database. Attention: If two or more of the Cassandra database services are down for more than 15 days, contact technical support, and do not proceed with the steps below.

42 42 StorageGRID Webscale 10.3 Troubleshooting Guide You can determine how long Cassandra has been down by charting it or by reviewing the servermanager log file. To chart Cassandra: a. Select Grid > site > Storage Node > SSM > Services > Reports > Charts. b. Select Attribute > Service: Status - Cassandra c. Enter a Start Data that is at least 16 days before today s date and for End Date, today s date. d. Click Update. If the chart shows Cassandra as being down for more than 15 days, rebuild the Cassandra database. The following chart example shows that Cassandra has been down for at least 17 days. To review the servermanager log file: i. At the Storage Node, access a command shell and log in as root using the password listed in the password.txt file. ii. Enter: cat /var/local/log/servermanager.log The contents of the servermanager log file are displayed. iii. In the servermanager log file, if Cassandra has been down for longer than 15 days, the following message is displayed: " :01: cassandra cassandra not started because it has been offline for longer than its 15 day grace period - rebuild cassandra Make sure the timestamp of this message is the time when you attempted restarting Cassandra as instructed in step 3 on page 41. There can be more than one entry for Cassandra; thus, you must locate the most recent entry. If Cassandra is down for longer than 15 days, you must rebuild the Cassandra database, by rebuilding the distributed key value store.

43 Troubleshooting Storage Nodes 43 After Cassandra is rebuilt, alarms should clear. If alarms do not clear, contact technical support. Related tasks Rebuilding the distributed key value store on page 43 Related information StorageGRID Webscale 10.3 Administrator Guide Rebuilding the distributed key value store If a DDS service s distributed key value store (Cassandra database) for a Storage Node is offline for more than 15 days, you must rebuild the DDS service s distributed key value store. Choosing to rebuild the database means that the database is deleted from the grid node and rebuilt from other grid nodes. About this task When Cassandra is down, the CASA and SVST alarms are triggered. Warning: This procedure should never be performed on multiple Storage Nodes concurrently as it may result in data loss. Attention: If two or more of the Cassandra database services are down for more than 15 days, contact technical support, and do not proceed with the steps below. Steps 1. From the service laptop, log in to the Storage Node as root using the password listed in the Passwords.txt file. 2. Rebuild the Cassandra database. Enter: rebuild-cassandra-data When prompted, enter: y. DC1-S1 #: rebuild-cassandra-data Running: rebuild-cassandra-data Rebuild the Cassandra database for this Storage Node. ATTENTION: Do not execute this script when two or more Storage Nodes have failed or been offline at the same time. Doing so may result in data loss. Contact technical support. ATTENTION: Do not rebuild more than a single node within a 15 day period. Rebuilding 2 or more nodes within 15 days of each other may result in data loss. Enter 'y' to rebuild the Cassandra database for this Storage Node. Rebuild may take hours. Do not stop or pause the rebuild. If the rebuild was stopped or paused, re-run this command. [y/n]? y Removing Cassandra commit logs Removing Cassandra SSTables Updating timestamps of the Cassandra data directories. starting service ntp starting service cassandra Running nodetool rebuild. Done. Cassandra database successfully rebuilt. Rebuild was successful. Starting storage services. Storage services started

44 44 StorageGRID Webscale 10.3 Troubleshooting Guide 3. Monitor the status of nodetool rebuild. After Cassandra is rebuilt, alarms should clear. If alarms do not clear, contact technical support. Troubleshooting replication queues As the ingest rate increased, the number of ingests pending and the number of ILM evaluations pending increased because the LDR and CMS services could not keep up with the increased ingest rate. Trends suggest that something must have changed in the StorageGRID Webscale deployment s environment to account for the increased number of objects being ingested into the system. Did new client applications start ingesting objects? Is legacy data being migrated to the StorageGRID Webscale system? Data migration can cause queues to build up to a point where operations are disrupted. For guidelines on how to manage data migration, see the Administrator Guide. Queues should be resolved early by a prompt response to alarms with a severity of Notice. The following figure shows where ingest and replication queues are likely to build up and where alarms are likely to be triggered. LDR storage available If an Insufficient Free Space alarm is triggered, the amount of available space available has dropped below the Swapout No Create Watermark. The Insufficient Free Space alarm can be found on the Grid Management Interface LDR > Storage screen. Object replication queue Following ingest, copies of object data are made according to the StorageGRID Webscale system s ILM policy. During periods of high ingest, queues might form as it can become difficult for the CMS and LDR services to keep up with the required ILM activities. For LDR services, content replication queues are likely to form on both the source and destination LDR services. The number of Inbound Replications Queued (RIRQ) on the destination LDR service and of Output Replications Queued (RORQ) on the source LDR service are likely to increase and failures can trigger alarms. The Inbound Replications Failed (RIRF) and Outbound Replications Failed (RORF) alarms should clear after system activity goes back down. The replication attributes can be found for each of the Storage Nodes on the Grid Management Interface LDR > Replication screen.

45 Troubleshooting Storage Nodes 45