Power Vault in EMC Symmetrix DMX-3 and Symmetrix DMX-4 Systems

Power Vault in EMC Symmetrix DMX-3 and Symmetrix DMX-4 Systems Applied Technology Abstract This white paper is an overview of Power Vault operation in EMC Symmetrix DMX-3 and Symmetrix DMX-4 environments. June 2008

Copyright 2007, 2008 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number H2892.1 Applied Technology 2

Table of Contents Executive summary...4 Introduction...4 Audience... 4 Power subsystem overview...4 DMX-3 and DMX-4 system bay... 5 DMX-3 950 and DMX-4 950 system bay... 6 Storage bay... 6 Power Vault overview...7 Power Vault operations... 7 Power Vault save operation... 7 Power Vault restore operation... 8 Ability to vault (ATV)... 8 Need to vault (NTV)... 8 Need to power down (NTPD)... 9 Best practices for site and system service activities...9 Best practices for customers during site PDU maintenance... 9 Best practices for EMC service personnel... 10 Online DAE or disk drive upgrades... 11 Powerdown procedure... 11 Powerup procedure... 12 Configuration options and rules... 12 Power Vault Wait Time... 13 Power Zone Task... 13 Vault device configuration rules... 13 Conclusion...14 References...14 DMX-3... 14 DMX-3 950... 14 DMX-4... 14 DMX-4 950... 14 Other... 14 Applied Technology 3

Executive summary A key to Symmetrix DMX s resiliency is its ability to maintain data integrity during power outages. Each EMC Symmetrix storage array comes with battery backup in order to safely power down the array in the event of power loss. Ensuring that all data tracks in memory are successfully written to disk is essential to maintaining the consistency of application data stored on the Symmetrix. As cache size, disk size, and power requirements have grown, the time required to vault data in the event of a power outage has also increased. Power Vault is designed to limit the time needed to power off the system if it needs to switch to a battery supply. When the system detects the need to come offline or power down, the box stops all I/O and the disk adapters (DAs) start writing the contents of the appropriate areas of global memory to special vault devices located on each DA. The vault image is fully redundant, with the specified contents of global memory being saved twice to independent disks. When the DAs are done writing (saving) everything to these vault disks, the machine either finishes powering down or remains at the offline state if it doesn t need to power down. When the machine is powered back up, the DAs write all the saved info on the Power Vault devices back to all the correct locations in global memory. Write pending or format pending tracks that were in cache before the powerdown will be restored to cache. Introduction Vault save operations are triggered automatically by environmental conditions. Vaulting can also be initiated by manually powering down a system or taking the disk directors offline. When performing a vault operation, Symmetrix uses disk storage called Power Vault devices. Power Vault devices are volumes on designated physical disk drives that reserve a dedicated 5 GB space each for vaulting data, including metadata, from global memory during a powerdown operation. During powerup, the data is written back to global memory to restore the system. Power Vault devices are configured on the first four disk drives on each drive loop in the direct-attached storage bay. For each pair of disk directors 160 GB of total capacity is reserved for vault devices. This white paper describes the vault save and restore operations, configuration options and rules, and environmental monitoring. Best practice recommendations for powerdown and powerup procedures, as well as for preventing accidental triggering of a vault save operation, are also included. Audience This white paper is intended for technology professionals who configure and support Symmetrix systems. It is also intended for operations and data center staff who plan, install, and configure Symmetrix storage arrays in their IT environments. Power subsystem overview The power subsystem architecture provides two completely separate power zones (Power Zone A and Power Zone B), each of which can maintain power for an entire system bay or storage bay independent of the operation of the second zone. There is no redundancy within each zone; rather the redundancy is accomplished through the two-zone design. The Symmetrix system connects to two dedicated or isolated AC power lines. If AC power fails on one power zone, the system continues to operate through the other power zone. Symmetrix DMX-3 and DMX-4 systems have a modular power subsystem featuring a redundant architecture that facilitates field replacement of any component without any interruption in processing. The Symmetrix power subsystem supports a (2N) power zone for redundancy and drive expansion. If any Applied Technology 4

Symmetrix power supply fails, the remaining power supplies continue to share the load. The Symmetrix system senses the fault and reports it as an environmental error. The following section describes the power subsystem components for the system bays and storage bays. DMX-3 and DMX-4 system bay The system bay contains up to eight 1,800-watt power supplies that are split into two power zones (Power Zone A and Power Zone B) consisting of up to four power supplies each. The system bay also contains up to eight 2.2-kilowatt Battery Backup Unit (BBU) modules providing battery backup in the event an AC failure occurs. One BBU provides the AC line power and high voltage DC backup to one 1,800-watt power supply. The A side BBU modules receive their power from the A side Power Distribution Unit (PDU) and the B side BBU modules receive their power from the B side PDU. Two Power Distribution Panels (PDPs), one for each zone, provide a centralized cabinet interface and distribution control of the AC power input lines when connected to the system bay PDUs. The PDPs contain the manual on/off power switches, which are accessible through the rear door. Two Communications and Environmental Control Modules (XCM) communicate with the system bay BBU modules to determine the BBU status and run battery tests. The XCMs monitor and log environmental events across critical components and report any operational problems. Critical components include global memory directors, power supplies, fans, and various on/off switches. The XCM environmental control is capable of monitoring each component s local voltages, ensuring optimum power delivery. Temperature of global memory directors is also continuously monitored. Figure 1 gives a graphic illustration of the Symmetrix DMX-3 and DMX-4 system bay. *The Service Processor consists of the KVM and the server. **The Battery Backup Unit Assembly consists of tow Battery Backup Unit Modules. Figure 1. Symmetrix DMX-3 and DMX-4 system bay Applied Technology 5

DMX-3 950 and DMX-4 950 system bay Two 1800-watt redundant power supplies located at the front of the card cage enclosure supply the power to the card cage enclosure. The back of each drive enclosure contains two 12-volt power supplies (power and cooling modules) and two link control cards (LCCs). The two 12-volt power supplies (PS A, PS B) provide power and cooling to each drive enclosure. Each LCC (LCC A, LCC B) supports and controls one Fibre Channel loop and monitors the drive enclosure environment. The 2.2-kilowatt BBUs provide redundant backup for every four drive enclosures. Two BBU modules also provide redundant power backup to the card cage enclosure. The PDU, when connected to the PDP, provides a means of connecting AC input power to the system components from the independent power sources. The PDP provides a centralized cabinet interface and distribution control of the AC power input line when connected to the PDU. Each PDP contains a manual on/off power switch that is accessible through the rear door. The physical connection to a Symmetrix system channel interface occurs at the connectors on the frontend/back-end (FEBE) boards. The FEBE board provides an interface between the director and host channels. Additionally, the director boards monitor the card cage enclosure chassis environmental s PS1 and PS2 and the fan module through the FEBE boards. Figure 2 gives a graphic illustration of the Symmetrix DMX-3 950 and DMX-4 950 system bay. Figure 2. Symmetrix DMX3-950 and DMX-4 950 system bay Storage bay The storage bay power subsystem consists of the drive enclosure power supply/cooling modules and the BBU modules that provide the battery backup for the drive enclosures. The 2.2-kilowatt BBUs provide redundant backup for every four drive enclosures. The A side BBU modules receive their power from the A side PDU and support both the A and B side drive enclosures. The B side BBU modules receive their power from the B side PDU and support both the A and B side drive enclosures. The back of each drive enclosure also contains two LCCs. Each LCC (LCC A, LCC B) supports and controls one Fibre Channel loop and monitors the drive enclosure environment. Applied Technology 6

Two PDPs, one for each zone, provide a centralized cabinet interface and distribution control of the AC power input lines when connected to the storage bay PDUs. The storage bay s PDPs contain the manual on/off power switches, which are accessible through the rear door. Figure 3 gives a graphic illustration of the storage bay. Figure 3. Storage bay Further information on the power subsystems is available on Powerlink, EMC s password-protected customer- and partner-only extranet. Power Vault overview The Power Vault environmental algorithm continuously monitors the system s available power sources, available vault devices, and the battery holdup time in order to determine three status values: Ability to vault (ATV), Need to vault (NTV), and Need to power down (NTPD). These status values, the Vault Save operation, and the Vault Restore operation are explained in the following sections. Power Vault operations Vaulting operations are divided into two categories: Save operation Saving global memory to the vault disks. Restore operation Restoring global memory from the vault disks. The following sections describe these operations in more detail. Power Vault save operation When a system is powered down or transitioned offline, or when environmental conditions trigger a vault situation, a Power Vault save operation is initiated. The part of global memory being saved first reaches a consistent image (no more writes). The disk directors then write the appropriate sections of global memory to the vault disks, saving two copies of the logical data. Applied Technology 7

The BBU modules automatically transition to battery backup when the Symmetrix system detects loss of AC power. The BBU modules maintain power to the Symmetrix system for up to 5 minutes while the global memory is vaulted to the vault disk drives. When the disk directors are done writing to the vault disks, the system either finishes powering off or remains in the offline state if it doesn t need to power off. Power Vault restore operation When the Symmetrix system is powered on, the startup program does the following: Initializes the hardware and the environmental system Restores the global memory from the saved data while checking the integrity of the data Performs cleanup, data structure integrity, and reinitialization of needed global memory data structures At the end of the startup program, the system resumes normal operation when the BBUs are recharged enough to support another vault save. The system will not come back online until the BBU modules have a minimum of 300 seconds of holdup time. If any condition is not safe, the system will not resume operation and will call EMC Customer Support for diagnosis and repair. In this state, EMC Customer Support will be able to communicate with the Symmetrix system and find out the reason for not resuming normal operation. Note: A fully charged BBU will have 600 seconds of charge time, which is enough to support two vault save operations. Ability to vault (ATV) There must be enough vault devices available to save one full copy of global memory to disk. Enginuity will allow for the loss of a predetermined amount of vault devices depending on the configuration. If the minimum threshold is reached, the system will begin to vault. Need to vault (NTV) The following is a list of conditions that will initiate a vault save operation: The disk directors are taken offline Loss of a sufficient amount of vault drives Loss of a sufficient amount of disk directors Insufficient battery holdup time Loss of a sufficient amount of power supplies Loss of AC power to both power zones Loss of AC power to one power zone with insufficient battery holdup time remaining on the opposite zone Please note that multiple component failures within a zone are handled differently than a complete zone failure. If power is lost to an entire zone, the system will not begin to vault. However, if a single zone encounters multiple component failures that reduce the number of operational power supplies to an insufficient value, then the system will begin to vault. In general, the system will vault if half of the SPS modules in the system bay fail. This needs to be taken into consideration especially with smaller 24-slot systems (for example, the DMX-3 1500 and DMX-4 1500). Applied Technology 8

For example, a minimum configuration DMX-4 1500 is required to have only two SPS modules configured per zone in the system bay. But the system will begin to vault if two SPS modules fail in a single zone. A system with this configuration will also begin to vault if each zone were to lose a single SPS as both zones would have lost redundancy. Note: Systems with this type of configuration can be upgraded with additional SPS modules and power supplies to avoid these situations. Six-slot systems (DMX-3 950 and DMX-4 950) have fewer components in the system bay and are essentially protected only by the other zone. Therefore, these systems will not need to vault until both zones experience a failure. Need to power down (NTPD) The system will need to power down after performing the vault save operation if the system loses AC power to both power zones. The power subsystem s two-zone design ensures that if there is an AC power loss to one of the zones, power from the second power source maintains power to the affected bay or bays, provided that each system bay or storage bay s two Power Distribution Panels (PDP) are connected to two separate source Power Distribution Units (PDU) at the customer s site. Best practices for site and system service activities Power Vault is designed to maintain data integrity when the system needs to be taken offline or powered down manually. Environmental conditions may also trigger the system to begin a vault save operation. This must be taken into consideration when performing system and site power maintenance. The following sections describe best practice recommendations for customers and EMC service personnel to follow while performing power maintenance activities. Configuration options and rules as well as powerdown and powerup procedures are also discussed. Power Vault in Symmetrix DMX-3 and DMX-4 Systems: Operational Overview and Best Practices Technical Note contains detailed information about commands and tools available for following the guidelines below. This technical note is available only to restricted audiences on Powerlink and should be referenced by EMC service personnel prior to and while performing maintenance. Best practices for customers during site PDU maintenance It is highly recommended that EMC Customer Service personnel are onsite prior to and during any power maintenance activity. EMC Customer Service personnel have access to commands and tools to check for proper system health status. It is also highly recommended that EMC Customer Service personnel and site electricians verify that the AC power feeds to each rack of equipment come from redundant power sources and PDUs prior to beginning any power-related maintenance. The following guidelines should be followed by customer site personnel to avoid accidentally causing the system to vault while performing site maintenance: Verify system health status prior to beginning any power related maintenance. Discuss power maintenance plans with the local EMC Customer Service team prior to activity. Have the local EMC Customer Service team verify the Power Zone Task is disabled before beginning maintenance. See the Power Zone Task section on page 12 for more information. When installing a new system, apply power from two separate source PDUs at the site to each bay s two PDPs. For detailed information refer to the Physical Planning Guide for your specific model, also available on Powerlink. Applied Technology 9

Before beginning any power maintenance onsite or on previously installed systems, verify that each bay s PDPs are connected to separate source PDUs. Configure power redundantly to ensure that any PDU shut-off will not take power away from both power zones of any bay. If power has been removed from any bay s zone more than once, or if power has been removed from any combination of zones, do not remove power from the other zone until adequate battery recharge time has passed on the discharged BBUs. Charge time typically takes a minimum of 2 hours. It is recommended that EMC Customer Service personnel be contacted to utilize detailed commands and tools to check BBU charge status. Note: Enginuity 5773 introduces a Battery Conservation algorithm that will shut off the SPS modules in a zone that loses AC power as long as the opposite zone is fully operational. This helps prevent the SPS modules from being completely drained during a single-zone power failure. The loss of power to the zone will still drain the SPS modules of some battery capacity, but it increases the amount of single-zone power outages that can be experienced before the SPS modules drop below 300 seconds of holdup time. This functionality is also targeted for a future release of 5772 code. Best practices for EMC service personnel Power Vault in Symmetrix DMX-3 and DMX-4 Systems: Operational Overview and Best Practices Technical Note contains detailed information about commands and tools available for following the guidelines below. This technical note is available to restricted audiences on Powerlink and should be referenced by EMC service personnel prior to and while performing maintenance. The following guidelines should be followed by EMC service personnel to avoid accidentally causing the system to vault while performing system or site maintenance: Prior to starting any work, ensure that the power feeds to enclosures come from redundant sources. Ensure that DMX internal cable verification tests have been done. Verify the Power Zone Task is disabled before beginning maintenance. See the Power Zone Task section on page 12 for more information. Run a health check before power activity. Before removing power from a zone, check that the opposite zone has all power supplies on without faults. Before beginning any maintenance, check the SPS health. Each BBU provides power to four disk array enclosures (DAEs). Ensure that all four DAEs on the opposite side are healthy before maintenance. Check the battery test history along with charge state status before doing any maintenance. Whenever an action is taken on a power component, ensure that NTV is not set. If it is set, there is a 60-second window to undo the action that set it. When removing disk drives that contain vault devices, ensure that the NTV is not set. Applied Technology 10

Online DAE or disk drive upgrades Do not connect any cables from the new DAEs/drive bays to the existing system until all SPS units in the new DAEs/drive bays are fully charged. Ensure all SPS units in the new DAEs or new drive bays are fully charged prior to proceeding with the online upgrade. There should be no flashing green LEDs on any SPS module. Normal charge time is 2 hours minimum. Furthermore, do not power down any drive bays that are installed even if the disk drives in the bay are not yet configured. If there are any power or environmental issues in a daisy-chained environment, the vault devices that reside on that loop are discounted. If all the extended loops have power events due to the low SPS charge status, then Enginuity will discount all the drive loops and subsequently all the vault drives. If the number of available vault drives falls below a predetermined level as set by Enginuity, the system will begin to vault. Powerdown procedure The Symmetrix contains no user-serviceable parts. Therefore, the system bay and storage bays should not be opened for any reason by untrained personnel. If the Symmetrix is in need of repair, only qualified personnel familiar with safety procedures for electrical equipment and the Symmetrix should access components inside the unit. Note: The Symmetrix is designed to stay powered up for most all situations. Unless there is an emergency situation, first call the EMC Customer Support Center for assistance before powering down the Symmetrix. The method that EMC Customer Service field personnel should use to power down the Symmetrix is to execute a script. The script will ensure that the system bay is powered down prior to the storage bays, or else the system may not be able to save the vault image. The script instructs the service personnel to perform these steps to power down the Symmetrix: 1. Stop all processes to the Symmetrix. Note: If you are powering down the Symmetrix for an emergency condition, stopping all processes to the system is not necessary. 2. On the rear door of the system bay (Figure 4), press the Zone A and Zone B PDP power switches to the down O OFF position. 3. On the rear door of the storage bays (Figure 4), press the Zone A and Zone B PDP power switches to the down O OFF position. Applied Technology 11

Figure 4. System bay and storage bay power switches Powerup procedure Note: Do not use the following procedures for powering up the Symmetrix for the first time. If you have powered down the Symmetrix for an emergency condition, call the EMC Customer Support Center for assistance before powering up the system. Please reference the Physical Planning Guide and Quick Start Power Connection Guide for your specific model for power connection and configuration requirements. Perform these steps to power up the Symmetrix after it has been routinely powered down by the PDP power switches: 1. On the rear door of each storage bay (Figure 4), press the Zone A and Zone B PDP power switches to the up ON position. 2. On the rear door of the system bay (Figure 4), press the Zone A and Zone B PDP power switches to the up ON position. The Symmetrix system begins its initial microcode program load (IMPL) startup procedure. 3. Wait at least 30 minutes for the IMPL procedure to complete. Note: The actual IMPL procedure time varies by system type and configuration. Configuration options and rules There are two Power Vault-related options that can be configured to site specifications: Power Vault Wait Time Power Zone Task All other configuration rules are strictly defined by Enginuity. The options and rules are discussed in the following section. Applied Technology 12

Power Vault Wait Time The configuration file is customizable in order for the machine to be able to survive brownouts in areas where there are frequent power interruptions. It can be set for 1, 2, or 3 minutes to survive temporary power interruptions. After that time the front-end directors will be taken offline and the disk directors will begin the vault save. The default value for this setting is 1 minute. Contact your local EMC Customer Service team if changes to this setting may be desired. The exact system configuration must be taken into account before an increase to the wait time value is approved. The 3-minute value is an extreme case and is approved only in specific configurations. Also, an online configuration change is required to change this setting. Power Zone Task The Power Zone Task is optional under normal operation. However, it is highly recommended and considered best practice that the local EMC Customer Service team verifies this setting is disabled before beginning any power-related maintenance on the system and before the customer begins any power-related maintenance onsite. An online configuration change is not required to change this setting. When the Power Zone Task is enabled and the system loses AC power to a single power zone, a 20-hour timer is invoked. The system will call home to notify the EMC Customer Support Center of the event. When the timer counts down to 5 hours the system will again call home. Before the 20-hour period ends, the EMC Customer Engineer can choose one of three options: Repair the cause of the power fault Reset the 20-hour timer to continue the single-power zone operation Allow the system to vault, shutting down the system in an orderly manner Vault device configuration rules The following configuration rules apply to vault devices: Five GB on the first four drives of every drive loop is reserved for memory vaulting. Each disk director pair requires 32 such dedicated devices for a total of 160 GB of vault space per disk director pair. The vault devices can only use single-mirror data protection and cannot be configured with TimeFinder /Snap, virtual, or dynamic spare devices. The drive pool, virtual devices, or drive devices cannot reside in the 5 GB of vault space. However, they can reside on the same physical disk drive as the vault devices but not within the vault devices. The distribution of the vault devices across the disk directors, the back-end interfaces, and the physical disks should be such that a full vault save will be possible within the time frame dictated by the capacity of the battery (up to 5 minutes). The total capacity of all of the vault hypervolumes in the system will be at least sufficient to keep two logical copies of the persistent part of global memory. Physical drives that contain vault devices are candidates for permanent sparing if the following is true for each model: o o On the DMX-3, if there is an available spare on the same primary loop On the DMX-4, if there is an available spare on the same disk director processor Applied Technology 13

Conclusion This white paper explained Power Vault operations, described hardware features and environmental monitoring, and recommended best practices for performing power maintenance. For more information, please see the References section that follows. References The following documents can be found on Powerlink, EMC s password-protected customer- and partneronly extranet: DMX-3 Symmetrix DMX-3 Product Guide Symmetrix DMX-3 Physical Planning Guide Symmetrix DMX-3 Quick Start Power Connection Guide DMX-3 950 Symmetrix DMX-3 950 Product Guide Symmetrix DMX-3 950 Physical Planning Guide Symmetrix DMX-3 950 Quick Start Power Connection Guide DMX-4 Symmetrix DMX-4 Product Guide Symmetrix DMX-4 Physical Planning Guide Symmetrix DMX-4 Quick Start Power Connection Guide DMX-4 950 Symmetrix DMX-4 950 Product Guide Symmetrix DMX-4 950 Physical Planning Guide Symmetrix DMX-4 950 Quick Start Power Connection Guide Other Also helpful is EMC Knowledgebase solution emc119567, What is the correct power ON/OFF sequence for the Symmetrix DMX-3? Applied Technology 14