EMC CLARiiON CX4 Best Practices for Achieving Five 9s Availability

Transcription

1 EMC CLARiiON CX4 Best Practices for Achieving Five 9s Availability Best Practices Planning Abstract This white paper discusses how best to implement an EMC CLARiiON CX4 storage environment to reach five 9s availability levels. March 2010

2 Copyright 2007, 2010 EMC Corporation. All rights reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED AS IS. EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com All other trademarks used herein are the property of their respective owners. Part Number H Best Practices Planning 2

3 Table of Contents Executive summary...4 Introduction...4 Audience... 4 Terminology... 4 Overview...4 General availability and reliability...5 CX4 UltraFlex... 5 Improved write cache management... 6 UltraPoint enclosure... 6 Software enhancements... 7 Process and serviceability enhancements... 8 Best practices for enhancing availability...9 Storage system software Process Storage system hardware Conclusion...15 References...15 Best Practices Planning 3

4 Executive summary Storage is often a critical component of the data center because it stores data for applications that generate revenue or hold important customer information. In the past, the most critical information was often stored on the most expensive levels of storage available to the enterprise: those that guaranteed nearly continuous uptime. With the introduction of the CLARiiON CX4 line with UltraFlex technology, and the rewards of continuous improvements in every new FLARE (CLARiiON s operating environment) release, many customers are seeing the value of using what has traditionally been viewed as a small to midsize business platform for their critical business environments. CLARiiON s robust design integrates numerous patented data integrity features to provide the stringent reliability required by enterprise organizations for a price affordable to smaller companies. These characteristics make CLARiiON an attractive option for price-conscious customers with critical business applications - where downtime translates to costly business and operational losses. When following certain best practice recommendations, customers can be confident that their CLARiiON storage systems meet five 9s availability and protect their business operations. Introduction This white paper outlines EMC s recommended best practices for achieving five 9s availability on CLARiiON CX4 storage systems. It first discusses general availability numbers for the latest line of CLARiiON CX4 storage systems with UltraFlex technology including how EMC tracks availability across the field population and relevant architectural and product development process changes that were implemented to improve the availability levels of the CX4 series beyond those of the previous generations. Next, the paper discusses the top 10 best practices to implement in the environment, including hardware and software configuration recommendations and general process-related guidelines. These are intended as guidelines to ensure that the environment is properly maintained to continuously meet the required availability. Audience This white paper is for any EMC employee, partner, or customer personnel interested in understanding CLARiiON five 9s availability. It provides CLARiiON availability information and a set of guidelines for support personnel or customers looking to implement five 9s availability in their data center. Terminology Availability: The total amount of time that information stored on a computer system or network is accessible during a given period. Five 9s: The equivalent of five minutes of unplanned downtime a year, or percent system uptime or availability percent uptime, based on a 365-day year, translates to 5.26 minutes of downtime per year. Reliability: The resistance of a device or system to failure, or the probability that a system and its components will perform the required function for a specified interval of time under stated conditions 1. Overview EMC measures availability using population-wide statistical analysis based on installed storage systems cumulative runtimes and unplanned outages grouped into three categories: hardware, software, and 1 Wikipedia, Reliability engineering Best Practices Planning 4

5 processes. A system-wide availability rate of this level translates to less than an hour of downtime per 100,000 hours of CX4 runtime. Companies are storing more mission-critical applications on CLARiiON storage systems than ever before. This reliance has prompted EMC to improve the availability and reliability of its midrange CLARiiON line of storage systems. Through continuous improvement initiatives, great strides have been made over the last few years in these areas, culminating in the release of the CLARiiON CX4 storage system line with UltraFlex technology. The industries that use CLARiiON systems are varied, including health care, finance, telecommunications, and manufacturing. The companies in each industry maintain a variety of mission-critical, revenuegenerating applications, critical to running the business and generating revenue. Each of these industries can measure application downtime in terms of dollars, or revenue, lost. Each minute, or second, of equipment unavailability (meaning that revenue-generating applications are not available for transactions) relates directly to the bottom line. As an example, consider a retailer hosting an order entry system. If the application goes offline, customers cannot place an order for their desired items online or in stores. Many customers forgo the sale altogether and shop at a competitor s site. The business cost of lost revenue can run from thousands to tens of thousands of dollars per minute, depending on the industry and application. Regulatory requirements have changed significantly in recent years and downtime or loss of data can result in costly legal fees for certain institutions based on these requirements. Even an inaccessible system can result in costly lost employee productivity. There is a subtle distinction between the availability of a system, and the reliability of the system or its components. The frequency of downtime incidents or replacements experienced, whether one or many, is a measure of reliability. Hardware components can experience a number of reliability issues for example, numerous drive failures over the course of the year may indicate unacceptable reliability of a particular drive type but may not lead to unavailability of the storage system (provided that drives are not a single point of failure), allowing the overall system to still meet the % availability measure. Users should look to meet both high reliability and high availability in their storage environments, striving for higher values in both categories proportionate to the level of criticality of the applications stored on that hardware. Ensuring that components are reliable, hot swappable, modular, and not single points of failure contributes to a robust, highly available system. This paper discusses how best to configure the CLARiiON to meet five 9s availability. It also discusses the changes that EMC has made to its architecture, features, and processes to help you meet your five 9s availability goal. Details on how EMC measures availability across an installed population are outlined in the following section. General availability and reliability Every week, a team at EMC calculates the availability numbers for every CLARiiON storage system model in the field, based on total uptime divided by runtime. So, for example, if there are 2,000 CX4-240s currently installed in the field (maintained by EMC or authorized service providers), runtime is calculated as 2,000 units x 7 days x 24 hours/day, or 336,000 hours. If there are two outages during this period (for example, one caused by hardware and one by an improperly configured SAN) that add up to 1.5 hours, the uptime is 336, hours, or 335,998.5 hours. These values yield an availability rate for the week of % (uptime/runtime). Weekly values are then normalized, with each month comprising exactly four weeks, allowing EMC to equally weight and compare the rates for each month. CX4 UltraFlex The CLARiiON CX4 line with UltraFlex technology was designed in conjunction with the efforts of many continuous improvement programs and Six Sigma initiatives intended to positively impact availability and reliability and to allow EMC s midrange line to excel versus the competition in these areas. Since its introduction, the calculated availability rate across all installed CX4 series storage systems has tracked to Best Practices Planning 5

6 five 9s. An availability rate of this level translates to an hour or less of downtime per 100,000 hours of CX4 runtime. So why has the CLARiiON CX4 storage system line become the industry leader, in terms of availability rates, to achieve five 9s robustness? Why can no other direct competitor provide third-party validation for this level of availability covering hardware- and software-related issues? From the outset, a prominent design goal was to reach enterprise-level availability, specifically five 9s. The CX4 series is able to meet this level of availability through significant architecture, process, and software enhancements. For example, the UltraPoint disk array enclosure (DAE) allows for drive fault isolation and more proactive testing of components. Improved write cache management in CLARiiON CX4 series allows the write cache to stay enabled during specific failure scenarios and maintenance procedures. This avoids many potential data unavailability (DU) situations due to cache destaging and/or forced flushing scenarios. Process improvements in quality assurance and design testing have drastically reduced the number of postrelease incidents found in the field. The latest FLARE release implements many new features to enhance reliability, and many serviceability features have been introduced to speed problem resolution in the field. In conjunction with availability, EMC continuously monitors the reliability of individual field-replaceable units via mean time between product replacements (MTBPR). Improvements in reliability numbers ultimately translate to higher availability rates. The following section discusses the latest architectural changes that provide enhanced availability and reliability, the latest software features, and many of the internal process and serviceability changes that contribute to the robustness of the CX4 platform. Improved write cache management There are changes in how CLARiiON CX4 implements cache management. When various components failed in earlier generations of CLARiiON, the storage systems automatically flushed the write cache by writing the write cache data to the vault drives. However, CLARiiON CX4 storage systems maintain data in the write cache during certain failures including a single SP, vault drive, power supply, SPS, and fan failure. Data is also maintained in the write cache during the following maintenance procedures: Non- Disruptive Upgrade (NDU) of FLARE and layered applications, single SP restarts, or when a single SP or UltraFlex I/O module is physically removed. To protect the array from data loss or data unavailable conditions, write cache content is still flushed to the vault drives in the following scenarios: loss of power to the array on both the A and B power circuits, array overheating conditions affecting both the SPs, and multi-fan failures (on CX4-960). This improved write cache management algorithm in CLARiiON CX4 storage systems ensures that the array performance is maintained after component failures. For example, losing a single SPS results in no loss of performance. This is because the write cache now stays enabled during this failure. Since the write cache stays enabled, the CLARiiON is not forced to flush the write cache, and host I/O does not time out waiting for the CLARiiON to finish the flush. Furthermore, the increased amount of storage-system write cache also helps avoid forced cache flushing which increases improves performance and availability. UltraPoint enclosure The UltraPoint (DAE4P) enclosure contributes significantly to the robustness of the CX4 systems. It has also beneficially impacted the previous generation s (CX300, CX500, CX700, and CX3) availability rates in the 2 Gb/s version of those platforms. The UltraPoint back end offers significant enhancements for fault isolation and diagnosability by implementing a point-to-point architecture versus a loop between the link control card (LCC) and the individual drives. This architecture provides better isolation of error conditions while reducing the loop latency on the Fibre Channel protocol. A failing drive can be detected and isolated quickly. When a new drive is inserted into an UltraPoint DAE it is tested in an isolated manner. If it fails it is bypassed and not allowed to participate in back-end traffic. When an enclosure powers up, all drives are Best Practices Planning 6

7 independently tested in this fashion. The same initialization test is performed on any new or replacement drive; it is tested before it is brought online and bypassed if it does not pass. Another significant benefit of the CX4 UltraPoint enclosure is that all the latest drive types utilize this architecture. SATA II, Low-Cost Fibre Channel, and Fibre Channel drives all leverage the new UltraPoint architecture and take advantage of its benefits. Because the translation from Fibre Channel to SATA now takes place at the drive level and not at the enclosure level, both drive types benefit from the enhanced diagnosability and fault isolation advances of the UltraPoint point-to-point drive architecture. UltraPoint can run at 2 or 4 Gb/s, and a new LED color scheme quickly shows customers and service personnel the speed of the drives and the enclosure (green for 2 Gb/s and blue for 4 Gb/s). This simple visual indicator provides immediate notification of configuration issues that may cause newly added enclosures, or drives, to be offline (for example, a 2 Gb/s drive was swapped in for a 4 Gb/s drive), which allows a fast resolution to the problem. The 4 Gb/s enclosures underwent extensive automated fault injections, including power fail and power glitch tests, and drive faults and pulls. Lastly, a significant difference with the introduction of the 4 Gb/s drives and enclosures is the signal quality specification. CX4 UltraPoint was designed to a soft SCSI error rate of 1x10e-16, rather than the industry standard of 1x10e-12. The EMC 4 Gb/s Fibre Channel signal quality design specification is a significant differentiator in terms of robustness of the end product. This translates to significant gains in signal integrity, ultimately reducing the number of disk and loop faults from soft SCSI errors. Software enhancements The continuous evolution of new software features for the FLARE operating environment also contributes to the higher availability numbers on CX4, as well as to improvements in rates on the previous line. From FLARE release 19 onward, many features have been added related to error detection and handling, rebuilds on ATA drives, and Navisphere error reporting. There have been significant enhancements designed to decrease RAID group exposure to double drive faults: proactive hot sparing, rebuild logging, and RAID 6 technology. Proactive hot sparing allows a copy operation to a hot spare from a failing drive, eliminating the rebuild window and reducing exposure to a second drive fault. Rebuild logging allows FLARE to log I/O destined for a drive that may have been issuing timeouts (for example, due to a remap), allowing FLARE to rebuild only the ranges with pending writes if the drive becomes accessible again within the next few minutes. This feature significantly shortens the time needed for a rebuild on larger drives, avoiding the need for a lengthy full drive rebuild and thereby reducing the window of opportunity for a second fault. RAID 6 is generally described as data striped across a number of drives within a RAID group, with two independent parity fields maintained for redundancy. It can therefore tolerate two failed disks in a RAID group. More importantly, RAID 6 helps recovery if a hard media error occurs during a single disk rebuild. As drive sizes increase, the possibility of encountering an unrecoverable bit error during the rebuild process also increases. In the case of other RAID types like RAID 1/0, RAID 5, or RAID 3, an unrecoverable bit error encountered during a single drive rebuild results in an uncorrectable error. But when data is protected by RAID 6, parity information is available even after a single drive failure which is used to recover in the scenario mentioned above. In FLARE release 26, EMC introduced the Asymmetric Active/Active feature for CLARiiON storage systems. This feature changes how a host manages multiple communication paths to LUNs on the array (commonly referred to as path management) by permitting I/O to flow to either or both SPs. Using this feature, the host can send I/O to a LUN regardless of which SP owns the LUN this eliminates the need for an explicit trespass of the LUN in case of a path failure. This feature reduces data-unavailable conditions caused by misconfiguration of the host; it also prevents a boot from a SAN system from becoming unavailable if there is a path failure. For more details on implementation and benefits, refer to the EMC CLARiiON Asymmetric Active/Active Feature (ALUA) A Detailed Review white paper available on Powerlink. Best Practices Planning 7

8 Obviously, the latest release of FLARE also carries forward all of the previous robust data integrity mechanisms patented in early systems. These include, for example, patented sector-level data checksumming and other important sector-level metadata, such as SNiiFFER and background verify. CLARiiON (CX/CX3/CX4 arrays) sectors are formatted to 520 bytes, 8 bytes more than a normal 512-byte sector. The extra 8 bytes contain checksums of the sector and proprietary RAID stamps. This information allows the CLARiiON to protect against data corruption. These 8 bytes travel with the data while it is in the CLARiiON. This protects against memory and media errors as well as a power failure during a multistep write operation. SNiiFFER is a CLARiiON background process that reads all the data on all the RAID groups in a system at a constant rate. This is important because hard media errors are often latent and if the sector is never read, the error is never reported. SNiiFFER proactively checks for these errors and makes sure that the drives are in a readable condition even if an application is actively using that data. All of these features work together to ensure that failures, such as a power failure during a write to a degraded RAID group, do not corrupt the data residing on the affected drives. EMC has years of experience in providing the robustness required in overcoming many varied and complicated failure scenarios. Process and serviceability enhancements Many development process changes and serviceability enhancements were also implemented for the newest CX4 platform. Combined, these efforts delivered a platform with fewer issues found and reported in the field, faster issue resolution, and defined, easy-to-follow service procedures that led to the five 9s availability we see today. This section will highlight just a few of these contributors. Process improvements From the outset, it was decided to design and develop the CX4 platform with higher availability and reliability as a design goal. Learning from, and leveraging, many new or enhanced procedures and testing methodologies helped to ensure a more robust midrange line of products. For the CX4, existing single point of failure (SPOF) testing was supplemented with failure mode and effects analysis (FMEA). Using detailed FMEA tools helped determine which faults to inject, above and beyond those that were traditionally thought to be SPOFs. This testing led to performing twice the number of storage processor (SP) tests than on previous generations of hardware, and almost four times the number of DAE tests; this uncovered errors that would have led to outages in the field because they had not previously been judged to be SPOFs. A second key component of FMEA testing was to validate that all systems (array hardware and software, and Navisphere) operate together to correctly identify the root cause of the issue through accurate component failure reporting. This small change ensures that the root cause and the true faulted component can quickly be identified and replaced by service, shortening the duration of any outage. A second significant process improvement was to treat all early engineering systems, such as QA, as early field installs. Reporting issues as they arose through event monitoring in-house, and performing analyses on all these systems, provided CLARiiON engineering with trending before the actual ship date. Rather than relying solely on calculated MTBPR rates at shipment, CLARiiON had demonstrated and calculated rates to base reliability values upon. CLARiiON also implemented extensive highly accelerated life tests (HALT), in addition to typical eight corner tests, to determine the true breaking point of components. Serviceability improvements Serviceability tools and processes have also changed significantly in many areas. These include more granular reporting of faulted subcomponents (as identified via FMEA testing), more customer replaceable units, and tools designed to reliably and continuously replicate specific service processes. For example, to speed diagnosis and recovery times (thereby reducing outage windows), each SP now has full visibility into its peer. This subtle architecture change means that should one SP become unavailable, the remaining SP can phone home with exactly which component is faulted on its peer. This level of granularity extends to the boot process for the peer. Should one SP become stuck during the boot process, its peer knows exactly what caused the boot issue. Best Practices Planning 8

9 Small improvements, such as adding a new service LAN port to each SP, and making front-end optical receivers field-replaceable SFPs, yield large serviceability benefits. Service personnel no longer need to wait for access to the customer network to log in to the system via Navisphere, and when an optic fails an SFP can be replaced instead of an entire SP. Similarly, with the modular I/O complex in CX4 storage systems, only the faulty I/O modules can be replaced rather than replacing the entire SP. Lastly, EMC has developed new service utilities and incorporated them into the field service procedures. These utilities cover basic maintenance tasks and code upgrades, and help to ensure that service personnel follow best practices consistently, thereby eliminating potential process errors. EMC also enhanced the Navisphere Server Utility with new features that make it easier to validate the server configuration against the E-Lab Interoperability Navigator and check that the server has high-availability access to the storage system. Running these checks periodically on each server helps to prevent against configuration changes that may have inadvertently changed the connectivity to the storage system, or changed the server status from an E-Lab qualified configuration to one that is unsupported. Best practices for enhancing availability EMC takes a system-level approach to delivering highly available and highly reliable storage infrastructures. We focus on reducing planned downtime and eliminating unplanned downtime. To ensure that systems are tracking to the acceptable availability numbers, EMC determines current field availability rates at all levels of the environment for the entire CX4 system and attached hosts. As discussed in the previous section, every week the latest availability numbers for all CLARiiON systems are calculated based on rates and durations of any outage incidents. Outages caused by factors in any of the following three categories contribute to the availability rates: EMC CLARiiON hardware: The root cause was due to components such as a storage processor or a disk drive. EMC software: The root cause of the incident was found to be due to an EMC software component such as microcode. Process: The root cause of the incident was related to an EMC process issue. Note that scheduled downtime is not included in any of the availability calculations. Scheduled downtime can involve anything from disaster recovery testing, to CLARiiON storage system conversions or upgrades to a newer hardware platform model. Incidents in any of the above areas are tracked and contribute to the overall current availability rate of the CLARiiON CX4 family. EMC understands that components outside of the hardware system itself may cause data on the storage to become inaccessible, and attributes incidents in this category to the overall availability of the storage system. For example, an improperly configured PowerPath installation, or incorrect failovermode setting, may cause data on the storage system to become inaccessible in the event of a path failure. The root cause of the incident is logged in the process category and contributes to the overall availability rate for CLARiiON. A related best practice for customers and implementation specialists is to always properly configure path management software and run high-availability (HA) checks and tests; this ensures proper failover capabilities to the storage system. Even with process and other issues included in the reports, the EMC CLARiiON CX4 series system-wide still maintains a five 9s availability rate. By monitoring the root cause of incidents beyond the enclosure hardware and software, EMC can define best practices for implementation that reduce the occurrence of some of the most common issues in customer incident reports. Documenting and sharing best practices and guidelines for high-availability design helps to ensure that both EMC and the customer contribute to providing the most highly available systems on the market today. The following 10 best practices are based on EMC s stringent analysis of all logged incidents over years of accumulated runtime. Following these recommendations can help to minimize potential data unavailability events in your storage environment. The most common contributing factors to incidents in each of the three areas are tracked and translated to best practices that can be implemented in your environment to provide the highest levels of availability possible. The top 10 resulting best practices are outlined in the following section. Best Practices Planning 9

10 Storage system software Best Practice No. 1: Stay current on FLARE releases. The latest FLARE release will always offer the most robust features to date, as well as significant fixes to issues found in earlier versions. Customers seeking five 9s availability rates should be taking advantage of the features inherent to the latest FLARE release as discussed in the previous section. The latest FLARE releases also have useful serviceability features, such as Reactive SP Collect, that enables EMC Service to have instant diagnostics on the storage system when any event is logged and dialed home to EMC. This eliminates the time necessary to gather these reports manually when a call is opened. It is generally in a customer s best interest to stay current with FLARE releases, remaining within at least one release of the most recent. This often translates to maintaining at least one major release and one patch every year. Best Practice No. 2: Install and configure EMC Remote Access and Call Home capabilities. Configure a CLARalert station with access to EMC s call center, either via modem or , to monitor the CLARiiON storage systems in the environment and send notice of events back to EMC. Allow your EMC service representative to implement all capabilities of CLARalert to allow faster diagnostic file gathering for problem resolution. These features include the capability for EMC support to dial in to (or access over the web) the storage system in question, and also for SP Collects (diagnostic files) to be sent to support when an event is logged and dialed home. Lastly, configure Navisphere Event Monitor to notification to the storage administrator so that they are notified of all warning and critical events in the CLARiiON environment. Implementing features that enable service to more quickly resolve issues shortens any downtime window and translates directly to increasing overall availability. Aside from availability, it is also in the best interest of a customer to configure these rapid response tools so that any failure that results in performance degradation is quickly noticed so that the appropriate replacement parts may be brought on site quickly. Process Best Practice No. 3: Install supported path management software on every attached server. Supported path management software must be installed on every server that requires high-availability access to the storage system. EMC recommends PowerPath as the most robust path management software available on the market. PowerPath is designed to intelligently manage I/O across the data paths in a heterogeneous storage environment. However, other path management software is supported by EMC, provided it properly manages and works with the CLARiiON LUN Ownership Model, which trespasses a LUN from one controller (SP) to the other when access to the LUN via its default controller is lost. Always refer to the E-Lab Interoperability Navigator for the latest path management applications and revisions supported. A version of PowerPath, PowerPath SE, is included with every CLARiiON storage system and provides support for single HBA environments zoned to see both SPs of the storage system. A minimum license level of PowerPath SE is a critical component for successfully performing nondisruptive software upgrades on CLARiiON. Proper configuration of the path management software installed on the server is imperative. Many of the outages EMC tracks are directly related to improper path management configurations. By default, failovermode is set to 1, which is the appropriate selection for PowerPath. Implementing other software, however, may require changing this setting from the default to ensure that failover behaves properly with Best Practices Planning 10

11 CLARiiON. If all paths to an SP fail, but the LUN is not trespassed because the failover mode is not set properly, an outage will occur. The CLARiiON High Availability (HA) - Best Practices Planning white paper on EMC Powerlink provides details about required settings for failovermode, arraycommpath, and systemtype for each operating system and path management software combination in the environment. Properly setting these variables for the type of path management software used ensures that the trespass of a LUN is handled appropriately in the event of a failure. The Server Utility now offers an option to Verify High Availability on the server - this includes checking the installed path management software and verifying that the necessary settings are configured appropriately. Proactively running this utility on each attached server can help to highlight any failover issues before problems arise. Best Practice No. 4: Configure servers properly for high availability (HA) access to the storage system. To configure for high availability in the SAN means to eliminate all potential single points of failure between attached servers and applications and the storage system. An HA system can continue to access data in the event of a failure of a component, and in many cases, will be able to function despite multiple failures. The CLARiiON storage system itself is designed for redundancy with N+1 components at all levels of the system. In conjunction with implementing RAID protection for data (Best Practice No. 10), designing for high availability begins at the server. Due to the CLARiiON architecture, properly configuring the host to, at a minimum, see both SPs is imperative for maintaining access to data. To achieve the highest levels of availability between the server ports and the storage system ensure that you are configuring for true high availability, as illustrated in Figure 1. For true high availability, multiple HBAs or NICs should be implemented and Fibre Channel or iscsi paths configured for redundancy to each SP. Redundant switches should also be implemented. Configuring for high availability ensures that the CLARiiON SAN is designed to achieve the highest availability rates possible. Again, as with path management software, EMC often sees outages related to customers installing only a single HBA, or only a single dual-ported HBA. In the latter case, while the ports and paths may be redundant, the card itself is still a single point of failure. Best Practices Planning 11

12 Figure 1. Configuring for high availability For details on configuring a highly available environment refer to the CLARiiON High Availability (HA) - Best Practices Planning white paper on EMC Powerlink. Best Practice No. 5: Ensure all SAN components are listed in the E-Lab Interoperability Navigator. A listing of all SAN components tested and supported by EMC for interoperability with CLARiiON is available on EMC Powerlink in the E-Lab Interoperability Navigator. To ensure the highest levels of availability, check that all components, from the server hardware and operating system to the operating system level, are supported by EMC. Other important information, such as required OS patches and hot fixes, is also supplied. Commonly installed components that are not often checked for support by EMC are HBAs and drivers. Often, if these components have not been tested and qualified by EMC it is because they do not pass our stringent interoperability tests. At the very least, ensure that the installed versions of these components have been tested for interoperability by E-Lab. The Navisphere Server Utility will print a checklist of all relevant installed components for each server; you can compare this checklist to the list of supported components that is available through the new CLARiiON wizards on the E-Lab site. This checklist is another feature of the Verify High Availability option of the Server Utility. After performing hours of interoperability and integration testing on all the supported components in the matrix, EMC is confident that each component provides the level of required robustness for the enterprise when used in conjunction with your CX4 storage system. Ensuring E-Lab support mitigates your risk. Best Practice No. 6: Implement redundant data center components. Extend SAN high-availability best practices into the data center and eliminate potential single points of failure. Implement redundant power and cooling within the data center. Best Practice No. 7: Be proactive and storage-conscious. Best Practices Planning 12

13 Customers should have best practices in place and work proactively with EMC to encourage timely testing and upgrades; this ensures smooth transitions between FLARE revisions and consistent management of the storage systems. Customer best practices can be grouped into the following categories: Site management: Locate storage systems in a few large, centrally managed sites with dedicated and knowledgeable storage personnel who also have a through understanding of the environment. Keep your EMC or certified provider notified of any site information changes. Keep personnel up to date on their knowledge of the CLARiiON storage system and SAN concepts by regularly attending EMC or partner training. Details on EMC classes are listed on EMC Powerlink under the Training heading. Capacity management: Regularly forecast storage capacity requirements and growth to minimize the frequency of required upgrades. This can be accomplished by using Virtual Provisioning technology on the storage system. See the EMC CLARiiON Virtual Provisioning white paper available on Powerlink for more details. Change management: Manage change control by planning storage configuration changes and upgrades in conjunction with EMC or service partners. This includes thoroughly testing new components, firmware, and FLARE in a test environment before standardizing it in production; and collaborating with EMC or service partners to determine the best releases, ETAs, and patches to implement. This recommendation also includes planning upgrades centrally with EMC or the partner team to determine upgrade schedules conservatively and ahead of time. Configuration management: Maintain a consistent server configuration that has been validated against the E-Lab Interoperability Matrix. Understand the existing CLARiiON configuration by periodically running high-availability reports via the Server Utility, and reporting on capacity and layout via the reports in Navisphere. Regular performance checks can also ensure that the configuration is functioning as desired. Availability management: At a minimum, regularly run the Navisphere Server Utility with the Verify High Availability option to check that no changes have been inadvertently made in the connection between the server and the storage system (for example, a zoning change that has disrupted the visibility from the host to both SPs) that will affect failover. These health checks can verify that no changes have occurred (since implementation), at the host or storage system level, that have affected the high availability of the SAN. Storage system hardware This section details best practices to implement in the configuration of the CX4 storage system hardware. Best Practice No. 8: Leverage UltraPoint enclosures. First and foremost, leverage the UltraPoint design inherent to the CX4 family of storage systems. Because of the enhancements described previously, utilizing UltraPoint enclosures increases the availability of your system. The enhanced fault isolation ensures that a drive failing due to soft SCSI errors does not affect other neighboring drives on the bus, along with the other benefits described earlier. Ensure that 2 Gb/s enclosures and drives are segregated from 4 Gb/s enclosures by loop. Monitor loop speeds via the provided blue and green LEDs. Drives or enclosures not compatible with their peers will be bypassed - either until a component of the appropriate speed is installed in its place or until the Bus Reset Wizard is used (in a planned maintenance window) to change the speed of any back-end bus when necessary. Best Practice No. 9: Allocate one hot spare for every 30 drives in the storage system. To ensure there are sufficient hot spares available for continuous data availability, always allocate at least one hot spare for every 30 data drives in system. There should also be hot spares configured of appropriate size, equivalent to at least the largest drive in the system. Lastly, consider drive speed and type when allocating hot spares, so that performance of the RAID group is not degraded when a hot spare is participating. (For example, a 10k rpm drive sparing into a RAID group comprised of 15k rpm drives will limit performance to this lowest common denominator.) Best Practices Planning 13

14 Type, outside of performance limitations, is not as much a factor as in earlier systems. FC drives can now spare for SATAII drives, and vice versa. ATA drives may still only spare for ATA drives. Similarly, for performance reasons Enterprise Flash Drives can be spared only by other Enterprise Flash Drives. The EMC CLARiiON Global Hot Spares and Proactive Hot Sparing - Best Practices Planning white paper on EMC.com and EMC Powerlink provides details on properly laying out hot spares within the storage system. It also details the hot spare selection algorithm used by FLARE. Best Practice No. 10: Implement mirrored or parity level RAID protection. This best practice may seem obvious but ensure that you are configuring RAID groups appropriate for both performance and availability. This means selecting a RAID type that meets the specific performance needs of the application, in terms of bandwidth or throughput, and provides a level of redundancy that protects against loss of data in the event of a drive failure. Protection options include mirrored RAID (RAID 1 and RAID 1/0 with a minimum two-drive configuration) and parity-protected RAID 3, RAID 5, and RAID 6. RAID 4+1 is generally an overall good performer and may suffice for the majority of applications. RAID 1/0 should be implemented when availability and redundancy are required. For example, critical log files should be placed on RAID 1/0 LUNs to add another level of redundancy beyond RAID 5. RAID 6 provides protection from dual drive failures and unrecovered bit errors during rebuilds. Also pay particular attention to how data is laid out across the LUNs that have been configured. For example, avoid putting two highly accessed tables or file systems on LUNs that share the same physical drives. Situations such as this may create hot spots or contention for the physical resources and can result in performance issues. As time progresses, hot spots may naturally develop. EMC recommends monitoring performance regularly, and using virtual LUN technology to move contentious LUNs seamlessly to new spindles. LUN migration is an included component of FLARE that enables this movement. A Fully Automated Storage Tiering (FAST) feature called FAST LUN Migrator is available for CLARiiON CX4 storage arrays with FLARE release 29. This feature can analyze the hot spots and move the LUNs to the appropriate storage tier with minimal user input. Also, keep performance in mind when configuring metaluns. Expansion via striping maintains or enhances the original performance characteristics, preserving the RAID type chosen at initial setup. The EMC CLARiiON Storage System Fundamentals for Performance and Availability - Applied Best Practices and the EMC CLARiiON Best Practices for Performance and Availability white papers on EMC.com and EMC Powerlink provide specific implementation recommendations and full characteristics of each RAID type. Best Practices Planning 14

15 Conclusion Data is critical to the enterprise, and any interruption in access translates quickly to costly losses, whether in lost revenue, restricted employee productivity, or an inability to meet regulatory requirements. The EMC CLARiiON CX4 series is committed to providing the highest levels of availability in the market today. By implementing the 10 best practices outlined in this white paper, in conjunction with the robust architecture of the UltraScale and the continuous improvement efforts EMC has implemented internally, you can drive availability rates to five 9s, or less than five and a half minutes of downtime per year for each storage system in your environment. References This white paper references several detailed documents, all available on EMC.com and Powerlink: EMC CLARiiON Global Hot Spares and Proactive Hot Sparing - Best Practices Planning white paper EMC CLARiiON Storage System Fundamentals for Performance and Availability - Applied Best Practices white paper EMC CLARiiON Best Practices for Performance and Availability - Applied Best Practices white paper CLARiiON High Availability (HA) - Best Practices Planning white paper EMC CLARiiON Asymmetric Active/Active Feature (ALUA) A Detailed Review white paper EMC CLARiiON Virtual Provisioning Applied Technology white paper For further details on some of the features and products discussed within this white paper refer to: Introduction to the EMC CLARiiON CX4 Series Featuring UltraFlex Technology white paper EMC PowerPath Product Guide Best Practices Planning 15