Dell EMC PowerMax Reliability, Availability, and Serviceability Technical White Paper

Transcription

1 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper This technical white paper explains the Reliability, Availability, and Serviceability hardware and sftware features f Dell EMC PwerMax strage arrays Dell Engineering May 2018 A Dell EMC Technical White Paper

2 Revisins Date May 2018 Descriptin Initial release The infrmatin in this publicatin is prvided as is. Dell Inc. makes n representatins r warranties f any kind with respect t the infrmatin in this publicatin, and specifically disclaims implied warranties f merchantability r fitness fr a particular purpse. Use, cpying, and distributin f any sftware described in this publicatin requires an applicable sftware license. Cpyright May 2018 Dell Inc. r its subsidiaries. All Rights Reserved. Dell, EMC, and ther trademarks are trademarks f Dell Inc. r its subsidiaries. Other trademarks may be the prperty f their respective wners. Published in the USA [4/20/2018] [Technical White Paper] H17064 Dell believes the infrmatin in this dcument is accurate as f its publicatin date. The infrmatin is subject t change withut ntice. 2 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

3 Table f cntents Revisins... 2 Executive summary Intrductin Dell EMC PwerMax System Family Overview PwerMax engine and directr cmpnents Channel frnt-end redundancy Glbal memry technlgy verview Physical memry errr verificatin and errr crrectin PwerMax NVMe Back-end Smart RAID RAID RAID Drive sparing Data at Rest Encryptin (D@RE) Drive mnitring and crrectin InfiniBand fabric switch Redundant pwer subsystem Vaulting Vault triggers Pwer-dwn peratin Pwer-up peratin Remte Supprt Supprtability thrugh the Management Mdule Cntrl Statin Secure Service Credential (SSC), secured by RSA Cmpnent-level serviceability Dell EMC internal QE testing Nndisruptive PwerMaxOS upgrades TimeFinder and SRDF replicatin sftware Lcal replicatin using TimeFinder Remte replicatin using SRDF Cascaded SRDF and SRDF/Star supprt SRDF/Metr supprt Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

4 11 Unisphere fr PwerMax System Health Check Cnclusin References Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

5 Executive summary Tday s missin-critical envirnments demand mre than redundancy. They require nn-disruptive peratins and upgrades, and being always nline. They require high-end perfrmance, handling all wrklads, predictable r nt, under all cnditins. They require the added prtectin f increased data availability prvided by lcal snapsht replicatin and cntinuus remte replicatin. Dell EMC PwerMax strage platfrms deliver all f these needs. The intrductin f NVMe drives raises the perfrmance expectatins and pssibilities f high-end arrays. A simple, service level-based prvisining mdel simplifies the way users cnsume strage, taking the fcus away frm the back-end cnfiguratin steps and allwing them t cncentrate n ther key rles. While perfrmance and simplificatin f strage cnsumptin is critical, ther features als create a pwerful platfrm. Redundant hardware cmpnents and intelligent sftware architecture deliver extreme perfrmance while als prviding high availability levels achieving six-nines ( %) availability with remte replicatin. This cmbinatin prvides exceptinal reliability, while als leveraging cmpnents in new ways that decrease the ttal cst f wnership f each system. Imprtant functinality such as lcal and remte replicatin f data, used t deliver business cntinuity, must cpe with mre data than ever befre withut impacting prductin activities. Furthermre, at the end f the day, all f these challenges must be met while cntinually imprving data center ecnmics. Reliability, availability, and serviceability (RAS) features are crucial fr enterprise envirnments requiring always-n availability. PwerMax platfrms are architected t prvide six-nines ( %) availability in the mst demanding, missin-critical envirnments. The many redundant features discussed in this dcument are taken int accunt in the calculatin f verall system availability. This includes: Redundancy in the back-end, cache memry, frnt-end and fabric. The types f RAID prtectins given t vlumes n the back-end. Lcal and remte replicatin sftware features such as TimeFinder and Symmetrix Remte Data Facility (SRDF). Calculatins may als include time t replace failed r failing FRUs (field replaceable units). In turn, this als takes int accunt custmer service levels, replacement rates f the varius FRUs, and ht sparing capability in the case f drives. Figure 1 PwerMax RAS highlights 5 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

6 1 Intrductin PwerMax arrays include enhancements that imprve reliability, availability, and serviceability. This makes PwerMax arrays ideal chices fr critical applicatins and 24x7 envirnments demanding uninterrupted access t infrmatin. PwerMax array cmpnents have a mean time between failure (MTBF) f several hundred thusand t millins f hurs fr a minimal cmpnent failure rate. A redundant design allws systems t remain nline and peratinal during cmpnent replacement. All critical cmpnents are fully redundant, including directr bards, glbal memry, internal data paths, pwer supplies, battery backup, and all NVMe back-end cmpnents. Peridically, the system tests all cmpnents. PwerMaxOS reprts errrs and envirnmental cnditins t the hst system as well as t the Custmer Supprt Center. PwerMaxOS validates the integrity f data at every pssible pint during the lifetime f the data. Frm the pint at which data enters an array, the data is cntinuusly prtected by errr detectin metadata, data redundancy, and data persistence. This prtectin metadata is checked by hardware and sftware mechanisms any time data is mved within the subsystem, allwing the array t prvide true end-t-end integrity checking and prtectin against hardware r sftware faults. Data redundancy and persistence allws recvery f data where the integrity checks fail. The prtectin metadata is appended t the data stream, and cntains infrmatin describing the expected data lcatin as well as CRC representatin f the actual data cntents. The expected values fund in prtectin metadata are stred persistently in an area separate frm the data stream. The prtectin metadata is used t validate the lgical crrectness f data being mved within the array any time the data transitins between prtcl chips, internal buffers, internal data fabric endpints, system cache, and system disks. PwerMaxOS supprts industry standard T10 Data Integrity Field (DIF) blck cyclic redundancy cde (CRC) fr track frmats. Fr pen systems, this enables a hst-generated DIF CRC t be stred with user data and used fr end-t-end data integrity validatin. Additinal prtectins fr address/cntrl fault mdes prvide increased levels f prtectin against faults. These prtectins are defined in user-definable blcks supprted by the T10 standard. Address and write status infrmatin is stred in the extra bytes in the applicatin tag and reference tag prtin f the blck CRC. The bjective f this technical nte is t prvide an verview f the architecture f PwerMax arrays and the reliability, availability, and serviceability (RAS) features within PwerMaxOS. 6 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

7 2 Dell EMC PwerMax System Family Overview The Dell EMC PwerMax 2000 and Dell EMC PwerMax 8000 are the first Dell EMC hardware platfrms with a Nn-Vlatile Memry Express (NVMe) back-end fr custmer data. NVMe is the prtcl that runs n the PCI Express (PCIe) transprt interface, used t efficiently access strage devices based n Nn-Vlatile Memry (NVM) media, including tday s NAND-based flash alng with future, higher-perfrming, Strage Class Memry (SCM) media technlgies such as 3D XPint and Resistive RAM (ReRAM). NVMe als cntains a streamlined cmmand set used t cmmunicate with NVM media, replacing SCSI and ATA. NVMe was specifically created t fully unlck the bandwidth, IOPS, and latency perfrmance benefits that NMV ffers t hst-based applicatins which are currently unattainable using the SAS and SATA strage interfaces. The NVMe back-end cnsists f a 24-slt NVMe DAE using 2.5 frm factr drives cnnected t the Brick via dual-prted NVMe PCIe Gen3 (8 lane) back-end I/O interface mdules, delivering up t 8GB/sec f bandwidth per mdule. In additin t the all-nvme strage density and scale which prvide high back-end IOPS and lw latency, the Dell EMC PwerMax arrays als intrduce a mre pwerful data reductin mdule capable f perfrming inline hardware data cmpressin, deduplicatin, and adaptive tiering t lwer TCO by using aut data placement. Highlights f the PwerMax 2000 system include: 1-2 engines per system 12-cre Intel Bradwell CPUs yielding 96 cres per engine Up t 2TB f DDR4 cache per engine Up t 64 FE prts per system Up t 1 PBe per system f PCIe Gen3 NVMe strage Highlights f the PwerMax 8000 system include: 1-8 engines per system 18-cre Intel Bradwell CPUs yielding 144 cres per engine Up t 2TB DDR4 cache per engine Up t 256 FE prts per system Up t 4 PBe per system f PCIe Gen3 NVMe strage The primary benefits that the PwerMax platfrms ffer Dell EMC custmers are: Massive scale with lw latency NVMe design Mre strage IOPS density per system in a much smaller ftprint Future prf technlgy - ready fr next generatin strage media such as 3D XPint and NVMe ver Fabric (NVMe-F) infrastructure Applied machine learning t lwer TCO by using intelligent data placement Imprved data efficiency and data reductin capabilities with inline dedupe and cmpressin 7 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

8 3 PwerMax engine and directr cmpnents The engine is the critical building blck f PwerMax systems. It primarily cnsists f tw redundant directr bards that huse glbal memry, frnt-end cnnectivity, back-end cnnectivity, internal netwrk cmmunicatins and envirnmental mnitring cmpnents. Each directr bard has a dedicated pwer and cling system. Even single-engine cnfiguratins are fully redundant. A PwerMax system may have between ne and eight engines depending n mdel and cnfiguratin. Table 1 lists the cmpnents within an engine, cunt per directr, and defines their purpses. Table 1 PwerMax engine and directr cmpnents Directr Cunt Cmpnent (per directr) Purpse Pwer Supply 2 Prvide redundant pwer t a directr Fan 5 Prvide cling fr a directr Management Mdule 1 Manage envirnmental functinality NVMe Flash I/O Mdule Up t 4 Safely stre data frm cache during the vaulting sequence. Frnt-end I/O Mdule Up t 4 Prvide frnt-end cnnectivity t the array. There are different types f frnt-end I/O mdules that allw cnnectivity t varius interfaces, including SAN, FICON, SRDF, and embedded NAS (enas). PCIe Back-end I/O Mdule 2 Cnnect the directr bards t the back-end f the system, allwing I/O t the system s drives. Cmpressin and Deduplicatin I/O Mdule 1 Perfrm inline data cmpressin and deduplicatin Fabric I/O mdule 1 Prvides cnnectivity between directrs. In multi-engine PwerMax 8000 systems, the fabric I/O mdules are cnnected t an internal InfiniBand switch. Memry Mdule 16 Glbal memry cmpnent 8 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

9 Figure 2 displays the frnt view f a PwerMax engine. Figure 2 Frnt view f PwerMax 2000 and PwerMax 8000 engine The fllwing figures display rear views f engine cmpnents, with lgical prt numbering. Figure 3 Rear view f PwerMax 2000 engine with lgical prt numbering 9 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

10 Figure 4 Rear view f PwerMax 8000 multi-engine with lgical prt numbering Figure 5 Rear view f PwerMax 8000 single-engine with lgical prt numbering Nte that a single-engine PwerMax 8000 system requires fur NVMe Flash I/O mdules per directr cmpared t a multi-engine PwerMax 8000 which requires three NVMe Flash I/O mdules per directr. The fur NVMe Flash I/O mdules per directr cnfiguratin will remain even if additinal engines are added t the system. This must be cnsidered when rdering new systems as the additinal NVMe Flash I/O mdule reduces the number f external I/O mdules, thus reducing the ttal number f external prts. 3.1 Channel frnt-end redundancy Channel redundancy is prvided by cnfiguring multiple cnnectins frm the hst servers (direct cnnect) r Fibre Channel switch (SAN cnnect) t the system. With SAN cnnectivity, thrugh Fibre Channel switches, each frnt-end prt can supprt multiple hst attachments, enabling strage cnslidatin acrss a large number f hst platfrms. The multiple cnnectins are distributed acrss separate directrs t ensure 10 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

11 uninterrupted access in the event f a channel failure. A minimum f tw cnnectins per server r SAN t different directrs is necessary t prvide full redundancy. Hst cnnectivity t the frnt-end directr prts shuld be spread acrss physical cmpnents fr the mst efficient frm f redundancy. The fllwing are recmmended fr cnnecting a hst r cluster: 2-4 frnt-end paths are cnfigured in the prt grup fr masking and znes t the hst (single initiatr zning is recmmended). Fr cabling ptins, ne apprach is t cnnect all even-numbered prts t fabric A and all ddnumbered prts t fabric B. In single engine systems with this apprach, select 2 I/O prts spanning bth SAN fabrics n each directr, with each prt being n a separate I/O mdule. Example: Prt 4 & 24 n bth directrs 1 and 2. In a multi-engine system, distributing the paths further acrss directrs spanning different engines spreads the lad fr perfrmance and ensures fabric redundancy. Example: Prt 4 in directrs 1, 2, 3 and 4. Figure 6 SAN cnnectivity in a single engine envirnment Glbal memry technlgy verview Glbal memry is a crucial cmpnent in the architecture. All read and write peratins are transferred t and frm glbal memry. Transfers between the hst prcessr and channel directrs can be prcessed at much greater speeds than transfers invlved with physical drives. PwerMaxOS uses cmplex statistical prefetch algrithms which can adjust t prximate cnditins n the array. Intelligent algrithms adjust t the wrklad by cnstantly mnitring, evaluating and ptimizing cache decisins. PwerMax arrays can have up t 2TB f mirrred DDR4 memry per engine and up t 16TB mirrred per array. Glbal memry within an engine is accessible by any directr within the array. Dual-write technlgy is maintained by the array. Frnt-end writes are acknwledged when the data is written t mirrred lcatins in the cache. In the event f a directr r memry failure, the data cntinues t be 11 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

12 available frm the redundant cpy. If an array has a single engine, physical memry mirrred pairs are internal t the engine. Physical memry is paired acrss engines in multi-engine PwerMax 8000 arrays Physical memry errr verificatin and errr crrectin PwerMaxOS can crrect single-bit errrs and reprt an errr cde nce the single-bit errrs reach a predefined threshld. T prtect against pssible future multi-bit errrs, if single-bit errr rates exceed a predefined threshld, the physical memry mdule is marked fr replacement. When a multi-bit errr ccurs, PwerMaxOS initiates directr failver and calls ut the apprpriate memry mdule fr replacement. When a memry mdule needs t be replaced, the array ntifies Dell EMC supprt and a replacement is rdered. The failed mdule is then sent back t Dell EMC fr failure analysis. 12 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

13 4 PwerMax NVMe Back-end The PwerMax architecture incrprates an NVMe back-end that reduces cmmand latency and increases data thrughput while maintaining full redundancy. NVMe is an interface that allws hst sftware t cmmunicate with a nn-vlatile memry subsystem. This interface is ptimized fr Enterprise and Client slid state drives (SSDs), typically attached as a register-level interface t the PCI Express interface. The NVMe back-end subsystem prvides redundant paths t the data stred n slid state drives. This prvides seamless access t infrmatin, even in the event f a cmpnent failure and/r replacement. Each PwerMax Drive Array Enclsure (DAE) can hld NVMe SSDs. The DAE als huses redundant Canister Mdules (Link Cntrl Cards) and redundant AC/DC pwer supplies with integrated cling fans. Figure 7 and Figure 8 shw the frnt and rear views f the PwerMax DAE. Figure 7 PwerMax DAE (frnt) Figure 8 PwerMax DAE (rear) The directrs are cnnected t each DAE thrugh a pair f redundant back-end I/O mdules. The back-end I/O mdules cnnect t the DAEs at redundant LCCs. Each cnnectin between a back-end I/O mdule and an LCC uses a cmpletely independent cable assembly. Within the DAE, each NVMe drive has tw prts, each f which cnnects t ne f the redundant LCCs. The dual-initiatr feature ensures cntinuus availability f data in the unlikely event f a drive management hardware failure. Bth directrs within an engine cnnect t the same drives via redundant paths. If the sphisticated fencing mechanisms f PwerMaxOS detect a failure f the back-end directr, the system can prcess reads and writes t the drives frm the ther directr within the engine withut interruptin. 4.1 Smart RAID Smart RAID prvides active/active shared RAID supprt fr PwerMax arrays. Smart RAID allws RAID grups t be shared between back-end directrs within the same engine. Each back-end directr has access t every physical drive within the DAE but each TDAT n that physical drive will be primary t nly ne backend directr. 13 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

14 Smart RAID helps in cst reductin by allwing a smaller number f RAID grups while imprving perfrmance by allwing tw directrs t run I/O cncurrently t the same set f drives. Figure 9 illustrates Smart RAID cnnectivity between directrs, spindles, and TDATs. Figure 9 Smart RAID cnnectivity 4.2 RAID 5 RAID 5 is an industry-standard data prtectin mechanism with rtating parity acrss all members f the RAID 5 set. In the event f a physical drive failure, the missing data is rebuilt by reading the remaining drives in the RAID grup and perfrming XOR calculatins. PwerMax systems supprt tw RAID 5 cnfiguratins: RAID 5 (3+1) Data striped acrss 4 drives (3 data, 1 parity) RAID 5 (7+1) Data striped acrss 8 drives (7 data, 1 parity) 4.3 RAID 6 RAID 6 enables the rebuilding f data in the event that tw drives fail within a RAID grup. Dell EMC s implementatin f RAID 6 calculates tw types f parity. This is imprtant during events when tw drives within the same RAID grup fail, as it still allws the data in this scenari t be recnstructed. Hrizntal parity is identical t RAID 5 parity, which is calculated frm the data acrss all f the disks in the RAID grup. Diagnal parity is calculated n a diagnal subset f data members. Fr applicatins withut demanding perfrmance needs, RAID 6 prvides the highest data availability. PwerMax systems implement RAID 6 (6+2) Data striped acrss 8 drives (6 data, 2 parity) 14 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

15 4.4 Drive sparing PwerMaxOS supprts Universal Sparing t autmatically prtect a failing drive with a spare drive. Universal Sparing increases data availability f all vlumes in use withut lss f any data capacity, transparently t the hst, and withut user interventin. When PwerMaxOS detects a drive is failing, the data n the faulty drive is cpied directly t a spare drive attached t the same engine. If the faulty drive has failed, the data is rebuilt nt the spare drive thrugh the remaining RAID members. When the faulty drive is replaced, data is cpied frm the spare t the new drive. PwerMax systems have ne spare drive in each engine. The spare drives reside in dedicated DAE slts. In rder t allw all drives in the engine t share the spare drive, the spare drive type is the same as the highest capacity and perfrmance class as the ther drives in the engine. Slutins Enabler 9.0 prvides tls t view infrmatin related t spare drives in PwerMax arrays. The symcfg list v utput reprts ttal values fr Cnfigured Actual Disks, Cnfigured Spare Disks and Available Spare Disks in the system. The Number f Cnfigured Actual Disks field reprts nly nn-spare cnfigured disks, and Number f Cnfigured Spare Disks field reprts nly cnfigured spare disks. C:\>symcfg -sid XYZ list -v Symmetrix ID: XYZ (Lcal) Time Zne : Eastern Standard Time Prduct Mdel Symmetrix ID Micrcde Versin (Number) : PwerMax_8000 : XYZ : 5978 (175A0000) < TRUNCATED > Number f Cnfigured Actual Disks : 64 Number f Cnfigured Spare Disks : 2 Number f Available Spare Disks : 2 Figure 10 symcfg list -v The symdisk list dskgrp_summary by_engine reprts spare cverage infrmatin per Disk Grup per Engine. The Ttal and Available spare disk cunts fr each Disk Grup include bth spare disks that are in the same Disk Grup in the same Engine, as well as shared spare disks in anther Disk Grup in the same Engine that prvide acceptable spare cverage. These shared spares are als included in the ttal disk cunt fr each Disk Grup in each Engine. Therefre, the cumulative values f all Disk Grups in all Engines in this utput shuld nt be expected t match the values reprted by the symcfg list v cmmand that were described in the previus example. 15 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

16 Ttal Disk Spare Cverage percentage fr a particular Disk Grup is the spare capacity in cmparisn t usable capacity shwn in the utput. C:\>symdisk list -dskgrp_summary -by_engine Symmetrix ID: XYZ Disk Hyper Usable Capacity Spare Cverage Flgs Speed Size Ttal Ttal Avail Grp Eng Cnt LT (RPM) (MB) Disk (%) (MB) Disk (%) Disk (%) IE IE IE Ttal Legend: Disk (L)catin: I = Internal, X = External, - = N/A (T)echnlgy: S = SATA, F = Fibre Channel, E = Enterprise Flash Drive, - = N/A Figure 11 symdisk list dskgrp_summary by_engine Hwever, Spare Cverage as reprted by the symdisk list v and symdisk shw cmmands indicates whether the disk currently has at least ne available spare; that is, a spare disk that is nt in a failed state r already invked t anther disk. C:\>symdisk -sid XYZ list -v Symmetrix ID : XYZ Disks Selected : 66 Directr : DF-1C Interface : C Target ID : 0 Spindle ID : < TRUNCATED > Spare Disk Spare Cverage : N/A : True Figure 12 symdisk list v 16 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

17 4.5 Data at Rest Encryptin Data at Rest Encryptin prtects data cnfidentiality by adding back-end encryptin t the entire array. D@RE prvides hardware-based, n-array, back-end encryptin. Back-end encryptin prtects infrmatin frm unauthrized access when drives are remved frm the system. D@RE prvides encryptin n the back-end that incrprate XTS-AES 256-bit data-at-rest encryptin. These I/O mdules encrypt and decrypt data as it is being written t r read frm a drive. All cnfigured drives are encrypted, including data drives, spares, and drives with n prvisined vlumes. D@RE incrprates RSA Embedded Key Manager fr key management. With D@RE, keys are selfmanaged, and there is n need t replicate keys acrss vlume snapshts r remte sites. RSA Embedded Key Manager prvides a separate, unique Data Encryptin Key (DEK) fr each drive in the array, including spare drives. By securing data n enterprise strage, D@RE ensures that the ptential expsure f sensitive data n discarded, misplaced, r stlen media is reduced r eliminated. As lng as the key used t encrypt the data is secured, encrypted data cannt be read. In additin t prtecting against threats related t physical remval f media, media can readily be repurpsed by destrying the encryptin key used fr securing the data previusly stred n that media. D@RE: Is cmpatible with all PwerMaxOS features. Allws fr encryptin f any supprted lcal drive types r vlume emulatins. Delivers pwerful encryptin withut perfrmance degradatin r disruptin t existing applicatins r infrastructure. D@RE can als be deplyed with external key managers using Key Management Interperability Prtcl (KMIP) that allw fr a separatin f key management frm PwerMax arrays. KMIP is an industry standard that defines message frmats fr the manipulatin f cryptgraphic keys n a key management server. External key manager prvides supprt fr cnslidated key management and allws integratin between a PwerMax array with an already existing key management infrastructure. Fr mre infrmatin n D@RE, refer t the Dell EMC PwerMax Data at Rest Encryptin White Paper. 4.6 Drive mnitring and crrectin PwerMaxOS mnitrs media defects by bth examining the result f each data transfer and practively scanning the entire drive during idle time. If a blck is determined t be bad, the directr: Rebuilds the data in physical memry if necessary. Remaps the defective blck t anther area n the drive set aside fr this purpse. Rewrites the data frm physical memry back t the remapped blck n the drive. The directr maps arund any bad blck(s) detected, thereby aviding defects in the media. The directr als keeps track f each bad blck detected. If the number f bad blcks exceeds a predefined threshld, the 17 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

18 primary MMCS invkes a sparing peratin t replace the defective drive and then autmatically alerts Custmer Supprt t arrange fr crrective actin. 18 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

19 5 InfiniBand fabric switch Multi-engine PwerMax 8000 systems emply tw 18-prt Infiniband fabric switches t carry cntrl, metadata, and user data thrugh the system. This technlgy cnnects all f the engines in the system t prvide a pwerful frm f redundancy and perfrmance. This allws the engines t share resurces and act as a single entity while cmmunicating. Fr redundancy, each directr has a cnnectin t each switch. Each switch has redundant, ht pluggable pwer supplies. Figure 13 and Figure 14 shw the frnt and rear views f the InfiniBand switches. Figure 13 Frnt view f InfiniBand switch Figure 14 Rear view f InfiniBand switch Nte: Since the purpse f the dynamic virtual matrix is t create a cmmunicatin intercnnectin between all f the engines, single-engine systems and dual-engine PwerMax 2000 systems d nt require a fabric switch. 19 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

20 6 Redundant pwer subsystem A mdular pwer subsystem features a redundant architecture that facilitates field replaceme nt f any f its cmpnents withut any interruptin in prcessing. The pwer subsystem has tw pwer znes fr redundancy. Each pwer zne cnnects t a separate dedicated r islated AC pwer line. If AC pwer fails n ne zne, the pwer subsystem cntinues t perate thrugh the ther pwer zne. If any single pwer supply mdule fails, the remaining pwer supplies cntinue t share the lad. PwerMaxOS senses the fault and reprts it as an envirnmental errr. Each directr is cnfigured with a management mdule that prvides lw-level, system-wide cmmunicatins and envirnmental cntrl fr running applicatin sftware, mnitring, and diagnsing the system. The management mdules are respnsible fr mnitring and reprting any envirnmental issues, such as pwer, cling, r cnnectivity prblems. Envirnmental infrmatin is carried thrugh tw redundant Ethernet switches. Each management mdule cnnects t ne switch, except fr the MMCS mdules in Engine 1 which cnnect t bth Ethernet switches. Management mdule A cnnects t Ethernet switch A, and management mdule B cnnects t Ethernet switch B. Each management mdule als mnitrs ne f the system standby pwer supplies (SPS) thrugh an RS232 cnnectin. Standard PwerMax 8000 racks have LED bars that are cnnected t the management mdules and are used fr system/bay identificatin during service activities. Figure 15 illustrates management mdule cnnectivity. LED Bar Figure 15 Management mdule cnnectivity The internal Ethernet cnnectivity netwrk mnitrs and lgs envirnmental events acrss all critical cmpnents and reprts any peratinal prblems. Critical cmpnents include directr bards, glbal memry, pwer supplies, pwer line input mdules, fans, and varius n/ff switches. This netwrk s envirnmental cntrl capability is able t mnitr each cmpnent s lcal vltages, ensuring ptimum pwer delivery. Temperature f directr bards and memry are als cntinuusly mnitred. Failing cmpnents can be detected and replaced befre a failure ccurs. The AC pwer main is checked fr the fllwing: 20 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

21 AC failures Pwer lss t a single pwer zne DC failures Current sharing between DC supplies DC utput vltage Specific ntificatin f vervltage cnditin Current frm each DC supply Vltage drps acrss majr cnnectrs Figure 16 illustrates the internal Ethernet cnnectivity. Figure 16 Internal Ethernet cnnectivity 6.1 Vaulting As cache size has grwn, the time required t mve all cached data t a persistent state has als increased. Vaulting is designed t limit the time needed t pwer ff the system if it needs t switch t a battery supply. Upn cmplete system pwer lss r transitining a system t an ffline state, PwerMaxOS perfrms a vault f cache memry t dedicated I/O mdules knwn as flash I/O mdules. The flash I/O mdules use NVMe technlgy t safely stre data in cache during the vaulting sequence. Lithium-in standby pwer supply (Li-In SPS) mdules prvide battery backup functinality during the vault peratin. Tw SPS mdules are cnfigured per engine. The SPS mdules als prvide back-up pwer t the InfiniBand switches in applicable cnfiguratins Vault triggers State changes that require the system t vault are referred t as vault triggers. There are tw types f vault triggers: internal availability triggers and external availability triggers Internal availability triggers Internal availability triggers are initiated when glbal memry data becmes cmprmised due t cmpnent unavailability. Once these cmpnents becme unavailable, the system triggers the Need t Vault (NTV) state, and vaulting ccurs. There are three internal triggers: Vault flash availability The NVMe flash I/O mdules are used fr strage f metadata under nrmal cnditins, as well as string any data that is being saved during the vaulting prcess. PwerMax systems 21 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

22 can withstand failure and replacement f flash I/O mdules withut impact t prcessing. Hwever, if the verall available flash space in the system is reduced t the minimum t be able t stre the required cpies f glbal memry, the NTV prcess triggers. This is t ensure that all f the data is saved befre a ptential further lss f vault flash space ccurs. Glbal memry (GM) availability When any f the mirrred directr pairs are bth unhealthy either lgically r envirnmentally, NTV triggers because f GM unavailability. Fabric availability When bth the fabric switches are envirnmentally unhealthy, NTV triggers because f fabric unavailability External availability triggers External availability triggers are initiated under circumstances when glbal memry data is nt cmprmised, but it is determined that the system preservatin is imprved by vaulting. Vaulting in this cntext is used as a mechanism t stp hst activity, facilitate easy recvery, r act as an attempt t practively take actin t prevent ptential data lss. There are three external triggers: Input pwer If pwer is lst t bth pwer znes, the system vaults. Engine trigger If an entire engine fails, the system vaults. DAE trigger If the system has lst access t the whle DAE r DAEs, including dual-initiatr failure, and lss f access causes cnfigured RAID members t becme nn-accessible, the system vaults. 6.2 Pwer-dwn peratin When a system is pwered dwn r transitined t ffline, r when envirnmental cnditins trigger a vault situatin, a vaulting prcedure ccurs. First, the part f glbal memry that is saved reaches a cnsistent image (n mre writes). The directrs then write the apprpriate sectins f glbal memry t the flash I/O mdules, saving multiple cpies f the lgical data. The SPS mdules maintain pwer t the system during the vaulting prcess fr up t 5 minutes. 6.3 Pwer-up peratin During pwer-up, the data is written back t glbal memry t restre the system. When the system is pwered-n, the startup prgram des the fllwing: Initializes the hardware and the envirnmental system Restres the glbal memry frm the saved data while checking the integrity f the data. This is accmplished by taking sectins frm each cpy f glbal memry that was saved during the pwer-dwn peratin and cmbining them int a single cmplete cpy f glbal memry. If there are any data integrity issues in a sectin f the first cpy that was saved, then that sectin is extracted frm the secnd cpy during this prcess. Perfrms a cleanup, data structure integrity, and initializatin f needed glbal memry data structures 22 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

23 At the end f the startup prgram, the system resumes nrmal peratin when the SPS mdules are recharged enugh t supprt anther vault peratin. If any cnditin is nt safe, the system des nt resume peratin and calls Custmer Supprt fr diagnsis and repair. In this state, Dell EMC Custmer Supprt can cmmunicate with the system and find ut the reasn fr nt resuming nrmal peratin. 23 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

24 7 Remte Supprt Remte supprt is an imprtant and integral part f Dell EMC Custmer Supprt. Every PwerMax system has tw integrated Management Mdule Cntrl Statins (MMCS) that cntinuusly mnitr the PwerMax envirnment. The MMCS mdules can cmmunicate with the Custmer Supprt Center thrugh a netwrk cnnectin t the EMC Secure Remte Supprt (ESRS) Gateway. Thrugh the MMCS, the system actively mnitrs all I/O peratins fr errrs and faults. By tracking these errrs during nrmal peratin, PwerMaxOS can recgnize patterns f errr activity and predict a ptential hard failure befre it ccurs. This practive errr tracking capability can ften prevent cmpnent failures by fencing ff, r remving frm service, a suspect cmpnent befre a failure ccurs. T prvide remte supprt capabilities, the system is cnfigured t call hme and alert Dell EMC Custmer Supprt f a ptential failure. An authrized Dell EMC Technical Supprt Engineer can run system diagnstics remtely fr further trubleshting and reslutin. Cnfiguring Dell EMC prducts t allw inbund cnnectivity als enables Dell EMC Custmer Supprt t practively cnnect t the systems t gather needed diagnstic data r t attend t identified issues. The current cnnect-in supprt prgram fr the system uses the latest digital key exchange technlgy fr strng authenticatin, layered applicatin security, and a centralized supprt infrastructure that places calls thrugh an encrypted tunnel between Custmer Supprt and the MMCS lcated inside the system. Befre anyne frm Custmer Supprt can initiate a cnnectin t a system at the custmer site, that persn must be individually authenticated and determined t be an apprpriate member f the Custmer Supprt team. Field-based persnnel wh might be knwn t the custmer must still be prperly assciated with the specific custmer s accunt. An essential part f the design f the cnnectivity supprt prgram is that the cnnectin must riginate frm ne f several specifically designed Remte Supprt Netwrks at Dell EMC. Within each f thse Supprt Centers, the necessary netwrking and security infrastructure has been built t enable bth the call-hme and call-device functins. 7.1 Supprtability thrugh the Management Mdule Cntrl Statin Each PwerMax system has tw management mdule cntrl statins (MMCS) in the first engine f each system (ne per directr). The MMCS cmbines the management mdule and cntrl statin (service prcessr) hardware int a single mdule. It prvides envirnmental mnitring capabilities fr pwer, cling, and cnnectivity. Each MMCS mnitrs ne f the system standby pwer supplies (SPS) thrugh an RS232 cnnectin. Each MMCS is als cnnected t bth internal Ethernet switches within the system as part f the internal cmmunicatins and envirnmental cntrl system. The MMCS als prvides remte supprt functinality. Each MMCS cnnects t the custmer s lcal area netwrk (LAN) t allw mnitring f the system, as well as remte cnnectivity fr the Dell EMC Custmer Supprt team. Each MMCS can als be cnnected t an external laptp r KVM surce. The MMCS lcated in directr 1 is knwn as the primary MMCS, and the MMCS lcated in directr 2 is knwn as the secndary MMCS. The primary MMCS prvides all cntrl statin functinality when it is perating nrmally, while the secndary MMCS prvides a subset f this functinality. If the primary MMCS fails, the 24 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

25 secndary MMCS is put in an elevated secndary state, which allws mre functinality fr the duratin f this state. Bth MMCS are cnnected t the custmer netwrk, giving the system the redundant ability t reprt any errrs t Dell EMC Custmer Supprt, as well as allwing Dell EMC Custmer Supprt t cnnect t the system remtely. The MMCS is used in the fllwing supprt and maintenance tasks: PwerMaxOS upgrade prcedures Hardware upgrade prcedures Internal scheduler tasks that mnitr the health f the system Errr cllectin, lgging, and reprting thrugh the call-hme feature Remte cnnectivity and trubleshting by Dell EMC Custmer Supprt Cmpnent replacement prcedures The MMCS als cntrls the LED bars n the frnt and back f each standard PwerMax 8000 rack. These can be used fr system identificatin purpses by remte and n-site Dell EMC service persnnel. Figure 17 illustrates MMCS cnnectivity. LED Bar Figure 17 MMCS cnnectivity 7.2 Secure Service Credential (SSC), secured by RSA The Secure Service Credential technlgy applies exclusively t service prcessr activities and nt hstinitiated actins n array devices. These service credentials describe wh is lgging in, the capabilities they have, a time frame that the credential is gd fr, and the auditing f actins the service persnnel perfrmed which can be fund in the symaudit lgs. If these credentials are nt validated, the user cannt lg in t the MMCS r ther internal functins. SSC cvers bth n-site and remte lgin. Sme f the security features are transparent t the custmer, such as service access authenticatin and authrizatin by Dell EMC Custmer Supprt and SC (user ID infrmatin) restricted access (MMCS and Dell EMC Custmer Supprt internal functins). Access is definable at a user level, nt just at a hst level. All user ID infrmatin is encrypted fr secure strage within the array. 25 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

26 MMCS-based functins hnr Slutins Enabler Access Cntrl settings per authenticated user in rder t limit view/cntrl f nn-wned devices in shared envirnments such as SRDF-cnnected systems. 26 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

27 8 Cmpnent-level serviceability PwerMax systems prvide full cmpnent-level redundancy t prtect against a cmpnent failure and ensure cntinuus and uninterrupted access t infrmatin. This nn-disruptive replacement capability allws the Custmer Supprt Engineer t install a new cmpnent, initialize it if necessary, and bring it nline withut stpping system peratin, taking unaffected channel paths ffline, r pwering the unit dwn. A mdular design imprves serviceability by allwing nn-disruptive cmpnent replacements, shuld a failure ccur. This lw parts cunt minimizes the number f failure pints. PwerMax systems feature nn-disruptive replacement f all majr cmpnents, including: Engine cmpnents: Directr bards I/O Mdules Fibre Channel (frnt-end) Embedded NAS (enas) PCIe (back-end) Flash (Vault) SRDF Cmpressin Inline Cmpressin/Deduplicatin Fabric Management mdules/management mdule cntrl statins Pwer supplies Fans Drive Array Enclsure (DAE) cmpnents: NVMe drives Link Cntrl Cards (LCC) Pwer supplies PCIe cables Cabinet Cmpnents InfiniBand switches Ethernet switches Standby Pwer Supplies (SPS) Pwer Distributin Units (PDU) 8.1 Dell EMC internal QE testing Dell EMC s Quality Engineering (QE) Teams perfrm thrugh testing f all FRUs. Each FRU is tested multiple times fr each cde level with very specific pass/fail criteria. Standard tests perfrm verificatin f the GUI-based scripted replacement prcedures that are used by Dell EMC field persnnel. The tests are designed t verify the replaceability f each FRU withut any adverse effects n the rest f the system, and t verify the functinality and ease-f-use f the scripted prcedures. These tests are straightfrward replacement prcedures perfrmed n peratinal cmpnents. 27 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

28 Nn-standard tests are als perfrmed n cmpnents that have failed either by errr injectin r ht remval f the cmpnent r its pwer surce. These tests als incrprate negative testing by intentinally causing different failure scenaris during the replacement prcedure. Please nte that remving a drive ht will nt cause sparing t invke. This behavir is ptimal as the system knws the device has nt gne bad. The crrect curse f actin is t recver the drive rather than g thrugh needless sparing and full rebuild prcesses. Negative tests are designed t make sure that the replacement prcedure prperly detects the errr and that the rest f the system is nt affected. Sme examples f negative tests are: Replacing the wrng cmpnent Replacing cmpnent with an incmpatible cmpnent Replacing cmpnent with a faulty cmpnent Replacing cmpnent with a new cmpnent that has lwer cde that needs t be upgraded Replacing cmpnent with a new cmpnent that has higher cde that needs t be dwngraded Replacing cmpnent with the same cmpnent and make sure script detects and alerts the user that the same cmpnent is being used Imprperly replacing a cmpnent (miscabled, unseated, etc) Initiating a system vault save (system pwer lss) peratin during a replacement prcedure Bth the standard and nn-standard tests are perfrmed n all system mdels and varius cnfiguratins with custmer-like wrklads running n the array. Tests are als perfrmed repeatedly t verify there are n residual issues left unreslved that culd affect subsequent replacements f the same r different cmpnent(s). Cmpnents that are knwn t fail mre frequently in the field, drives fr example, as well as cmplex cmpnent replacements, are typically tested mre frequently. 28 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

29 9 Nndisruptive PwerMaxOS upgrades Interim updates f PwerMaxOS can be perfrmed remtely by the Remte Change Management (RCM) grup. These updates prvide enhancements t perfrmance algrithms, errr recvery and reprting techniques, diagnstics, and PwerMaxOS fixes. They als prvide new features and functinality fr PwerMaxOS. During an nline PwerMaxOS cde lad, a member f the RCM team dwnlads the new PwerMaxOS cde t the MMCS. The new PwerMaxOS cde lads int the EEPROM areas within the directrs, and remains idle until requested fr a ht lad in the cntrl stre. The system lads executable PwerMaxOS cde within each directr hardware resurce until all directrs are laded. Once the executable PwerMaxOS cde is laded, internal prcessing is synchrnized and the new cde becmes peratinal. The system des nt require custmer actin during the perfrmance f this functin. All directrs remain nline t the hst prcessr, thus maintaining applicatin access. 29 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

30 10 TimeFinder and SRDF replicatin sftware 10.1 Lcal replicatin using TimeFinder TimeFinder sftware delivers pint-in-time cpies f vlumes that can be used fr backups, decisin supprt, data warehuse refreshes, r any ther prcess that requires parallel access t prductin data. TimeFinder SnapVX is highly-scalable, highly-efficient, and easy t use. SnapVX prvides very lw impact snapshts and clnes fr data vlumes. SnapVX supprts up t 256 snapshts per surce vlume, which are tracked as versins with less verhead and simple relatinship tracking. Users can assign names t their snapshts, and have the ptin f setting autmatic expiratin dates n each snapsht. SnapVX prvides the ability t manage cnsistent pint-in-time cpies fr strage grups with a single peratin. Up t 1024 target vlumes can be linked per surce vlume, prviding read/write access as pinter-based r full cpies. Users can als create secure snapshts that prevent a snapsht frm being terminated until a specified retentin time has been reached. Fr mre infrmatin n TimeFinder SnapVX, refer t Dell EMC PwerMaxOS TimeFinder Lcal Replicatin Technical Ntes Remte replicatin using SRDF Symmetrix Remte Data Facility (SRDF) slutins prvide industry-leading disaster recvery and data mbility slutins. SRDF replicates data between 2, 3 r 4 arrays lcated in the same rm, n the same campus, r thusands f kilmeters apart. SRDF synchrnus (SRDF/S) Maintains a real-time cpy at arrays lcated within 200 kilmeters. Writes frm the prductin hst are acknwledged frm the lcal array when they are written t cache at the remte array. SRDF asynchrnus (SRDF/A) Maintains a dependent-write, cnsistent cpy at arrays lcated at unlimited distances. Writes frm the prductin hst are acknwledged immediately by the lcal array. Thus replicatin has n impact n hst perfrmance. Data at the remte array is typically nly secnds behind the primary site. SRDF disaster recvery slutins use active remte mirrring and dependent-write lgic t create cnsistent cpies f data. Dependent-write cnsistency ensures transactinal cnsistency when the applicatins are restarted at the remte lcatin. SRDF can be tailred t meet varius Recvery Pint Objectives/Recvery Time Objectives. SRDF can be used t create cmplete slutins t: 30 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

31 Create real-time (SRDF/S) r dependent-write-cnsistent (SRDF/A) cpies at 1, 2, r 3 remte arrays. Mve data quickly ver extended distances. Prvide 3-site disaster recvery with; Business cntinuity Zer data lss Disaster restart SRDF integrates with ther Dell EMC prducts t create cmplete slutins t: Restart peratins after a disaster with: Business cntinuity Zer data lss Restart peratins in clustered envirnments. Fr example, Micrsft Cluster Server with Micrsft Failver Clusters. Mnitr and autmate restart peratins n an alternate lcal r remte server. Autmate restart peratins in VMware envirnments Cascaded SRDF and SRDF/Star supprt Cascaded SRDF cnfiguratins use 3-site remte replicatin with SRDF/A mirrring between sites B and C, delivering additinal disaster restart flexibility. Figure 18 shws an example f a Cascaded SRDF slutin. Figure 18 Cascaded SRDF SRDF/Star is cmmnly used t deliver the highest resiliency in disaster recvery. SRDF/Star is cnfigured with three sites enabling resumptin f SRDF/A with n data lss between the tw remaining sites, prviding cntinuus remte data mirrring and preserving disaster-restart capabilities. Figure 19 shws examples f Cascaded and Cncurrent SRDF/Star slutins. 31 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

32 Figure 19 SRDF/Star SRDF/Metr supprt SRDF/Metr significantly changes the traditinal behavir f SRDF Synchrnus mde with respect t the remte (R2) device availability t better supprt hst applicatins in high-availability envirnments. With SRDF/Metr, the SRDF R2 device is read/write accessible t the hst and takes n the federated (such as gemetry and device WWN) persnality f the primary R1 device. By prviding this federated persnality n the R2 device, bth R1 and R2 devices then appear as a single virtual device t the hst. With bth the R1 and R2 devices being accessible, the hst r hsts (in the case f a cluster) can read and write t bth R1 and R2 devices with SRDF/Metr ensuring that each cpy remains current, cnsistent, and addressing any write cnflicts that may ccur between the paired SRDF devices. Figure 20 shws examples f SRDF/Metr slutins. Figure 20 SRDF/Metr 32 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

33 On the left is an SRDF/Metr cnfiguratin with a standalne hst that has read/write access t bth arrays (R1 and R2 devices) using multi-pathing sftware such as PwerPath. This is enabled by federating the persnality f the R1 device t ensure that the paired R2 device appears, thrugh additinal paths t the hst, as a single virtualized device. On the right is a clustered hst envirnment where each cluster nde has dedicated access t an individual array. In either case, writes t the R1 r R2 devices are synchrnusly cpied t its SRDF paired device. Shuld a cnflict ccur between writes t paired SRDF/Metr devices, the cnflicts are internally reslved t ensure a cnsistent image between paired SRDF devices is maintained t the individual hst r hst cluster. SRDF/Metr may be selected and managed thrugh Slutins Enabler, Unisphere fr PwerMax, and REST API. SRDF/Metr requires a separate license n bth arrays t be managed. Fr mre infrmatin n SRDF, refer t the Dell EMC PwerMax Family Prduct Guide, and Intrductin t SRDF/Metr White Paper. 33 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper

34 11 Unisphere fr PwerMax System Health Check Unisphere fr PwerMax has a system health check prcedure that interrgates the health f the array hardware. The prcedure checks varius aspects f the system and reprts the results as either pass r fail. The results are reprted at a high level with the intent f either telling the user that there are n hardware issues present, r that issues were fund and the user shuld cntact Dell EMC Custmer Supprt fr further investigatin. The health check prcedure is accessed frm the System Health Dashbard as Figure 21 shws. Figure 21 Unisphere System Health Dashbard The test takes several minutes t cmplete. When cmplete, clicking the Run Health Check link displays test results in the frmat shwn in Figure 22: 34 Dell EMC PwerMax Reliability, Availability, and Serviceability Technical White Paper