EMC VPLEX METRO CONTINUOUS AVAILABILITY AND CLUSTER WITNESS

Transcription

1 EMC VPLEX METRO CONTINUOUS AVAILABILITY AND CLUSTER WITNESS IMPLEMENTATION AND DESIGN GUIDE ABSTRACT This technical nte is targeted fr EMC field persnnel, partners, and custmers wh will be cnfiguring, installing, supprting and managing EMC VPLEX Metr fr Cntinuus Availability. This dcument is designed t shw all users hw VPLEX Metr is deplyed with and withut VPLEX Witness and explains hw t achieve seven 9 s availability thrugh prper cnfiguratin. February 2015 EMC WHITE PAPER

2 T learn mre abut hw EMC prducts, services, and slutins can help slve yur business and IT challenges, cntact yur lcal representative r authrized reseller, visit r explre and cmpare prducts in the EMC Stre Cpyright 2014 EMC Crpratin. All Rights Reserved. EMC believes the infrmatin in this publicatin is accurate as f its publicatin date. The infrmatin is subject t change withut ntice. The infrmatin in this publicatin is prvided as is. EMC Crpratin makes n representatins r warranties f any kind with respect t the infrmatin in this publicatin, and specifically disclaims implied warranties f merchantability r fitness fr a particular purpse. Use, cpying, and distributin f any EMC sftware described in this publicatin requires an applicable sftware license. Fr the mst up-t-date listing f EMC prduct names, see EMC Crpratin Trademarks n EMC.cm. Part Number H

3 TABLE OF CONTENTS PREFACE... 4 AUDIENCE... 4 RELATED PUBLICATIONS FOR REFERENCE... 4 VPLEX METRO DEPLOYMENT SCENARIOS... 5 PLANNED APPLICATION MOBILITY... 5 DISASTER RESTART... 6 VPLEX METRO ARCHITECTURE... 7 VPLEX METRO... 7 VPLEX WITNESS... 7 FOUNDATIONS OF VPLEX METRO BEHAVIOR AND CONSISTENCY... 8 CONSISTENCY GROUPS... 9 FAILURE HANDLING WITHOUT VPLEX WITNESS FAILURE HANDLING WITH VPLEX WITNESS THE A-HA! MOMENT APPENDIX CLI EXAMPLE OUTPUTS

4 Preface This EMC Engineering technical nte describes and prvides an insightful discussin n hw implementatin f VPLEX Metr with the inclusin f the VPLEX Witness will give custmers seven 9 s availability and assurance f business cntinuity they expect. As part f an effrt t imprve and enhance the perfrmance and capabilities f its prduct lines, EMC peridically releases revisins f its hardware and sftware. Therefre, sme functins described in this dcument may nt be supprted by all versins f the sftware r hardware currently in use. Fr the mst up-t-date infrmatin n prduct features, refer t yur prduct release ntes. If a prduct des nt functin prperly r des nt functin as described in this dcument, please cntact yur EMC representative. AUDIENCE This white paper is intended fr the fllwing readers: EMC Pre-Sales Organizatin fr utlining and describing the architecture fr their custmers prir t purchase. EMC Glbal Services Applicatin Supprt fr effectively intrducing the prduct int the envirnment and assuring that the implementatin is specifically riented t the custmers needs and negates any pssible DU/DL and/r applicatin failure r misunderstanding f such failures. EMC custmers interested in deplying VPLEX r have deplyed VPLEX and need a slid understanding f hw VPLEX Metr and VPLEX Witness behaves under different cnditins EMC Supprt reference when issues d get reprted, s that they can be quickly triaged under nrmal cnditins as described in the technical playbk. It is expected this dcument be shared with custmers as a tl fr guiding right decisins while implementing EMC VPLEX Metr fr Cntinuus Availability. Readers shuld be familiar with VPLEX as a prduct, has previus training and are respnsible fr the success f prduct implementatin. RELATED PUBLICATIONS FOR REFERENCE The fllwing dcuments are lcated n EMC.COM and shuld be used as additinal reference depending n envirnment and applicatin fcus: EMC VPLEX AND VMWARE TECHNICAL CONTINUOUS AVAILABILITY FAILURE HANDLING EMC OVERVIEW AND GENERAL BEST PRACTICES EMC VPLEX SAN CONNECTIVITY EMC VPLEX HOST MULTIPATHING EMC VPLEX METRO CROSS-CONNECT HOST CLUSTERING ORACLE REAL APPLICATION CLUSTERS (RAC) ON EXTENDED DISTANCE CLUSTERS WITH EMC VPLEX METRO BEST PRACTICES PLANNING 4

5 VPLEX Metr Deplyment Scenaris VPLEX Metr 5.0 (and abve) intrduced high availability cncepts beynd what is traditinally knwn as physical high availability. Intrductin f the VPLEX Witness t a high availability envirnment, allws the VPLEX slutin t increase the verall availability f the envirnment by arbitrating a pure cmmunicatin failure between tw primary sites and a true site failure in a multi-site architecture. EMC VPLEX is the first prduct t bring t market the features and functinality prvided by VPLEX Witness prevents failures and asserts the activity between clusters in a multi-site architecture. Thrugh this technical nte, administratrs and custmers gain an understanding f the high availability slutin that VPLEX prvides them: VPLEX witness is a game changer t the way cntinuus availability is achieved Active/active use f bth f their data centers Increased availability fr their applicatins (n single pints f strage failure, aut-restart) Fully autmatic failure handling Better resurce utilizatin Lwer CapEx and lwer OpEx as a result Bradly speaking, when ne cnsiders legacy envirnments ne typically sees highly available designs r active/active applicatins implemented within a data center, and disaster recvery r replicatin type functinality deplyed between data centers. One f the main reasns fr this is that within data centers cmpnents generally perate in active/active with autmatic failver whereas between data centers legacy replicatin technlgies use active/passive techniques which require manual failver t use the passive cmpnent. When using VPLEX Metr active/active replicatin technlgy in cnjunctin with VPLEX Witness, the lines between lcal high availability and lng distance disaster recvery are nw cmbined since it enables High Availability (HA) applicatins t be stretched beynd the data center walls. Since the riginal publicatin f this technical nte (frmerly, TechBk), VPLEX Metr implementatins have becme the mst implemented ptin fr custmers using VPLEX. As custmers deply Metr, it is indicative and required the VPLEX Witness be installed t enable all ut cntinuus availability cnsumers t achieve seven 9 s availability. PLANNED APPLICATION MOBILITY An nline planned applicatin mbility event is defined as when clustered applicatins r virtual machines can be mved fully nline withut disruptin frm ne lcatin t anther in either the same r remte data center. This type f mvement can nly be perfrmed when all cmpnents that participate in this mvement are available (e.g., the running state f the applicatin r VM exists in vlatile memry which wuld nt be the case if an active site has failed) and if all participating hsts have read/write access at bth lcatin t the same blck strage. Additinal a mechanism is required t transitin vlatile memry data frm ne system/hst t anther. When perfrming planned nline mbility jbs ver distance a prerequisite is the use f an active/active underlying strage replicatin slutin (VPLEX Metr nly at this publicatin). An example f this nline applicatin mbility wuld be VMware vmtin where a virtual machine wuld need t be fully peratinal befre it can be mved. It may sund bvius but if the VM was ffline then mvement culd nt be perfrmed nline (This is imprtant t understand and is the key difference ver applicatin restart). When vmtin is executed all live cmpnents that are required t make the VM functin are cpied elsewhere in the backgrund befre cutting the VM ver. Since these types f mbility tasks are ttally seamless t the user sme f the use cases assciated are fr disaster avidance where an applicatin r VM can be mved ahead f a disaster (such as, Hurricane, Tsunami, etc.) as the running state is available t be cpied, r in ther cases it can be used t enable the ability t lad balance acrss multiple systems r even data centers. Due t the need fr the running state t be available fr these types f relcatins these mvements are always deemed planned activities. 5

6 VPLEX Family and Use Case Overview DISASTER RESTART Disaster restart is where an applicatin r service is re-started in anther lcatin after a failure (be it n a different server r data center) and will typically interrupt the service/applicatin during the failver. A gd example f this technlgy wuld be a VMware HA Cluster cnfigured ver tw gegraphically dispersed sites using VPLEX Metr where a cluster will be frmed ver a number f ESX servers and either single r multiple virtual machines can run n any f the ESX servers within the cluster. If fr sme reasn an active ESX server were t fail (perhaps due t site failure) then the VM can be re-started n a remaining ESX server within the cluster at the remte site as the data stre where it was running spans the tw lcatins since it is cnfigured n a VPLEX Metr distributed vlume. This wuld be deemed an unplanned failver which will incur a small utage f the applicatin since the running state f the VM was lst when the ESX server failed meaning the service will be unavailable until the VM has restarted elsewhere. Althugh cmparing a planned applicatin mbility event t an unplanned disaster restart will result in the same utcme (i.e., a service relcating elsewhere) it can nw be seen that there is a big difference since the planned mbility jb keeps the applicatin nline during the relcatin whereas the disaster restart will result in the applicatin being ffline during the relcatin as a restart is cnducted. Cmpared t active/active technlgies the use f legacy active/passive type slutins in these restart scenaris wuld typically require an extra step ver and abve standard applicatin failver since a strage failver wuld als be required (i.e. changing the status f write disabled remte cpy t read/write and reversing replicatin directin flw). This is where VPLEX can assist greatly since it is active/active therefre, in mst cases, n manual interventin at the strage layer is required, this greatly reduces the cmplexity f a DR failver slutin. If best practices fr physical high available and redundant hardware cnnectivity are fllwed the value f VPLEX Witness will truly prvide custmers with Abslute availability! Other applicatins aside frm VMware that benefit frm planned and unplanned events with VPLEX Metr are Oracle RAC, Micrsft Hyper-V, RedHat Linux, Pwer HA t name sme; but all applicatins benefit while strage is accessible. In additin refer t the VPLEX EMC Simple Supprt Matrix fr up t date supprted applicatins, strage, etc. 6

7 VPLEX METRO ARCHITECTURE VPLEX METRO VPLEX Metr systems cntain tw clusters, each cluster having ne, tw, r fur engines. The clusters in a VPLEX Metr deplyment need nt have the same number f engines. Fr example, a VPLEX Metr system culd be cmpsed f ne cluster with tw engines and the ther with fur. The tw clusters f a VPLEX Metr must be deplyed within synchrnus cmmunicatin distance f each ther (abut 5-10ms f RTT cmmunicatin latency). VPLEX Metr systems are ften deplyed t span between tw data centers that are clse tgether but they can als be deplyed within a single data center fr applicatins requiring a high degree f lcal availability. VPLEX WITNESS With VPLEX Metr, VPLEX virtual vlumes can be mirrred between the VPLEX clusters, allwing a hst t have access t the data thrugh either cluster. This prvides added resiliency in the case f an entire cluster failure. In such a deplyment, n a percnsistency grup basis, ne cluster is designated as the preferred cluster fr data availability which will be explained in the next few sectins in detail. But at a high level; shuld the redundant cmmunicatin between the VPLEX clusters be lst and cnnectivity with the VPLEX Witness retained, the VPLEX Witness will indicate t the clusters that the preferred cluster shuld cntinue prviding service t the vlumes in the cnsistency grup. In this situatin, the nn-preferred cluster will stp servicing the vlumes, until such time as the link is restred, and the mirrrs are re-established. Shuld the preferred cluster f a cnsistency grup fail, the VPLEX Witness will indicate this failure t the nn-preferred cluster, which will cntinue t prvide access t the vlumes in the grup. Likewise, in the event f the failure f the nn-preferred cluster, the Witness will direct the preferred cluster t cntinue t service the vlumes. This prevents a partitin between the tw clusters frm allwing the state f the vlumes t diverge; this avids the well-knwn split-brain prblem. The use f the Witness is required fr Cntinuus Availability since it prvides zer need fr RTO f data in the presence f these failures. When the VPLEX Witness is nt deplyed, the system will suspend I/O t a vlume when that vlume s preferred cluster fail; again t be explained in sectin [VPLEX METRO WITHOUT VPLEX WITNESS]. VPLEX Witness functinality applies nly t distributed vlumes that are placed in cnsistency grups. Distributed vlumes that are nt placed in a cnsistency grup have their wn independent bias settings as determined by the administratr during initial set-up. These vlumes will have their I/O suspended when their preferred cluster fails as previusly mentined. VPLEX Witness cnsists f tw cmpnents: Cluster Witness Server a VM installed n a custmer s ESX server cnnected t bth clusters in a VPLEX Metr r Ge cnfiguratin. Cluster Witness CLI CLI cmmands t cnfigure, manage, and diagnse VPLEX Witness and its functinality. VPLEX Witness is installed as a virtual machine (VM) perating in a custmer s ESX server deplyed in a failure dmain separate frm either f the VPLEX clusters. This ensures that the VM is nt affected by faults that impact the VPLEX clusters. A failure dmain is a cllectin f entities affected by the same fault. If yu are installing the VPLEX Witness n a cnfiguratin running GeSynchrny 5.1 r later, befre deciding t install and enable VPLEX Witness, read the VPLEX Administratin Guide t understand VPLEX Witness peratin in a VPLEX Ge r VPLEX Metr envirnment. It is very imprtant t deply the VPLEX Witness int a failure dmain that is independent f each f the failure dmains cntaining the tw VPLEX clusters, t ensure that a single failure impacts n mre than ne f these entities. Custmers wh have mre than (2) failure dmains shuld perate VPLEX Witness n-prem. Custmers have and will ask if it is pssible t deply VPLEX Witness at a service prvider ff-prem r clud based VM. The answer is yes, supprted clud-based deplyments will becme available ver time and supprted by EMC in future release cycles. 7

8 VPLEX Witness cnnects t bth VPLEX clusters ver a VPN tunnel ver the IP management netwrk. Lw bandwidth health-check heartbeats are used t bserve the status f bth clusters. VPLEX Witness recnciles the status reprted by the clusters with its wn bservatins and prvides guidance back t the clusters, if necessary. VPLEX Witness is applicable nly t VPLEX Metr cnfiguratins1. Befre the Cluster Witness Server VM is deplyed, VPLEX sftware that supprts the VPLEX Witness must be installed first. Ensure that yu fllw VPLEX installatin instructins fr Release 5.0 r later befre deplying and cnfiguring VPLEX Cluster Witness Server VM. Figure 1. Standard EMC VPLEX Metr Envirnment FOUNDATIONS OF VPLEX METRO BEHAVIOR AND CONSISTENCY Where des cnsistency begin in the VPLEX Metr envirnment? It begins with the hardware redundancy, apprpriatin f paths; the implementatin f multipath capabilities and f curse applicatin lad balancing relatin t writes and reads prperly thrugh cache cherency intelligence. Befre ne implements Metr, an understanding f hw Metr cache wrks is essential t ptimize the envirnment and lad balance efficiently reads and writes but als highlights why Witness is such a big part f the whle entity that is this CA slutin. The individual memry systems f each VPLEX directr are cmbined t frm the VPLEX distributed cache. Data structures within these memries in cmbinatin with distributed algrithms achieve the cherency and cnsistency guarantees prvided by VPLEX virtual strage. This guarantee ensures that the I/O behavir bserved by hsts accessing VPLEX strage is cnsistent with the behavir f a traditinal disk. The VPLEX distributed algrithms are designed t minimize inter-directr messaging and take advantage f I/O lcality in the placement f key data structures. 1 VPLEX Ge is supprted in additin hwever fr purpses f this technical nte, nt cvered 8

9 The design is truly distributed: Any directr within a cluster is able t service an I/O request fr a virtual vlume served by that cluster. Each directr within a cluster is expsed t the same set f physical strage vlumes frm the back-end arrays and has the same virtual-t-physical strage mapping metadata fr its vlumes. The distributed design extends acrss a VPLEX Metr system t prvided cache cherency and cnsistency fr the glbal system. This ensures that a hst accesses t a distributed vlume always receive the mst recent cnsistent data fr that vlume. This increases speed fr read intensive applicatins and assures that writes are being driven t the nearest path in the Initiatr Target Lun (ITL) relatinship. VPLEX Metr uses the write-thrugh cache mde. (Lcal des as well, but we fcus this tpic n Metr itself) With write-thrugh caching as a write request is received frm a hst t a virtual vlume, the data is written thrugh t the back-end strage vlume(s) that map t the vlume. When the array(s) acknwledge this data, an acknwledgement is then sent back frm VPLEX t the hst indicating a successful write. This prvides an especially strng guarantee f data durability in the case f a distributed mirrr where the back-end strage vlumes supprting the mirrr can be placed in different data centers. Business cntinuity is extended by the VPLEX Lcal RAID 1 technlgy, allwing applicatins t cntinue prcessing in the presence f array failures and maintenance peratins. The distributed RAID 1 technlgy extends this prtectin further, allwing clustered active/active applicatins t leverage the capabilities f VPLEX t ride thrugh site disasters as previusly assured. The distributed RAID 1 features cupled with VPLEX s distributed cherent cache is the cre technlgies that prvides the fundatin f distributed mbility, availability, and cllabratin acrss distance. If yu think abut hw distributin with a single cluster behaves and interdirectr messaging ccurs fr cherent messaging based n writes and reads; the same applies when a DR1 is created s glbal systems as described als ccurs acrss distance. Stability and a level f respnsibility are wed t this underlying infrastructure. Yu the cnsumer OWE yur data a VPLEX Witness t be implemented. CONSISTENCY GROUPS As previusly mentined, in rder fr VPLEX Witness t maintain cnsistency fr VPLEX Metr all virtual vlumes need t be placed int cnsistency grups. These can be carved up hwever is lgical fr the custmer but in general, grupings accrd t applicatins, IOPs and characteristics f the data shuld be cnsidered. DR1s will benefit frm having like array characteristics in additin as VPLEX respnse t IO frm B/E will always be a fast as the slwest link. Each cnsistency grup will have prperties applied t it in the cntext f preference rules. These rules are: Winner (Site A) - any cnsistency grups that have this marking will retain activity shuld there be an utage at nn-preferred site withut witness Winner (Site B) any cnsistency grups that have this marking will retain activity shuld there be an utage at nn-preferred site withut witness N Autmatic Winner there is n winner, suspensin ccurs at bth sides withut witness Essentially, the architecture defined by having n VPLEX witness is still an Active/Passive type f envirnment hwever the dependency f the rules will dictate if IOs will r will nt cntinue at survival site. Once VPLEX Witness is implemented, the ntin f being at the mercy f the preferred site rule r static bias n lnger applies. 9

10 FAILURE HANDLING WITHOUT VPLEX WITNESS The fllwing sectin discusses several disruptive scenaris at a high level t a multiple site VPLEX Metr cnfiguratin withut VPLEX Witness. The purpse f this sectin is t prvide the custmer r slutins architect the ability t understand site failure semantics prir t the deplyment f VPLEX Witness. This sectin is nt designed t highlight flaws in high availability but t demnstrate what is ACHIEVABLE by intrducing witness technlgy as ppsed t mre active/passive apprach t availability. In Figure 2., fr purpses f setting example f the fllwing exercises, this shws a Metr slutin with Site A and Site B. Currently, there is n VPLEX Witness and IO is flwing actively at bth sites. Figure 2. Sample template, n VPLEX Witness High level Site A failure Suppse that a clustered applicatin was running nly in Site A at the time f the incident it wuld nw need t be restarted at the remaining Site B. Reading this dcument, yu knw this since yu have an external perspective being able t see the entire diagram. Hwever, if yu were lking at this purely frm Site B s perspective, all that culd be deduced is that cmmunicatin has been lst t Site A. Withut an external independent bserver f sme kind, it is impssible t distinguish between full Site A failure vs. the inter-cluster link failure. Nw, with VPLEX as previusly described, there are rule-sets inherently cnfigured t the vlumes. In this picture, they are applied t Site A. Therefre, IO is suspended t the winner r Bias Rules and B will suspend. Figure 3. Site utage Site A, Bias Rules Winner A utage, B suspends 10

11 High level Site B failure Suppse that a clustered applicatin was running nly in Site B at the time f the incident it wuld nw need t be restarted at the remaining Site A. Reading this dcument, yu knw this since yu have an external perspective being able t see the entire diagram. Hwever, if yu were lking at this purely frm Site A s perspective, all that culd be deduced is that cmmunicatin has been lst t Site B. Withut an external independent bserver f sme kind, it is impssible t distinguish between full Site B failure vs. the inter-cluster link failure. Hwever, because preference rules (Bias Rules Winner A) are applied t the Site A vlumes, IO will cntinue in this scenari, Site B will remain suspended. Figure 4. Site utage Site B, Bias Rules Winner A IO cntinues, B retains utage Inter-cluster link failure It is very imprtant t have redundant stable inter-cluster links. In the event that there is an inter-site link utage n the FC DWDM r IP WAN cnfiguratin, withut witness there will be suspensin n the Site B nn-preferred site, much like the Site B failure incident. In Site A, as in the Site B failure, it will cntinue t serve up IO and cntinue t perate but there will be n mre simultaneus writes ccurring until restratin f the links. Figure 5. Inter-cluster utage, all nn-preferred Site CGs suspend 11

12 FAILURE HANDLING WITH VPLEX WITNESS VPLEX Witness failure semantics As seen in the previus sectin VPLEX Witness will perate at the cnsistency grup level fr a grup f distributed devices and will functin in cnjunctin with the detach rule set (nn-preferred rule) within the cnsistency grup. Belw represents the architecture f the cnfiguratin itself. Yu can see with the dtted line a 3-way VPN is als represented. Figure 6. Metr representatin with VPLEX Witness Inter-cluster link failure with Witness If the inter-cluster link were t fail in this scenari VPLEX Witness wuld still be able t cmmunicate with bth VPLEX clusters since the management netwrk that cnnects the VPLEX Witness server t bth f the VPLEX clusters is still peratinal. The behavir f the utage in this case desn t change hwever frm Site B being suspended. The rules will still apply as cmmunicatin between the tw management servers and clusters are n lnger available. Figure 7. Inter-cluster link failure 12

13 VPLEX Witness and static preference after Cluster partitin The next example shws hw VPLEX Witness can assist if yu have a site failure at the preferred site. As discussed abve, this type f failure withut VPLEX Witness wuld cause the vlumes in the surviving site t g ffline. This is where VPLEX Witness greatly imprves the utcme f this event and remves the need fr manual interventin. Figure 8. Preferred Site A detaches Witness 91Intrductin t VPLEX Witness VPLEX Witness diagram shwing Cluster 2 failure As discussed in the previus sectin, when a site has failed then the distributed vlumes are nw degraded. Hwever, unlike ur previus example where there was a site failure at the preferred site and the static preference rule was used frcing vlumes int a suspend state at cluster 1, VPLEX Witness will nw bserve that cmmunicatin is still pssible t cluster 1 (but nt cluster 2). Additinally since cluster 1 cannt cntact cluster 2, VPLEX Witness can make an infrmed decisin and guide cluster 1 t verride the static rule set and prceed with I/O. Figure 9. Nn-preferred site utage 13

14 VPLEX Witness is lst nw what? Abslutely nthing changes, again the IO will cntinue t flw as recvery t the witness returns. If hwever, a failure t either Site A r Site B ccurs while Witness is unavailable then VPLEX perates against the nrmal preference site rules. It is imprtant t mnitr the VPLEX witness fr availability via alerts in rder t maintain its integrity. Figure 9. Lss f witness Failure Scenari Recap with VPLEX Witness 14

15 THE A-HA! MOMENT When it cmes t achieving the highest pssible f availability fr strage envirnments and respective clustered applicatins, VPLEX Metr with Cluster Witness is the best ptin fr custmer in the industry t date. The amunt f physical, lgical and cntrllable redundancy and mnitred behavir tied t cache cherency make the prduct unparalleled. The mst imprtant aspect f the implementatin as depicted in the fregrund f this paper entitles that the cnsumer is aware f the way VPLEX handles failure and what the best ptins are when cnfiguring fr critical business applicatins and sustainability f nging transactins and uptime. If yu did nt get an a-ha mment frm this nte and still need answers, re-read it and draw it ut. A fine piece f paper f bar napkin wrks great. Als, refer t dcuments mentined in the beginning f the tech nte fr mre infrmatin r cntact yur lcal EMC specialists. Witness 15

16 APPENDIX CLI EXAMPLE OUTPUTS On systems where VPLEX Witness is deplyed and cnfigured, the VPLEX Witness CLI cntext appears under the rt cntext as "cluster-witness." By default, this cntext is hidden and will nt be visible until VPLEX Witness has been deplyed by running the cluster-witness cnfigure cmmand. Once the user deplys VPLEX Witness, the VPLEX Witness CLI cntext becmes visible. The CLI cntext typically displays the fllwing infrmatin: VPlexcli:/> cd cluster-witness/ VPlexcli:/cluster-witness> ls Attributes: Name Value admin-state enabled private-ip-address public-ip-address Cntexts: cmpnents VPlexcli:/cluster-witness> ll cmpnents/ /cluster-witness/cmpnents: Name ID Admin State Operatinal State Mgmt Cnnectivity cluster-1 1 enabled in-cntact k cluster-2 2 enabled in-cntact k server - enabled clusters-in-cntact k VPlexcli:/cluster-Witness> ll cmpnents/* /cluster-witness/cmpnents/cluster-1: Name Value admin-state enabled diagnstic INFO: Current state f cluster-1 is in-cntact (last state change: 0 days, secs ag; last message frm server: 0 days, 0 secs ag.) id 1 management-cnnectivity k peratinal-state in-cntact /cluster-witness/cmpnents/cluster-2: Name Value admin-state enabledtin t VPLEX Witness diagnstic INFO: Current state f cluster-2 is in-cntact (last state change: 0 days, secs ag; last message frm server: 0 days, 0 secs ag.) 16

17 id 2 management-cnnectivity k peratinal-state in-cntact /cluster-witness/cmpnents/server: Name Value admin-state enabled diagnstic INFO: Current state is clusters-in-cntact (last state change: 0 days, secs ag.) (last time f cmmunicatin with cluster-2: 0 days, 0 secs ag.) (last time f cmmunicatin with cluster-1: 0 days, 0 secs ag.) id - management-cnnectivity k peratinal-state clusters-in-cntact Eefer t the VPLEX CLI guide fund n Pwerlink fr mre details arund VPLEX Witness CLI. 97 Intrductin t VPLEX Witness 17