Protecting remote site data SvSAN clustering - failure scenarios

Transcription

1 White paper Protecting remote site data SvSN clustering - failure scenarios Service availability and data integrity are key metrics for enterprises that run business critical applications at multiple remote locations. To ensure applications remain available, the infrastructure (servers and storage) which they run on, is grouped or clustered together, eliminating single points of failure within the infrastructure. Clustering is primarily used to protect against unplanned downtime enabling redundant infrastructure components to take over the application/service workload, following a component failure. Equally, clustering can be used to minimize service disruption for planned downtime where IT operations are restricted, in order to perform upgrades, carry out repairs, etc. To make clustering possible, some form of shared storage is required. SvSN provides this capability using internal server storage, eliminating the need for expensive SN switches and external storage arrays. SvSN has its own clustering mechanism built-in, to ensure that the shared storage remains available while protecting the integrity of the data at all times. One scenario that exists in clustered environments and can affect both the availability and integrity of the data is split-brain. Split-brain occurs when the clustered, synchronously mirrored nodes lose contact with one another, becoming isolated. The nodes then operate independently from one another, and the data on each node diverges becoming inconsistent, ultimately leading to data corruption and potentially loss. To prevent splitbrain scenarios from occurring a quorum is used. quorum acts as an arbitrator or tiebreaker, providing a majority vote in the event of a cluster leader election process, ensuring there is only one cluster leader. If quorum cannot be determined the storage under the cluster control is taken offline, preventing data corruption. StorMagic call this quorum service the Neutral Storage Host (NSH). The : cts as an arbitrator or tie-breaker, providing a majority vote in the cluster leader election process Is a passive element of a SvSN configuration and does not service any I/O requests for data Maintains the cluster and mirror state Has the ability to provide arbitration for thousands of SvSN mirrors Can be local to the storage or at a remote location Used over a wide area network (WN) link Can tolerate high latencies and low bandwidth network links Is an optional SvSN component It is possible to have SvSN configurations that do not use a NSH however implementation best practices must be followed and this is outside the scope of this white paper The following diagram shows a typical 2-node SvSN configuration with NSH: VM VM SvSN SvSN Data Mirroring SvSN This white paper describes the more common failure scenarios and explains the expected cluster behavior during the failure and subsequent recovery. VM NSH VM StorMagic. Copyright ll rights reserved. 1

2 SvSN clustering - failure scenarios This section discusses the common failure scenarios, relating to 2-node SvSN configurations with a Neutral Storage Host. SvSN in this configuration is designed to withstand failures for a single infrastructure component. However, for some scenarios it is possible to tolerate multiple failures. Scenario #1 Network link failure between VS & NSH This scenario occurs when the network link between a single server/vs (VS1) and the NSH is interrupted, as shown in the following diagram: Each scenario describes what happens during the failure and subsequently what happens when the infrastructure is returned to the optimal state. The scenarios include: Scenario #1 Network link failure between SvSN virtual storage appliance (VS) & NSH Scenario #2 Mirror network link interruption Scenario #3 Server failure Scenario #4 NSH failure Scenario #5 Network isolation Scenario #6 Mirror network link & NSH failure Scenario #7 Server failure followed by NSH failure Scenario #8 NSH failure followed by a server failure For all the failure scenarios the following assumptions are made: The cluster/mirror leader is VS1 SvSN is in the optimum state before the failure occurs There are multiple, resilient mirror network links between servers/vss Optimum state The following diagram shows the optimum SvSN cluster state The VSs and NSH remain fully operational with the VSs continuing to serve I/O requests, without degradation to performance ll mirror targets remain synchronized ensuring that the data is fully protected and that the required service availability is maintained During the network interruption, VS1 continues to remain as the cluster leader and the quorum is maintained VS1 makes periodic attempts to connect to the NSH and recover the network connection When the network connectivity is restored: Communication between VS1 and NSH is re-established. s the network interruption did not affect the environment, operation continues as normal In the optimal cluster state ll servers, VSs, (NSH) and network links are fully operational Quorum is determined and one of the VSs (VS1) is elected the cluster leader I/O can be performed by any of the VSs Mirror state is synchronized StorMagic. Copyright ll rights reserved. 2

3 Scenario #2 Mirror network link interruption The mirror traffic network link between the servers is interrupted, as shown below: If this is a recovery from total communication failure between servers: The node with offline plexes brings them online and its mirror state will be unsynchronized The VSs will perform a fast resynchronization of the Scenario #3 Server failure This occurs when a single server (Server ) fails, as shown in the diagram below: Where there are multiple redundant network connections between VS1 and VS2: oth VSs and NSH remain fully operational and the VSs continue to serve I/O requests Mirror traffic is automatically redirected over the alternate network links if permitted During the network interruption, VS1 continues to remain as the cluster leader and the quorum is maintained The mirror state remains synchronized! Potential performance issues could arise if the alternative networks links do not have the same characteristics (speed and bandwidth) as the primary mirror network. Using alternative network links for SvSN mirror traffic could potentially affect other applications or users on the same network. oth of these are especially important when there is a high rate of change of data (e.g. full mirror re- synchronization). In the event that all network communication between VS1 and VS2 is lost (multiple failures), but they are able to communicate with the NSH: One of the mirror plexes will be taken offline to prevent split-brain from occurring and avoid data corruption or loss When the network links are recovered: Mirror traffic will automatically fail back and use the primary mirror traffic network The surviving VS (VS2) and the NSH remain fully operational VS2 is promoted to cluster leader and the NSH is updated to reflect the state change Only the surviving VS (VS2) can perform I/O Virtual machines that were running on the failed server (Server ) will be restarted on the surviving server (Server ) Virtual Machines running on Server continue to run uninterrupted The mirror state becomes unsynchronized When Server is recovered: VS1 re-joins the cluster. Its mirror state is marked as unsynchronized VS2 remains as cluster leader The VSs will perform a fast resynchronization of the On completion, the mirror state is marked as synchronized If the failure was caused by the total loss! of storage, then this will require a full resynchronization of the data The virtual machines remain running on Server Virtual machines can be moved to Server manually (vmotion/live Migration) or automatically (VMware Distributed Resource Scheduler or Microsoft Hyper-V Dynamic Optimization) StorMagic. Copyright ll rights reserved. 3

4 Scenario #4 NSH failure This occurs when the NSH fails, as shown below: oth servers (Server & Server ) remain fully operational I/O requests can be serviced by both VSs without disruption to service Quorum is maintained, with VS1 remaining as cluster leader The VSs periodically retry to connect to the NSH Mirror state remains synchronized The NSH is recovered VS1 and VS2 reconnect to the NSH Current cluster state is propagated to the NSH Scenario #5 Network isolation This scenario leads to server isolation when multiple network links fail between the servers and NSH and the server (Server ) remains operational. This is shown below: VS1 continues as normal accepting and servicing I/O requests VS1 remains as cluster leader s VS2 cannot contact either VS1 or the NSH, it identifies itself as being isolated VS2 takes its mirror plexes offline to stop updates to the storage and to prevent a split-brain condition occurring. VS1 marks VS2 mirror plexes as unsynchronized. VS2 experiences loss of quorum and has its volumes taken offline until quorum is restored. The virtual machines that were running on Server experience a H event and are restarted on Server. When the network connectivity to Server is restored: VS2 re-joins the cluster. Its mirror state is marked as unsynchronized The VSs will perform a fast resynchronization of the On completion, the mirror state is marked as synchronized The virtual machines remain running on Server Virtual machines can be moved to Server manually (vmotion/live Migraton) or automatically (VMware Distributed Resource Scheduler or Microsoft Hyper-V Dynamic Optimization) Scenario #6 Mirror network link & NSH failure This multiple failure scenario explains what happens when the mirror network link and the NSH fails. s shown in the following diagram: StorMagic. Copyright ll rights reserved. 4

5 oth servers remain online, with VS1 remaining as the cluster leader If there are other network links between the VS1 and VS2 The mirror traffic will be redirected to utilize those links and the mirror state will remain synchronized Either server can perform I/O requests oth VSs periodically poll for the presence of the NSH If all the links between the servers (Server and Server ) are severed ll storage will be immediately taken offline to prevent data corruption and split-brain scenarios occurring When the network links are recovered: The VSs negotiate leadership Storage is brought back online The VSs will perform a fast resynchronization of the On completion, the mirror state is marked as synchronized. Guest virtual machines will be restarted on the servers fter the failure of Server If VS2 was able to update the cluster state on the NSH before it failed VS2 remains online and is promoted to leader The mirror state becomes unsynchronized I/O requests are serviced by VS2 without service interruption If VS2 was NOT able to update the cluster state on the NSH before it failed VS2 takes its mirror plexes offline experiencing loss of quorum Scenario 1 - Server is recovered first, followed by the NSH VS1 re-joins the cluster; its storage will be in an unsynchronized state Mirrors are automatically resynchronized When the NSH returns to service, it is updated with the cluster and mirror state Scenario 2 - NSH is recovered first, followed by Server The NSH is updated with the cluster and mirror status VS1 re-joins the cluster Mirrors are resynchronized Scenario #7 Server failure followed by NSH failure This scenario occurs when multiple infrastructure components fail. Here one of the servers (Server ) fails followed by a subsequent failure of the NSH or failure of communication to the NSH. Scenario #8 NSH failure followed by a server failure This scenario occurs when multiple infrastructure components fail. Here the NSH or the link to the NSH fails first followed by a server (Server ) failure. StorMagic. Copyright ll rights reserved. 5

6 The remaining VS (VS2) is unable to contact either its partner server or the NSH. VS2 assumes it has become isolated and takes its mirror plexes offline to prevent data corruption Service disruption occurs - no I/O requests are serviced by the VSs Scenario 1 - Server is recovered first, followed by the NSH Servers renegotiate cluster leadership The storage is brought back online and the mirrors are resynchronized When the NSH is returned to service, it is updated with the cluster and mirror state Scenario 2 - NSH is recovered first, followed by Server VS2 elects itself as leader and brings its mirror plexes online NSH updated with the cluster and mirror status VS1 re-joins the cluster and mirrors are resynchronized Conclusion SvSN has been developed to withstand single infrastructure component failures. However, for some scenarios it is possible to tolerate multiple component failures, ensuring that service availability is maintained wherever possible. For single component failure scenarios, SvSN preserves cluster stability, avoiding split-brain conditions. When the failure is rectified, SvSN automatically recovers and returns the infrastructure back to the optimal state, performing a fast resynchronization of the mirrors where possible reducing the time frame and exposure to subsequent failures and avoiding potential service disruptions. s shown throughout this white paper, SvSN protects the integrity of the data at all costs during an infrastructure failure, while keeping the storage available. bout StorMagic Established in 2006, StorMagic is a leading provider of enterprise-class software-defined storage for one to many lean IT environments. It delivers shared storage through SvSN, a software solution which enables enterprises to eliminate downtime and ensure high availability of business critical applications at remote sites where disruptions directly equate to losses in revenue and service. StorMagic. Copyright ll rights reserved. 6