CIT 668: System Architecture

Transcription

1 CIT 668: System Architecture Availability Topics 1. What is availability? 2. Measuring Availability 3. Failover 4. Failover Configurations 5. Linux HA Availability Availability is the ratio of the time a service is available for use during an interval to the length of the entire interval. Fault tolerance is the capability of a system to continue normal operation despite the presence of hardware or software faults. 1

2 Single Point of Failure A single point of failure (SPOF) is a component which will cause the entire system to fail if it fails. A fault tolerant system cannot have a single point of failure, and so has redundant components. Fault Tolerance Techniques Replication Providing multiple identical instances of the same system, querying them in parallel, and choosing the correct answer on the basis of a quorum. Redundancy Providing multiple identical instances of the same system and switching to one of the remaining instances in case of a failure (failover). Diversity Providing multiple different implementations of the same specification and using them as in replication. Load Balancing and High Availability Load Balancing High Availability 2

3 Load Balancing and High Availability Load Balancing High Availability round-robin DNS data partitioning reverse proxy wackamole heartbeat hot spare Measuring Availability A = Availability MTBF = Mean Time Between Failures MTTR = Maximum Time To Resolution Example: MTBF=1000 hours, MTTR=1 hour A = MTBF MTBF + MTTR = = 99.9% 3

4 Nines Hardware Cannot Be Reliable Enough If servers are 99% reliable, then a system with 10 servers is % reliable a system with 100 servers is % reliable ^n Google Uptime Example 4

5 Causes of Availability Loss Power Outages AWS Outages February 15, 2008: AWS Availability Event July 20, 2008: S3 Availability Event December 10, 2009: EC2 Power Outage December 12, 2010: EC2 Ireland February 26, 2011: EC2 us-east-1d 5

6 Cost of Downtime Example Downtime Costs Failover 6

7 Failover Requirements Transparency Failover should not be noticeable by clients. Speed Failover should happen quickly, so that there is only a short downtime while it occurs. Automatic Failover should not require sysadmin to intervene. Consistent Clients should see same data on failover server as they saw on the original server prior to failover. Failover Components Move from failed server to the failover server Network identity: DNS name, IP address, or MAC address must be transferred, depending on what layer of protocol stack service functions on. Data: Usually must be accomplished by a shared storage system: NAS or SAN. Processes: Server processes associated with data must be restarted once data and network identity are transferred. Heartbeat Heartbeat is the signal that failover pairs use to communicate their status with each other. Heartbeat signals are typically sent over a dedicated network between the pair, so that other network issues won t interfere. When heartbeat signals are not received by the secondary host for a specified time, it will start the failover process. 7

8 Reasons for Heartbeat Failure Primary server is down Heartbeat NIC has failed Heartbeat cable broken or removed Network congestion stops heartbeat Heartbeat service failed Remote server too slow General network failure Failover Configurations Asymmetric Failover Pair (Active/Passive) Shared Redundant Storage Physically connected, but not used until failover. Primary Server Secondary Server (idle) Heartbeat Network Production Network 8

9 Symmetric Failover Pair (Active/Active) Shared Redundant Storage Primary Server Secondary Server Heartbeat Network Production Network Complex Failover Configurations N-to-1: One dedicated failover node for a cluster of N servers. Dedicated node can perform tasks of all services. If N=1, asymmetric. N+1: Failover node is used as a normal part of cluster to improve performance until failover occurs. If N=1, same as symmetric. N+M: Having multiple (M) failover nodes instead of just one as in N+1. N-to-N: Similar to N+M, but with services redistributed among cluster nodes if a node fails to balance load. N-to-N Cluster Failover 9

10 N-to-N Cluster Failover Split Brain Cluster Shared Redundant Storage Primary Server Secondary Server Heartbeat Network Production Network Split-Brain Syndrome Servers lose contact without being down. Primary continues serving content. Secondary tries to take over primary s role. The Problem: Data corruption Both services can write to shared storage simultaneously, corrupting it. Solution: Fencing STONITH I/O reservations 10

11 Linux HA Linux HA Cluster Architecture Pacemaker Active/Passive Cluster 11

12 Pacemaker Components Resource Agents Agents abstract services provided by cluster. Classes of agents: Linux Standards Base (LSB) scripts /etc/init.d scripts: start, stop, reload Open Cluster Framework (OCF) agents /usr/lib/ocf/resource.d Configuration via environment variables See for documentation Resource Types Primitives Most services. Groups Set of resources that must be located together, started sequentially, and stopped in reverse order. Clones Resources that can be active on multiple hosts, e.g. cluster filesystems. Masters Clones with two states: master and slave. 12

13 Resource Group Example Key Points Availability is MTBF/(MTBF + MTTR) measured in 9s not Load Balancing High availability requires fault tolerance No single point of failure Failure detection (heartbeat) and failover mechanism Failover transfers Network identity Data Processes Failover configurations Asymmetric (active/passive) Symmetric (active/active) Split-brain syndrome (causes data corruption) 13