Linux-HA Version 3. Open Source Data Center April 30 th, 2009

Size: px

Start display at page:

Download "Linux-HA Version 3. Open Source Data Center April 30 th, 2009"

Abigail Francis
5 years ago
Views:

1 MultiNET Services GmbH Linux-HA Version 3 Open Source Data Center April 30 th, 2009 Dr. Michael Schwartzkopff, MultiNET Services GmbH MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 1

2 Contents Availability in general Basics and Architecture of Clusters Resources and Constraints Administration: GUI and Command Line Operation MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 2

Hardlink Sorry, only available in German. Why: ask_tim@oreilly.

3 Hardlink Sorry, only available in German. Why: Contact me: MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 3

4 What is high availability? Our IT department should offer the services always What are services? What does always mean? MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 4

5 What does always mean? The standard is 7 x 24, which is 365,2425 days/year. In real live this will only be achieved approximately. The degree of achievement is expressed in figures of % availability does not exist. Really! Samples 99% ~ 7,2 h / month not available 99,999% ~ 5,26 min / year (!) MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 5

6 Redundancy in the Linux OS: Linux-HA A standby server takes over in case of failure of the active system. This includes the takeover of IP addresses. Load Sharing: In principle no, but... Active / Passive System with two servers. Also possible: Active / Active: The resources are distributed on to the two servers. or N to M systems: The N resources are distributed on the M servers of the cluster. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 6

7 What does Linux-HA offer? No 100% availability! Cluster do protect against single failures, not against several simultaneous failures, that perhaps depend on each other. Outages will be short: From seconds to minutes. Like magic: If the hand is faster the the eye, everything is OK. Generally you can add a 9 to your basic availability. Beware: Complexity is the enemy of reliability! MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 7

8 Linux HA: Principle (I) network heartbeat IP and MAC: 1 IP and MAC: 2 Cluster IP: 3, MAC: 1 MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 8

9 Linux HA: Principle (II) network fail over heartbeat Attention: New MAC for IP 3! IP and MAC: 1 IP and MAC: 2 Cluster IP: 3, MAC: 2 MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 9

10 Well known... Starting resources through agents works like the init process. Enhancements Optional passing of parameters to the script Runs on more than one computer Rules for Order for Start / Stop of services Relationship amongst services Time based constraints for services HA systems are a little like init on steroids. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 10

11 ... and news Cluster architectures can cause problems: Split Brain Quorum Fencing The problem of shared data does not exist in single servers, but is critical in server clusters. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 11

12 Split Brain Problems in the communication between nodes can lead to separate parts of the cluster. If every part of the cluster wants to take control over the resources you have a problem. This is called Split-Brain. Imagine every node wants to start the cluster IP address. Split-Brain is really a problem and can lead to total loss of data. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 12

13 Quorum The concept of quorum is protects against the fatal results of a Split-Brain in most of the situations. The quorum does allow only one part of the cluster to control the resources. Common method is voting: Only the part of the cluster with more than N/2 of the originally nodes can host resources. Quorum doesn't work in clusters with 2 nodes. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 13

14 Fencing (I) Fencing tries prevent defective nodes to access resources like memory, network, or power. Fencing does not rely on the assumption of a correct behaviour of the defective nodes. Linux HA uses STONITH Shoot The Other Node In The Head Self-Fencing, i.e. IBM ServeRAID MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 14

15 Fencing (II) Guarantees the integrity of the whole cluster system in problematic cases. Since one part of the cluster cannot know anything about the actual state of the other part, STONITH is used to make the reality reflect the nodes imagination of reality. No communication with the other node node dead. Simple implementation: Switch off power. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 15

16 Shared data No: In many scenarios no data need to be shared between the nodes. This includes all applications that sync data amongst cluster members. Replication: In many cases a slow replication is enough. This can be achieved with rsync. Full Sync:Distributed Redundant Block Devices (DRBD). Think about RAID on two separate computers. Outsourcing of the problem: NAS, SAN MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 16

17 Linux HA: Background Development since 1998 A user survey in 2003 showed ~ systems in use. Extrapolation of the sales figures of the book leads to higher numbers. Active development by SuSE/Novell Included in the most distributions. No hardware necessary, everything runs in user space (no kernel module) MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 17

18 Linux-HA version 2: Background Version 2 introduced the Central Resource Manager (CRM) in 2005 Uses heartbeat for the cluster communication: Detection of node membership state. Communication between the cluster nodes. Utilises XML for the representation of the configuration in the Cluster Information Base (CIB): Configuration State MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 18

19 Linux-HA version 3 Version 3 in 2007 removed the CRM code from the heartbeat package into the new pacemaker project. Uses heartbeat only for the communication. But also can use OpenAIS for these tasks. Since the code is the same, configuration is the same. New, usable GUI. Future: pacemaker + OpenAIS MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 19

20 Linux-HAv2/3: Features (I) Up to 16 nodes in a cluster. Communication: Serial, UDP (bcast, mcast, ucast). Redundant lines possible. Fail over when problems occur: The node has problems. The service has problems The network has problems Others, arbitrary conditions. Distribute resources on your nodes as you like it. Build-in monitoring of the resources. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 20

21 Linux-HAv2/3: Features (II) Supports the Open Cluster Framework (OCF) standard. Complex model for constraints for the resources. Configuration XML-based. GUI for configuration and monitoring Supports OCFSv2 Multi-State (Master/Slave) resources MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 21

22 Linux-HAv2/3: Features (III) Will have clvm resource agents. XEN and KVM virtual machines managed though the virtual domain resource agent. Full support of sandboxes to test configurations. OpenAIS is the preferred cluster stack. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 22

23 Installation Included in most of the distributions. SuSE Build Service: download.opensuse.org/repositories/server:/ha clustering/ Packages for debian, fedora, mandriva, SLES, CentOS, SLES, opensuse, ubuntu heartbeat 2.99 pacemaker SLES documentation on: MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 23

24 First Configuration Files ha.cf and authkeys are necessary. Copy samples from /usr/share/doc/heartbeat to /etc/ha.d Nodes and cluster communication are defined in ha.cf: bcast eth0 autojoin any crm on Setup password in authkeys MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 24

25 The rest of the configuration is defined in the CIB File: /var/lib/heartbeat/crm/cib.xml Never edit manually! At the first system start an empty CIB is created. Display the contents of the CIB: cibadmin Q MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 25

26 The CIB The CIB contains: Information about the configuration Information about the nodes Information about the resources Information about the constraints Information about the state of the cluster: Which nodes are up / down? All attributes of the nodes Which resources run on what nodes? The administrator is only responsible for the configuration. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 26

27 Variables in the CIB The configuration in the CIB allocates values to names. Linux HA uses <nvpair> tags The syntax is: <nvpair id= uuid name= name value= value /> Other languages than XML do the same with: name = value MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 27

28 Resources: Simple resources Resources are everything that is managed by the CRM. Agents are the link between the CRM and the executable binary. You can compare them with the init scripts that are called at system start. Simple resources are called primitive resources. There also exist complex (composed) resources. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 28

29 Resource Objects Version 2 supports the following resource objects : Primitive resources: OCF (open cluster framework), heartbeat, or LSB resource agent Scripts clones: n primitive resource objects somewhere in the cluster. groups: A group of primitive resources with implicit constraints. Multi-state resources (master/slave): Represents resources that can exist in a master/slave state, i.e. (DRBD,...) MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 29

30 Simple constraints There are no constraints, which resource should run where. If the administrator does not like this default, he can configure constraints for resources: Order: A resource is started / stopped with respect to another resource. Co-location: A resource is co-located with respect to an other resource. Location: A resource should (not) run on a specific node. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 30

31 Scores The administrator assigns points for a constraint met. The CRM evaluates the constraints for every resource in the context of every node and counts the points for every node. The resource will run on that node, where is achieves the most points. In case of a tie the CRM distributes the resources even on all nodes. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 31

32 Advanced constraints Arbitrary constraints can be configured. Samples: Constraints in respect with multi state resources. Constraints that refer to attributes of the nodes (CPU load, disc free,...). Time limits for constraints. Reachability of reference points in the network. Arbitrary scriptable conditions. Fail over only after N failures of the resource. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 32

33 Operation: The GUI There exists a GUI for managing the cluster. Resources and constraints can be configured via the GUI. Good for monitoring, but no role concept. For firewalls: The GUI uses port TCP/5560. Ideal are identical versions of the GUI and pacemaker. Run with haclient.py MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 33

34 The GUI MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 34

35 Command Line Interface The most important commands are: crm_mon Display the state of the cluster. Cibadmin Interface to the CIB. crm_resource Interface to the CRM crm_failcountinterface to the failcounter ocf tester Checks a ocf resource crm_verify Checks the CIB stonith Interface to STONITH system MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 35

36 Why pacemaker? Simple configuration according to version 1 are still possible. But complex configurations also. Monitoring of the resources included in the cluster software. Dynamical modifications of resources are possible while the cluster is running. Bug fixes and support by the developers and community! MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 36

37 Monitoring of a cluster External monitoring is a MUST. Use existing network management or build new. Simple Network Management Protocol (SNMP) is the classic tool. Linux-HA comes with its own subagent that connects into net snmp. MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 37

38 Thank you very much for your attention! Application HOWTO: High Available Firewall with State Table Sync and management by the fwbuilder GUI More Information: Mail: Seminar at Heinlein-Support (Berlin) MultiNET Services GmbH, OSDC, 09/04/30: LinuxHAv2: page 38

CIT 668: System Architecture

CIT 668: System Architecture Availability Topics 1. What is availability? 2. Measuring Availability 3. Failover 4. Failover Configurations 5. Linux HA Availability Availability is the ratio of the time