HA for OpenStack: Connecting the dots

Transcription

1 HA for OpenStack: Connecting the dots Raghavan Rags Srinivas Rackspace OpenStack Meetup, Washington DC on Jan. 23 rd 2013

2 Rags Solutions Architect at Rackspace for OpenStack-based Rackspace Private Cloud Speaker at JavaOne, RSA conferences, Sun Tech Days, JUGs and other developer conferences Trying to help make OpenStack more App Developer friendly

3 Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA on Public Cloud Resources and Summary

4 OpenStack Design Tenets Scalability and elasticity are our main goals Any feature that limits our main goals must be optional Everything should be asynchronous a) If you can't do something asynchronously, see #2 All required components must be horizontally scalable Always use shared nothing architecture (SN) or sharding a) If you can't Share nothing/shard, see #2 Distribute everything a) Especially logic. Move logic to where state naturally exists. Accept eventual consistency and use it where it is appropriate. Test everything RACKSPACE HOSTING 4

5 What is HA? Minimization of system downtime Minimization of data/transaction loss In case of multiple (or interrelated) failures, minimization of data loss is preferred over minimization of system downtime HA as Nines Downtime/Year 99% (two nines) 3.65 days 99.9% 8.76 hours 99.99% minutes % 5.26 minutes % (six nines) 31.5 seconds

6 Implementing HA Elimination of Single Point of Failure (SPOFs) Redundancy of network components such as switchers and routers Redundancy of applications and automatic service migrations Redundancy of storage components Redundancy of facilities services such as power, AC, etc.

7 Components (High Level) Client VIP NODE 1 Replication Services Health Check Cluster Communication NODE 2 Replication Services Health Check Cluster Communication

8 Concepts State Description Example Stateless There is no dependency between requests No need for data replication/synchronization. Failed request may need to be restarted on a different node. Apache web server, Nova API, Nova Scheduler, etc. Stateful An action typically comprises multiple requests Data needs to be replicated and synchronized between redundant services (to preserve state and consistency) MySQL, RabbitMQ, etc.

9 More Concepts Terminology Failover Description Migration of a service from the primary to the secondary Failback Migration of service back to the primary Switchover Migration is initiated manually

10 Much more concepts Active/Passive o There is a single master o Load balance stateless services using a VIP and a load balancer such as HAProxy o For Stateful services a replacement resource can be brought online. A separate application monitors these services, bringing the backup online as necessary o After a failover the system will encounter a speed bump since the passive node has to notice the fault in the active node and become active Active/Active o Multiple masters o Load balance stateless services using a VIP and a load balancer such as HAProxy o Stateful Services are managed in such a way that services are redundant, and that all instances have an identical state o Updates to one instance of database would propagate to all other instances o After a failover the system will function in a degraded state

11 HA for OpenStack OpenStack APIs (nova, cinder, etc.) RabbitMQ MySQL Cinder, Swift, and so on Heat (still Work in Progress) Application running on OpenStack (Application dependent)

12 Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA on Public Cloud Resources and Summary

13 HA on OpenStack Overall Philosophy (Don t reinvent the wheel) Leverage time-tested Linux utilities such as Keepalived, HAProxy and Virtual IP (using VRRP) Leverage Hardware Load Balancers Leverage replication services for RabbitMQ/MySQL such as RabbitMQ Clustering, MySQL master-master replication, Corosync, Pacemaker, DRBD, Galera and so on

14 Keepalived vrrp_script rabbitmq {! }! Based on Linux Virtual Server (IPVS) kernel module providing layer 4 Load Balancing Implements a set of checkers to maintain health and Load Balancing HA is implemented using VRRP Protocol script usr/sbin/service rabbitmq-server status" # Check the service status! interval 5 weight -2 rise 2 fall -2 # check every 5 seconds! # adjust priority by -2 if OK! # required number of failures for KO switch! # required number of successes for OK switch!

15 HAProxy Load Balancing and Proxying for HTTP and TCP Applications Works over multiple connections

16 HA with Keepalived, VRRP & HAProxy Application VRRP Network Layer Host1 HAProxy Keepalived Host2 Backup Application Layer Realserver1 Realserver2

17 HA on Rackspace Private Cloud INTERNET VIP(Keepalived, VRRP) HAProxy Controller 1 Controller 2 Active-Passive Infrastructure services (MySQL, Rabbit) Active-Active Infrastructure services (API services) Heartbeat Redundant Active-Passive Infrastructure services Redundant Active-Active Infrastructure services Compute Node 1 Compute Node 2 Compute Node N VMs Instantiated

18 HA on Rackspace Private Cloud (switchover) INTERNET VIP(HAProxy) Controller 1 Controller 2 Active-Passive Infrastructure services (MySQL, Rabbit) Heartbeat Infrastructure services Compute Node 1 Compute Node 2 Compute Node N VMs Instantiated

19 Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA on Public Cloud Resources and Summary

20 RabbitMQ HA VRID Ethernet Controller 1 VRID 13 IP address: Master (Active) RabbitMQ RabbitMQ Clustering RabbitMQ Backup (Passive) Controller 2 VRID 13 IP address:

21 Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA on Public Cloud Resources and Summary

22 MYSQL HA: MASTER/MASTER REPLICATION

23 MySQL Master/Master Replication VRID Ethernet Controller 1 VRID 12 IP address: Master (Active) MySQL Master/Master MySQL Backup (Passive) Controller 2 VRID 12 IP address:

24 MySQL Master/Master Replication simplified

25 MYSQL HA: COROSYNC, PACEMAKER AND DRBD

26 Pacemaker, Corosync and DRBD Image from: RACKSPACE HOSTING 26

27 Pacemaker, Corosync, DRBD Pacemaker Corosync DRBD High availability and load balancing stack for the Linux platform Totem single-ring ordering and membership protocol Synchronizes data at the block device Interacts with applications through Resource Agents (RA) UDP and InfiniBand based messaging, quorum, and cluster membership to Pacemaker Uses a journaling system (such as ext3 or ext4)

28 DRBD Service Service FILE SYSTEM FILE SYSTEM BUFFER CACHE BUFFER CACHE DRBD RAW DEVICE TCP/IP RAW DEVICE TCP/IP DRBD DISK SCHED DISK SCHED DISK DRIVER NIC DRIVER NIC DRIVER DISK DRIVER DISK NIC NIC DISK

29 MYSQL HA: GALERA

30 Galera Synchronous multi-master cluster technology for MySQL/InnoDB CLIENTS MySQL patched for wsrep (Write Set REPlication) Transparent Connections Active/active multi-master topology Read and write to any cluster node DBMS DBMS DBMS True parallel replication, in row level wsrep API wsrep API wsrep API No slave lag or integrity issues Galera Replication

31 Multi-master replication Based on Optimistic Concurrency Control In case of two transactions modifying the same row on different nodes, one of the transactions will abort Victim transaction will get Deadlock Error Application needs to handle this error

32 Multi-master Replication read & write read & write read & write MySQL Multi-master cluster looks like one big database with multiple entry points

33 Multi-master conflicts write write MySQL MySQL MySQL GALERA REPLICATION

34 Multi-master conflicts write write MySQL MySQL MySQL Conflict detected GALERA REPLICATION

35 Multi-master conflicts OK write Deadlock error MySQL MySQL MySQL GALERA REPLICATION

36 OpenStack and Galera Image from

37 Galera on Rackspace Private Cloud/OpenStack A How To: OFFICIALLY UNSUPPORTED 1. Install Rackspace Private Cloud on 2 controllers with HA mode (Haproxy, Keepalived and VRRP is already installed) 2. Install Galera (with ws-rep) on 3 separate nodes 3. Mysqldump from controller nodes to Galera node 4. Grant privileges to OpenStack (nova, glance, etc.) and haproxy users 5. Update keepalived and haproxy and OpenStack configuration files on controller/compute 6. Stop/Uninstall MySQL services on controller nodes and restart controller nodes

38 Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA on Public Cloud Resources and Summary

39 A PEEK INTO HA ON PUBLIC CLOUD

40 HA on the Public Cloud

41 Agenda What is HA? HA of OpenStack APIs HA of RabbitMQ MySQL HA A Peek into HA on Public Cloud Resources and Summary

42 HA methods Infrastructure Clustering/Replication Technique Characteristics OpenStack APIs None required (Stateless) HA also serves as scale out using HAProxy RabbitMQ RabbitMQ Clustering RabbitMQ Clustering is setup for single/ multiple nodes Heat TBD Application Dependent (No standard methods yet). MySQL Many Discussed later slide

43 HA methods for MySQL Clustering Method Replication Technique Characteristics Pacemaker/Corosync/DRBD Mirroring on Block Devices Well tested, more complex to setup. Split brain possibility Keepalived/HAProxy/VRRP Works on MySQL master-master replication Simple to implement and understand. Works for any storage system. Master-master replication does not work beyond 2 nodes. Galera Others Based on write-set Replication (wsrep) MySQL Cluster, RHCS with DAS/ SAN Storage No Slave lag Needs at least 3 nodes Deadlock erros on hotspot rows. Relatively new Some relatively new (GTID) Some well test More complex setup

44 Resources OpenStack HA guide Other Resources

45 Book

46 Summary In general leverage existing methods of HA There are several time-tested and more recent methods for implementing MySQL HA. Rackspace Private Cloud provides Chef cookbooks and recipes for implementing HA via Keepalived, HAProxy and VRRP. Galera is gaining more popularity. Since it s Active/Active it does scale out and is HA. Few steps to get from Rackspace Private Cloud to MySQL with Galera (officially unsupported). Corosync/Pacemaker/DRBD is recommended by Oracle/MySQL. OpenStack HA guide goes through all these options in more detail.

47

48 Thank you! Raghavan Rags Srinivas Solutions Architect Rackspace