SUSE Linux Enterprise High Availability Extension

Transcription

1 A Xen Cluster Success Story Using the SUSE Linux Enterprise High Availability Extension Mark Robinson Director, MrLinux

2 Case Study: R M Donaldson APC High Power Partner My first commercial SUSE client 2 Deployed SUSE Professional in 1999 for File/Print ISDN router Proxy Mail server Moved to SUSE Linux Standard Server Moved to SUSE Linux Enterprise Server when Standard Server was retired Still happy running SUSE today!

3 Growth 3 Started with a single server Needed to expand as business grew Added dedicated server for Lotus Domino Needed to introduce additional services Blackberry Enterprise Server Sophos Enterprise Manager Pervasive SQL for accounts

4 Introduction of Xen Couldn't afford to buy servers for each service Sophos Enterprise Console was installed on a PC instead Pervasive SQL installed on SUSE Linux Enterprise Server file server 4 Performance/stability wasn't up to scratch, were told to move it to a Windows Server Installed SUSE Linux Enterprise Server 10 - Xen technology IBM x346 Lots of local disk Lots of memory (16GB!)

5 Xen/Hardware Issues Upgrading RAM requires downtime Tape drive attached to server Hardware instability No way to take server down to fix tape issues without stopping RMD's business RAID controller failure 5 Lucky escape reboot and firmware update fixed it!

6 The Move to High Availability

7 Business Drivers Business was still growing More software packages required Increased number of VMs for service separation Hardware refresh required Servers now soooo cheap! 7 Could buy three powerful x3550 M3 servers for 1.2 x price of original Xen host! 16G per server, but that's still 3x previous RAM capacity Let's not even talk about how many more cores now!

8 Business Drivers 8 Cannot tolerate downtime during normal working hours for office based staff Extended working hours due to nature of their business: Engineers working on customers' sites OOH Customers in many timezones Heavy reliance on /bes/lotus Traveler/Sametime

9 So, They Need a Cluster! 9 Shared datastore for storing VMs All nodes are active to spread load Live migrate VMs to allow a host to be taken down with no impact on service Automatic restart of VMs if a host fails

10 Software SUSE Linux Enterprise Server 11 High Availability Extension 11 VMDP for control of Windows VMs SMT for consistent patching SEP Sesam for backups 10 Agents for Domino etc

11 Hardware SAN was required for a cluster FC deemed too expensive for just three servers iscsi over multiple 1GB ethernet should be enough QNAP chosen for reliability/cost reasons 11 Two of them for DR purposes Switch stack for server cabinet iscsi Management LAN trunk to user switched upstairs

12 Design

13 High Level Design Nodes run minimal SUSE Linux Enterprise Server / Xen Hypervisor / High Availability Extension VMs are cluster resources VM storage (OCFS2) is also managed by cluster Live migrate configured in Xend, but managed by cluster Utilisation settings for VMs 13 autoyast for consistent builds Memory only

14 Storage Design Primary QNAP SAN carries all running VMs Secondary QNAP SAN for nightly DR copies iscsi with 2 x 1GB bonded NIC OCFS2 filesystems on both SANs Disk to disk backups to third QNAP 14 Mounted by cluster resources Housed in a data safe!

15 Design Overview VM VM xenstore 1 OCFS2 Filesystems xenbackup 1 xenstore 2 xenbackup 2 xenstore 3 Node 1 VM xenbackup 3 Node 2 Node 3 iscsi LAN Primary QNAP 15 Secondary QNAP

16 Node Network Config eth eth Bond Client LAN Bridge VM Node 1 OCFS2 Bond eth iscsi LAN 16 eth Switch

17 Cluster Design Active/Active - all nodes run VMs Use cluster integrated O2CB/DLM Six OCFS2 VM stores Use SBD for STONITH 17 Just one as there's only one SAN/iSCSI target Use csync2 for config file synchronisation

18 Cluster Design 18 Resource order constraints OCFS2 after O2CB/DLM VM after VMstore that it resides on! Use groups to reduce complexity Use clonesets Set resource (VM) utilisation for Memory to balance across hosts No (co)location constraints

19 Disaster Recovery No mirroring between SANs at the moment Rsync copy to second SAN 19 Pause VMs during rsync for some consistency

20 The Running Cluster

21 Current 21 Three nodes Six OCFS filesystems 11 VMs

22 22 Screenshots of grm_gui/hawk

23 23 Screenshots of grm_gui/hawk

24 24 Screenshots of grm_gui/hawk

25 25 Screenshots of grm_gui/hawk

26 Benefits

27 Benefits Increased performance vs single server Much better availability RAM upgrade to 32G per node with no VM downtime Three short outages in two years Two caused by me! One SAN reboot. Everything came back perfectly without intervention. Scalability Could easily add nodes 27 Clonesets take care of almost everything!

28 Benefits 28 SBD is very stable and easy to configure Csync2 works well HAWK for management Very easy to use, just requires a browser! Very clear status view No need to know about XEN at all once VMs are configured Can start/stop/migrate VMs without using CLI Wizard for adding new OCFS2 resources

29 Future Enhancements

30 Future Enhancements Upgrade cluster to SUSE Linux Enterprise Server 11 SP3 Introduce mirroring between SANs 30 Rolling cluster upgrade no downtime required! DRDB Snapshots instead of pausing VMs Install/configure PSSH

31 31 Corporate Headquarters (Worldwide) Join us on: Maxfeldstrasse Nuremberg Germany

32 Unpublished Work of SUSE. All Rights Reserved. This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability. General Disclaimer This document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.