Building Elastix-1.3 High Availability Clusters with Redfone fonebridge2, DRBD and Heartbeat Disclaimer DRBD and Heartbeat are not programs maintained or supported by Redfone Communications LLC. Do not contact Redfone for support on these programs. Use the information in this document at your own risk. Redfone disavows any potential liability for the contents of this document. Use of the concepts, examples, and/or other content of this document are entirely at your own risk. All copyrights are owned by their owners, unless specifically noted otherwise. Use of a term in this document should not be regarded as affecting the validity of any trademark or service mark. You are strongly recommended to take a backup of your system before major installation and backups at regular intervals. Credits Special thanks to Telesoft Integrando Technologies whose earlier documentation on doing DRBD in Elastix was a great reference for this work; http://asterisk.aplitel.info/files/asteriskcluster.pdf
Operational Overview What is DRBD? DRBD refers to block devices designed as a building block to form high availability (HA) clusters. This is done by mirroring a whole block device via an assigned network. It is shown as network raid-1- DRBD. In the illustration above, the two orange boxes represent two servers that form an HA cluster. The boxes contain the usual components of a Linux kernel: file system, buffer cache, disk scheduler, disk drivers, TCP/IP stack and network interface card (NIC) driver. The black arrows illustrate the flow of data between these components. The orange arrows show the flow of data, as DRBD mirrors the data of a high availably service from the active node of the HA cluster to the standby node of the HA cluster. (Source www.drbd.org) In our implementation we will be creating a DRBD synchronized partition on /dev/sda3 called replica. This partition will contain only those directories and files we want synchronized between our primary and secondary server. Namely, the important Asterisk and Elastix related directories and files.
What is Heartbeat? The upper part of this picture shows a cluster where the left node is currently active, i.e., the service's IP address is currently on the left node, and the client machines are talking to the service via its service's IP address on the active (left) node. The service, including its IP address, can be migrated to the other node at any time, either due to a failure of the active node or as an administrative action. The lower part of the illustration shows a degraded cluster. In HA speak the migration of a service is called failover, the reverse process is called failback and when the migration is triggered by an administrator it is called switchover. (Source www.drbd.org) In our implementation we will utilize Heartbeat to monitor the state of two servers and during a failover mount our synchronized partition on the secondary server and start up the following resources/applications; fonulator, asterisk, mysql and http. The fonulator utility will re-configure the fonebridge to reroute T1/E1 traffic to the secondary server as specified in /etc/redfone.conf and asterisk, mysql and http start up all aspects of Elastix on the secondary. During failover our floating IP address will move from the primary to the secondary server. This IP address should be used to register SIP and other VoIP endpoints against.
Equipment Overview This installation scenario assumes two servers, each with three Ethernet interfaces and a single SATA hard drive. You may have a different type of hard drive (IDE, SCSI, etc) and therefore some of these steps may need to be modified to better reflect your environment. A fonebridge2 (providing T1/E1 connectivity), interconnected with the two servers via a dedicated Ethernet switch or on a shared managed switch with an isolated VLAN for fonebridge to Asterisk traffic. It is also possible to implement this with only two Ethernet interfaces. Under those circumstances it would be advisable to use one Eth for the fonebridge and the other for IP, heartbeat and DRBD traffic. Network Diagram
DRBD Install and Configuration The following steps are to be performed on both primary and secondary servers; 1. Boot Elastix-1.3 Install CD 2. From boot menu, type advanced and enter. 3. During install routine choose to manually partition hard drive. The following is based on a 160.0 GB SATA Create root (/), ext3 partition with 6144MB (sda1) Create swap partition with 3072MB (sda2) 4. The remainder of the install routine is standard. 5. After installation and booting perform upgrade yum y update 6. Ensure /boot/grum/menu.1 st to boot non-xen kernel unless you need the Xen kernel. default=1 7. Create partition that will contain the replicated data fdisk /dev/sda Add a new partion (n) Primary (p) Partition number (3) Press enter until returned to fdisk command prompt NOTE: if your servers have two different sized hard drives it is imperative that the third partition is identical in size or they will never synchronize over DRBD. Do this by accepting the default first cylinder and then specifying the Last cylinder with the +sizem option. Ex. +6048M. Make these same specifications on both servers. Press t to change the partition system ID Press 3 to choose partition number Choose HEX 83 for type Press w to save changes RESTART SERVER 8. Format newly made partition mke2fs j /dev/sda3 9. Now we delete the file system from the disk we just created dd if=/dev/zero bs=1m count=1 of=/dev/sda3; sync 10. Install DRBD, Heartbeat and dependencies with yum. (or install_utilities script) yum install drbd yum install kmod-drbd82 yum install OpenIPMI-libs yum install heartbeat-pils yum install openhpi
yum install heartbeat yum install heartbeat-stonith 11. To ensure proper host name to IP resolution it is recommended that you manually update the /etc/hosts file to reflect proper host-to-ip mapping. 10.1.1.1 primary.yourdomain.com 10.1.1.2 secondary.yourdomain.com 127.0.0.1 primary.yourdomain.com ::1 localhost6.localdomain6 localhost6 12. DRBD allows us to write the declared partition in both members of the cluster. In our case /dev/sda3. To make this happen we need to create a virtual partition called /dev/drbd0. 13. Edit /etc/drbd.conf on both primary and secondary servers. drbd.conf file must be identical on both servers. Modify this sample to meet your particular needs. resource "r0" { protocol A; disk { on-io-error pass_on; } startup { wfc-timeout 5; degr-wfc-timeout 3; } syncer { rate 100M; } on primary.yourdomain.com { device /dev/drbd0; disk /dev/sda3; address 10.1.1.1:7789; meta-disk internal; } on secondary.yourdomain.com { device /dev/drbd0; disk /dev/sda3; address 10.1.1.2:7789; meta-disk internal; } } 14. Before continuing, change the name of the server in the Elastix web interface. 15. Now we create the virtual partition /dev/drdb0 on both servers drbdadm create-md r0 16. Start drbd service on both servers to begin synchronization process. service drbd start 17. Verify sync process with the following command service drbd status
18. Initially, both servers will be secondary. We need to assign who is the primary. Enter the following command on the primary server only drbdsetup /dev/drbd0 primary -o 19. Check drbd status again with service drbd status. You should obtain a status similar to the following; # service drbd status drbd driver loaded OK; device status: version: 8.0.3 (api:86/proto:86) SVN Revision: 2881 build by buildsvn@c5-i386-build, 2007-05-13 08:22:43 0: cs:connected st:primary/secondary ds:uptodate/uptodate A r--- ns:464976 nr:0 dw:0 dr:464976 al:0 bm:170 lo:0 pe:0 ua:0 ap:0 resync: used:0/31 hits:28977 misses:85 starving:0 dirty:0 changed:85 act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0 20. We can determine the role of a server by executing the following; drbdadm state r0 The primary server should return; # drbdadm state r0 Primary/Secondary 21. Primary Server Only: Now we can mount the virtual partition /dev/drbd0, but first we must format the partition with ext3 using the following command mke2fs j /dev/drbd0 mkdir /replica mount /dev/drbd0 /replica 22. Primary Server Only: Now we will copy all of the directories we want synchronized between the two servers to our new partition, remove the original directories and then create symbolic links to replace them. Directories of interest are; /etc/asterisk /var/lib/asterisk /usr/lib/asterisk /var/spool/asterisk /var/lib/mysql /var/log/asterisk /var/www 23. Copy, remove and link. (Or create_directories script which automates this step) cd /replica tar -zcvf etc-asterisk.tgz /etc/asterisk tar -zxvf etc-asterisk.tgz
tar -zcvf var-lib-asterisk.tgz /var/lib/asterisk tar -zxvf var-lib-asterisk.tgz tar -zcvf usr-lib-asterisk.tgz /usr/lib/asterisk/ tar -zcvf var-www.tgz /var/www/ tar -zxvf usr-lib-asterisk.tgz tar -zcvf var-spool-asterisk.tgz /var/spool/asterisk/ tar -zxvf var-spool-asterisk.tgz tar -zcvf var-lib-mysql.tgz /var/lib/mysql/ tar -zxvf var-lib-mysql.tgz tar -zcvf var-log-asterisk.tgz /var/log/asterisk/ tar -zxvf var-log-asterisk.tgz tar -zxvf var-www.tgz rm -rf /etc/asterisk rm -rf /var/lib/asterisk rm -rf /usr/lib/asterisk/ rm -rf /var/spool/asterisk rm -rf /var/lib/mysql/ rm -rf /var/log/asterisk/ ln -s /replica/etc/asterisk/ /etc/asterisk ln -s /replica/var/lib/asterisk/ /var/lib/asterisk ln -s /replica/usr/lib/asterisk/ /usr/lib/asterisk ln -s /replica/var/spool/asterisk/ /var/spool/asterisk ln -s /replica/var/lib/mysql/ /var/lib/mysql ln -s /replica/var/log/asterisk/ /var/log/asterisk ln -s /replica/var/www /var/www 24. Restart mysql server service mysqld restart Heartbeat Configuration 1. Remember to stop any boot up services that should be controlled by heartbeat. These services will be controlled by heartbeat on the server that is in control. chkconfig asterisk off chkconfig mysqld off chkconfig httpd off
2. Edit the /etc/ha.d/ha.cf file. This is a basic config file that provides all of our required functionality. debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 keepalive 2 deadtime 20 warntime 10 initdead 40 udpport 694 bcast eth1 auto_failback off node primary.yourdomain.com node secondary.yourdomain.com 3. Edit /etc/ha.d/haresources. This is where we specify which is our primary server, the name of the drbd partition we will mount and where, our floating IP address (10.1.1.3 in this example on eth1 with a 24 bit subnet mask) and the init.d scripts that will be started during a failover or switchover (fonulator, asterisk, mysqld, httpd) primary.yourdomain.com drbddisk::r0 Filesystem::/dev/drbd0::/replica::ext3 IPaddr::10.1.1.3/24/eth1/10.1.1.255 fonulator asterisk mysqld httpd 4. Edit /etc/ha.d/authkeys. This provides a level of security/authentication between the nodes. auth 1 1 sha1 B!gb@dUndUglIp@$$werd 5. Change permissions on the /etc/ha.d/authkeys file chmod 600 /etc/ha.d/authkeys 6. Configure heartbeat to initialize at boot chkconfig --add heartbeat 7. Restart drbd on both servers service drbd restart 8. Verify that both servers are currently in secondary state drbdadm state r0 Result should be Secondary/Secondary on both 9. Restart heartbeat service heartbeat restart
10. Wait a few moments are re-check the state of the servers. drbdadm state r0 Result this time should be Primary/Secondary on the primary server 11. Execute df h on the primary to confirm that our /dev/drbd0 partition is mounted and in use. # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 5.9G 1.9G 3.8G 33% / tmpfs 1.5G 0 1.5G 0% /dev/shm /dev/drbd0 138G 305M 131G 1% /replica 12. You can also monitor the synchronization process with the following command watch -n 1 cat /proc/drbd Install and configure fonebridge Support 1. Install fonulator utility (install_utilities script) rpm -ivh http://support.red-fone.com/downloads/fonulator/fonulator-2.0.0-36.i386.rpm 2. Download sample redfone.conf file 1. cd /etc/ 2. wget http://support.red-fone.com/downloads/fonulator/redfone.conf 3. Edit redfone.conf to match your requirements Refer to fonebridge general install guide for details and options 4. Download fonulator initd script cd /etc/init.d wget http://support.redfone.com/downloads/fonulator/old/fonulator_initd_script mv fonulator_initd_script fonulator chmod +x fonulator 5. Reboot both servers and confirm functionality. Under normal operation your primary server should boot, mount drbd partition and start up those init scripts specified in /etc/ha.d/haresources.
6. Verify drbd partition is mounted by executing df h. You should see something similar to the following. # df -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 5.9G 1.8G 3.8G 32% / tmpfs 125M 0 125M 0% /dev/shm /dev/drbd0 5.7G 409M 5.0G 8% /replica Troubleshooting 1. Split brain or nodes running in standalone mode. Ex; cs:standalone st:secondary/unknown ds:uptodate/dunknown, drbdadm -- --discard-my-data connect all DRBD documentation http://www.drbd.org/docs/about/ Heartbeat documentation http://www.linux-ha.org/heartbeat Centos DRBD/Heartbeat Howtos http://wiki.centos.org/howtos/ha-drbd http://www.howtoforge.com/drbd-on-centos-4.5 Last updated December 23, 2008