LINBIT DRBD Proxy Configuration Guide on CentOS 6. Matt Kereczman 1.2,

Transcription

1 LINBIT DRBD Proxy Configuration Guide on CentOS 6 Matt Kereczman 1.2,

2 Table of Contents 1. About this guide Two Node Cluster using DRBD Proxy Assumptions Installing DRBD and DRBD Proxy Configure the DRBD Resource using Proxy Create DRBD Metadata Connect the Nodes Start the Initial Sync Mount and Test the DRBD Device Conclusion Three Node Stacked Cluster using DRBD Proxy Assumptions Install DRBD on Charlie Install DRBD Proxy on All Nodes Configuring the Stacked Resource Testing the Disaster Recovery Node Failing Over to a Disaster Recovery Node Install and Configure Pacemaker on DR node Simulate a Failure of Alpha and Bravo Resuming Services on the DR Node Failback to Primary from DR site Starting Alpha and Bravo Manually Start and Connect DRBD Resume Services on HA cluster Tuning DRBD Proxy Determining a Proxy Buffer Size Configuring ko-count and timeout Configuring Congestion Policies Configure Compression with DRBD Proxy Speed up Initial Connection Example configurations Conclusion Feedback Additional information and resources

3 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 1. About this guide 1. About this guide This guide covers two common DRBD Proxy implementations. The first method uses DRBD proxy in a newly configured two-node DRBD cluster to replicate data to a peer over a high latency WAN connection; this method is common for customers looking for a disaster recovery solution. The second method demonstrates adding a third off site (stacked) node, to a preexisting high availability cluster using DRBD Proxy; this method is very common for customers who are already using DRBD for high availability, and would like to add disaster recovery capabilities. We will also cover procedures for failing-over and failing-back to a DR (Disaster Recovery) site; using our three-node cluster as an example. Finally, we will conclude the guide with a section on how to begin tuning DRBD and DRBD Proxy for different workloads and WAN connections. 2. Two Node Cluster using DRBD Proxy DRBD Proxy makes data replication over a high latency, low bandwidth, WAN connection possible. Without DRBD Proxy (even with DRBD s protocol A), our network throughput and latency will become our disk s throughput and latency. DRBD Proxy uses a configurable amount of system memory to act as a send buffer for our replicated writes; this allows writes to be buffered and flushed to the peer as fast as the replication link allows, without reducing local disk performance Assumptions This guide assumes the following: This is a new two node cluster with no preexisting data Both nodes have freshly installed CentOS 6.x operating system The node s hostnames are alice and bob Both nodes have two network interface: eth0 on alice has an IP address of: eth1 on alice has an IP address of: eth0 on bob has an IP address of: eth1 on bob has an IP address of: Both nodes have separate storage for their root filesystem and DRBD replicated data /dev/vda holds the root filesystem on both nodes /dev/vdb is a freshly installed drive with no existing data that will be used for DRBD replicated data Both nodes have their firewalls configured to allow traffic on TCP ports 7788, 7789, and Installing DRBD and DRBD Proxy Configure DRBD s Yum Repository We will begin our installation by setting up LINBIT s yum repositories. As a customer, we have access to all of LINBIT s certified binaries using the distributions package manager or direct download through LINBIT s customer portal. If you are not a customer, 30-day trial licenses are available; visit linbit.com for contact information. To enable LINBIT s repositories, create /etc/yum.repos.d/linbit.repo with the following contents, replacing <HASH> with the custom hash value received from LINBIT. 1

4 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 2.3. Configure the DRBD Resource using Proxy [LINBIT-DRBD] baseurl= gpgcheck=1 [LINBIT-DRBD-Proxy] baseurl= gpgcheck=1 Before installing packages from LINBIT s repositories, we must import the public key used to sign LINBIT s RPM packages. Use the following commands to do so: # wget # rpm --import gpg-pubkey-53b3b b6e23.asc # rm gpg-pubkey-53b3b b6e23.asc DRBD and DRBD Proxy Package Installation Use the commands below to install the drbd, kmod-drbd, and drbd-proxy-3.0 packages, load DRBD s kernel module, and ensure DRBD Proxy starts at boot on both nodes. # yum install drbd kmod-drbd drbd-proxy-3.0 # modprobe drbd # chkconfig drbdproxy on At the time of writing this document, choosing the drbd-proxy-3.0 package will install the latest version of DRBD Proxy 3.x. This naming convention may change in future releases be be more precise. We can use yum to search for all currently available DRBD Proxy packages to ensure we are downloading the most current one using the following command: # yum search drbdproxy The kmod-drbd package might not be for the exact kernel version we are running. As long as kmod-drbd is the closest version available from LINBIT s repository to the version of our kernel without going over, everything should work as expected. The Packages in LINBIT s repository are tested on systems running the latest kernel available from the vendor before being released. We were provided with a license file for DRBD Proxy. We must copy or move this file into the /etc/ directory, and change it s ownership to drbdpxy, before DRBD Proxy will start. Since we are using proxy on both nodes, these steps should be completed on both nodes. # cp drbd-proxy.license /etc/drbd-proxy.license # chown drbdpxy: /etc/drbd-proxy.license For the examples in this guide, we will install DRBD Proxy on the cluster nodes. This is the most common way to install Proxy, but it should be noted that it is possible to install DRBD Proxy on a separate machine outside of the cluster Configure the DRBD Resource using Proxy Now that we have DRBD and DRBD Proxy installed, we can begin creating our first resource using DRBD Proxy. Our DRBD configuration files will be stored in the /etc/drbd.d/ directory, and have the.res extension. Any resource files within this directory with the.res extension will be activated automatically at boot time. We recommend using 2

5 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 2.4. Create DRBD Metadata naming conventions for consistency between resource configuration files and the resources defined within them. In our example we will define a resource r0 in the resource configuration file r0.res. Matching a resource s name with that of its configuration file is not a requirement. Also, it is possible to define more than one resource in a single resource file, though it is highly recommended to keep these resource files as granular as possible, for simplicity and sanity. The exception to this rule is stacked resources; stacked resource configurations should be placed in the same configuration file as the lower-level resource. Below is the configuration of the resource r0, which will be stored in the resource configuration file /etc/drbd.d/r0.res. Notice that the proxy on stanza for each node has an inside and an outside interface; since we are running proxy locally on both nodes, the inside interface will be the loopback, while the outside interface is the address ping-able by the peer. The configuration also includes a proxy stanza; inside this stanza is where one would configure the proxy memory buffer, and the compression plugin. The example below is using zlib compression at the maximum level of 9, and has a memlimit (send buffer) of 100 megabytes. Selecting an appropriate memlimit depends upon the expected write workloads, throughput of our connections, and acceptable maximum amount of data that may be lost in the event of a complete site loss. It is suggested to configure an initial value higher than we think is needed, and dial it back later if monitoring shows it s not needed. All nodes in the cluster must have the exact same resource configuration files. Any modification made to a resource file, needs to be made on both nodes. resource r0 { protocol A; device minor 0; disk /dev/vdb; meta-disk internal; proxy { memlimit 100M; plugin { zlib level 9; on alice { address :7789; proxy on alice { inside :7788; outside :7790; on bob { address :7789; proxy on bob { inside :7788; outside :7790; 2.4. Create DRBD Metadata Now that the resource has been defined, we need to create the metadata used by DRBD to keep track of the written blocks on both nodes. Since the DRBD backing device, /dev/vdb, contains no preexisting data, it is safe to simply 3

6 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 2.5. Connect the Nodes create the metadata letting DRBD determine the amount of space needed and write it to the end of the disk. Create the metadata by running the following command on both nodes: # drbdadm create-md r0 On production systems with preexisting data some steps must be taken to make room at the end of the backing device for DRBD s metadata. There are equations in the DRBD User s guide for determining exactly how much space is needed for metadata; this depends on the size of our backing disk (DRBD User s Manual). In most practical applications there is a rule of thumb that can be used to determine the required amount of space: 32MB for metadata per 1TB of storage Connect the Nodes Both nodes have created their metadata and are ready for the initial sync. Before beginning the sync, we need to start the drbdproxy service and bring up r0 on both nodes. # service drbdproxy start # drbdadm up r0 We can now check the status of our DRBD device. If all is working as expected, we should see output similar to that below when executing the following commands: # cat /proc/drbd version: (api:1/proto:86-101) GIT-hash: 8ca20467bb9e10aa ac1d65f2b54b4 build by buildsystem@linbit 0: cs:connected ro:secondary/secondary ds:inconsistent/inconsistent A r----- ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos: # drbd-proxy-ctl -c 'show hconnections' Name Status LAN WAN Compr Up since r0-bob-alice Up 1.0K 3.6K Fri Jun 27 14:10: connection(s) listed. If we see cs:wfconnection instead of the expected cs:connected, be sure the firewall is not blocking port DRBD Proxy can take a short period of time to establish a connection depending on the replication link. We can use the command watch -n1 cat /proc/drbd to monitor the status of the connection state Start the Initial Sync If our devices are in the Connected state, we can now choose a node to become Primary; enabling us to use our DRBD device. Since this guide assumes there is no preexisting data that needs preservation, this choice is arbitrary. I will choose alice as the Primary in my cluster with the following command: In a production environment where data preservation is required, the node where the data currently exists MUST be selected as the Primary to avoid data loss. # drbdadm primary --force r0 4

7 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 2.7. Mount and Test the DRBD Device That will start the initial sync. The initial sync is a full sync and can take a long time depending on the size of our device. We can view the rate and progress of the sync by using the watch -n1 cat /proc/drbd command (here done on the Secondary): version: (api:1/proto:86-101) GIT-hash: 8ca20467bb9e10aa ac1d65f2b54b4 build by buildsystem@linbit 0: cs:synctarget ro:secondary/primary ds:inconsistent/uptodate A r----- ns:0 nr: dw: dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos: [===>...] sync'ed: 20.8% ( / )K finish: 0:01:26 speed: 19,144 (13,100) want: 16,800 K/sec The device can be used on the Primary node during all sync processes. By default, DRBD dynamically throttles the IO available to the sync process in preference of incoming application IO. We do not need to wait for the initial sync to complete to proceed - on the contrary, if the mkfs tool that s being used sends "Trim" (alternatively called "Discard") requests for creating the filesystem, DRBD can use that information to shorten the initial sync! So, the ext4 filesystem being created later on will be synchronized to the remote within a very short time - nearly independent of the filesystem size. We can also watch the DRBD Proxy buffer fill and flush using the watch -n1 "drbd-proxy-ctl -c 'show memusage'" command on the Primary node: b-alice normal prio 38 of 104 MiB used, 0 persistent [***************...] / bytes [...] 1024/ bytes 2.7. Mount and Test the DRBD Device We now have a usable DRBD device. This section of the guide will have us format and mount the new device to run some tests verifying the cluster is performing as expected Mount the DRBD Device On the Primary node we should see that there is a new device, /dev/drbd0. To use this new device, we will format it as ext4, create a mount-point, and mount the DRBD device to it: We would not want to create a filesystem on a device that already contains data. mkfs.ext4 should only be run on new, previously empty, resources. # mkfs.ext4 /dev/drbd0 # mkdir /mnt/drbd_mount # mount -t ext4 -o noatime,nodiratime,discard /dev/drbd0 /mnt/drbd_mount Running the command, mount, should now show that /dev/drbd0 is mounted on /mnt/drbd_mount as a read/write ext4 filesystem. # mount < snipped output > /dev/drbd0 on /mnt/drbd_mount type ext4 (rw,noatime,nodiratime,discard) 5

8 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 2.8. Conclusion Be sure to create the mountpoint on both nodes Test Write Speeds To test that the new resource is writing at speeds roughly equal (less than 3% slower) to that of it s backing device, we can perform a write test with the DRBD devices connected, then again while disconnected. This will prove that the DRBD Device is not being affected by the latency/throughput of our replication link. Run the command below to write a 100MB file to the mounted DRBD device, and note the rate that is output when the dd command completes: # dd if=/dev/zero of=/mnt/drbd_mount/dd_test_file bs=1m count=100 oflag=direct records in records out bytes (105 MB) copied, s, 43.5 MB/s Now we will disconnect the DRBD device from the Primary node putting it into the StandAlone state. This will write directly to the DRBD device, without replicating any data over DRBD Proxy, and we should see that we are achieving the same speed as we were when the devices were connected. # drbdadm disconnect r0 # dd if=/dev/zero of=/mnt/drbd_mount/dd_test_file bs=1m count=100 oflag=direct records in records out bytes (105 MB) copied, s, 46.4 MB/s The systems used to write this guide are Virtual Machines with limited resources on shared hardware. Performance on physical hardware should be much better. The above demonstrates that our network speed is not limiting our disk speed. Reconnect the device using the following command on the Primary node: # drbdadm connect r Conclusion With a few easy steps we ve set up a data replication solution for disaster recovery. As a final tip: Repeat the command from above, to see whether DRBD Proxy is really compressing the data (which should be very efficient, as only NULL bytes have been written by dd: # drbd-proxy-ctl -c 'show hconnections' Name Status LAN WAN Compr Up since r0-bob-alice Up 204.2M 82.1K Fri Jun 27 14:40: connection(s) listed. 3. Three Node Stacked Cluster using DRBD Proxy This section of the guide covers adding a third (disaster recovery) node to an already existing high availability cluster. This is a common procedure, but it needs to be executed carefully as there is preexisting data that needs to remain 6

9 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.1. Assumptions intact. Also, in order to add DRBD Proxy to a preexisting HA cluster, there will be a brief period of downtime when we reconfigure HA services to use the new stacked resource. It is recommended to perform dry-runs of this procedure in test environments (when possible) to limit the amount of downtime required to make the necessary changes Assumptions This guide assumes the following: All systems are running CentOS 6.x All systems have a separate device for DRBD replicated storage The existence of a local high availability NFS cluster One DRBD resource, formated ext4, mounted to /mnt/drbd_data/, exported via NFS Cluster contains data which must be preserved Hostnames: Alpha and Bravo Alpha and Bravo have a service IP of /24 for NFS DRBD 8.4.4, Pacemaker, and Heartbeat, are already installed and configured A third, remote node named Charlie, is to be added to the cluster as a disaster recovery node Charlie has a new drive /dev/vdb that will be used for the replicated data All three nodes use LVM as their DRBD backing devices, and have room available to grow All nodes have their firewalls configured to allow cluster related traffic All nodes have nfs-utils installed The primary site and DR site are connected via a Layer 2 WAN link Install DRBD on Charlie Configure LINBIT s Yum Repository To enable LINBIT s repositories, create /etc/yum.repos.d/linbit.repo with the following contents, replacing <HASH> with the custom hash value we received from LINBIT. [LINBIT-DRBD] baseurl= gpgcheck=1 [LINBIT-DRBD-Proxy] baseurl= gpgcheck=1 [LINBIT-Pacemaker] baseurl= gpgcheck=1 Before installing packages from LINBIT s repositories, we must import the public key used to sign LINBIT s RPM packages. Use the following commands to do so: # wget # rpm --import gpg-pubkey-53b3b b6e23.asc # rm gpg-pubkey-53b3b b6e23.asc 7

10 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.3. Install DRBD Proxy on All Nodes Install the drbd, and kmod-drbd packages from the LINBIT repositories, load the kernel module, and make sure DRBD is set to start at boot. # yum install drbd kmod-drbd # modprobe drbd # chkconfig drbd on 3.3. Install DRBD Proxy on All Nodes Install DRBD Proxy on all three nodes using yum, and ensure it is set to start at boot time only on charlie; we will use Pacemaker to start and stop DRBD Proxy on alpha and bravo. On all three nodes: # yum install drbd-proxy-3.0 # chkconfig drbdproxy off Then on charlie only: # chkconfig drbdproxy on We were provided with a license file for DRBD Proxy. We must copy or move this file into the /etc/ directory, and change it s ownership to drbdpxy, before DRBD Proxy will start. Since we are using proxy on all three nodes, these steps should be completed on all three nodes. # cp drbd-proxy.license /etc/drbd-proxy.license # chown drbdpxy: /etc/drbd-proxy.license 3.4. Configuring the Stacked Resource Three node clusters require resource stacking. Resource stacking is exactly what it sounds like, creating a DRBD resource on top of another DRBD resource. When adding a third node to a previously existing two node cluster, we will configure a new resource that uses the already existing DRBD device as its backing disk. We will need to stop services using the existing DRBD device while configuring our stacked resource, as we never want to access the backing disk of a DRBD resource [1: Because DRBD wouldn t see that data changes, and so couldn t replicate to the other node.]. Once our configuration is in place, we will need to reconfigure our services to use the new stacked resource before starting them, ensuring changes are replicated to all three nodes. In our example, we are adding a DR node to an already existing HA NFS cluster managed by Pacemaker. In order to make the needed changes, we will need stop all these services that require the DRBD device or filesystems on top of it, leaving only the Master/Slave set for the DRBD device running; then put the cluster into maintenance mode, so it doesn t attempt to manage any cluster services until we re done Putting Pacemaker into Maintenance Mode In our simple NFS cluster comprised of alpha and bravo, the crm_mon utility shows that the group, g_nfs, contains all our clustered services: 8

11 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource Last updated: Fri Jun 27 10:48: Last change: Fri Jun 27 16:11: via crm_attribute on alpha Stack: heartbeat Current DC: bravo (8001d23c-c0ef-462e-82b3-c983967df5b3) - partition with quorum Version: hg bf8451.el6-9bf Nodes configured, unknown expected votes 6 Resources configured. Online: [ alpha bravo ] Resource Group: g_nfs p_fs_drbd (ocf::heartbeat:filesystem): Started alpha p_lsb_nfsserver (lsb:nfs): Started alpha p_exportfs_drbd (ocf::heartbeat:exportfs): Started alpha p_virtip_nfs (ocf::heartbeat:ipaddr2): Started alpha Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ alpha ] Slaves: [ bravo ] Running crm configure show shows us the currently running cluster configuration. Our example configuration looks like this: 9

12 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource node $id="32e7f01c a34-97e6-efec5733ed1d" alpha node $id="8001d23c-c0ef-462e-82b3-c983967df5b3" bravo primitive p_drbd_r0 ocf:linbit:drbd \ params drbd_resource="r0" \ op monitor interval="29" role="master" \ op monitor interval="30" role="slave" primitive p_exportfs_drbd ocf:heartbeat:exportfs \ params fsid="1" directory="/mnt/drbd_data" \ options="rw,sync,mountpoint" clientspec=" / " \ wait_for_leasetime_on_stop="false" \ unlock_on_stop="true" \ op monitor interval="30s" primitive p_fs_drbd ocf:heartbeat:filesystem \ params device="/dev/drbd0" directory="/mnt/drbd_data" \ options="noatime,nodiratime,discard" fstype="ext4" \ op monitor interval="10s" primitive p_lsb_nfsserver lsb:nfs \ op monitor interval="30s" primitive p_virtip_nfs ocf:heartbeat:ipaddr2 \ params ip=" " cidr_netmask="24" \ op monitor interval="10s" timeout="20s" group g_nfs p_fs_drbd p_lsb_nfsserver p_exportfs_drbd p_virtip_nfs ms ms_drbd_r0 p_drbd_r0 \ meta master-max="1" master-node-max="1" clone-max="2" \ clone-node-max="1" notify="true" colocation c_nfs_with_drbd inf: g_nfs ms_drbd_r0:master order o_drbd_before_nfs inf: ms_drbd_r0:promote g_nfs:start property $id="cib-bootstrap-options" \ stonith-enabled="false" \ no-quorum-policy="ignore" \ dc-version=" c7312c689715e096b716419e2ebc12b " \ cluster-infrastructure="heartbeat" \ last-lrm-refresh=" " rsc_defaults $id="rsc-options" \ resource-stickiness="200" In the example Pacemaker configuration above, there are no STONITH devices configured and the stonith-enabled cluster property is disabled. However, in a production cluster, STONITH should always be configured and enabled. Looking at the configuration we can confirm that the group g_nfs holds all our cluster services, including our filesystem, so that is what we must stop within Pacemaker. To stop the group, issue the command below at a shell prompt on either cluster node: # crm resource stop g_nfs Put the cluster into maintenance mode by issuing the command below at a shell prompt on either cluster node: # crm configure property maintenance-mode=true 10

13 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource Preparing Disks for Stacked Resource s Meta-data Since we are stacking a new DRBD device on top of an existing one, we need to make room for the stacked devices' metadata on alpha and bravo, then create a logical volume of the same size on charlie. On alpha and bravo we need to extend the logical volume enough to fit the new metadata. Though there are equations to determine the exact number of sectors needed for metadata, in most practical implementations it is satisfactory to use the rule of thumb: 32MB for metadata per 1TB on the backing disk. This rule scales 1:1; 32KB per 1GB. Please remember to always round the size up, never down. After extending the logical volume on both alpha and bravo, we need to resize the DRBD device we will be stacking on top of so we replicate that metadata between the lower level DRBD devices. We must use internal metadata with the stacked resource since we must have the metadata replicated if the resource switches from one HA node to the other, the new Primary needs to know which data hasn t reached the third node yet. In our example, we are using a logical volume named drbd_data in the volume group VolGroup00. VolGroup00 has 1GB free. drbd_data is currently 1GB in size, requiring us to grow the logical volume by only 32KB. We will grow the logical volume by 1 physical extent (4MB) on alpha and bravo to accommodate the metadata using the following command: # lvextend -l +1 /dev/volgroup00/drbd_data Then, on the current Secondary node, we resize the DRBD resource using the following command: # drbdadm resize r0 We need to know the new number of extents used by drbd_data on alpha and bravo, so we can create a logical volume of the same size on charlie. To do this we can run the following command: # lvdisplay /dev/volgroup00/drbd_data grep "Current LE" Current LE 257 We now know that we need to create a LV on charlie that is 257 extents in size. We will create VolGroup00 on the new disk /dev/vdb on charlie, as well as the appropriately sized drbd_data logical volume using the following commands: # pvcreate /dev/vdb # vgcreate VolGroup00 /dev/vdb # lvcreate -n drbd_data -l 257 /dev/volgroup Configure the Stacked DRBD Resource File For reference, here is the previously configured DRBD resource r0, defined in the file r0.res, which we will be stacking on top of: 11

14 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource resource r0 { protocol C; device /dev/drbd0; meta-disk internal; on alpha { disk /dev/volgroup00/drbd_data; address :7788; on bravo { disk /dev/volgroup00/drbd_data; address :7788; Convention is to define the new stacked resource with -u appended to the name of the resource we will stack on top of [2: -u for upper.]. We will define the new stacked resource r0-u within the same file as the lower level resource [3: since we are stacking on top of this resource, we will now refer to r0 as the lower level resource] r0, defined in the r0.res resource file. Following this convention will keep things as simple as possible and becomes more important when there are many resources. The contents of r0.res will now look like this: 12

15 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource resource r0 { protocol C; device /dev/drbd0; meta-disk internal; on alpha { disk /dev/volgroup00/drbd_data; address :7788; on bravo { disk /dev/volgroup00/drbd_data; address :7788; resource r0-u { protocol A; device /dev/drbd10; meta-disk internal; proxy { memlimit 100M; stacked-on-top-of r0 { address :7789; proxy on alpha bravo { inside :7799; outside :7779; on charlie { disk /dev/volgroup00/drbd_data; address :7789; proxy on charlie { inside :7799; outside :7779; Looking at the new configuration we can see that our new stacked device will take the device name /dev/drbd10, while our original device is still /dev/drbd0. These minor numbers can be whatever we would like, however for manageability, we should follow the convention of using a separate number range, e.g. an unused tens or hundreds range above all the non-stacked resources Create Metadata for the Stacked Device Now that we ve made room for the stacked device s metadata and configured the DRBD resource, we can create the metadata. On the node where the lower level device is Primary, we can run the following commands to initialize the resource: # drbdadm create-md --stacked r0-u # drbdadm up --stacked r0-u # service drbdproxy start 13

16 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource Since we ve made space for the new metadata in previous steps, there should be no errors when issuing the drbdadm create-md --stacked r0-u command. If we receive a message about overwriting existing data, we ll need to ensure we ve made enough room for the new metadata [5: before proceeding, failing to do so could result in overwriting our production data. On charlie, we will run the following commands: # drbdadm create-md r0-u # drbdadm up r0-u # service drbdproxy start Now, if we cat /proc/drbd on charlie we should see something like this: version: (api:1/proto:86-101) GIT-hash: 8ca20467bb9e10aa ac1d65f2b54b4 build by buildsystem@linbit 10: cs:wfconnection ro:secondary/unknown ds:inconsistent/dunknown A r----s ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos: On the lower level Primary there should be a similar output, with lines for the minor device 0 added. The resource will remain in the WFConnection connection state, until we resume the virtual IP in Pacemaker that we stopped earlier. Before we can reconfigure and resume services in Pacemaker using the new stacked resource, we need to first promote the resource to Primary on alpha or bravo, whichever is currently the lower level Primary, with the stacked resource running. Run the drbdadm primary command using the --stacked and --force directives as shown below to accomplish this: # drbdadm primary --stacked r0-u --force We should now see that we have an UpToDate connection state on the peer selected as the sync source in the command above Modify the Pacemaker Configuration and Restart Services We must now configure Pacemaker to manage the stacked resource. This involves adding a new Master/Slave definition, as well as modifying the current filesystem definition to mount the stacked resource instead of the lower level DRBD resource. Failing to modify the current filesystem definition will result in no data replication to the new peer, as we would be writing to the stacked resources backing disk. We also need to configure the DRBD Proxy resource. Depending upon the network in a particular environment, it may be necessary to add another floating IP via an IPaddr2 resource in order to connect to the disaster recovery node. However, this is outside the scope of this document. To add the new DRBD and DRBD Proxy resources to the current Pacemaker configuration, run the command crm configure to drop into a crm shell, then define the new primitives, master/slave resource, colocation, and ordering constraint: 14

17 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource # crm configure crm(live)configure# primitive p_drbd_r0-u ocf:linbit:drbd \ params drbd_resource="r0-u" \ op monitor interval="31s" role="master" \ op monitor interval="29s" role="slave" crm(live)configure# ms ms_drbd_r0-u p_drbd_r0-u \ meta master-max="1" master-node-max="1" clone-max="1" \ clone-node-max="1" notify="true" crm(live)configure# primitive p_lsb_proxy lsb:drbdproxy \ op monitor interval="30s" timeout="30s" crm(live)configure# colocation c_drbd0_upper_with_lower \ inf: ms_drbd_r0-u ms_drbd_r0:master crm(live)configure# order o_drbd_r0_lower_before_upper \ inf: ms_drbd_r0:promote ms_drbd_r0-u:start Now we need to edit the filesystem definition so that we re mounting the stacked DRBD device instead of the lower level device. We also need to modify the colocation and ordering constraints for the NFS resources so they start with the stacked DRBD device and start DRBD Proxy after the virtual IP resource. To do that, we will drop into edit mode by entering the edit command from the crm configure prompt. crm(live)configure# edit This will drop us into vi where we can modify the filesystem definition, location constraint, and ordering constraint so that they look like this: primitive p_fs_drbd ocf:heartbeat:filesystem \ params device="/dev/drbd10" directory="/mnt/drbd_data" \ options="noatime,nodiratime,discard" fstype="ext4" \ op monitor interval="10s" group g_nfs p_fs_drbd p_lsb_nfsserver p_exportfs_drbd p_virtip_nfs p_lsb_proxy colocation c_nfs_with_drbd inf: g_nfs ms_drbd_r0-u:master order o_drbd_before_nfs inf: ms_drbd_r0-u:promote g_nfs:start Write the changes to the Pacemaker configuration and quit vi. Use the show command from the crm configure prompt to show the current configuration. It should look similar to this: 15

18 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.4. Configuring the Stacked Resource node $id="32e7f01c a34-97e6-efec5733ed1d" alpha node $id="8001d23c-c0ef-462e-82b3-c983967df5b3" bravo primitive p_drbd_r0 ocf:linbit:drbd \ params drbd_resource="r0" \ op monitor interval="29" role="master" \ op monitor interval="30" role="slave" primitive p_drbd_r0-u ocf:linbit:drbd \ params drbd_resource="r0-u" \ op monitor interval="31s" role="master" \ op monitor interval="29s" role="slave" primitive p_exportfs_drbd ocf:heartbeat:exportfs \ params fsid="1" directory="/mnt/drbd_data" \ options="rw,sync,mountpoint" \ clientspec=" / " \ wait_for_leasetime_on_stop="false" unlock_on_stop="true" \ op monitor interval="30s" primitive p_fs_drbd ocf:heartbeat:filesystem \ params device="/dev/drbd10" directory="/mnt/drbd_data" \ options="noatime,nodiratime,discard" fstype="ext4" \ op monitor interval="10s" primitive p_lsb_nfsserver lsb:nfs \ op monitor interval="30s" primitive p_lsb_proxy lsb:drbdproxy \ op monitor interval="30s" timeout="30s" primitive p_virtip_nfs ocf:heartbeat:ipaddr2 \ params ip=" " cidr_netmask="24" \ op monitor interval="10s" timeout="20s" group g_nfs p_fs_drbd p_lsb_nfsserver p_exportfs_drbd p_virtip_nfs p_lsb_proxy ms ms_drbd_r0 p_drbd_r0 \ meta master-max="1" master-node-max="1" clone-max="2" \ clone-node-max="1" notify="true" ms ms_drbd_r0-u p_drbd_r0-u \ meta master-max="1" master-node-max="1" clone-max="1" \ clone-node-max="1" notify="true" colocation c_drbd0_upper_with_lower inf: ms_drbd_r0-u ms_drbd_r0:master colocation c_nfs_with_drbd inf: g_nfs ms_drbd_r0:master order o_drbd_before_nfs inf: ms_drbd_r0:promote g_nfs:start order o_drbd_r0_lower_before_upper inf: ms_drbd_r0:promote ms_drbd_r0-u:start property $id="cib-bootstrap-options" \ stonith-enabled="false" \ no-quorum-policy="ignore" \ dc-version=" c7312c689715e096b716419e2ebc12b " \ cluster-infrastructure="heartbeat" \ last-lrm-refresh=" " rsc_defaults $id="rsc-options" \ resource-stickiness="200" Once we have verified everything looks correct, we can bring the cluster out of maintenance mode and commit our changes: crm(live)configure# property maintenance-mode=false crm(live)configure# commit Running the crm_mon command should now show us that the new Master/Slave Set, the DRBD Proxy resource, as well 16

19 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 3.5. Testing the Disaster Recovery Node as all the previously configured services, have started. It should now look similar to this: Last updated: Fri Jun 27 08:35: Last change: Fri Jun 27 16:34: via crmd on bravo Stack: heartbeat Current DC: bravo (8001d23c-c0ef-462e-82b3-c983967df5b3) - partition with quorum Version: hg bf8451.el6-9bf Nodes configured, unknown expected votes 7 Resources configured. Online: [ alpha bravo ] Resource Group: g_nfs p_fs_drbd (ocf::heartbeat:filesystem): Started alpha p_lsb_nfsserver (lsb:nfs): Started alpha p_exportfs_drbd (ocf::heartbeat:exportfs): Started alpha p_virtip_nfs (ocf::heartbeat:ipaddr2): Started alpha p_lsb_proxy (lsb:drbdproxy): Started alpha Master/Slave Set: ms_drbd_r0 [p_drbd_r0] Masters: [ alpha ] Slaves: [ bravo ] Master/Slave Set: ms_drbd_r0-u [p_drbd_r0-u] Masters: [ alpha ] Now that the virtual IP is running, DRBD Proxy should be connected, or trying to connect. We can check the status of DRBD by looking at the contents of /proc/drbd. Once Proxy has established a connection, the full synchronization will begin. We can monitor this process using watch -n1 cat /proc/drbd/: Every 1.0s: cat /proc/drbd version: (api:1/proto:86-101) GIT-hash: 8ca20467bb9e10aa ac1d65f2b54b4 build by buildsystem@linbit 10: cs:synctarget ro:secondary/primary ds:inconsistent/uptodate A r----- ns:0 nr: dw: dr:0 al:0 bm:0 lo:0 pe:5 ua:0 ap:0 ep:1 wo:f oos:98560 [==============>...] sync'ed: 77.8% (98560/439896)K finish: 0:02:53 speed: 560 (540) want: 760 K/sec 3.5. Testing the Disaster Recovery Node Now that we have our Disaster Recovery node up and connected via DRBD Proxy, we will want to test that it is working as it should. To do this, mount our NFS share on a client machine, and write some test data using the dd utility. We will also need to create a directory inside the mountpoint where nfsnobody has write access (assuming we run the tests from the client as root), since root_squash is a default NFS option. While we are performing these write tests, we can monitor proxy on the node that is currently the Primary for the stacked resource, using drbd-proxyctl -c "show memusage" with the watch utility. Use the command below to accomplish this: On the current Primary for the stacked resource: # mkdir /mnt/drbd_data/world # chmod 777 /mnt/drbd_data/world # watch -n1 'drbd-proxy-ctl -c "show memusage"' 17

20 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 4. Failing Over to a Disaster Recovery Node The output of the above command will update every second, and will look similar to the output below when there is no activity on the device. This is what an empty DRBD Proxy buffer looks like: r0-u-charlie-alpha_bravo 0 of 104 MiB used, 0 persistent normal [...] 1024/ bytes prio [...] 1024/ bytes Leave the watch command running in one terminal. Open a new terminal on a client system or the Secondary cluster node and mount the NFS share. Then write some test data to the mount point using dd: # mkdir /mnt/abc_proxy # mount -t nfs :/mnt/drbd_data /mnt/abc_proxy # dd if=/dev/zero of=/mnt/abc_proxy/world/dd_test_file bs=1m count=50 oflag=direct 50+0 records in 50+0 records out bytes (52 MB) copied, s, 15.6 MB/s We should see that our test file was written as quickly as our disks would allow, and was not limited by the throughput of our replication link. Without using DRBD Proxy, we would see that our disk throughput would drop to match that of our WAN s network throughput. On our test systems, the WAN replication link is 1MB/s, but our disk throughput is still 15.6MB/s. Perform the test again, this time pay attention to the watch command running on the Primary node s terminal. Notice the memory buffer fill and flush after the dd has already finished. This is what a partially full DRBD Proxy buffer looks like: r0-u-charlie-alpha_bravo 62 of 104 MiB used, 0 persistent normal [************************...] / bytes prio [...] 1024/ bytes 4. Failing Over to a Disaster Recovery Node Making the decision to failover to a disaster recovery node is something that we recommend be performed manually. Typically, we would not want to use automated logic to determine when to perform failover because of the inherent unreliability of most WAN links. Consider the following: a routing issue temporarily affects the route between our two data centers, interrupting the communications between the Primary and Secondary node. The Primary node is still being accessed by systems taking a different network path than the replication/monitoring traffic. Since the Secondary can t reach the Primary, it believes the primary site is down; so it becomes Primary as well, creating a splitbrain. Using multiple out-of-band links between the two sites can overcome such issues, but it is typically still recommended to use manual failover; it is best practice to have a human confirm that a site is in fact down, before failing over services Since we are using Pacemaker to manage clustered services in the local HA cluster, we can easily configure a standalone Pacemaker cluster at the DR site to make bringing up services as simple as issuing a single command. We will outline the general steps to accomplish such a configuration in the sections following this introduction Install and Configure Pacemaker on DR node Since we ve already configured yum to pull from LINBIT s Pacemaker repositories, we can run the following commands to install Pacemaker, Heartbeat, and the CRM shell: # yum install pacemaker-linbit pacemaker-linbit-crmsh heartbeat 18

21 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 4.1. Install and Configure Pacemaker on DR node Once the installation completes, we will need to configure Heartbeat. Create the file, /etc/ha.d/ha.cf with the following contents on charlie: udpport 694 mcast eth autojoin none warntime 5 deadtime 15 initdead 60 keepalive 2 node charlie pacemaker respawn Configure the authentication key for charlie by creating the file /etc/ha.d/authkeys with the following contents: auth 1 1 sha1 ThisIsWhereWePutOurSuperSecretKeyFile Set the authkeys file s permissions so only root has read and write access, and start Heartbeat and Pacemaker: # chmod 600 /etc/ha.d/authkeys # /etc/init.d/heartbeat start After a few moments, our crm_mon should look like this: Last updated: Fri Jun 27 10:20: Last change: Fri Jun 27 09:16: via crmd on charlie Stack: heartbeat Current DC: charlie (00212d60-e5e b55-48a3bf685ba8) - partition with quorum Version: hg bf8451.el6-9bf Nodes configured 0 Resources configured Online: [ charlie ] Now we can begin preparing our standalone Pacemaker configuration for use as our HA cluster s DR solution. We will start by disabling STONITH and placing the cluster into maintenance-mode by issuing the following commands: # crm configure property stonith-enabled="false" # crm configure property maintenance-mode="true" We must set stonith-enabled="false" in a standalone Pacemaker configuration, as there will be no peer. We set maintenance-mode="true", so that services do not start until we want them to; we will set maintenance-mode="false" to start all services on the DR node in the event that we need to failover to the DR site. Configuring Pacemaker from this point forward, should be as simple as copy-and-pasting the configuration from our HA cluster s Pacemaker configuration. We will omit anything that does not apply to the DR site; in our example this includes the lower-level DRBD resource, master/slave definition for the lower-level DRBD resource, DRBD Proxy service, and all colocation constraints. We must also remove or edit any references to the removed primitives from all 19

22 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 4.1. Install and Configure Pacemaker on DR node resource groups, ordering constraints, etc. Use the command, crm configure edit to modify the Pacemaker configuration, paste the standalone configuration below into the editor beneath the configurations already present. Once everything is in place, write the configuration and quit. primitive p_drbd_r0-u ocf:linbit:drbd \ params drbd_resource="r0-u" \ op monitor interval="31s" role="master" \ op monitor interval="29s" role="slave" \ op start interval="0" timeout="240s" \ op stop interval="0" timeout="100s" primitive p_exportfs_drbd ocf:heartbeat:exportfs \ params fsid="1" directory="/mnt/drbd_data" \ options="rw,sync,mountpoint" clientspec=" / " \ wait_for_leasetime_on_stop="false" unlock_on_stop="true" \ op start interval="0" timeout="40s" \ op stop interval="0" timeout="15s" \ op monitor interval="30s" timeout="20s" primitive p_fs_drbd ocf:heartbeat:filesystem \ params device="/dev/drbd10" directory="/mnt/drbd_data" fstype="ext4" \ options="noatime,nodiratime,discard" \ op start interval="0" timeout="60s" \ op stop interval="0" timeout="60s" \ op monitor interval="10s" timeout="40s" primitive p_lsb_nfsserver lsb:nfs \ op start interval="0" timeout="30s" \ op stop interval="0" timeout="30s" \ op monitor interval="30s" timeout="20s" primitive p_virtip_nfs ocf:heartbeat:ipaddr2 \ params ip=" " cidr_netmask="24" \ op start interval="0" timeout="20s" \ op stop interval="0" timeout="20s" \ op monitor interval="10s" timeout="20s" group g_nfs p_fs_drbd p_lsb_nfsserver p_exportfs_drbd p_virtip_nfs \ meta target-role="started" ms ms_drbd_r0-u p_drbd_r0-u \ meta master-max="1" master-node-max="1" clone-max="1" clone-node-max="1" notify="true" order o_drbd_before_nfs inf: ms_drbd_r0-u:promote g_nfs:start Once we ve saved our configuration, our crm_mon output should look similar to this: Last updated: Fri Jun 27 10:20: Last change: Fri Jun 27 09:16: via crmd on charlie Stack: heartbeat Current DC: charlie (00212d60-e5e b55-48a3bf685ba8) - partition with quorum Version: hg bf8451.el6-9bf Nodes configured 5 Resources configured Online: [ charlie ] Master/Slave Set: ms_drbd_r0-u [p_drbd_r0-u] (unmanaged) p_drbd_r0-u (ocf::linbit:drbd): Slave charlie (unmanaged) 20

23 LINBIT DRBD Proxy Configuration Guide on CentOS 6: 4.2. Simulate a Failure of Alpha and Bravo 4.2. Simulate a Failure of Alpha and Bravo To simulate a failure of alpha and bravo, there are many things we can do including simply pulling the power plugs on our nodes. This section of the guide will walk us through accomplishing that using Linux s /proc/sysrq-trigger. Echo ing values to the sysrq-trigger allows us to do a number of useful things on Linux systems; for the purposes of this guide I will cover only one, the poweroff option. Using the command, echo o > /proc/sysrq-trigger, will immediately poweroff the node. This is one of the best ways to simulate a failure of the primary site, since the system will not automatically come back online and attempt to resume cluster services. In a three node cluster similar to the one configured in the last section, we could poweroff the current Primary, which will fail services over to the Secondary. Once services are migrated to the Secondary, poweroff the new Primary, then test our failover procedure to the third (DR) node. To begin our testing, poweroff the current Primary using the following command: # echo o > /proc/sysrq-trigger We can watch the services failover to the Secondary using the crm_mon utility. We should see that all services failover to the Secondary, and DRBD Proxy reconnects the new Primary node to the DR node. On the new Primary, and last node remaining in the HA cluster, run the same command we ran above. At this point, only the DR node should be responsive. Run the following command to view the status of DRBD on the DR node, it should look similar to the output below: # cat /proc/drbd version: (api:1/proto:86-101) GIT-hash: 8ca20467bb9e10aa ac1d65f2b54b4 build by buildsystem@linbit 10: cs:wfconnection ro:secondary/unknown ds:uptodate/dunknown A r----- ns:0 nr: dw: dr:1689 al:15 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos: Resuming Services on the DR Node Since we have a standalone Pacemaker cluster with all our clustered services defined, just waiting in maintenancemode at the DR site, we simply need to login to charlie and run the following command to resume our services: # crm configure propery maintenance-mode="false" If we watch the output of crm_mon, we can see our services starting up on the DR node. They should all start within a few seconds and clients can continue using services on the DR node. The output of crm_mon will look like this if everything started successfully: 21