High Availability of IBM Security Directory Server using Heartbeat A highly available authentication system

High Availability of IBM Security Directory Server using Heartbeat A highly available authentication system Prabir Meher IBM India Software Lab, Pune 2014/02/20, 1.0 Abstract: The purpose of this article is to demonstrate the use of ISDS (IBM Security Directory Server), formerly known as ITDS (IBM Tivoli Directory Server), using Heartbeat for creating a highly available authentication system with fail-over mechanism. High availability is very critical for enterprise authentication services since consolidating any service on a particular server is not at all reliable. Depending on a single server eventually creates a "single point of failure" (SPOF), which can break the entire organization's authentication system. You will see here one method of creating a reliable authentication server using IBM Security Directory Server, which can be adapted by many different applications. We will use the Heartbeat package from the Linux HA project (see the "Resources" section) to design a highly available authentication system using IBM Security Directory Server.

Table of Contents 1Introduction...4 2Environment setup...5 3About IBM Security Directory Server (ISDS)...5 4About Heartbeat...5 5Prerequisite...6 6IBM Security Directory Server replication...7 6.1Log in to Web Administration Tool...8 6.2WAT introduction page...8 6.3Adding servers to WAT...9 6.4Log in to the ISDS instance to configure replication...11 6.5Setting up the replication...11 6.5.1Check replication-related entries in ISDS using command line...23 6.6Synchronize data between two replication servers...23 6.6.1Exporting data...24 6.6.2Exporting data using salt and seed of the destination server...24 How to obtain salt and seed of an ISDS server...25 6.6.3Importing data...25 6.7Resume replication queue...26 7Heartbeat setup...28 7.1Installing Heartbeat...28 7.2Configuring Heartbeat...28 7.2.1The authkeys file...28 7.2.2The ha.cf file...29 7.2.3The haresources file...30 7.3Propagating configuration files to peer servers...30 7.4Starting Heartbeat...31 7.4.1Network configuration before starting heartbeat...31 7.4.2Network configuration after starting heartbeat...34 7.4.3Checking Heartbeat status...34 8Testing high-availability scenario...35 8.1.1Checking the network configuration on ldaphost2...35 9DNS hack...37 10Query ISDS server using virtual IP...37 11Resources...38 Table of Figures Figure 1: Directory operations under normal condition...4 Figure 2: Directory operations under failover condition...5 Figure 3: WAT Console administration login...8 Figure 4: WAT console introduction page...8 Figure 5: Adding servers to WAT...9 Figure 6: Adding servers to WAT(2)...9 Figure 7: WAT servers list...10 Figure 8: WAT Directory server login...11 Figure 9: WAT Introduction page...12 Figure 10: Add replication subtree...12 Figure 11: Browse subtree(1)...13

Figure 12: Select replication suffix...13 Figure 13: Browse subtree (2)...13 Figure 14: Show topology...14 Figure 15: Show topology for o=ibm,c=in...15 Figure 16: Replication pop-up menu...15 Figure 17: Add peer server to replication topology...16 Figure 18: Add credentials (1)...16 Figure 19: Add credentials (2)...17 Figure 20: Add credentials (3)...17 Figure 21: Select credential...18 Figure 22: Add master server...18 Figure 23: Add credential on peer server...19 Figure 24: Collect peer agreement information...19 Figure 25: Enter ldaphost1 credential information...20 Figure 26: Replication setup success information...23 Figure 27: Resume replication queue from ldaphost1 to ldaphost2...27 Figure 28: Resume replication queue from ldaphost2 to ldaphost1...27 Table of Listings Listing 1: "idsilist" output from ldaphost1...6 Listing 2: "idsilist" output from ldaphost2...7 Listing 3: ISDS and admin server running on ldaphost1...7 Listing 4: ISDS and admin server running on ldaphost2...7 Listing 5: ISDS instance list...10 Listing 6: Information added to DIT after adding replication suffix...14 Listing 7: Bind credential...18 Listing 8: Supplier bind credential information...19 Listing 9: Information added to server ldaphost1...21 Listing 10: Information added to server ldaphost2...22 Listing 11: Exporting data with seed and salt...24 Listing 12: ISDS server's crypto salt...25 Listing 13: Copy the exported LDIF file to the target server...25 Listing 14: Importing data into the server...26 Listing 15: View Heartbeat's document...28 Listing 16: The /etc/ha.d/authkeys file...29 Listing 17: authkeys permission...29 Listing 18: Sample ha.cf file...29 Listing 19: Sample haresources file...30 Listing 20: ha_propagate script location...30 Listing 21: ha_propagate...31 Listing 22: ifconfig output before starting heartbeat...32 Listing 23: Starting heartbeat...32 Listing 24: log snippet from /var/log/ha-log on ldaphost1...33 Listing 25: ifconfig output after starting heartbeat...34 Listing 26: Stopping heartbeat...35 Listing 27: ifconfig output from ldaphost2 after stopping heartbeat on ldaphost1...35 Listing 28: log snippet from /var/log/ha-log on ldaphost2 after stopping heartbeat on ldaphost1...36 Listing 29: /etc/hosts file from server ldaphost1 and ldaphost2...37

Listing 30: idsldapsearch using virtual IP/hostname...38 1 Introduction As organizations add applications and services, centralized authentication services can increase security and decrease administrative tasks. However, being dependent on a single server eventually creates a "single point of failure" (SPOF), which can break the entire organization's authentication system. To overcome this SPOF, we will be discussing here how to configure IBM Security Directory Server (ISDS) with Heartbeat to deliver a highly available authentication system. In this article, we will demonstrate one method to create a reliable authentication server using ISDS, which can be adapted by many organization-wide applications. We will use the Heartbeat package from the Linux HA project. Starting with two identical ISDS servers (peer-to-peer replication), several configurations can be used. First, we could do a "cold standby" where the master ISDS server has a virtual IP and a running ISDS instance. The secondary ISDS server sits idle. When the master server fails, the ISDS instance and virtual IP move to the cold node (secondary server). This is a very simple setup to implement. See Figure 1: Directory operations under normal condition. However, the data synchronization between the master and secondary servers could be a problem. To solve that, we will configure the servers with live ISDS instances running on both the servers. In this way, updates to the master server are immediately replicated to the secondary server. Figure 1: Directory operations under normal condition Failure of the master ISDS server leaves our secondary ISDS server available to respond to client queries. See Figure 2: Directory operations under failover condition

Figure 2: Directory operations under failover condition 2 Environment setup The example described in this article is based on a setup that requires: Two ISDS v6.3.1 Servers, installed on RHEL 6.5 64-bit, configured in a peer-to-peer or master-master replication. Server1 hostname ldaphost1.in.ibm.com - 192.168.56.20 Server2 hostname ldaphost2.in.ibm.com - 192.168.56.21 Heartbeat version 3.0.4 installed on both servers (i.e., ldaphost1 and ldaphost2) Note: See the "Resources" section to download ISDS and Heartbeat packages. 3 About IBM Security Directory Server (ISDS) IBM Security Directory Server is an enterprise identity management software that uses the Lightweight Directory Access Protocol (LDAP) and provides a trusted identity data infrastructure for authentication. See the "Resources" section to find out more about IBM Security Directory Server. 4 About Heartbeat In an enterprise environment, certain servers (such as an authentication server) must always be up and running for the business to keep functioning smoothly. These servers provide services that need to be always available. A cornerstone of any mission-critical service that always needs to be up with no downtime is being able to transfer the service from one system to another gracefully. The magic that makes this possible is a service called Heartbeat. Heartbeat is the main product of the High-Availability Linux project.

In this article, we have tested only an active/passive method with two ISDS peer servers, where the active server provides the services and the passive server waits to take over in case the active server goes down. The best part of this method is that you do not need any hardware devices, which tend to be expensive, to build a highly available authentication system. 5 Prerequisite In this section, I assume that you have already installed ISDS on both systems (i.e., ldaphost1 and ldaphost2). I also assume that you have created an ISDS instance dsrdbm01 on both the servers, which runs on port 389. If you have set up the instances properly, then you will see output from both servers similar to the following two listings. [root@ldaphost1 ~]# idsilist -a Directory server instance(s): -------------------------------------Instance 1: Name: dsrdbm01 Version: 6.3.1 Location: /home/dsrdbm01 Description: IBM Security Directory Server Instance V6.3.1 IP Addresses: All available Port: 389 Secure Port: 636 Admin Server Port: 3538 Admin Server Secure Port: 3539 Type: Directory Server Listing 1: "idsilist" output from ldaphost1 Listing 1 shows sample output from ldaphost1.in.ibm.com. Listing 2 shows sample output from ldaphost2.in.ibm.com.

[root@ldaphost2 ~]# idsilist -a Directory server instance(s): -------------------------------------Instance 1: Name: dsrdbm01 Version: 6.3.1 Location: /home/dsrdbm01 Description: IBM Security Directory Server Instance V6.3.1 IP Addresses: All available Port: 389 Secure Port: 636 Admin Server Port: 3538 Admin Server Secure Port: 3539 Type: Directory Server Listing 2: "idsilist" output from ldaphost2 To configure the replication, both the instances on server ldaphost1 and ldaphost2 must be up and running. To confirm that they are, type the following command on both servers: [root@ldaphost1 ~]# netstat -tnlp grep ibm Listing 3shows sample output from ldaphost1: [root@ldaphost1 ~]# netstat -tnlp grep ibm tcp 0 0 :::389 :::* tcp 0 0 :::3538 :::* LISTEN LISTEN 1755/ibmslapd 29489/ibmdiradm LISTEN LISTEN 14158/ibmslapd 4581/ibmdiradm Listing 3: ISDS and admin server running on ldaphost1 Listing 4 shows sample output from ldaphost2: [root@ldaphost2 ~]# netstat -tnlp grep ibm tcp 0 0 :::389 :::* tcp 0 0 :::3538 :::* Listing 4: ISDS and admin server running on ldaphost2 6 IBM Security Directory Server replication Replication is a technique used by directory servers to improve performance, availability, and reliability. The replication process keeps the data in multiple directory servers synchronized. We will set up a peer-to-peer replication using the Web Administration Tool (WAT), which uses a graphical user interface to administer ISDS servers. If you do not have WAT installed on your machine, see "Setting up Tivoli Directory Server replication using the command line," an excellent article on developerworks that presents a step-by-step method of configuring replication in an ISDS server using command-line tools. It can be accessed through the link

provided below in "Resources." For the current article, I have set up WAT on machine ldaphost1.in.ibm.com. 6.1 Log in to Web Administration Tool You can log in to the Web Administration Tool (WAT) using the following url: http://ldaphost1.in.ibm.com:12100/idswebapp/. You will need to change the hostname of the server where you have deployed the WAT. If the above url doesn't work, try the direct login page: http://ldaphost1.in.ibm.com:12100/idswebapp/idsjsp/login.jsp Figure 3: WAT Console administration login Use the default username superadmin and password secret to log in to the WAT. 6.2 WAT introduction page After you have logged in to the WAT for the first time, click on Manage Console Servers to begin adding the servers. Figure 4: WAT console introduction page

6.3 Adding servers to WAT Click on Add... to add your servers to the Web Administration Tool to manage from a remote location or servers. Figure 5: Adding servers to WAT In the Server name field, enter a descriptive name for your ISDS instance. The best practice in creating a server name is to use the hostname followed by the instance name (i.e., hostnameinstance_name). To use only the hostname, leave the field blank. Hostname is the real server's name on which TDS instance is running, and the hostname must be resolvable by your DNS, otherwise WAT will fail to contact the server. On the other hand, Server name is for end user reference to give a meaningful name to the instance. Server name is confined to the WAT portal only, whereas Hostname must be resolvable to a specific IP address. Figure 6: Adding servers to WAT(2)

The panel in Figure 6 also prompts you for a Port number and an Administration port number. Use the command line tool idsilist -a to find out these values. See, for example, the output in Listing 5. [root@ldaphost1 ~]# idsilist -a Directory server instance(s): -------------------------------------Instance 1: Name: dsrdbm01 Version: 6.3.1 Location: /home/dsrdbm01 Description: IBM Security Directory Server Instance V6.3.1 IP Addresses: All available Port: 389 Secure Port: 636 Admin Server Port: 3538 Admin Server Secure Port: 3539 Type: Directory Server Listing 5: ISDS instance list After you have added each server, press OK. Add the rest of the servers that you want to administer remotely using WAT. After all the servers have been added, you can display the list in the Manage console servers page. Figure 7: WAT servers list Now click Logout to log out of the WAT.

6.4 Log in to the ISDS instance to configure replication You can now log in to any of the ISDS servers that you added in the "Adding servers to WAT" section. Here we will log in to the instance dsrdbm01 on server ldaphost1. The LDAP Server Name drop-down field will pop out a list of all the servers that you have added. See Figure 8: WAT Directory server login. Select ldaphost1-dsrdbm01 as the LDAP Server Name and enter the ISDS administrative user ID (in most cases, it is cn=root) and password, which was configured when the ISDS instance dsrdbm01 was created on hostldaphost1. Note: If you do not know the administrative user ID and password, you cannot log in to WAT as a privileged user and cannot configure the replication. Please refer to the command line tool idsdnpw for more information on how to configure the administrative DN and password for an ISDS instance. Figure 8: WAT Directory server login 6.5 Setting up the replication When you click Login see Figure 8 you are presented with the introduction page of the Web Administration Tool, where you can administer your ISDS instance from a local or remote system. See Figure 9: WAT Introduction page.

Figure 9: WAT Introduction page Near the top of Figure 9, two values have been outlined in red. The left box signifies the physical server name and the port on which the ISDS instance is running along with the server name in brackets, which you configured in the "Adding servers to WAT" section. The right box indicates the current logged-in user, which in this case is cn=root (ISDS root administrator). Click on Manage Replication to configure the replication for a particular suffix or subtree. See Figure 10: Add replication subtree Figure 10: Add replication subtree Click on Add subtree... to add the suffix or subtree that you want to replicate. You are presented with a screen similar to Figure 11: Browse subtree(1) to add your subtree.

Now click on Browse... to navigate through the Directory Information Tree (DIT) of your ISDS server. Accept the default value in the Master server referral LDAP URL field. Figure 11: Browse subtree(1) After clicking on OK, as shown in Figure 11: Browse subtree(1), you are presented with a screen similar to Figure 12: Select replication suffix, on which to choose the suffix or subtree to replicate. Figure 12: Select replication suffix Click the radio button on the left to choose the suffix that you want to replicate. Click on Select. You are redirected to a page similar to Figure 13: Browse subtree (2). This screen resembles the one displayed in Figure 11, except that the Subtree DN value is filled in with the selected subtree name: in this case, o=ibm,c=in. Next, click on the OK button to continue. WAT adds the following three replication stanzas (shown in Listing 6: Information added to DIT after adding replication suffix) to the DIT of instance dsrdbm01, which is Figure 13: Browse subtree running on host ldaphost1.in.ibm.com. (2)

Figure 15: Show topology for o=ibm,c=in Click on the Topology for selected subtree button to get a pop-up menu, as shown in Figure 16: Replication pop-up menu. This menu allows you to add a peer master or a replica or even manage your gateway servers. Since we are setting up a peer-to-peer replication topology,we need to add a master server under the server ldaphost1.in.ibm.com which is also a master server for the suffix o=ibm, c=in. Click on Add master to add another peer server to Figure 16: Replication pop-up menu ldaphost1.in.ibm.com. After clicking on Add master, you are presented with a screen similar to that in Figure 17: Add peer server to replication topology:

Figure 17: Add peer server to replication topology Click on the drop-down list for the field Server hostname:port to select your peer master server. Click on Get server ID to get the server ID of ldaphost2.in.ibm.com. You will now need bind credentials which your server ldaphost1.in.ibm.com will use to bind to the server ldaphost2.in.ibm.com and make the necessary changes during replication. If any entry information on server ldaphost1.in.ibm.com changes by any client, then the same operation will be replicated to server ldaphost2.in.ibm.com using these bind credentials. Click Select as shown Figure 17. You will see a screen similar to Figure 18: Add credentials (1). No credentials have yet been created; therefore, the Select credential field is empty. Select the replication suffix (o=ibm,c=in) radio button under Location for credentials and click on Add credentials. Figure 18: Add credentials (1)

You now see a screen similar to Figure 19: Add credentials (2), where you need to enter a CN (Common Name) under which your bind credentials will be stored. Click on Next to continue. Figure 19: Add credentials (2) You now see a screen similar to Figure 20: Add credentials (3), where you need to enter the bind DN and a password. The DN will be used to authenticate the replication changes from server ldaphost1.in.ibm.com to ldaphost2.in.ibm.com. Optionally, you can also enter a description for the bind DN. Figure 20: Add credentials (3) Click Finish on Figure 20: Add credentials (3). You now see a screen similar to Figure 21: Select credential.

Figure 21: Select credential A stanza has been added to your DIT under DN ibm-replicagroup=default,o=ibm,c=in as shown in Listing 7: Bind credential. cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in replicacredentials=manager description=bind to replication server ldaphost2 objectclass=ibm-replicationcredentials objectclass=ibm-replicationcredentialssimple objectclass=top replicabinddn=cn=manager cn=bindcreds Listing 7: Bind credential Refer to "Check replication-related entries in ISDS using command line" to find out the Listing 6 information for your server. Click OK on Figure 21. You see a screen similar to that shown in Figure 22: Add master server. Figure 22: Add master server

Click on Additional to configure the peer replication server (ldaphost2.in.ibm.com). You are presented with a new screen. Scroll down to the Consumer section and add the details as shown in Figure 23: Add credential on peer server. Figure 23: Add credential on peer server Click the Add credential information on consumer check box and type in the admin DN for the peer replication server. In this case, the admin DN is cn=root. Enter the password root. Click OK. The following supplier bind credential information, as shown in Listing 8: Supplier bind credential information, for server ldaphost1.in.ibm.com will be added to server ldaphost2.in.ibm.com. cn=supplier1397407251698, cn=configuration cn=supplier1397407251698 ibm-slapdmasterdn=cn=manager ibm-slapdmasterpw={aes256}ekp6vmxelqjhsnalgywukw== ibm-slapdreplicasubtree=o=ibm, C=IN objectclass=ibm-slapdconfigentry objectclass=ibm-slapdsupplier objectclass=top Listing 8: Supplier bind credential information You now see a screen similar to Figure 24: Collect peer agreement information. Figure 24: Collect peer agreement information Click OK.You are asked to enter the credential information to be added to server ldaphost1.in.ibm.com. See Figure 25: Enter ldaphost1 credential information.

Figure 25: Enter ldaphost1 credential information Enter all the information requested in Figure 25. Click OK. The following stanzas have now been added to server ldaphost1.in.ibm.com (see Listing 9: Information added to server ldaphost1):

cn=supplier1397416499356, cn=configuration cn=supplier1397416499356 ibm-slapdmasterdn=cn=manager ibm-slapdmasterpw={aes256}mu4i4chmyt7oytfazn+hia== ibm-slapdreplicasubtree=o=ibm, C=IN objectclass=ibm-slapdconfigentry objectclass=ibm-slapdsupplier objectclass=top cn=ldaphost2.in.ibm.com:389,ibm-replicagroup=default,o=ibm,c=in objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=7cf882c0-46f0-1033-8038-b508ef5ab1d7 ibm-replicationserverismaster=true cn=ldaphost2.in.ibm.com:389 cn=ldaphost1.in.ibm.com:389,cn=ldaphost2.in.ibm.com:389,ibmreplicagroup=default,o=ibm,c=in ibm-replicamethod=1 ibm-replicaconsumerid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0 ibm-replicationonhold=true ibm-replicacredentialsdn=cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in ibm-replicaurl=ldap://ldaphost1.in.ibm.com:389 objectclass=ibm-replicationagreement objectclass=top cn=ldaphost1.in.ibm.com:389 cn=ldaphost2.in.ibm.com:389,cn=ldaphost1.in.ibm.com:389,ibmreplicagroup=default,o=ibm,c=in ibm-replicamethod=1 ibm-replicaconsumerid=7cf882c0-46f0-1033-8038-b508ef5ab1d7 ibm-replicationonhold=true ibm-replicacredentialsdn=cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in ibm-replicaurl=ldap://ldaphost2.in.ibm.com:389 objectclass=ibm-replicationagreement objectclass=top cn=ldaphost2.in.ibm.com:389 Listing 9: Information added to server ldaphost1 In addition, the following information has been added to server ldaphost2.in.ibm.com (Listing 10: Information added to server ldaphost2):

o=ibm,c=in objectclass=organization objectclass=top objectclass=ibm-replicationcontext o=ibm ibm-replicareferralurl=ldap://ldaphost1.in.ibm.com:389 ibm-replicagroup=default,o=ibm,c=in ibm-replicagroup=default objectclass=ibm-replicagroup objectclass=top cn=ldaphost1.in.ibm.com:389,ibm-replicagroup=default,o=ibm,c=in objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0 ibm-replicationserverismaster=true cn=ldaphost1.in.ibm.com:389 cn=ldaphost2.in.ibm.com:389,ibm-replicagroup=default,o=ibm,c=in objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=7cf882c0-46f0-1033-8038-b508ef5ab1d7 ibm-replicationserverismaster=true cn=ldaphost2.in.ibm.com:389 cn=ldaphost1.in.ibm.com:389,cn=ldaphost2.in.ibm.com:389,ibmreplicagroup=default,o=ibm,c=in ibm-replicamethod=1 ibm-replicaconsumerid=de8e2ec0-bfa2-1032-950c-8d69c0d06df0 ibm-replicationonhold=true ibm-replicacredentialsdn=cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in ibm-replicaurl=ldap://ldaphost1.in.ibm.com:389 objectclass=ibm-replicationagreement objectclass=top cn=ldaphost1.in.ibm.com:389 cn=ldaphost2.in.ibm.com:389,cn=ldaphost1.in.ibm.com:389,ibm-replicagroup=default ibm-replicamethod=1 ibm-replicaconsumerid=7cf882c0-46f0-1033-8038-b508ef5ab1d7 ibm-replicationonhold=true ibm-replicacredentialsdn=cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in ibm-replicaurl=ldap://ldaphost2.in.ibm.com:389 objectclass=ibm-replicationagreement objectclass=top cn=ldaphost2.in.ibm.com:389 cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in replicacredentials=manager description=bind to replication server ldaphost2 objectclass=ibm-replicationcredentials objectclass=ibm-replicationcredentialssimple objectclass=top replicabinddn=cn=manager cn=bindcreds Listing 10: Information added to server ldaphost2

Next, you are presented with a screen similar to that in Figure 26: Replication setup success information. Figure 26: Replication setup success information At this point, you have successfully configured the peer-to-peer replication between host ldaphost1.in.ibm.com and ldaphost2.in.ibm.com. Note: If you already have data under suffix o=ibm,c=in, then you need to synchronize both servers manually. When WAT is used to set up the replication, WAT doesn't synchronize the data between the hosts but replicates only the configuration information essential for replication to work. By default, the replication queue is in suspended mode. You need to resume the queue manually to allow the changes made to both the servers to be replicated. Please refer to the section "Resume replication queue" to find out how to resume the queue using WAT. 6.5.1 Check replication-related entries in ISDS using command line Sometimes it is useful to check the replication-related entries in ISDS to find out the basic reason why the replication is not working: whether an replication entry is missing or whether you have input the replication hostname incorrectly. Use the following command to find out all the replication related entries in ISDS. idsldapsearch -h <hostname/ip> -p <port> -D <admindn> -w <admindn_pw> -s sub -b "" objectclass=ibm-repl* For example, idsldapsearch -h ldaphost1.in.ibm.com -p 389 -D cn=root -w root -s sub -b "" objectclass=ibm-repl* Note: You don't have to specify the port if it's 389, because 389 is the default port for LDAP. If the port number is other than 389, then you must specify the port or else you'll receive an error message. Use the following command to find out the replication-related entries within a subtree. For example, to find out the replication-related entries under suffix "O=IBM,C=IN" idsldapsearch -h ldaphost1.in.ibm.com -D cn=root -w root -s sub -b "O=IBM,C=IN" objectclass=ibm-repl* 6.6 Synchronize data between two replication servers Suppose you already have data under the suffix o=ibm,c=in. After OK on Figure 26: Replication setup success information is clicked, any changes made to or under suffix o=ibm, c=in will be replicated to server ldaphost2.in.ibm.com. But what about the data that was there under o=ibm, c=in on server ldaphost1.in.ibm.com before the replication was set up? No need to worry. All you have to do is to export the data under o=ibm,c=in from server ldaphost1.in.ibm.com into the server ldaphost2.in.ibm.com.

Note: For better performance, synchronize the directory servers that are taking part in the replication cryptographically. See "Resources" for a link to Appendix J in the IBM Security Directory Server Administration Guide. 6.6.1 Exporting data You can export the data from a particular suffix using the command line tool idsdb2ldif. This will create an LDIF file that will contain all the data under a particular suffix. Syntax: idsdb2ldif -I <instance_name> -o <outfile.ldif> Note: If your server instances are not synchronized cryptographically, then exported data (in this case <outfile.ldif> from server ldaphost1.in.ibm.com) cannot be imported into another server ldaphost2.in.ibm.com. In that case, you need to export the data from server ldaphost1.in.ibm.com using the salt and seed of the destination server ldaphost2.in.ibm.com. See "How to obtain salt and seed of an ISDS server" to get the salt value (-t); the seed value should be known to the person who originally created the ISDS instance. 6.6.2 Exporting data using salt and seed of the destination server After you have obtained the seed and salt value of your destination server in this case, ldaphost2.in.ibm.com perform the following command to export the data from server ldapohost1.in.ibm.com. Syntax: idsdb2ldif -I <instance_name> -k <seed_value> -t <salt_value> -o <outfile.ldif> For example, see Listing 11: Exporting data with seed and salt. [root@ldaphost1 sbin]# pwd /opt/ibm/ldap/v6.3.1/sbin [root@ldaphost1 sbin]#./idsdb2ldif -I dsrdbm01 -k abc0123456789 -t 'umu"myj4oct0' -o dsrdbm01_fullbackup.ldif GLPCTL113I Largest core file size creation limit for the process (in bytes): '0'(Soft limit) and '-1'(Hard limit). GLPCTL119I Maximum Data Segment(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 262144. GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is 2097152. GLPCTL122I Maximum Open Files soft ulimit for the process is 1024 and the prescribed minimum is 500. GLPCTL122I Maximum Stack Size(Kbytes) soft ulimit for the process is 10240 and the prescribed minimum is 10240. GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 1048576. GLPSRV221I Replication of security attributes feature is disabled. GLPSRV200I Initializing primary database and its connections. GLPD2L011I 61 entries have been successfully exported from the directory. Listing 11: Exporting data with seed and salt

How to obtain salt and seed of an ISDS server Obtain the encryption salt value by performing the following search: ldapsearch -D <admindn> -w <adminpw> -b "cn=crypto,cn=localhost" objectclass=* ibm-slapdcryptosalt For example, to obtain the salt value of ldaphost2.in.ibm.com, execute the following command: [root@ldaphost1 ~]# idsldapsearch -h ldaphost2.in.ibm.com -L -D cn=root -w root -s base -b cn=crypto,cn=localhost objectclass=* ibm-slapdcryptosalt dn: cn=crypto,cn=localhost ibm-slapdcryptosalt: umu"myj4oct0 Listing 12: ISDS server's crypto salt Note: For security purposes, the encryption seed value is not stored anywhere in the server. While creating the ISDS instance, be sure to make note of it so that you can remember it later. 6.6.3 Importing data You can import an LDIF file into an ISDS server using a command line tool called idsldif2db. Now you can copy the newly generated LDIF file to the destination server ldaphost2.in.ibm.com so that you can import it into the ISDS instance. Remote copy the LDIF file as shown in Listing 13: Copy the exported LDIF file to the target server (on a Linux system): [root@ldaphost1 ~]# scp dsrdbm01_fullbackup.ldif ldaphost2.in.ibm.com:~ The authenticity of host 'ldaphost2.in.ibm.com (192.168.56.21)' can't be established. RSA key fingerprint is ab:20:99:59:74:25:dc:96:20:47:e5:ae:7e:9c:a5:90. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ldaphost2.in.ibm.com' (RSA) to the list of known hosts. root@ldaphost2.in.ibm.com's password: dsrdbm01_fullbackup.ldif 100% 44KB 43.7KB/s 00:00 [root@ldaphost1 ~]# Listing 13: Copy the exported LDIF file to the target server

On the destination server, ldaphost2.in.ibm.com, execute the command shown in Listing 14: Importing data into the server to import the LDIF file. [root@ldaphost2 sbin]#./idsldif2db -I dsrdbm01 -i ~/dsrdbm01_fullbackup.ldif GLPCTL113I Largest core file size creation limit for the process (in bytes): '0'(Soft limit) and '-1'(Hard limit). GLPCTL119I Maximum Data Segment(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 262144. GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is 2097152. GLPCTL122I Maximum Open Files soft ulimit for the process is 1024 and the prescribed minimum is 500. GLPCTL122I Maximum Stack Size(Kbytes) soft ulimit for the process is 10240 and the prescribed minimum is 10240. GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is 1048576. GLPCOM022I The database plugin is successfully loaded from libback-config.so. GLPSRV221I Replication of security attributes feature is disabled. GLPSRV200I Initializing primary database and its connections. GLPRPL137I Restricted Access to the replication topology is set to false. GLPRDB052E Entry CN=IBMPOLICIES already exists. GLPRDB052E Entry IBM-REPLICAGROUP=DEFAULT,CN=IBMPOLICIES already exists. GLPRDB052E Entry globalgroupname=globaladmingroup,cn=ibmpolicies already exists. GLPRDB052E Entry cn=pwdpolicy,cn=ibmpolicies already exists. GLPRDB052E Entry CN=REPLICATION,CN=IBMPOLICIES already exists. GLPRDB052E Entry o=ibm,c=in already exists. GLPRDB052E Entry ibm-replicagroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=ldaphost1.in.ibm.com:389,ibm-replicagroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=bindcreds,ibm-replicagroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=ldaphost2.in.ibm.com:389,ibm-replicagroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=ldaphost1.in.ibm.com:389,cn=ldaphost2.in.ibm.com:389,ibmreplicagroup=default,o=ibm,c=in already exists. GLPRDB052E Entry cn=ldaphost2.in.ibm.com:389,cn=ldaphost1.in.ibm.com:389,ibmreplicagroup=default,o=ibm,c=in already exists. GLPRDB002W ldif2db: 49 entries have been successfully added out of 61 attempted. Listing 14: Importing data into the server Note: The ISDS instance must be stopped before importing the LDIF file; otherwise it will not allow you to import the data into it. Also, you can ignore the error message GLPRDB052E shown in Listing 14, as the entries were already present on the destination server, ldaphost2.in.ibm.com. The idsldif2db utility will not change these entries; it will simply skip them. At this point, both the source server (ldaphost1.in.ibm.com) and the destination server (ldaphost2.in.ibm.com) are perfectly in sync. Now you can go ahead and resume the replication queue to allow both the servers to replicate data to each other. 6.7 Resume replication queue Log in to the instance that is running on server ldapohost1.in.ibm.com and click on Replication management on the left pane. Under it, click on Manage queues.

Figure 27: Resume replication queue from ldaphost1 to ldaphost2 Click on Suspend/resume to resume the replication queue. If ldaphost2.in.ibm.com is displayed in the Replica column, the replication queue flow is from server ldaphost1.in.ibm.com to ldaphost2.in.ibm.com. Next, log in to the instance that is running on server ldaphost2.in.ibm.com and click on Replication management on the left pane. Under it, click on Manage queues. Figure 28: Resume replication queue from ldaphost2 to ldaphost1 Click on Suspend/resume to resume the replication queue. If ldaphost1.in.ibm.com is displayed in the Replica column, the replication queue flow is from server ldaphost2.in.ibm.com to ldaphost1.in.ibm.com. Now both the servers are perfectly in sync and the replication queue is in Ready state. We will now

configure high availability (HA) for these servers. 7 Heartbeat setup Heartbeat is very flexible and powerful. In this article, I have touched on only the basic active/passive mode with two servers, where the active server (ldaphost1.in.ibm.com) is providing the service and the passive server (ldaphost2.in.ibm.com) is waiting to take over if necessary. 7.1 Installing Heartbeat Installing the Heartbeat package from the yum repository is always easier than building it from the source. The Heartbeat package is available in the EPEL (Extra Package for Enterprise Linux) repository. See "Resources" for download information. Execute the following command to install Heartbeat: yum install heartbeat --enablerepo=epel 7.2 Configuring Heartbeat You need three configuration files to work with Heartbeat. Each goes into the directory /etc/ha.d/ authkeys File containing keys for mutual node authentication ha.cf Main Heartbeat configuration file haresources Resource configuration file The ha.cf and hareources files may be readable by everyone, but the authkeys file must not be. The good news is that sample versions of these files may be found in the documentation directory. If you installed Heartbeat using yum, then the following command will show you where they are on your system: rpm -q heartbeat -d For example, see Listing 15: View Heartbeat's document. [root@ldaphost1 ~]# rpm -q heartbeat -d...... /usr/share/doc/heartbeat-3.0.4/authkeys /usr/share/doc/heartbeat-3.0.4/ha.cf /usr/share/doc/heartbeat-3.0.4/haresources...... Listing 15: View Heartbeat's document 7.2.1 The authkeys file The authkeys file must be owned by root and be set to chmod 600. The actual format of the authkeys file is very simple; it consists of only two lines. The first line contains an auth directive with an associated method ID number, and the second line identifies the authentication method and the key that go with the auth directive.

Three authentication methods are supported: CRC, MD5, and SHA1. CRC doesn't accept a key. You normally have only one authentication method listed in a CRC file. It adds no security, except from packet corruption, and should be used only on physically secure networks. Of the remaining two, SHA1 is usually considered to be the best, followed by MD5. Listing 16: The /etc/ha.d/authkeys file shows an example. Make the key long as it will improve security and you will not have to type it again. auth 1 1 sha a9e2189843e9c3fb7c5a1bf81587fc11 Listing 16: The /etc/ha.d/authkeys file Check the permission on the file /etc/ha.d/authkeys. If the permission is not 600, then change it to 600 as shown in Listing 17: authkeys permission: [root@ldaphost1 ~]# ls -lh /etc/ha.d/authkeys -rw-r--r-- 1 root root 46 Apr 19 01:51 /etc/ha.d/authkeys [root@ldaphost1 ~]# chmod 600 /etc/ha.d/authkeys [root@ldaphost1 ~]# ls -lh /etc/ha.d/authkeys -rw------- 1 root root 46 Apr 19 01:51 /etc/ha.d/authkeys Listing 17: authkeys permission 7.2.2 The ha.cf file This is the main Heartbeat configuration file. The content of this file should be same on both the servers with a couple of exceptions. Heartbeat ships with an example file in the documentation directory that is well worth a look. Also, when creating your ha.cf file, the order in which things appear matters. Do not move them around! debugfile /var/log/ha-debug logfile /var/log/ha-log logfacility local0 bcast eth0 warntime 5 deadtime 15 initdead 60 keepalive 2 auto_failback on node ldaphost1.in.ibm.com node ldaphost2.in.ibm.com Listing 18: Sample ha.cf file The auto_failback on command tells Heartbeat to prefer ldaphost1.in.ibm.com. If ldaphost1.in.ibm.com should go down, ldaphost2.in.ibm.com will take over the Virtual IP Address and start the highly available services. When ldaphost1.in.ibm.com comes back online, ldaphost2.in.ibm.com will transfer (automatically failback) the Virtual IP and continue serving the

clients without any downtime. You should familiarize yourself with the sample ha.cf file that comes with the Heartbeat package. Refer to the Listing 15: View Heartbeat's document to find out more about ha.cf options. Also see the "Resources" section for a link to the official Heartbeat documentation. 7.2.3 The haresources file This file should be exactly the same on both the servers. ldaphost1.in.ibm.com IPaddr::192.168.56.252/24/eth0:1/192.168.56.255 Listing 19: Sample haresources file ldaphost1.in.ibm.com is the hostname of our first or primary node, and IPaddr::192.168.56.252/24/eth0:1/192.168.56.255 is the virtual IP address with a subnet mask 24. It will be attached to an Ethernet interface eth0:1 with a broadcast address of 192.168.56.255. See the official Heartbeat documentation to learn more about the options available. 7.3 Propagating configuration files to peer servers The Heartbeat package provides a script ha_propagate to copy the configuration files over to peer servers that are listed in the ha.cf file with a node directive. By default, the script ha_propagate copies only the two files authkeys and ha.cf. However, I have tweaked my version of ha_propagate to copy all the three files to the peer servers. See the "Resources" section to download the tweaked version of the script. If you want to find out where the default ha_propagate script is located, then execute the following command: # rpm -ql heartbeat grep -i ha_propagate /usr/share/heartbeat/ha_propagate Listing 20: ha_propagate script location Since we have created all the three files (authkeys, ha.cf, and haresources) in ldaphost1.in.ibm.com, we need to copy them over to ldaphost2.in.ibm.com. Execute the downloaded script to copy the three configuration files. See Listing 21: ha_propagate

[root@ldaphost1 heartbeat]# pwd /usr/share/heartbeat [root@ldaphost1 heartbeat]#./ha_propagate Propagating HA configuration files to node ldaphost2.in.ibm.com. ha.cf 100% 259 0.3KB/s 00:00 authkeys 100% 47 0.1KB/s 00:00 haresources 100% 69 0.1KB/s 00:00 Setting HA startup configuration on node ldaphost2.in.ibm.com....... heartbeat 0:off 1:off 2:on 3:on 4:on 5:on 6:off httpd 0:off 1:off 2:off 3:off 4:off 5:off 6:off ip6tables 0:off 1:off 2:off 3:off 4:off 5:off 6:off...... Listing 21: ha_propagate When you execute the ha_propagate script, it may ask you for the password to the remote system to which the configuration files are being copied. Provide the password to continue. In the above case, since both the servers are configured for passwordless ssh, ha_propagate didn't prompt for any password. Note: ha_propagate uses scp to copy the files to the remote server. If you have set up passwordless ssh, it will not ask you for the remote system's password. 7.4 Starting Heartbeat Now that both the servers (ldaphost1 and ldaphost2) have the same Heartbeat configuration files, you are almost ready to start the Heartbeat service on both. 7.4.1 Network configuration before starting heartbeat Before starting Heartbeat, check your network configuration with the command line tool ifconfig. See Listing 22: ifconfig output before starting heartbeat

[root@ldaphost1 ~]# ifconfig eth0 Link encap:ethernet HWaddr 08:00:27:63:97:31 inet addr:192.168.56.20 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe63:9731/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:27950 errors:0 dropped:0 overruns:0 frame:0 TX packets:23545 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5230588 (4.9 MiB) TX bytes:5613926 (5.3 MiB) lo Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:239275 errors:0 dropped:0 overruns:0 frame:0 TX packets:239275 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:34065653 (32.4 MiB) TX bytes:34065653 (32.4 MiB) Listing 22: ifconfig output before starting heartbeat Now start the Heartbeat service on server ldaphost1.in.ibm.com and ldaphost2.in.ibm.com. [root@ldaphost1 ~]# service heartbeat start Starting High-Availability services: INFO: Resource is stopped Done. [root@ldaphost2 ~]# service heartbeat start Starting High-Availability services: INFO: Resource is stopped Done. Listing 23: Starting heartbeat Next, check the log file /var/log/ha-log on the primary server, which is ldaphost1.in.ibm.com in this case. Refer to Listing 18: Sample ha.cf file.

Apr 20 00:26:47 ldaphost1.in.ibm.com heartbeat: [30705]: info: Pacemaker support: false... Apr 20 00:26:47 ldaphost1.in.ibm.com heartbeat: [30705]: info: Configuration validated. Starting heartbeat 3.0.4 Apr 20 00:26:47 ldaphost1.in.ibm.com heartbeat: [30706]: info: heartbeat: version 3.0.4...output omitted...... Apr 20 00:26:49 ldaphost1.in.ibm.com heartbeat: [30706]: info: Local status now set to: 'up' Apr 20 00:26:49 ldaphost1.in.ibm.com heartbeat: [30706]: info: Link ldaphost1.in.ibm.com:eth0 up. Apr 20 00:26:57 ldaphost1.in.ibm.com heartbeat: [30706]: info: Link ldaphost2.in.ibm.com:eth0 up. Apr 20 00:26:57 ldaphost1.in.ibm.com heartbeat: [30706]: info: Status update for node ldaphost2.in.ibm.com: status up...output omitted...... ResourceManager(default)[30916]: 2014/04/20_00:27:17 info: Acquiring resource group: ldaphost1.in.ibm.com ResourceManager(default)[30916]: 2014/04/20_00:27:21 info: Running /etc/ha.d/resource.d/ipaddr 192.168.56.252/24/eth0:1/192.168.56.255 start IPaddr(IPaddr_192.168.56.252)[31072]: 2014/04/20_00:27:24 INFO: Adding inet address 192.168.56.252/24 with broadcast address 192.168.56.255 to device eth0 (with label eth0:1) IPaddr(IPaddr_192.168.56.252)[31072]: 2014/04/20_00:27:24 INFO: Bringing device eth0 up...output omitted...... /usr/lib/ocf/resource.d//heartbeat/ipaddr(ipaddr_192.168.56.252)[31046]: 2014/04/20_00:27:25 INFO: Success Apr 20 11:06:01 ldaphost1.in.ibm.com heartbeat: [30706]: info: No pkts missing from ldaphost2.in.ibm.com! Listing 24: log snippet from /var/log/ha-log on ldaphost1

7.4.2 Network configuration after starting heartbeat [root@ldaphost1 ~]# ifconfig eth0 Link encap:ethernet HWaddr 08:00:27:63:97:31 inet addr:192.168.56.20 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe63:9731/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:34202 errors:0 dropped:0 overruns:0 frame:0 TX packets:29170 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6692040 (6.3 MiB) TX bytes:7078873 (6.7 MiB) eth0:1 Link encap:ethernet HWaddr 08:00:27:63:97:31 inet addr:192.168.56.252 Bcast:192.168.56.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:239275 errors:0 dropped:0 overruns:0 frame:0 TX packets:239275 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:34065653 (32.4 MiB) TX bytes:34065653 (32.4 MiB) Listing 25: ifconfig output after starting heartbeat If you see the above ifconfig output, a new Ethernet alias eth0:1 has been added with an IP address of 192.168.56.252 as specified in the file /etc/ha.d/haresources. 7.4.3 Checking Heartbeat status You can check the status of the Heartbeat service from any server listed in the node directive as specified in the ha.cf file. Use the command line tool /usr/bin/cl_status,which is provided by the Heartbeat package. Refer to Listing 18: Sample ha.cf file. Check the list of the servers that are configured to run heartbeat [root@ldaphost1 ~]# cl_status listnodes ldaphost2.in.ibm.com ldaphost1.in.ibm.com Check the heartbeat status on a particular server [root@ldaphost1 ~]# cl_status hbstatus Heartbeat is running on this machine. [root@ldaphost2 ~]# cl_status hbstatus Heartbeat is stopped on this machine. [root@ldaphost1 ~]# cl_status nodestatus ldaphost1.in.ibm.com active [root@ldaphost1 ~]# cl_status nodestatus ldaphost2.in.ibm.com dead If you see the above output, since Heartbeat is running on ldaphost1, it is shown as being "active." Server ldaphost2 is shown as being "dead."

Now start Heartbeat on server ldaphost2. See Listing 23. Check the Heartbeat status on server ldaphost2: [root@ldaphost2 ~]# cl_status hbstatus Heartbeat is running on this machine. [root@ldaphost1 ~]# cl_status nodestatus ldaphost2.in.ibm.com active 8 Testing high-availability scenario At this point, Heartbeat is configured on both the servers and is running successfully as we saw in the section "Checking Heartbeat status." In order to test the high-availability scenario, we can simulate a system failure by stopping Heartbeat on one server (ldaphost1.in.ibm.com) so that the peer server (ldaphost2.in.ibm.com) will not receive any Heartbeat response and will therefore assume that the system is dead. Stop the Heartbeat service on server ldaphost1.in.ibm.com. [root@ldaphost1 ~]# service heartbeat stop Stopping High-Availability services: Done. Listing 26: Stopping heartbeat 8.1.1 Checking the network configuration on ldaphost2 [root@ldaphost2 ~]# ifconfig eth0 Link encap:ethernet HWaddr 08:00:27:03:6B:CD inet addr:192.168.56.21 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe03:6bcd/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:28666 errors:0 dropped:0 overruns:0 frame:0 TX packets:24660 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6159029 (5.8 MiB) TX bytes:6471877 (6.1 MiB) eth0:1 Link encap:ethernet HWaddr 08:00:27:03:6B:CD inet addr:192.168.56.252 Bcast:192.168.56.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 lo Link encap:local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:155 errors:0 dropped:0 overruns:0 frame:0 TX packets:155 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:21318 (20.8 KiB) TX bytes:21318 (20.8 KiB) Listing 27: ifconfig output from ldaphost2 after stopping heartbeat on ldaphost1