ITDS Replication August, By: Roy Spencer. IBM Tivoli Software 09/24/11

Transcription

1 IBM Tivoli Software ITDS Replication August, 2011 By: Roy Spencer Support Technical Exchange Web site

2 Introduction Abstract: This STE will discuss ITDS 6.3 replication and will cover topics such as planning your topology, implementing replication in environment, and troubleshooting replication errors Objectives: 1. Planning an effective replication topology 2. Implementation of a replication topology via both webadmin and command line 3. Best replication debugging practices 2

3 ITDS 6.3 Replication Presentation Road Map Part 1 Preparing the instances that will be used in the topology Part 2 Configuring Replication via ITDS 6.3 WebAdmin and command line Part 3 Advanced configuration Topics Part 4 Replication Troubleshooting 3

4 Agenda Part 1 1. Useful Links 2. Why Replicate? 3. Planning an effective replication topology 4. Preparing the systems for replication 5. Deep Dive into the concept of synchronizing two-way cryptography between server instances 6. Questions 4

5 Useful Resources before we get started Info Center for ITDS toc=/com.ibm.ibmds.doc/toc.xml Admin Guide: doc/admin_gd.htm Troubleshooting Replication: Example of 6.0 Configurations: Command line Replication Config. Webadmin Replication Config. 5

6 Why would I use replication? To provide failover functionality in case of a catastrophic hardware or software failure - Redundancy Balance read traffic Allow offline backup without taking an outage Application balancing depending on level of read or write level access required Example: Read only to the public, read-write to the private networks 6

7 Planning an effective replication topology 1. Identify systems for use which meet the requirements to run ITDS 6.3 (consult the ITDS 6.3 Systems Requirements Guide) 2. Plan the correct physical layout - Firewall vs Public - Application needs 3. Effectively plan how to direct traffic at a SINGLE master for write traffic - Load Balancer configuration - Correct application configuration 4. It is always a best practice to apply the recommended version of ITDS to ALL ITDS systems in a topology: 7

8 Types of Toplogies There are 4 basic replication topologies: Peer to Peer Supplier to Consumer Gateway Replication Forwarders For this STE we will discuss the most common topologies of Peer to Peer and the Supplier to Consumer configurations 8

9 Peer to Peer vs Master/Consumer In a Peer to Peer relationship each ITDS server has both read and write access to the data Should Peer1 fail Peer2 will keep track of all changes made and will update Peer1 when it is recovered (note: application stack would need to be pointed to Peer2 via either proxy or load balancer) In a Supplier/Consumer relationship only the Supplier has read and write access to the data, while the Consumer is a read-only server. 9

10 The boss walks in and says We need to have two ITDS 6.3 servers behind a Load Balancer on the intranet, but we also need a read only copy of this data on the external network. So we need 2 peers and one consumer for this example. 10

11 First we need to establish our Authoritative Master This means we need to pick 1 of the 3 servers which has the most up-todate data. We will use that system to build the other two. For this example I chose my server named: - peer1.austin.ibm.com Assumptions made about peer1 for this example: - That peer1 is has all desired schema changes - cn=ibmpolicies default replication agreements are clean (optional) - User Data is up to date - Peer1 system is production ready - The ibm-slapdserverid is set to PEER 1 serverid (optional) - That ITDS has been applied to all servers and that webadmin is at version: ITDS is now available Lets quickly dig into these assumptions 11

12 Instance Prep - Schema on Authoritative Master Assumption 1 That peer1 is has all desired schema changes Validating schema The schema is essentially the way ibmslapd will construct and store data to the repository (db2). For more information check out the Managing the IBM Directory schema section of the 6.3 Admin Guide Example of the schema used on peer1.austin.ibm.com 12 -rw-rw----rw-rw----rw-rw----rw-rw----rw-rw----rw-rw----rw-r-----rw-rw----rw-rw----rw-rw----rw-rw----rw-rw----rw-rw----rw-rw----rw-rw----rwsr-s---rwxrwx---rw-rw----rw-rw peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap 1 root idsldap 1 peer1 idsldap 1 peer1 idsldap 1 peer1 idsldap Jul 13 11:24 V3.config.at Jul 13 11:24 V3.config.oc Jul 13 11:24 V3.ibm.at Jul 13 11:24 V3.ibm.oc 3425 Jul 13 11:24 V3.ldapsyntaxes 3262 Jul 13 11:24 V3.matchingrules 443 Aug 29 08:16 V3.modifiedschema Jul 13 11:24 V3.system.at Jul 13 11:24 V3.system.oc Jul 13 11:24 V3.user.at Jul 13 11:24 V3.user.oc Aug 29 08:07 ibmslapd.conf Aug 29 07:47 ibmslapd.conf.save 104 Jul 13 11:24 ibmslapdcfg.ksf 104 Jul 13 11:24 ibmslapddir.ksf Jul 13 11:24 idsrun 68 Jul 13 11:24 ldapdb.properties 2131 Jul 13 11:24 perftune_input.conf 1142 Jul 13 11:24 tdsdelref.conf All instances in a replication topology will need to have the same schema

13 A quick review of customized schema on Peer1 #cat /home/peer1/idsslapd-peer1/etc/v3.modifiedschema objectclasses { ( roys-oc-oid NAME 'roys-objectclass' DESC 'my replication ste objectclass' SUP top AUXILIARY MAY roys-custom-attr ) } attributetypes { ( roys-cust-attr-oid NAME 'roys-custom-attr' DESC 'roys custom attribute for ITDS 6.3 replication STE' SYNTAX SINGLE-VALUE ) } IBMattributeTypes { ( roys-cust-attr-oid DBNAME ( 'royscustattr' 'royscustattr' ) EQUALITY ) } ldapsyntaxes { } matchingrules { } Note: In order to use the 'roys-objectclass' objectclass or the 'roys-custom-attr' on Peer2/Replica1, I will first have to provide Peer2/Replica1 with the schema definition set in the instances V3.modifiedschema file. We will dig into the importance of this topic in more detail when we discuss crypto sync later in this presentation in slide 40 13

14 Instance Prep cn=ibmpolicies - webadmin Assumption 2: cn=ibmpolicies default replication agreements are clean (optional) This step is optional. When we configure ITDS 6.3 instances for the first time the cn=ibmpolicies replicationcontext is already present. 14

15 Instance Prep cn=ibmpolicies clean up via webadmin WebAdmin -> Login as cn=root/pwd -> Replication Management -> Manage topology -> select cn=ibmpolicies -> drop menu to Delete subtree -> Go Click OK 15

16 Instance Prep cn=ibmpolicies clean up via command line We can also accomplish the clean up via command line by first determinging the cn=ibmpolicies replication related entries with : #idsldapsearch D cn=root w? b cn=ibmpolicies s sub objectclass=ibm-repl* CN=IBMPOLICIES cn=ibmpolicies objectclass=container objectclass=top objectclass=ibm-replicationcontext IBM-REPLICAGROUP=DEFAULT,CN=IBMPOLICIES objectclass=top objectclass=ibm-replicagroup ibm-replicagroup=default Commands for removal. 16

17 Instance Prep cn=ibmpolicies clean up via command line continued 1. delete the child object: #idsldapdelete -D cn=root -w **** -p 389 k IBM-REPLICAGROUP=DEFAULT,CN=IBMPOLICIES Should return: Deleting entry IBM-REPLICAGROUP=DEFAULT,CN=IBMPOLICIES 2. Remove the ibm-replicationcontext from cn=ibmpolicies: #idsldapmodify D cn=root w **** -p 389 k dn: cn=ibmpolicies changetype: modify delete: objectclass objectclass: ibm-replicationcontext Should return: Operation 0 modifying entry CN=IBMPOLICIES 17

18 Instance Prep cn=ibmpolicies - Validate Via command line our ldapsearch for the ibm-repl* filter should now return zero results when cn=ibmpolicies (or any subtree) is the base: #idsldapsearch D cn=root w? b s sub objectclass=ibm-repl* Or via the webadmin: 18

19 Instance Prep User Data up to date. Assumption 3 - User Data is up to date For example: This is simply saying that application or user related data should be up to date prior to replication configuration. For example: Stop ITDI feeds Stop recon s etc 19

20 Instance Prep Production Ready? Assumption 4: Instance is production ready This assumption basically states that the system we have defined as our Authoritative Master already contains the data required for day to day operation. For example, in this case the peer1.austin.ibm.com instance currently contains 50 sample entries. This is simply saying that since part of this process will consist of backing up these 50 entries on peer1 and importing that data to peer2, it would be appropriate to have the entries on peer1 before we begin. 20

21 Instance Prep What is the ibm-slapdserverid Assumption 5 (Optional). The ibm-slapdserverid is set to peer1 One of the items used as part of a replication topology configuration in ITDS is the identification of each instance. The servers will rely on a setting in the ibmslapd.conf of: ibm-slapdserverid=(some value) For instance in the lab I see: ibm-slapdserverid: 1e e94-102f-87d8-eab7a0cc83e2 That id was randomly generated when this instance was started for the first time. I want to change this to the value PEER 1 serverid as that is a name I will recognize. Note: the ibm-serverid must be unique to each system in the topology Method to accomplish this goal 21

22 Instance Prep Setting the ibm-slapdserverid to peer1 General process is: Stop ibmslapd User your favorite editor to modify the instances ibmslapd.conf Update the value for ibm-slapdserverid to desired value ( peer1 in this example) Save the ibmslapd.conf Restart the instance Command to validate the change took effect: #idsldapsearch -D cn=root -w secret -s base objectclass=* ibm-serverid Should return the value: ibm-serverid=peer 1 serverid Note: This must be done prior to replication configuration as the ibmslapdserverid is used as part of a replication agreement. 22

23 Instance Prep Validating ITDS 6.3 Versions Assumption 6 ITDS Server and WebAdmin Versions are current (or later) ITDS # idsslapd -v GLPSRV033I Server version is (Jun :10:44). Validate Webadmin #cd /opt/ibm/ldap/v6.3/idstools./deploy_idswebapp -v /opt/ibm/ldap/v6.3/idstools/deploy_idswebapp: Command Version: Command Date: 01/18/ / opt/ibm/ldap/v6.3/appsrv/profiles/tdswebadminprofile/installedapps/defaultnode/idswebapp.war.e ar/idswebapp.war: <app-version>6.0003</app-version> <build-date>mon 06/20/2011</build-date>

24 Validating product versions: The following document contains the version of ITDS and Webadmin shipped with each release: The recommended fixes page contains the version currently suggested 24

25 Taking stock of where we are The only system defined in our topology is peer1 The instance is production ready The instances serverid is set to the value: PEER 1 serverid Peer1 system is ready to configure replication! 25

26 Preparing the TARGET systems for replication There are 5 tasks which must be taken on each peer/consumer prior to configuring replication: 1. New ITDS 6.3 instances configured on Peer2 and Replica1 2. Instances cryptographically synced with Peer1 3. Schema files on Peer2/Replica1 match Peer1 (note: slides 12 and 13) 4. Removal of default cn=ibmpolicies replication agreements (optional slides 14 18) 5. Set the ibm-slapdserverid to something recognizable (optional slide 21) Lets quickly discuss each step 26

27 New instances configured on Peer2 and Replica1 Before we begin our replication configuration we must configure new and blank instances on Peer2 and Replica1 If an existing instance is on this system we need to drop the data from the database and reconfigure Resyncing Replication If this is a newly installed system we need to configure for first time use. The steps to accomplish this can be found in the ITDS 6.3 Install and Configuration Guide: Or you could check out the STE we did on install at ITDS 6.3: Attached to this presentation are the commands use in file: ITDS6.3.Replication.CheatSheet.txt (see slide: 123) 27

28 Configuring Peer/Consumer Instances. Instance Info: Instance Info: Name: replica1 Name: peer2 Version: 6.3 Version: 6.3 Location: /home/replica1 Location: /home/peer2 Description: IBM Tivoli Directory Server Instance V6.3 Description: IBM Tivoli Directory Server Instance V6.2 IP Addresses: All available IP Addresses: All available Port: Port: Secure Port: Secure Port: Admin Server Port: 3572 Admin Server Port: 3562 Admin Server Secure Port: 3573 Admin Server Secure Port: 3563 Type: Directory Server Type: Directory Server In this case the value used for the salt was randomly generated. #idsicrt -I peer2 -e seedforpeer2 -t peer2 -l /home/peer2 Ibmslapd.conf: ibm-slapdserverid: PEER 2 serverid (see slide 21-22) In this case the instance was created using a different seed and salt than peer1: idsicrt -I replica1 -e seedforrepl2 -t replica1 -l /home/replica1 28 Ibmslapd.conf: ibm-slapdserverid: replica (see slide 21-22) YOU MUST CRYPTO SYNC THIS INSTANCE WITH PEER1 BEFOR YOU START THE peer2 INSTANCE for the first time!!! STILL NEED TO CRYPTO SYNC because the same seed and salt values as peer1 were not used on instance creation (slide ) Since the same seed/salt values were not used on instance creation a sync will be required for this instance Operating System: Linux Operating System: AIX

29 Cryptographically Syncing Database Instances What the heck does cryptographically syncing mean and why would I do it? In a nutshell this is simply the way sensitive data is encrypted and stored within the directory when using a two way encryption scheme. By syncing this cryptography method we save overhead and make the data more secure in transfer between Peers and Replicas (i.e. we send encrypted sensitive data as opposed to decrypting, transmitting in clear text and re-encrypting) 29

30 How do I cryptographically sync??? There are several good resources for instructions on this topic: IBM Tivoli Directory Server, Version Installation, instance creation, configuration, and upgrade overview: Appendix J. Synchronizing two-way cryptography between server instances: %2Fadmin_gd538.htm&path=8_4_7_9 Notes on the ibmslapdcfg.ksf file, ibmslapddir.ksf file and cn=crypto,cn=localhost Syncing data between two servers Note: the ibmslapdcfg.ksf is unique to each system 30

31 So we have synchronized our two-way cryptography But WHY?!? In ITDS 6.X if you export data from one system and that data contains any AES encrypted data, you must crypto sync before the data (ldif) will load. For example if you try to load an ldif into an instance which has not been synced you will get the error: #idsldif2db -I someinst -i full_backup_8.ldif GLPCRY007E The directory key stash file is inconsistent with the associated encrypted data. GLPL2D017E ldif2db unable to run because of initialization error. As we will see in the next section of this presentation, the ability to load the data taken from our authoritative master will be very important 31

32 Crypto Sync Deeper examination Lets use an example to help solidify the understanding of this function. dn: o=sample To the left lets assume we have a very simple ldif file of a o=sample parent entry and a single child: dc: com objectclass: top objectclass: domain dn: cn=example,o=sample objectclass: top objectclass: inetorgperson cn: example sn: example userpassword: examplepwd 32

33 Crypto Sync Deeper examination slide 2 What is actually exported to ldif when we run the idsdb2ldif command? dn: cn=example,o=sample control: false:: MIQAAADVMIQAAAAmCgEAMIQAAAAdBAxjcmVhdG9yc05hbWUxhAAAAAkEB0NOPVJPT1QwhAAAAD gkaqawhaaaac8ed2nyzwf0zvrpbwvzdgftcdgeaaaagaqwmjaxmta4mjkxnjizmtuunti5nzcxwj CEAAAAJwoBADCEAAAAHgQNbW9kaWZpZXJzTmFtZTGEAAAACQQHQ049Uk9PVDCEAAAAOAoBADCEAA AALwQPbW9kaWZ5VGltZXN0YW1wMYQAAAAYBBYyMDExMDgyOTE2MjMxNS41Mjk3NzFa objectclass: top objectclass: inetorgperson objectclass: organizationalperson objectclass: person cn: example sn: example userpassword: {AES256}gfIcRHA1JMcTnj61CfzJ3A== ibm-entryuuid: f0f05cc0-66a e7-f513f4a2b4cb In order to import this LDIF data into PEER2, we must be able to read the AES encrypted values, this is why cryto sync is required. 33

34 Cryptographically syncing with Command - Seed The first thing I must know is the original seed value that was used when Peer1 was created This was the command used to initially configure the instance: # idsicrt -I peer1 -e passwd4eseed -g passwd4esalt -t peer1 l /home/peer1 The encryption seed that was used was: passwd4eseed NOTE: It is VERY important to understand that the encryption seed is NOT stored anywhere. Support will not be able to recover this value as it cannot be searched. 34

35 Crypto Syncing The Salt The next thing I will require is the salt value used on my authoritative master or peer1. To find this out I run: idsldapsearch h peer1 -D cn=root -w secret -b cn=crypto,cn=localhost objectclass=* cn=crypto,cn=localhost cn=crypto objectclass=ibm-cryptoconfig objectclass=ibm-slapdconfigentry objectclass=top ibm-slapdcryptosync=yp/c0o9ncuon6ill ibm-slapdcryptosalt=t[oqobvqupg@ 35

36 IDSGENDIRKSF So with our salt and encryptseed values we are now ready to build the key file for the Replica1 instance on Linux. Command syntax: idsgendirksf [-s salt [-e encryptseed] -l location [-d debuglevel] [-b outputfile] [-q] [-n]] -v -? My Instance on replica1 was called replica1 My Peer1 Seed: passwd4eseed My Peer1 Salt: Note: the salt value may contain special characters this may effect your shell and the characters may need to be escaped. 36

37 Creating the key with idsgendirksf So based on that information we generate our key on the Peer2 instance: Note: it is best practice to backup the ibmslapddir.ksf file before running this command # idsgendirksf -s t[oqobvqupg@ -e seedforpeer1 -l /home/replica1/idsslapd-replica1/etc/ -n You have chosen to perform the following actions: GLPKEY009I The following directory key stash file will be created: '/home/replica1/idsslapd-replica1/etc/ibmslapddir.ksf'. GLPKEY011I Creating directory key stash file: '/home/replica1/idsslapd-replica1/etc/ibmslapddir.ksf'. GLPKEY012I Created directory key stash file: '/home/replica1/idsslapd-replica1/etc/ibmslapddir.ksf'. 37

38 Checking that the key is owned correctly. For example, I was root when I ran that command so when I check the directory listing: -rw-rw idsldap idsldap 104 Apr 26 15:18 ibmslapdcfg.ksf -rw-r root root 104 Aug 18 20:52 ibmslapddir.ksf So I need to change ownership that: #chown peer2:idsldap ibmslapddir.ksf 38

39 Copying the key file if on the same OS When both peers and replicas are on the same OS we can simply copy the key file from the authoritative master. In this example if I had the same AIX version of OS on both Peer1 and Peer2 I could have simply run: On Peer1: cd /home/ inst_name /idsslapd- inst_name /etc/ cp ibmslapddir.ksf ibmslapddir.ksf.masterkey ftp Peer2 cd /home/ inst_name /idsslapd-inst_name/etc/ bin put ibmslapddir.ksf.masterkey Bye On Peer2: cd /home/ inst_name /idsslapd- inst_name /etc mv ibmslapddir.ksf ibmslapddir.ksf.orig mv ibmslapddir.ksf.masterkey ibmslapddir.ksf 39

40 Copying customized schema As described in slide 13, there is a customized user in this environment which uses customized schema: cn=customeruser,o=sample objectclass=inetorgperson objectclass=organizationalperson objectclass=person objectclass=top objectclass=roys-objectclass sn=customeruser cn=customer cn=customeruser roys-custom-attr=change 6 userpassword=test 40 In order to add that entry, and others like it, I will need the same V3.modifiedschema file in the instance directory on both Peer2 and Replica1 For example: /home/replica1/idsslapd-replica1/etc ls -al V3.mod* -rw-r--r-- 1 replica1 idsldap 443 Aug 29 16:39 V3.modifiedschema

41 Once we have the keys and schema in sync After synchronizing the key cryptographically we are ready to: 1. Start ibmslapd on peer2/replica1 2. Clean up cn=ibmpolicies replication agreements (optional) 3. Update the ibm-slapdserverid entry on peer2/replica1 (optional) 4. Stop ibmslapd Start ibmslapd on peer2/replica1 5. Configure replication 41

42 Start ibmslapd on peer2/replica1 We have several alternatives available to us for starting the ibmslapd process on peer2 or replica1: idsslapd I inst_name Starting via webadmin With ibmdiradm running we can issue: ibmdirctl -D cn=root w ***** start Note: why this startup is important: The first start up of the ibmslapd process creates several objects: - serverid - creation of cn=localhost/cn=ibmpolicies etc - verification of the instances normal startup 42

43 At versions of ITDS less than ITDS 6.3, Clean up cn=ibmpolicies replication agreements By default the cn=ibmpolicies when created has bad replication agreements created, please see technote on this issue: Errors replicating/modifying CN=IBMPOLICIES (or errors modifying the schema) ITDS 6.3 does not have this same concern. However, we cleaned the agreement up in slides

44 Cleaning up cn=ibmpolicies replication agreements in versions <

45 Cleaning up cn=ibmpolicies replication agreements The webadmin will then prompt you to make sure you want to delete the agreement. Click ok 45

46 Cleaning up cn=ibmpolicies replication agreements We can now remove the cn=ibmpolicies replication topology totally: 46

47 The webadmin always wants to make sure Before most tasks will complete in ITDS client or the web admin there is usually a prompt making sure we want to accomplish the task. The same is true for the removal of this replication topology: Click on OK 47

48 What we are left with is a completely clean replication topology on Peer1 : We are now prepared to begin configuring replication on Peer1. 48

49 Update the ibm-slapdserverid entry on peer2/replica1 The reason we want to update the serverid on Peer2 and Replica1 is to make it easier for us to recognize the systems. For example: ibm-slapdserverid: PEER 2 serverid is much easier to recognize than say ibm-slapdserverid: 12d74a40-66ae-102b-964fafea13b025c4 49

50 What the update will look like in the ibmslapd.conf To implement the change in serverid after the instance has been started for the first time we simply edit: #vi /home/inst_name/idsslapd-inst_name/etc/ibmslapd.conf Changing the stanza: dn: cn=configuration cn: Configuration ibm-slapdserverid: 12d74a40-66ae-102b-964f-afea13b025b4 To ibm-slapdserverid: PEER 50 2 serverid

51 Stopping ibmslapd on Peer2/Replica1 We have a few alternatives for stopping the ibmslapd process on peer2/replica1: idsslapd I inst_name k ibmdirctl -D cn=root w ***** stop Why do we stop ibmslapd? Any time we make a change in schema or the config file we must restart ibmslapd before the change will take effect. 51

52 We are now ready to configure replication Lets take stock of where we are. At this time, all three servers should be started using the commands from slide 51 At this time we have 1 authoritative master running (peer1) and two clean and synchronized systems ready to become a peer (peer2) and a replica (replica1) The user data is loaded on peer1 and we have cleaned up any unneeded replication agreements 52

53 The best tool for replication as simple as ldapsearch When trying to determine your topology it is always a good idea to start by running the following search: #ldapsearch h hostname D cn=root w ***** -s sub objectclass=ibm-repl* This will show us any configured replication agreement currently on the system. At this time this entry should return blank from: peer1, peer2 and replica1 53

54 Configuring replication from the web admin tool Login to the webadmin as cn=root on Peer1 (Authoritative Master) Replication Management Manage Topology Add Subtree 54

55 Next, we have to select our tree There are two options in this case: 1. We can manually type in our subtree 2. We can browse and select our subtree In this example we are going to click on Browse 55

56 Select the subtree you want to replicate You will notice the selection screen is very similar to the Directory Management section of webadmin Select the subtree you wish to replicate, and click the Select button 56

57 Checking our subtree We have selected our subtree, and the last thing we need to check is to make sure we are not using another system for our Master Server Referral LDAP URL!!! Click OK 57

58 So, what is actually taking place under the covers? #ldapsearch -D cn=root -w secret -s sub -b " objectclass=ibm-repl* Everything that we just did via the webadmin adds specific entries to the directory database itself. peer1_cn=ibmpol_repl_topology.ldif: CN=IBMPOLICIES cn=ibmpolicies objectclass=container objectclass=top objectclass=ibm-replicationcontext Lets start with the top level object cn=ibmpolicies ibm-replicareferralurl=ldap://peer1:389 ibm-replicagroup=default,cn=ibmpolicies ibm-replicagroup=default objectclass=ibm-replicagroup objectclass=top cn=peer1:389,ibm-replicagroup=default,cn=ibmpolicies Note: the ibm-replicaserverid is the value set in slide 22 objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=peer 1 serverid ibm-replicationserverismaster=true cn=peer1:389 58

59 Defining the replica group Once the top level entry is set, then we must create an object where all replication related data will be stored. This is called the replicagroup and looks like: ibm-replicagroup=default,cn=ibmpolicies ibm-replicagroup=default objectclass=ibm-replicagroup objectclass=top 59

60 Next we have the definition of the master This is a very important entry as it tells Peer1 that he is actually a master for this section of the tree and looks like: cn=peer1:389,ibm-replicagroup=default,cn=ibmpolicies objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=peer 1 serverid ibm-replicationserverismaster=true cn=peer1:389 60

61 This is what the basic topology will look like And we begin to add our additional Masters/Replicas 61

62 Add a Master screens The final step is to click on the Select option under Credential object so we can select our outbound replication credential. 62

63 Adding the credential object OUTBOUND! Ok, so in this panel what we are doing is setting our OUTBOUND credential. This will be the bind dn that is used when PEER1 tries to replicate to any other system. Select the Add credentials button 63

64 Adding the credential Naming your credential object In this first screen you can name the credential object anything you want (in this case cn=replicationcreds): Click Next 64

65 Setting your bind credential. The key to this dn is that it MUST NOT BE the AdminDN (for example cn=root). In fact, this dn should not match any real user on your system. In my case I will use cn=replbind Click Finish 65

66 Command Line taking stock of what was actually added to the DB: We can see by running our replication ldapsearch (slide - 58) #ldapsearch -D cn=root -w secret -s sub -b " objectclass=ibm-repl* The new added entry is: cn=replicationcreds,cn=replication,cn=ibmpolicies replicacredentials=replbind (note: this is the repl pwd) description=pwd is "replbind" objectclass=ibm-replicationcredentials objectclass=ibm-replicationcredentialssimple objectclass=top replicabinddn=cn=replbind cn=replicationcreds 66

67 The view of our newly created credential We can now see the credential object we created stored under the cn=ibmpolicies tree: Click OK 67

68 We are back in the Add Master screen We can now see the credential object we created as part of this agreement. Next we must click on the Additional tab: 68

69 The Additional Tab: Select the Add credential.. and provide the Admin DN and PW for the target consumer (peer2) and click OK 69

70 Lets look at what is actually added to the consumer: When we add the credential to the consumer for INBOUND replication the update is not made to the database, but is actually stored within the ibmslapd.conf file: dn: cn=supplier , cn=configuration cn: Supplier ibm-slapdmasterdn: cn=replbind ibm-slapdmasterpw: {AES256}cBGYg4eKxmysw9EQpIlhCQ== ibm-slapdreplicasubtree: CN=IBMPOLICIES objectclass: ibm-slapdconfigentry objectclass: ibm-slapdsupplier objectclass: top This method can become complicated quickly if we are replicating multiple subtrees. Note: In addition to adding the inbound credential object to PEER2 s ibmslapd.conf, the previous step also adds the outbound credential (cn=replicationcreds,cn=replication,cn=ibmpolicies) as well as the replication agreements cn=peer1:389,ibm-replicagroup=default,cn=ibmpolicies cn=peer2 :12389,ibm-replicaGroup=default,CN=IBMPOLICIES cn=peer1:389,cn=peer2 :12389,ibm-replicaGroup=default,CN=IBMPOLICIES cn=peer2 :12389,cn=peer1:389,ibm-replicaGroup=default,CN=IBMPOLICIES 70 We will simplify the inbound credential in slide 75

71 Building the replication agreements back from Peer2 to Peer1 In this screen we will first be told that in order for us to continue the replication configuration, additional steps will need to be taken Click OK 71

72 The Select credential screen This screen is where we define the credential object that the supplier (PEER2 in this case) will use when it connects back to PEER1 Because we added the credential object in the previous screen, the cred will show up when we click Show credentials button Click OK 72 Note: in this case we do not need to add the credential as the consumer in this case already had the credential created in slide 62-65

73 One more informational message Finally the replication management system will let us know our topology is complete, but reminds us that we still need to synchronize the Peer2 with Peer1 s data: Click OK 73

74 What a peer to peer topology looks like At this point we only have a peer to peer topology defined between Peer1 and Peer2.This is what that topology looks like: 74

75 Simplified Inbound Replication Credentials, and disabling Conflict Resolution (CR) To simplify the Inbound credentials, we set the default credential on each system in the topology and disable CR: Webadmin -> Repl Mgt -> Manage replication properties Select Default credentials and referral and click Edit 75

76 Simplified Inbound Credentials - Default credentials and referral screen Disable CR by changing the drop down to False Specify the credential object that the server (peer1 in this example) will allow to bind for replication only. cn=replbind replbind Click OK 76

77 Notes about the Default credentials and referral screen Viewing the actual change made to the instances ibmslapd.conf Inbound Credential : dn: cn=master Server, cn=configuration cn: Master Server ibm-slapdmasterdn: cn=replbind ibm-slapdmasterpw: {AES256}LXkc0fthIwexuXjNVqRxyQ== ibm-slapdnoreplconflictresolution: true objectclass: ibm-slapdconfigentry objectclass: ibm-slapdreplication objectclass: top 77

78 So lets take another dive under the covers Using the ldapsearch we spoke of in slide 53 we can review how each of these tasks we took show up in the directory. #ldapsearch h hostname D cn=root w ***** -s sub objectclass=ibm-repl* First lets look at the credential object Outbound Credential. cn=replicationcreds,cn=replication,cn=ibmpolicies cn=replicationcreds description=pwd is "replbind" objectclass=ibm-replicationcredentials objectclass=ibm-replicationcredentialssimple objectclass=top replicabinddn=cn=replbind replicacredentials=replbind 78

79 Stanza which defines a system s role in the topology dn: Name, Subtree cn=peer1:389,ibm-replicagroup=default,cn=ibmpolicies objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=peer 1 serverid ibm-replicationserverismaster=true cn=peer1:389 cn=peer2 :12389,ibm-replicaGroup=default,CN=IBMPOLICIES objectclass=ibm-replicasubentry objectclass=top ibm-replicaserverid=peer 2 serverid ibm-replicationserverismaster=true cn=peer2 :

80 Understanding the actual replication agreements. There are two agreements that make up the replication between Peer1 and Peer2. It is important to understand how these agreements look, and the function of each The key: Write it down. Consumer(port), Supplier(port), Subtree For example: cn=peer1:389,cn=peer2 :12389,ibmreplicaGroup=default,CN=IBMPOLICIES Is actually, peer2 on port supplying peer1 using port 389 for the cn=ibmpolicies subtree 80

81 Peer1 to Peer2 agreement This is how the agreement will appear where Peer2 is acting as a consumer (replica) while Peer1 is acting as a suppler (master): cn=peer2 :12389,cn=peer1:389,ibm-replicaGroup=default,CN=IBMPOLICIES ibm-replicamethod=1 ibm-replicaconsumerid=peer 2 serverid ibm-replicationonhold=true ibm-replicacredentialsdn=cn=replicationcreds,cn=replication,cn=ibmpolicies ibm-replicaurl=ldap://peer2:12389 objectclass=ibm-replicationagreement objectclass=top cn=peer2 :12389 A way or understanding this may be to say: When a change is made to cn=ibmpolicies, this agreement will be read on Peer1 and ITDS will examine this stanza and use the settings defined to try and connect to the replica (via the ibm-replicaurl) using the credentials defined in cn=replicationcreds,cn=replication,cn=ibmpolicies (see slide 77) 81

82 And the Peer2 to Peer1 agreement Basically the same as the previous slide but in the reverse direction: cn=peer1:389,cn=peer2 :12389,ibm-replicaGroup=default,CN=IBMPOLICIES ibm-replicamethod=1 ibm-replicaconsumerid=peer 1 serverid ibm-replicationonhold=true ibm-replicacredentialsdn=cn=replicationcreds,cn=replication,cn=ibmpolicies ibm-replicaurl=ldap://peer1:389 objectclass=ibm-replicationagreement objectclass=top cn=peer1:389 82

83 Lather rinse repeat We use the same steps to add our other subtrees. In this example we add the O=SAMPLE subtree to replication using the exact same steps used for CN=IBMPOLICIES 83

84 Configuring a replica The steps are basically the same, only we start by clicking on Add Replica 84

85 Adding a replica Input the same information as we did in the case of a peer configuration, but no need to fill any information in the Additional Tab Click OK Note: It is important to understand the default inbound credential and disabling CR preparation tasks were already completed on Replica1 using the method in slides 75 and 76 85

86 We would need to repeat this process for each subtree Add replica1 to peer2 Notes: keep in mind that only PEER 1 is currently in an authoritative mode. That is to say only PEER1 knows not only the entire replication topology, but also has all user related data. 86

87 What our final topology will look like on Peer1 87

88 Now what? We have our credentials, our topology and all customer data on our authoritative master. What s next? We need to take a full backup from Peer1 and import this data on Peer2 and Replica1. Commands to accomplish: idsdb2ldif/idsldif2db idsdbback/idsdbrestore We will use idsldif2db for this presentation 88

89 Placing the Authoritative Master in read-only The key, especially when you have multiple applications hitting a master, is to place the server into a read-only mode before taking the ldif. Why? We need to make sure that all servers (in this case peer2 and replica1) will have the EXACT same data set. The only way to guarantee this is to place the master in Read-Only Mode Note: This means that authentications will still work, only tasks such as changing a user password will fail 89

90 Before exporting data, always confirm there are no pending changes: In this example, due to how we configured, there are pending changes For example, peer1 wants to tell peer2 to add replica1 as a consumer. Because we are taking a full backup, these changes will not be relevant as we will bring them over with the full data. 90

91 Skip all blocking entries: In the queue management screen select a subtree and click on Queue Management Select the Pending Changes tab and select the Skip All button Click OK then OK on the number of skipped entries, and finally OK on the pending changes screen to return to the replication queues screen. 91

92 What your queues should look like prior to backup We should have no pending changes for any consumer: 92

93 Placing subtrees in read-only mode Quiesce/unquiesce 93

94 Backing up the ITDS data to ldif on Peer1 We are going to take all the entries that are stored in the DB2 database and store them in a flat text file in ldif format. You need to pass the instance name in the command #idsdb2ldif I peer1 o /tmp/full_peer1_backup.ldif Note: this command can be run while peer1 ibmslapd instance is running in read-only (or write) mode. 94

95 This is what is actually seen on Peer1 when the ldif is exported #idsdb2ldif -I peer1 o full_peer1_backup.ldif GLPCTL113I Largest core file size creation limit for the process (in bytes): ' '(Soft limit) and '1'(Hard limit). GLPCTL122I Maximum Data Segment(Kbytes) soft ulimit for the process is and the prescribed minimum is GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is GLPCTL122I Maximum Open Files soft ulimit for the process is 7000 and the prescribed minimum is 500. GLPCTL119I Maximum Physical Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is GLPCTL121I Maximum Stack Size(Kbytes) soft ulimit for the process was and it is modified to the prescribed minimum GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is GLPSRV200I Initializing primary database and its connections. GLPD2L011I 71 entries have been successfully exported from the directory. Note: the 21 extra entries from the original 50 we imported, are those which define the replication topology 95

96 Remember to make your Master writable when the ldif completes!!! 96

97 I now need to transfer this data to Peer1/Replica1 I can use ftp/scp or whatever utility I am most comfortable with to transfer the ldif file from Peer1 to Peer2 or Replica1 It is important to note this is an ascii file and to avoid problems you should transfer in ascii mode (avoid the ^M issue) Because Peer2 and Replica1 are already cryptographically synced we can begin the data load 97

98 Options for loading data We have two options for loading the data: 1. The idsldif2db/ldif2db utility 2. The idsbulkload/bulkload utility 98 The idsbulkload utility is optimized for large data loads, where idsldif2db is more useful for smaller loads. Note: the db2 transaction log may need to be increased for larger data loads.

99 FULL Resync It is best practice to make certain when adding data for replication that a drop db, reconfigure and resync #ibmslapd -I peer2 -k #idsucfgdb -I peer2 -r -n # idscfgdb -I peer2 -a peer2 -w peer2 -t peer2 -l /home/peer2 n Notes: There is no need to crypto sync or import the schema as the unconfigure of the database does not delete those files (V3.modifiedshema, ibmslapddir.ksf, etc) In addition, any changes that were in the ibmslapd.conf are retained such as the serverid, the default inbound credential and the admindn. The following article is helpful on: Syncing data between two servers 99

100 Loading Peer2 with idsldif2db For this example I am going to load my data on Peer2 with the idsldif2db utility: Stop ibmslapd # idsslapd -I peer2 k GLPSRV121I Stopped directory server instance: 'peer2'. # idsldif2db -I peer2 -i full_peer1_backup.ldif GLPCTL113I Largest core file size creation limit for the process (in bytes): '-1 '(Soft limit) and '-1'(Hard limit). GLPCTL119I Maximum Data Segment(Kbytes) soft ulimit for the process is -1 and th e prescribed minimum is GLPCTL119I Maximum File Size(512 bytes block) soft ulimit for the process is -1 and the prescribed minimum is GLPCTL122I Maximum Open Files soft ulimit for the process is 8000 and the prescr ibed minimum is 500. GLPCTL119I Maximum Physical Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is GLPCTL122I Maximum Stack Size(Kbytes) soft ulimit for the process is and the prescribed minimum is GLPCTL119I Maximum Virtual Memory(Kbytes) soft ulimit for the process is -1 and the prescribed minimum is GLPCOM022I The database plugin is successfully loaded from libback-config.a. GLPSRV200I Initializing primary database and its connections. GLPRPL137I Restricted Access to the replication topology is set to false.. GLPRDB002W ldif2db: 71 entries have been successfully added out of 71 attempted. Notice that the number of entries exactly match that exported from Peer1 in slide

101 Loading Replica1 with the bulkload utility: This example will load the replica (Replica1) using the bulkload utility. #bulkload -I replica -i full_peer1_backup.ldif Number of rows read =1 Number of rows skipped =0 Number of rows loaded =1 Number of rows rejected =0 Number of rows deleted =0 Number of rows committed =1 + [[ 0 -ne 0 ]] + db2 terminate DB20000I The TERMINATE command completed successfully. + RC=0 + [[ 0 -ne 0 ]] + echo 0 + 1> db2load.rc + exit 0 GLPBLK073I Bulkload completed. 101

102 Start of each consumer server. We can start ibmslapd on PEER2 and Replica1 at this point. Make sure the instances start in normal mode. Validate the replication credentials can bind from PEER1 to PEER2/Replica1 102

103 Let me give you an example The best method for testing if the credential object you used is going to work is a simple ldapsearch from PEER1: # idsldapsearch -h peer2 -p D cn=replbind -w replbind -s base objectclass=* ibmslapdisconfigurationmode Will return: ibm-slapdisconfigurationmode=false The return of the value indicates the bind credentials were valid. We can run the same test to replica1. 103

104 Almost done last step is to resume replication By default (and as we could see in slide 92) the replication agreements are suspended, and we must go to each peer and resume the queues. For example: ibm-replicationonhold=true We resume the replication on Peer1/Peer2 by using the webadmin or an ldap extended operation. 104

105 Resuming replication using the webadmin Replication Management Manage queues select subtree click on suspend/resume button To: All Replication Queues should show a Last Result of OK and a State of Ready 105

106 Messages in ibmslapd.log when replication resumes From PEER1 08/30/11 07:24:40 GLPRPL108I Replication for DN 'CN=PEER2 : 12389,CN=PEER1:389,IBM-REPLICAGROUP=DEFAULT,CN=IBMPOLICIES' will use the single threaded, synchronous method. 08/30/11 07:24:40 GLPRPL108I Replication for DN 'CN=REPLICA1:17389,CN=PEER1:389,IBMREPLICAGROUP=DEFAULT,CN=IBMPOLICIES' will use the single threaded, synchronous method. 08/30/11 07:24:41 GLPRPL108I Replication for DN 'CN=PEER2 : 12389,CN=PEER1:389,IBM-REPLICAGROUP=DEFAULT,O=SAMPLE' will use the single threaded, synchronous method. 08/30/11 07:24:41 GLPRPL108I Replication for DN 'CN=REPLICA1:17389,CN=PEER1:389,IBMREPLICAGROUP=DEFAULT,O=SAMPLE' will use the single threaded, synchronous method. 106

107 Explanation on synchronous method of replication With the reliance on ITDS replication to keep our systems in sync, the changes made to a writable master are sent sequentially to the other consumers in the topology If any change fails, all pending changes between that supplier and consumer will block. 107

108 Basic Debugging - I cannot stress firmly enough the first rule of replication! As clearly documented in the Hitchhikers Guide to the Galaxy 108

109 What the Other Admin did in this case on Replica1 Sometimes App developers want to make modifications to a read only system and test some application, in this case the developer made the following change: 1. The customized user was removed on the replica idsldapdelete -p D cn=root -w secret -k cn=customeruser,o=sample 109 However, generally the root cause of the replication failure is unknown

110 Using the Webadmin to resolve replication errors In this case, what we see is not actually an error. Let s look again at our queue: In the Manage queues panel we can see that the last result is "Error -32", however the root cause cannot be deduced. In order to learn more about the failure we need to click on Queue Details 110

111 Queue Details 111

112 A little log analysis shows the root cause. The FIRST place to start on any replication problem is the ibmslapd.log Default location: /home/inst_name/idsslapd-instname/logs in this case: /home/peer1/idsslapd-peer1/logs File Name: ibmslapd.log A little research shows the change we have stored in queue: 08/30/11 08:04:07 GLPRPL033E Error No such object occurred for replica 'CN=REPLICA1:17389,CN=PEER1:389,IBMREPLICAGROUP=DEFAULT,O=SAMPLE': modify failed for entry 'CN=CUSTOMERUSER,O=SAMPLE' change ID 5. 08/30/11 08:04:07 GLPRPL118E Replication for replica 'CN=REPLICA1:17389,CN=PEER1:389,IBMREPLICAGROUP=DEFAULT,O=SAMPLE' will continue to retry the same failed update with change ID 5 until it is successful. 112

113 So what does Error-32 mean? The key to all return codes is the error is documented in the appendix A of the admin guide: om.ibm.ibmds.doc/admin_gd507.htm?path=8_4_7_0#error1 113

114 Our options to resolve? We can deal with this one of two ways: 1. Do Nothing! There will be no more changes that will flow to replica1 Danger Replication Queue build up 2. Use the Skip Blocking Entry button in the Queue Details section of the WebAdmin or Use an extended op to skip the blocking entry Danger of corrupting data integrity between supplier and consumer 3. FIX the failure! 4. Resynchronize the failing node (Replica1 in this example) 114

115 Viewing pending changes via the WebAdmin As we can see in this case, a skip of the blocking entry really would not help, as the next pending changes in replication are all on the problem dn (cn=customeruser,o=sample in this example) 115

116 Viewing Queue Details via the command line. The key techdoc: rs=767&uid=swg In this case we are interested in the agreement between Peer1 and Replica1 for the o=sample subtree. Using what we learned: 1. Slide 58 on the ldapsearch to determine replication agreements 2. That the agreement between Peer1 and Replica1 for o=sample is (Slide 80-81) : cn=replica1:17389,cn=peer1:389,ibmreplicagroup=default,o=sample 116

117 Using ldapsearch to determine Queue details Knowing the replication agreement (RA) we can gain the same information seen in the webadmin via: idsldapsearch -D cn=root -w secret -b cn=replica1:17389,cn=peer1:389,ibmreplicagroup=default,o=sample objectclass=* ++ibmrepl cn=replica1:17389,cn=peer1:389,ibm-replicagroup=default,o=sample ibm-replicationchangeldif= dn: CN=CUSTOMERUSER,O=SAMPLE changetype: modify replace:roys-custom-attr roys-custom-attr: change 1 control: false control: false:: MIQAAACyMIQAAAAnCgECMIQAAAAeBA1tb2RpZmllcnNOYW1lMYQAAAAJBAdDTj1ST09UMIQAAA A4CgECMIQAAAAvBA9tb2RpZnlUaW1lc3RhbXAxhAAAABgEFjIwMTEwODMwMTUwNDA3LjA2MjY2OF owhaaaaeekaqiwhaaaadgegfjlcgxpy2f0aw9uqmfzzvrpbwvzdgftcdgeaaaagaqwmjaxmta4mj kxnte3ntauoda2mju1wg== ibm-replicationlastactivationtime= z ibm-replicationlastchangeid=4 ibm-replicationlastfinishtime=n/a ibm-replicationlastresult= z 0 81 connect NULL ibm-replicationlastresultadditional=simple bind: Can't contact LDAP server ibm-replicationnexttime=n/a ibm-replicationpendingchangecount=6 ibm-replicationstate=connecting ibm-replicationfailedchangecount=0 ibm-replicationperformance=n/a ibm-replicationpendingchanges=5 modify CN=CUSTOMERUSER,O=SAMPLE ibm-replicationpendingchanges=6 modify CN=CUSTOMERUSER,O=SAMPLE ibm-replicationpendingchanges=7 modify CN=CUSTOMERUSER,O=SAMPLE ibm-replicationpendingchanges=8 modify CN=CUSTOMERUSER,O=SAMPLE ibm-replicationpendingchanges=9 modify CN=CUSTOMERUSER,O=SAMPLE ibm-replicationpendingchanges=10 modify CN=CUSTOMERUSER,O=SAMPLE 117

118 Fixing the failure on Replica1 In this case the error condition is easy to resolve by re-adding the failing entry to replica1 idsldapadd -p D cn=root -w secret -k -f user.ldif Where user.ldif reads: dn: cn=customeruser,o=sample control: false:: MIQAAADVMIQAAAAmCgEAMIQAAAAdBAxjcmVhdG9yc05hbWUxhAAAAAkEB0NOPVJPT1QwhAAAAD gkaqawhaaaac8ed2nyzwf0zvrpbwvzdgftcdgeaaaagaqwmjaxmta4mjkxnte3ntauoda2mju1wj CEAAAAJwoBADCEAAAAHgQNbW9kaWZpZXJzTmFtZTGEAAAACQQHQ049Uk9PVDCEAAAAOAoBADCEAA AALwQPbW9kaWZ5VGltZXN0YW1wMYQAAAAYBBYyMDExMDgyOTE1MTc1MC44MDYyNTVa roys-custom-attr: My Custom Attribute Value objectclass: inetorgperson objectclass: organizationalperson objectclass: person objectclass: top objectclass: roys-objectclass sn: customeruser cn: customer cn: customeruser ibm-entryuuid: cd74e d b ef23e6 Will return: Operation 0 adding new entry cn=customeruser,o=sample 118

119 What your queues will look like when everything is perfect All subtrees should show: - Last result: OK - State: Ready - Queue size: 0 119

120 A quick test adding a new user to Peer1 I want to test my replication to make sure its working as I expect. 1. Add and entry to Peer1: ldapadd h Peer1 D cn=root w secret f /tmp/add.ldif Where /tmp/add.ldif reads: dn: CN=TESTUSER,o=sample objectclass: inetorgperson objectclass: organizationalperson objectclass: person objectclass: top sn: TEST cn: TEST cn: TESTUSER description: user added to Peer1 Will return: Operation 0 adding new entry CN=TESTUSER,o=sample 120

121 A quick test Checking for that user on Peer2/Replica1 I would run a simple search to each of the hosts to see if that user exsists: ldapsearch -h peer2 -D cn=root -w secret -b CN=TESTUSER,o=ibm,c=us objectclass=* And: ldapsearch -h replica1 -D cn=root -w secret -b CN=TESTUSER,o=ibm,c=us objectclass=* Both should return: CN=TESTUSER,o=ibm,c=us objectclass=inetorgperson objectclass=organizationalperson objectclass=person objectclass=top sn=test cn=test cn=testuser description=user added to Peer1 I could further test this by adding another user to Peer2 and checking for the user on Peer1 and so on. 121

122 Leveraging logs from both sides of the topology for troubleshooting. Any time I hit an error the first place I check is the ibmslapd.log on BOTH servers involved. There may not be useful information as to why the error is occurring on the supplier, but the consumer may clearly point out root cause. It is a good idea to copy off the ibmslapd.log so you are only dealing with the errors effecting us at this time 122

123 Other Resources In addition to these slides, I have attached to this presentation: ITDS_6.3_replication_attachements.zip Contains examples used in this presentation: add.ldif final_full_topology.ldif final_ibmpo_topology.ldif final_sample_topology.ldif full_peer1_backup.ldif ITDS6.3.Replication.CheatSheet.txt peer1_cfg_commands.txt peer1_ibmpol_repl_topology.ldif peer1_ibmslapd.conf peer1_starting_repl_topology.ldif user.ldif V3.modifiedschema 123

124 LDAP Extended Opperations - idsldapexop Anything that can be done from the WebAdmin can be done in some way via command line. One of the utilities we use for command line replication is: idsldapexop Documentation on the command: m.ibmds.doc/commandref59.htm 124

125 Replication Agreement extended details via ++ibmrepl A very useful ldapsearch command with available replication related operational attributes: This gives the current status and info about the replication agreement. idsldapsearch -p 389 -D cn=root -w secret -b "cn=replica1:17389,cn=peer1:389,ibmreplicagroup=default,o=sample" objectclass=* + +ibmrepl See slide 117 for and example of use 125

126 Using ldapexop - an extended op to quiesce a subtree ldapexop - control replication extended operation: The ldapexop command can be used for a variety of controls such as placing a subtree in read-only mode. ldapexop -h hostname -D cn=root -w <password> -action {quiesce unquiesce replnow wait} -rc <ReplicationContext> Where <action> can be one of the four values: quiesce - No further updates are allowed, except by replication. unquiesce - Resume normal operation, client updates are accepted. replnow - Replicate all queued changes to all replica servers as soon as possible, regardless of schedule. wait - Wait for all updates to be replicated to all replicas. e.g.: ldapexop -D cn=root -w secret -op quiesce -rc o=sample 126

127 Skipping blocking entries with an extended op ldapexop - control queue extended operation: ldapexop -h hostname -D cn=root -w <password> -op controlqueue -skip {all change-id} -ra <ReplicationAgreement> Where "all" indicates to skip all pending changes for this agreement and "change-id" identifies the single change to be skipped. If the server is not currently replicating this change, the request fails. e.g.: ldapexop -D cn=root -w secret -op controlqueue -skip all -ra "cn=replica1:17389,cn=peer1:389,ibm-replicagroup=default,o=sample 2 changes skipped. 127

128 Controling the queue - Unsuspend ldapexop - control replication extended operation: ldapexop -h hostname -D cn=root -w <password> -op controlrepl -action {suspend resume replnow} {-rc <ReplicationContext> -ra <ReplicationAgreementD>} If -rc <ReplicationContext> is provided then the action is performed for all agreements for this context. e.g.: ldapexop -D cn=root -w secret -op controlrepl -action resume -ra "cn=replica1:17389,cn=peer1:389,ibm-replicagroup=default,o=sample" Operation completed successfully. 128