Gabriella Davis Paul Mooney

Transcription

1 How to build a better cluster - from a standing start Gabriella Davis Paul Mooney 1

2 About us Gabriella Davis The Turtle Partnership Technical Director Security, System Design, Deployment of Lotus software since 1989 Yes, I m much older than Paul but feel free to be surprised by that Paul Mooney Bluewave Technology Senior Architect 14 years experience with Domino Geek / Biker / Jogger 2

3 Let s get legal! This slide presentation may contain the following copyrighted, trademarked, and/or restricted terms: IBM Lotus Domino, IBM Lotus Notes, IBM Lotus Symphony, LotusScript Microsoft Windows, Microsoft Excel, Microsoft Office Linux, Java, Adobe Acrobat, Adobe Flash 3

4 Simple Math... We have 250 slides We have 105 minutes 250 /105 = 2.3 SPM (slides per minute) Oh. S*&t. 4

5 What are you going to learn today What is a cluster and why you want it How does it work in the Domino world Other types of clustering that aren t as good How to set up Domino clustering correctly How to monitor and tune your new cluster environment How clustering behaves for clients Some examples of cluster topologies Things to watch out for If we have time How to juggle How to attend a session at lotusphere that does not include the word strategic How the cloud is evil How your mother never loved you How to win the lottery 5

6 How are we going to teach you? A Bit Of Tell before the Show Then the Show Starting with a single domino server (one we built earlier) Register an additional server Install the additional server Setup clustering Test clustering from a client, let s take a server down live! How to tune your cluster to do what you want not what it wants Terrible Terrible things that can happen if you don t heed our advice 6

7 Our Dilemma We want to step you through all our tricks for setting up a cluster LIVE We want to make sure you have all the information you need to complete the work yourself when you get back to the office To get all this content over we re going to be covering a lot of ground very quickly That s exciting, isn t it! 7

8 What Is A Cluster And Why Do You Want It? 8

9 So What s This All About? Wouldn t it be great when a server gets very busy or goes offline if your users don t notice If mail doesn t backup queuing for delivery on an offline mail server If out of office still works If mail still routes if everything doesn t get r-e-a-l s-l-o-w If your Blackberry or Traveler server automatically knows to look elsewhere Altogether now... Yes it would! Follow us into this magical world called clustering 9

10 What Is Clustering? Grouping together of resources so they appear as one resource instead of many to the user Creating a single virtual server experience In Domino Terms Separate Domino servers that operate individually Each server is cluster aware i.e. it knows it belongs to a cluster and what the other servers are Each cluster aware server shares its data in real time with the rest of the cluster The configuration of each server is separate and independent (everything from hardware and OS to Domino Tasks) 10

11 What Other Types Of Clustering Are There? The sharing of disk and storage Single point of failure being the disk RAID isn t clustering and neither are SANs despite what hardware companies will tell you MS Clustering Active/Active Active/Passive Can t do load balancing only failover Must be identical OS, software and versions All or nothing everything must be clustered Bit-level clustering e.g. Neverfail Active/Passive Data transferred beneath OS level to passive box 11

12 Why Use Domino Clustering vs OS Level More flexible Cluster different hardware, Win32, Linux, AIX together Partially cluster selected data Locate cluster servers anywhere with sufficient bandwidth connecting them More stable Load balancing means that users failover within the cluster before a server is pushed to its limits If an entire server fails there is an entire duplicate of data and services waiting 12

13 How Does It Work In The Domino World? 13

14 Clustering In Domino Just Works Yes it does, it s smart, it learns and dynamically adjusts itself to the current load It s incredibly useful and self maintaining Temptation is to not work with it until you need it but that would be a mistake One of the single best features of Domino Now available for Express Customers too (you lucky, lucky people) Often implemented poorly and ignored or not implemented at all The beauty of Domino clustering lies in being able to selectively choose which databases you want to cluster across which servers, you don t have to create mirror copies of each server 14

15 What is clustering? Replication ish It is NOT replication and doesn t use the same processes or threads It is event driven and in memory not scheduled It s designed to be very fast so it doesn t waste time and just sends everything Single, one-way push of data writes from one server to another Since reading a document can be a write So unread marks can cluster correctly Occurs almost immediately Depending on server load but should be less than 10 secs Data entered in server A goes to Server B Update is sent by Server A s cluster replicator Data entered in Server B goes to Server A Update is sent by Server B s cluster replicator Been around since R4x days and constantly enhanced and improved 15

16 What is clustering? Version independent Clustering an R6.x server with R8.x server works Platform independent You can cluster from and between Win2k, Win2k3, Win2k8, Linux, AIX, OS400, Solaris, etc etc Stunning feature of domino No other product Gab and I have ever come across does clustering so well 16

17 Components of clustering Cluster Manager Cluster Replicator (clrepl) Cluster Database Directory (cldbdir) Cluster Database Directory Manager Cluster Administrator 17

18 Cluster Manager Not a visible task What servers are in the cluster How busy are those servers Activating workload balancing Managing and logging failover Directing client connections 18

19 Cluster Replicator Cluster Replicator task CLREPL It is not a listed task in SERVERTASKS line by default Since R7 days It can be there though (especially if it is older or customised) Cluster Manager will load it automatically if the server is a cluster You can have multiple cluster replicator threads Typically equal to total members in a cluster less one Monitor cluster statistics to see if you need to do this (more later) Lotus supports up to 6 servers in a cluster You can have more Do you really need more? Let us answer this for you. No. Changes to be sent to cluster mates are held in memory by the CLRepl Task Tell CLRepl Quit loses in-memory changes Tell CLRepl Pause / Resume retains and accumulates in memory changes 19

20 More About Cluster Replication and onwards Streaming cluster replication Data transfers from buffer of domino server to cluster mates Doesn't wait to write to disk, then write to other servers Faster Broken (supposedly fixed in FP1 and disabled in 8.0.2) DAOS and clustering One or both servers can be using DAOS If both are DAOSified Both have their own.nlo store If you have DAOSified cluster servers over Cluster replication is DAOS aware, it won t resend a file if it already exists on the destination server CLREPL will check to see if the attachment already exists before sending MASSIVE performance gain in clustering and reduction in network traffic 20

21 Cluster Database Directory Cluster directory task cldbdir Creates and maintains the cluster database directory cldbdir.nsf One document per database per server Each document contains the replica id of the database, the server where it is located and its cluster state CLDBDir.Nsf is replicated throughout the cluster Maintains a list of all replicas on each cluster server If a server replica is not listed there... it WILL NOT cluster as the Cluster Manager won t know to failover a user requesting that database to that server The ClRepl tasks uses the Cldbdir.nsf db to verify which databases are to be clustered and where to failover users requesting that database Can be used to mark a database as out of service Force a failover 21

22 Cluster Administrator Not a visible task Adds CLRepl, CLDBDir to the servertasks when a server is added to a cluster Sets up calendaring for the cluster Starts the clustering tasks on the server Starts Adminp if it isn t already Reverses all those changes when a server is removed from a cluster 22

23 Components Work Together Cluster Database Directory keeps a record of all databases and their state This verifies if a db is in place on a server and if it s in an available state for user access The Cluster Database Directory Manager keeps the ClDDDir.nsf up to date The Cluster Replicator ensures all data changes on any server are pushed to the other clustermates immediately 23

24 Calendaring In A Cluster Usually Domino maintains freetime information for users on their home server in the file busytime.nsf If you want to check someone s schedule the server uses busytime.nsf to verify availability Since users can be on varying servers within a cluster and don t have a single home the busytime.nsf can t be used Busytime.nsf on individual servers becomes Clubusy.nsf replicated across all servers in a cluster Busytime.nsf is deleted from the server when it is added to a cluster by the Cluster Administrator Clubusy replicates using the cluster replicator tasks but should also be included in standard replication schedule as backup 24

25 What isn t clustering It isn t replication Replication formula is ignored by clustering Acting as designed Premise is that database cluster replicas should ALWAYS be identical Replication formulae could prevent that Scheduled replication will honour replication formulae It isn t good at managing deletion stubs It s in memory changes only If you quit the clrepl task then those in memory changes are lost Always have a backup replication document between server mates in a cluster but no more often than every 60 mins It works for NRPC calls only Notes clients Domino servers It is NOT web clustering We get to that later 25

26 How To Set Up Domino Clustering Correctly The Show Bit 26

27 In the beginning Single Domino server - atlantic/svr/demo DNS name - atlantic.snt-track.com Version of Domino FP1 OS - Windows 2003 Five mail files registered No implementation of anything - DAOS etc Default configuration document created Additional machine that will be the cluster - pacific/svr/demo pacific.snt-track.com Windows

28 Registering a new server in the domain Launch the administration client Go to the Configuration tab Click on Registration Select Server 28

29 Registering a new server in the domain Choose the atlantic as the registration server Select Server Navigate to the atlantic/svr/demo server Do NOT use local Select the Certifier id file Click Certifier id Navigate to the servers OU id file Click on OK Click on OK Enter the password 29

30 Registering a new server in the domain The register server dialog box appears Registration server Should still be atlantic Certifier Should still be /svr/demo Click on Continue 30

31 Registering a new server in the domain The Register new server dialog box opens Enter Pacific as the server name Enter title if desired Domino domain = Demo Click on the icon To select administrators Choose your administrator group Select OK 31

32 Registering a new server in the domain IN PRODUCTION Enter a password for the server id file Server id files SHOULD have passwords Keep password blank Storage of your id file Uncheck In Domino Directory Check in file Select the desktop Select the icon Click on REGISTER ALL Status bar updates 32

33 Registering a new server in the domain The server should now be visible in the domain 33

34 Let s Start By Installing Our 2nd Server We re going to install Domino bit in this instance 34

35 Run the installer file If you are running Windows 2008, run the installer as administrator 35

36 Security Warning The installer is an executable which will trigger an OS alert 36

37 Extract install files to a directory If you select default you will end up with 550+MB of install files in your server temp directory - you may want to clean those up! Paul s tip - I typically have a software folder created on every server that contains all software installed on that server (including rollback software) Yes it uses up space, but it gives Paul a warm fuzzy feeling 37

38 The Installer Will Now Run This should take only a few minutes (<10) If it takes longer, unless your hardware is very old, there may be something blocking the install such as evil virus protection. If you are upgrading and the install fails, existing files may be still in use Do a clean reboot and try again 38

39 The Installer GUI There is also a console installer for Windows On Linux there is also an optional GUI installer Click on NEXT 39

40 License Agreement Accept the license Click on NEXT 40

41 Installation Directory For Program Files When installing a 32bit application (as we ve chosen to do here) on a 64bit OS the default is \program files (x86) We don t want that - spaces, brackets, long directories. bad! Click on Browse 41

42 Better Installation Directory Change the install path to c:\ibm\lotus\domino Click on NEXT 42

43 Data Directory Domino assumes the data directory will be a subdirectory called data underneath the program directory You can accept that or choose another directory or even drive Click on NEXT 43

44 Server Type - Enterprise If you want to cluster your server then you must have an enterprise server license or CEO entitlement You can choose Customize and select specific Domino features to install, including clustering Click on NEXT 44

45 Confirmation Screen Read this carefully to confirm you have made the right selections Once installation is complete these settings are written to the registry Click on NEXT 45

46 Installer Running 46

47 Installer Complete! 47

48 Before Going Further - Let s Install A Fixpack It s tempting to jump right in, but a few minutes now to install the latest fixpack could save you hours later 48

49 Running the Fixpack Installer Again it will extract to a temporary directory Click on RUN 49

50 License Agreement Select ACCEPT Select NEXT 50

51 Confirming Install Directories If you have partitioned servers there will be multiple data directories to choose from You will have to run the updater for each data directory / partition you want to update Select the DATA directory and click on NEXT 51

52 First The Installer Verifies Source Version You can only use the incremental installer to update from a specific version If you aren t on you can t use the update installer Click on NEXT 52

53 Installer Complete Click on CLOSE 53

54 Starting Domino With Java Controller On Windows 2008 there is no visible Domino console Modify the properties of the desktop icon created by the installer Right click on the shortcut and select PROPERTIES 54

55 To Use Java Controller Select the SHORTCUT tab Add -jc to the program line eg c:\ibm \lotus\domino\nserver.exe -jc Select APPLY Select OK 55

56 Starting Domino The First Time We will need to configure it so we need on screen dialogs Right Click on the Shortcut and select RUN AS ADMINISTRATOR 56

57 It begins 57

58 First Setup Screen Click on NEXT 58

59 Choose Set Up An Additional Server You already have one server in your organisation It will expect you to already have a server ID It will expect to be able to connect to another server to get the configuration 59

60 Select A Server ID File Click on BROWSE 60

61 Select A Server ID File Copied onto the server from the workstation where we generated it Select the pacific server ID Click on SELECT 61

62 Confirmation Of Server Name The installer reads the ID file you gave it to find the server name Note it s not editable at this point as the ID file already exists Click on NEXT 62

63 What Services Do We Want The Server To Run LDAP will be selected by default, you can deselect it and should if you don t need it Don t choose options you don t need immediately you can always enable them later Click on NEXT 63

64 Configuring TCPIP These settings are written to the server document via the Adminp task There must be a fully qualified hostname and if there is an OS configured one, it will be selected by the installer Click on CUSTOMIZE 64

65 Modifying TCPIP Settings Both the hostname and fqhn must resolve on the server The server will use the hostname (or ipaddress if you enter one) to find itself Use a local hosts file to ensure it does resolve correctly Enter the correct hostname in the HOST NAME field Click on OK 65

66 Confirmation Of TCPIP Settings Click on NEXT 66

67 Connect To Another Server For Configuration This can be the server you intend to cluster with but doesn t have to be Usually we select the Admin server for the Domain to minimise the risk of replication conflicts in the server document Enter ATLANTIC/SVR/ DEMO in server name Enter ATLANTIC in optional network address Click on NEXT 67

68 Directory Information Choose the same configuration as on the server you want to cluster with 99% of the time, PRIMARY DOMINO DIRECTORY Click on NEXT 68

69 Security Setup Leave as is Anonymous accounted will be added with No Access to all databases/templates LocalDomainAdmins will be added with Manager access to all databases/ templates Select NEXT 69

70 Confirmation Screen - Verify Details Select SETUP 70

71 Files Installing The installer will fail if it can t find the Atlantic to find its configuration you already have files in place from a previous failed install 71

72 Ta-da! Select FINISH 72

73 A Brief Diversion Into Dangerous Territory When installing under Windows 2008 we want to alter the install Domino service to run Domino as java controller We have to modify the registry by using regedit (be v. careful!) Click on Windows Start button Type in REGEDIT Press Enter 73

74 Search For The Service Name Click on Edit/Find Type in Lotus Domino Server Click on Find next It will bring you to the service details 74

75 When You Find It - Double Click On ImagePath The registry reference IMAGEPATH will be under the Lotus Domino server registry directory. Double click on it 75

76 Add -jc To The End Of Value Data Add -jc to the end of the Value field Click on OK 76

77 Now Start The Domino Service Double click on the Lotus Domino server icon Start as a service Console will appear 77

78 If It Goes Wrong You can always delete, modify or add a new Domino service using the file ntsvinst in the Domino program directory 78

79 Testing Connectivity Between Servers We assume nothing Always test that: Clients can connect to servers Servers can connect to servers Both server names from resolve in the DNS from each server so Typing Pacific from the OS of Adriatic/Demo should return the correct ip for Pacific Typing Adriatic from the OS of Pacific/Demo should return the correct server ip for Adriatic But each server should also be able to resolve itself. Often the server needs to connect to itself, most notably when Delivering mail Running agents It first tries the setting in TCPIP address on the server document, then tries the FQHN on the basics tab Typing Pacific from the OS of Pacific/Demo should return the correct server ip for Pacific (itself) Typing Adriatic from the OS of Adriatic/Demo should return the correct ip for Adriatic (itself) 79

80 Curse you Windows! There have been persistent problems with Windows where it will simply forget its own hostname If you use a hostname in the TCPIP settings in the server document as the address of the server Even if the hostname still resolves outside of Domino Side effects include agent based mail not sending or even attempting to send If you suspect this is your problem you can confirm it by typing Trace xxx/xxx where xxx/xxx is the server name itself If it fails to connect to itself you ll need to restart Domino A regular Domino restart would usually avoid this happening (once a month) This was very common in older Windows OS versions 80

81 Ensuring Your Admin Client Can See The Server Your server is registered, but the admin client will not have it in the list of servers The admin client will not pick up the new server automatically Although you can always do File - Open Server and open the server itself, we don t want to do that You haven t switched to that server fully in the admin client If you run multiple domains you haven t switched to the domain fully 81

82 Ensuring Your Admin Client Can See The Server Select Administration/Refresh Server List / Current Domain (or All Domains if you run multiple Domains) The bookmarks should update and the new servers will now be visible 82

83 Confirm Connectivity From The Client From your admin client Select File / Preferences / User Preferences Select the Ports tab in Preferences 83

84 Trace From The Client Click on Trace Enter Pacific/svr/demo in the server field Click on the Trace button Trace runs successfully and reports the route take to connect to the server make sure the servername is resolvable and the connections isn t via passthru Passthru connections don t work with Full Access Administration which you ll need later 84

85 Tricky Client Connection Documents You can automatically create a connection document to a server by doing File - Application - Open then typing the ip address of the server in the server address field This is good! 85

86 Tricky Client Connection Documents However the connection document that is created will have a priority set as low so will be ignored this is bad! You will need to modify the connection document created in your local names.nsf and modify the priority on the Advanced tab of the document to say normal 86

87 Server Trace - Can The Servers See Each Other Select your Atlantic/Svr/Demo server Choose the server from the bookmarks bar Select the Server tab on the admin client Select the Server Console option in the sub-menu Choose the Live option 87

88 Server Trace - Can The Servers See Each Other You now have a live console In the console command bar, type in Trace pacific/svr/demo Click on SEND or Press Enter 88

89 Server Trace - Can The Servers See Each Other Trace will run on server You should see connected to server Success! 89

90 Trace From Other Cluster Mate Select your Pacific/Svr/Demo server Choose the server from the bookmarks Select the Server tab on the admin client Select the Server Console option in the sub-menu Choose the Live option 90

91 Trace From Other Cluster Mate You now have a live console In the console command bar, type in Trace atlantic/svr/demo Click on Send or Press Enter 91

92 Trace From Other Cluster Mate Trace will run on server You should see connected to server Success! 92

93 Creating Your Backup Replication Document Always always always Have a connection document replicating between server cluster mates Very good practice Helps if cluster fails Replicates stubs better Cluster replication is in-memory only so if one server is down the replication will never catch up Helps in the setup The creation of a cluster request has to replicate to the cluster servers It.makes.sense.anyway All servers in a domain should be replicating the system databases at the very least unless a proven, documented, business reason says otherwise but then you don t have a working cluster, you have something else 93

94 Create A Connection Document Select the atlantic server in the admin client Go to the Configuration tab Under Server, select Connections Connections may already exist from the setup Delete them 94

95 Create A Connection Document Click on ADD CONNECTION 95

96 Basics tab Connection type - Local Area Network Source server - leave as Atlantic/Svr/Demo Click on the button beside Destination Server - Select Pacific/Svr/Demo Click on OK Make sure both the Source and Destination Domains are correct, in this case Demo Select the button Select TCPIP (in most circumstances, more on ports later) Select OK 96

97 Replication/Routing tab Mostly left as is Choose the button beside Mail Routing and select -None- Click on OK You don t want your cluster replication backup connection to route mail, it s not necessary You may not want to replicate templates in which case at *.NTF in the Files To Exclude field 97

98 Schedule tab Leave Schedule as Enabled Change Connect at Times to 08:00-07:55 You don t want to replicate only during daylight hours This instructs replication to happen 24 hours a day (oh alright 23hrs 55mins a day) Change Repeat Interval Of to 60 minutes For this session we re going to use 5 minutes but cluster replication is in memory, if it s working you shouldn t need to replicate more than every 60 minutes and doing so more often can break your cluster replication tasks Leave Days of Week as Sun-Sat 98

99 Comments tab Enter details on the connection So you know why it is there What the purpose of it is Enter author details Who created it When 99

100 Checking Replication Click on Connection now visible in view 100

101 Checking Replication Select your Atlantic server Choose the server from the bookmarks bar Select the Server tab on the admin client Select the Server Console option in the sub-menu Choose the Live option 101

102 Checking Replication You now have a live console In the console command bar, type in sh sch or sh sched if you like a bit more typing Click on Send or Press Enter 102

103 Checking Replication You should get this output TIP - always good to wait and watch scheduled replication take place just to be sure Alternatively, check the replication log in log.nsf 103

104 Creating your first cluster Now we are ready to cluster the two servers And 103 slides already... we are doing well ;) In the admin client, select the Atlantic/Svr/Demo server Click on the Configuration tab Select the All Server Documents view / menu option on the left Click on the servers Atlantic/Svr/Demo and Pacific/Svr/Demo to select them 104

105 Now The Difficult Bit - Creating A Cluster Click on Add To Cluster button but that s not all

106 Now The Difficult Bit - Creating A Cluster Click YES on the verification dialog box to confirm that you did mean to click on the Add To Cluster button and that you do mean to apply this setting to every server you have selected Now would be a good time to look at your screen and triple check what servers you have selected 106

107 Creating Your First Cluster Select *Create New Cluster and click on OK Enter a name for your cluster the cluster name will be visible to administrators and in configuration documents but not to clients choose a name that makes sense and you don t mind keeping eg MailCluster Stay away from special characters For this SNT we re going to call our cluster OCEANS Click on OK 107

108 Using Adminp You will now get a dialog asking if you want to create the cluster immediately That s fine if you have access from your client to both servers and the connectivity is good and you have no issues making server side changes whilst the server is live However, choosing to create the cluster via the Administration Process will mean that all tasks are created in admin4.nsf and we can track and review them from there 108

109 Using Adminp For today s demo we want to track activity so we re choosing No and forcing the server to create requests that Adminp will process Click on OK to the next dialog which is confirming the requests have been generated 109

110 Using Adminp The requests you just generated will be processed by the Administration server for your domain That server is running the adminp task All your servers should be running adminp but if the administration server or any of the servers you want to cluster aren t, your cluster request won t work On your admin client select the Administration Server for your Domino domain, in our case that s Atlantic/Svr/DemoFiles The Administration server will be the first one listed in your domain bookmark With the key! 110

111 Using Adminp In your admin client select the Files tab Choose the admin4.nsf database (Administration Requests) which should appear near the top of the files list Double click on the database to open it If you see this screen, press Esc or close the tab to close it 111

112 Using Adminp Select the All Requests By Action view from the left hand menu Or All Requests By Time Initiated You will see your request Add Server To Cluster created as two entries, one for each server 112

113 Using Adminp Taking a look at one of the requests we can see that the Server To Perform The Action is the Administration Server for the Domino Directory this may not be (and often isn t) one of the servers you are clustering the admin4.nsf database must replicate the request to that server before anything will happen 113

114 Using Adminp Name to perform the action on is atlantic/svr/demo in this instance (there is also a document for pacific/svr/demo) This means the adminp task on your Administration server will update the server documents for those 2 servers when it receives this request If your new server documents aren t replicated to the Administration server, nothing will happen 114

115 It s All About The Replication Admin4 HAS to be replicating between your servers to be clustered and the Administration server for the domain or these requests you re generating will never be processed Only the Administration Server for the domain will process the request to add a server to the cluster Names.nsf HAS to be replicating between your servers to be clustered and the Administration server for the domain or these requests you re generating will never be processed If the server documents we just created aren t present in the directory the Administration server is using, the add to cluster request will fail If the updated server documents which are now assigned to a cluster are not replicated back to the original servers (atlantic/svr/demo and pacific/svr/demo) then those servers will never complete their cluster setup 115

116 It s All About The Replication If for some reason your request doesn t appear to work, don t just do it again, review your replication between servers You should always have a connection document that is able to replicate from your administration server to every other server, even indirectly, names.nsf and admin4.nsf (and events4.nsf but that s for another session) 116

117 Using Adminp Once the adminp task has processed the request the server documents in the directory on the Administration server will now have a cluster name Process is marked as completed SUCCESS! 117

118 Using Adminp Request in admin4.nsf is automatically marked Complete (with a tick) SUCCESS! 118

119 Using Adminp Now we just have to wait for those updated server documents to arrive into the directories on Atlantic/Svr/Demo and Pacific/Svr/Demo In this case there s no wait time for Atlantic since that is the Administration Server The server document update will trigger the cluster manager task on atlantic/ svr/demo and pacific/svr/demo Without you doing anything further, the servers will complete their own setup 119

120 On The Consoles On Atlantic you will see this The Cluster Database Directory task starts Cluster database directory created Attempts to replicate cldbdir.nsf with Pacific/Svr/Demo Fails The cluster request has not yet completed on Pacific/Svr/Demo yet so the cldbdir.nsf database doesn t yet exist The Cluster Replicator task starts 120

121 On The Consoles On Pacific you will see this Cluster task start Now cldbdir.nsf replicates correctly! You will see reference to busytime and clubusy - we will talk about calendaring in a bit 121

122 Cluster Database Directory Task cldbdir runs on any Domino server that is clustered The cldbdir tasks is responsible for keeping the cldbdir.nsf database up to date The cldbdir.nsf database contains the filename, replica id and status of every database and template on a server The cldbdir.nsf replicates between all servers in a single cluster When trying to open a database on a server, the cldbdir.nsf database can tell the cluster task if another server has a replica of the requested database for the user to be diverted to A user will never to sent to a cluster mate unless a replica of their requested database is available on the destination server Cldbdir.nsf is maintained via the cluster replicator task If cldbdir.nsf is not common across all cluster mates, and up to date, your cluster won t work properly 122

123 Cluster Directory Database Select the Files tab The cluster directory database is now visible Open the database by double clicking on it 123

124 Cluster Directory Database Select Databases By Server Both servers should be visible Select Databases by File Name Two documents per database, one representing each database replica on each of the two servers Not all databases have to be on all cluster mates. More on this later 124

125 Cluster Directory Database Open one of the documents - these aren t editable and are maintained by cldbdir tasks databases must share a replica id to cluster, but not a filename databases can be marked Out Of Service if you don t want them to cluster replicate 125

126 Check Tasks On The Server Select your Atlantic server Choose the server from the bookmarks bar Select the Server tab on the admin client Select the Server Console option in the sub-menu Choose the Live option 126

127 Check Tasks On The Server In the console command bar, type in sh ta Click on Send or press Enter 127

128 Check The Tasks On The Server You should see these tasks running 128

129 Ask The Cluster If It Is Running On the console, enter sh cl or show cluster 129

130 Cluster Security And Standardisation Consider what you want a cluster to do Both servers could be working with the applications Running agents Permitting user access Replicating All servers in a cluster should have identical security to Applications Code execution There are easy ways to do this 130

131 Server Groups Create a cluster group for your servers Navitage to the People and Groups tab on your admin client Click on the Groups view Click on Add Group 131

132 Server Groups Group Name Give it a meaningful name. We re using CLSVROceans (cluster servers oceans) Select the button beside Group Type and choose Server Group You don t want to use the server group you re creating here for mailing Select the button beside Category and choose Administration Or another category if you prefer Complete the Description field 132

133 Server Groups Select the icon beside Members Choose the servers you want from the list of all directory entries In this case atlantic/svr/demo and pacific/svr/demo Click on Add 133

134 Server Groups We re going to explicitly add this group to ACLs as a Server Group Don t use wildcards in the member fields Don t be tempted to use a client id such as an administrator s name in the member field People or wildcards added into a group containing server names will invalidate the group as a server group so its ACL rights won t apply 134

135 Adding The Group To All ACLs This is one of the best things we ll show you today First turn on the magic Full Access Administration Full Access Administration gives you Manager access to every database on the server but no roles In the Administrator client choose the Administration menu and the Full Access Administration 135

136 Adding The Group To All ACLs If you don t have rights to use Full Access Administration on that server you ll get a dialog box telling you Rights to use Full Access Administration are controlled on a server by server basis Configuration Tab - Current Server Document - Security Page 136

137 Adding The Group To All ACLs In the admin client, select the Files tab Click on the top level directory On the right Tools pane Expand Database Select Manage ACL 137

138 Adding The Group To All ACLs We are going to replace the existing server name in any database ACL with the new group name we just created server with the Group name So we re going to Modify ACL entries 138

139 Adding The Group To All ACLs Modify Dialog - first we choose the name we want to search for Then we select from the address dialog (or type, if you re sure you ll get it right) the name of the group we just created 139

140 Adding The Group To All ACLs After clicking OK you are brought back to the modify ACL dialog Click on the drop down beside User and change the type to Server Group Leave other settings as they are Click on OK You don t want access or roles to change 140

141 Adding The Group To All ACLs Each database on the server will now be scanned to see if Atlantic/Svr/Demo is in the ACL it isn t case sensitive but it won t match Atlantic only, the syntax must be correct When completed the following dialog or similar appears Click on OK 141

142 Adding The Group To All ACLs If your dialog shows x database processed / x errors the reasons could be If you don t have Full Access Administration turned on then you can only change databases that your ID already has Manager access to All other databases will throw an error and fail to update Any databases that don t have Atlantic/Svr/Demo in them will fail to update This isn t a problem. If the server name wasn t there then you don t need it replaced If you dialog shows some failures, click on the status bar at the bottom of your admin client to pull up a list of recent log messages 142

143 Standardising Your Server Configuration For Clusters to work well Settings on each server document should be identical where possible Good for user experience Both servers behave the same Good for applications Code executes correctly Good for clustering Servers can automagically create replicas on each other If a cluster is working properly and providing load balancing the user s should be unaware of its existence. Two users accessing the same application on different servers should have an identical experience 143

144 Server Level Rights For both server documents In the admin client go to the Configuration tab Go to the All Server Documents view or menu item Open the server document that you want to update using the button 144

145 Server Level Rights Go to the Security tab Scroll down to Server Access section (on the bottom half of the page) Select the field Create Databases and Templates Enter the values CLSVROceans and LocalDomainAdmins this grants both servers atlantic/svr/demo and pacific/svr/demo as well as any administrators, rights to create databases on both servers 145

146 Server Level Rights Repeat previous step for Create New Replicas, so each server can create replicas of their dbs on the other Create Master Templates Allowed to use Monitors For Trusted Servers Add the cluster group, so each server can run agents on the other You must grant Create new replicas rights as a minimum You don t need to do all of the other fields Its just good practice Click on SAVE AND CLOSE 146

147 Code Execution Rights For both server documents In the admin client go to the Configuration tab Go to the All Server Documents view or menu item Open the server document that you want to update using the button 147

148 Code Execution Rights Go to the Security tab Scroll down to Programability Restrictions section (on the right side of the form) Ensure that these fields match on both servers For this demo Enter LocalDomainAdmins in the following fields Sign or run unrestricted methods and operations Sign agents to run on behalf of someone else Sign agents or XPages to run on behalf of the invoker Sign or run restricted LotusScript/Java agents Enter */demo in Run Simple and Formula agents Click on SAVE AND CLOSE when complete 148

149 Putting Replicas Onto The Cluster Server There are many ways to put replicas onto the other server Manual Create Replica Enter each database and create a replica Slow (created from your client connection to the server, not server to server) Automated Create Replica Using Administration Process to create the replicas Faster (you can select lots of databases/directories in one go) Accelerated Create Replica Significantly faster version of Automated Create Replica, uses backup level APIs Physical database copy OS level copy Still a replica Replica ID Will need to be consistency checked (which can take a lot of time) Risk of corruption Cluster Database Directory won t know about its existence immediately 149

150 Replicating Multiple Database Over To New Cluster We want to use Accelerated Create Replica This isn t a specific option, if you send New Replica requests via your Admin client they will be added to admin4.nsf to be processed by adminp The request added to admin4.nsf is actually for Accelerated Create Replica so the adminp task will try to do that first If Accelerated Create Replica fails, adminp will generate a new Create Replica request to happen instead You don t choose Accelerated Create Replica vs Create Replica - the server does that for you 150

151 Replicating Multiple Database Over To New Cluster In Domino Administrator go to Files tab Select the Mail directory (or whichever directory you d like to replicate) Or manually select individual databases by holding down the CTRL key Click on Create Replica under the Tools - Database menu 151

152 Replicating Multiple Database Over To New Cluster On the Create Replica dialog check the box for show only cluster members for the cluster OCEANS You don t have to do this but if doesn t correctly show your cluster servers, then your cluster is not yet set up correctly and you need to stop and review All the requests being created are shown on the right hand side of the dialog 152

153 Replicating Multiple Database Over To New Cluster Creating these replicas correctly and quickly is key to our successful build of a cluster Ignoring errors or alerts that appear during replica creation will come back to haunt you later There are some tricks to minimise errors during replica creation but even when you think you have everything configured correctly TEST. TEST.TEST Select one single database and try a replica request against that Only when you get that working, try a few others Don t try running against every database on the server straightaway This isn t hard, but we ve both built countless clusters and would still take the time to test using single databases 30 minutes spent on testing could save you countless hours in deployment 153

154 Accelerated Create Replica Failing Accelerated Create Replica (ACR) requires a specific set of circumstances for it to work but it will save you at least 60% of initial replication time so ensuring your configuration meets these criteria will pay off in hours of saved time 154

155 What s Needed For ACR To Work Servers must be clustered Test this using the server command we discussed earlier Enter Sh Cl or Show Cluster on the console of each of the 2 servers you should see similar to the following note: output shows 2 cluster members and their names sh cl Cluster information: Cluster name: OCEANS, Server name: atlantic/svr/demo Server cluster probe timeout: 1 minute(s) Server cluster probe count: Server cluster default port: * Server cluster auxiliary ports: Server availability threshold: 0 Server availability index: 100 (state: AVAILABLE) Server availability default minimum transaction time: 3000 Cluster members (2): Server: atlantic/svr/demo, availability index: 100 Server: pacific/svr/demo, availability index:

156 What s Needed For ACR To Work Servers must both be v6 or higher ACR was introduced in v6 so it isn t supported if either the source or destination server is pre v6 If either server is pre v6, the admin process will use standard replica creation to complete the task 156

157 What s Needed For ACR To Work Destination server must be 6.03 or higher A new database property was introduced in entitled support reply/forward indicator Since that setting is a default for any databases created post 6.0.3, a source server that is or newer will most likely be home to databases with that property set A destination server that is earlier than will not be able to handle ACR creation of databases with a property it doesn t recognise 157

158 What s Needed For ACR To Work Source and Destination databases must share Full Text Index settings When creating replica requests you have a checkbox to create a matching Full Text Index. ACR will only work if the request matches the status of the source database Databases that are FT Indexed, must have replicas created with FT Indexes Default is that the box is unchecked which means do not create Full Text Index 158

159 What s Needed For ACR To Work DAOS must not be enabled for a source database Standard replicas can be created from databases that have file attachments stored in DAOS NLO s, but not Accelerated Create Replicas If you want to enable DAOS and you know you re building a cluster, build the cluster first then enable DAOS on both servers This is purely to make the initial replication of data from source to destination server, faster. With DAOS enabled the replication will still happen, just not using Accelerated Create Replica 159

160 What s Needed For ACR To Work Some slightly more obscure database settings that would prevent Accelerated Create Replica from working Source database must not use single copy object store (unlikely) Source database must not use a single copy template that doesn t exist on destination server So that s your ACR checklist. Spend 30 minutes verifying this and save yourself many many hours 160

161 Tip! A Nice Notes.ini Setting ADMINP_ACCELERATED_REPLICA_OVERRIDES is a notes.ini setting put in place on the source server from which you want to generate replicas ADMINP_ACCELERATED_REPLICA_OVERRIDES=4 Ignore checking that both servers are in a cluster. This very useful if you want to replicate large numbers of databases between 2 servers that aren t clustered ADMINP_ACCELERATED_REPLICA_OVERRIDES=1 Ignore checking that reply/forward indicators are supported. Useful if your destination or source server is prior to However we have seen the reply/forward error in later releases too so this is useful notes.ini setting to apply regardless ADMINP_ACCELERATED_REPLICA_OVERRIDES=5 Ignore both checking that source and destination servers are clustered and checking that reply/forward indicators are enabled All other ACR requirements are fixed and unavoidable 161

162 There s Always A But... Accelerated Create Replica will create a full replica of a database even if it has replication disabled To you it will look like everything has replicated perfectly however, that database will not accept any further updates, either from the cluster replicator task or from the scheduled replication Run the cluster analysis tool (more on this later on) to identify which databases on your source server have replication disabled Or ask the Domino Administrator team to add that as a column in the Files view, we ll be right behind you! (and whilst you re at it, can you ask them to add a column showing if there s a replication selection formula too - thanks) 162

163 Some Tips Prior To Creating Replicas Make sure you have enabled unread marks to replicate for all databases at least within the cluster, if not across all servers Remove Full Text Indexes from databases before you start and re-create them after replication is finished Rebuilding Full Text Indexes after a major upgrade is best practice anyway Create Full Text Indexes without indexing file attachments for better performance, stability and less disk usage 163

164 Enabling / Disabling Unread Mark Cluster Replication Unread mark architecture has changed a number of times in Domino Different releases/different solutions Standard fixed from 6.02/6.5.3 timeframe Unread marks are stored in a table When you read a document, it is written to the table It s a database overhead Unread marks can therefore replicate Consider the extra load on the cluster Do you need unread marks on certain databases? 164

165 Enabling / disabling unread mark cluster replication Navigate to the Files tab in the Administrator client Go to the Mail directory Right click on a mail file Select Properties 165

166 Enabling / Disabling Unread Mark Cluster Replication Select the Advanced tab Under the item Replicate Unread Marks Select Clustered servers only Click on the X to close the properties dialog box 166

167 Setting Unread Marks For Many Databases In the Administrator client choose the Administration menu and then Full Access Administration To ensure you have access to all the databases you are about to change Navigate to the Files tab in the Administrator client Go to the Mail directory Right Mouse Click - choose Advanced Properties Now select Replicate Unread Marks and check Clustered Servers Only 167

168 Removing Or Creating Full Text Indexes For Multiple Databases In the Administrator client choose the Administration menu and then Full Access Administration To ensure you have access to all the databases you are about to change Navigate to the Files tab in the Administrator client Go to the Mail directory Right Mouse Click - choose Full Text Index Choose to Create or Delete the Full Text Indexes 168

169 How Clustering Behaves For Clients 169

170 Clients And Clustering The Notes client is Cluster Aware When it communicates with a Domino server that is in a cluster, that server updates a file on the client with its cluster name and its cluster mates this makes the client aware that the server it wants is in a cluster and means even if the server is down, the client has built in information about where to go next if you add a domino server to a cluster and rapidly shut down the original server, the notes clients will never get a chance to learn about the cluster If a primary server fails, the Notes client will search the other cluster mates it knows about for a replica of the database it wants Searches cluster mates alphabetically not by availability this is very important in your planning Opens database on the first available cluster mate it finds Of course the Notes client has to be able to find these cluster mates So you need a connection document from the client to every cluster mate that could be used Or have the sever names registered in DNS 170

171 Cluster.NCF The cluster.ncf is a file in the data directory on the client It contains details of all clustered servers you have connected to from that client, along with their cluster mates there can be several clusters in your cluster.ncf If you delete the cluster.ncf the Notes client will forget all about any clusters it was aware of, but will relearn the first time you reconnect to a clustered server Time=21/12/ :57:53 (802577A4:0083A4BA) Oceans CN=Atlantic/OU=Svr/O=Demo CN=Pacific/OU=Svr/O=Demo 171

172 Notes Improved Client Clustering/Repilcas Client Transparency In the client, the ability to failover to a cluster mate without the user being notified or stopped in their work has been improved During the following activities, if the primary server goes down, the user will be invisibly transferred to one of the cluster mates Creating a new entry for PIM views (Mail message, Calendar entry, or To Do) Opening a document in an application Opening a different folder or view Replying or forwarding with attachment Creating a new mail message while multiple documents are selected Deleting a single or multiple document Sending a message Switching back to Primary server when it becomes available 172

173 Notes Managed Replicas Managed replicas are local replicas of mail files that behave as though they are server replicas Created by using a Mail Policy, a user assigned a managed replica will always be directed to that local replica even if they are on the network and the server is available If the managed replica can be smaller in size than the server replica, restricted by age of documents or simply not contain attachments attempts by the client to access older documents or retrieve attachments result in that information being transparently pulled from a server replica If a local managed replica becomes corrupt in some way, the client is transparently failed over to a server replica In this way Managed Replicas force the client to behave like a cluster mate for mail Managed replicas are intended for server based clients who have poor network connections to their home server or to reduce the number of concurrent clients connected to a home server. 173

174 Client Behaviour When clicking on a bookmark for mail the users will be sent to their home server if the home server is down or at its threshold level and there is another cluster mate available the client will be redirected there If the user explicitly clicks on a workspace icon pointing to a server, the database will open on that server unless the server is down completely No point building a cluster and leaving all your users with the same home server All users will be forced to connect to that server initially until it s so busy it has to redirect requests You are not getting good value for your hardware investment Not effective load balancing Move some users so their home server is a different cluster mate 174

175 Move Many Users At Once To New Server From the Administration server Navigate to the People and Groups tab on your admin client Select the People you want to move Choose Move To Another Server from the Tools menu on the right Move user ensures a mail file replica is on the new home server Updates the person document with the new home server Updates the client location settings when the user next logs in 175

176 Servers And Clusters Servers operate slightly differently to clients when deciding where to failover Domino server C will be aware that Server A and B are cluster mates As with the clients, the server has a cluster.ncf in its data directory with a maintained list of servers and clusters If Server A goes down.. Domino Server C will try to find a cluster mate to work with Alphabetically By availability (Server Availability Index - more on this later) Mail routing 176

177 Servers And Clusters When replicating/routing to a server in a cluster Create the connection TO the cluster name Suppose you want to replicate from ARTIC server to the cluster Let s create a connection to the cluster from ARTIC We create it on the administration server NOT on ARTIC Why? Centralised management of the directory is always a good thing 177

178 Servers And Clusters You can and should also utilise cluster awareness for replication Create connection documents to the cluster name, not server names By doing this you are telling atlantic/svr/demo in the connection below, to replicate with any server in the OCEANS cluster Since cluster replication should route all changes immediately, you should only need to replicate with one server in the cluster Using the cluster name as a destination means any not all! 178

179 Create A Connection Document Select the Administration server in the admin client, in this case that is Arctic/ Svr/Demo Go to the Configuration tab Under Server, select Connections Click on 179

180 Basics Tab Connection type - Local Area Network Source server - Change to Arctic/Svr/Demo Click in the field Destination Server - type in the Cluster name OCEANS Click on OK Select the Choose Ports button - Select TCPIP Confirm that both source and destination domain fields are set to demo Do not set a destination address 180

181 Replication/Routing Tab Mostly left as is Set Routing Task to be -None- 181

182 Schedule Tab Leave Schedule as Enabled Change Connect at Times to 08:00-07:55 You don t want to replicate only during daylight hours This instructs replication to happen 24 hours a day (oh alright 23hrs 55mins a day) Change Repeat Interval Of to 60 minutes For this session we re going to use 5 minutes but cluster replication is in memory, if it s working you shouldn t need to replicate more than every 60 minutes and doing so more often can break your cluster replication tasks Leave Days of Week as Sun-Sat 182

183 Comments Tab Enter details on the connection So you know why it is there What the purpose of it is Enter author details Who created it When 183

184 Checking Replication Click on SAVE AND CLOSE Connection now visible in view 184

185 Domino Mail Failover In Routing The Router tasks in Domino is cluster aware You can configure your servers to route mail to a cluster mate if the destination server isn t available This is done in the server configuration document Select the Administration server in the admin client, in this case that is Arctic/ Svr/Demo Go to the Configuration tab Under Server, select Configurations 185

186 Domino Mail Failover In Routing If there is a server configuration already for your server (in this case atlantic/svr/ demo) then we will edit it Alternatively click your server to add a new configuration document for We have an existing configuration for * which means [All Servers] so we re going to edit that 186

187 Domino Mail Failover In Routing Click on the Router/SMTP tab Click on the Advanced tab Click on the Controls tab 187

188 Domino Mail Failover In Routing Here we have a field called Cluster Failover Choices are: Enabled for last hop only Enabled for all transfers in domain Disabled 188

189 Domino mail failover in routing Enabled for last hop only If mail is sent from Arctic/Svr/Demo to Paul Mooney whose mail server is Atlantic/ Svr/Demo and Atlantic/Svr/Demo is not available, the router task will send the mail to Pacific/Svr/Demo instead However mail that would usually route through Atlantic/Svr/Demo for a user on a remote server, will not route through Pacific/Svr/Demo instead 189

190 Domino Mail Failover In Routing Enabled for all transfers in domain If mail is sent from Arctic/Svr/Demo to Paul Mooney whose mail server is Atlantic/ Svr/Demo and Atlantic/Svr/Demo is not available, the router task will send the mail to Pacific/Svr/Demo instead Mail that would usually route through Atlantic/Svr/Demo for a user on a remote server, will route through Pacific/Svr/Demo instead 190

191 Domino Mail Failover In Routing Enabled for all transfers in domain This setting makes your mail routing topology less controlled / structured because the router task will create its own best fit route from Server A to Server G if some of the intermediate servers are down. unexpected routing results difficult to troubleshoot 191

192 Domino Mail Failover In Routing Disabled If a server is down, even if it s in a cluster, mail for that server will queue up waiting on the sending server 192

193 So Let s Test It! Fail server Test failover - try and open a database on the server that is now down Test home mail - click on mail bookmark to open mail Bring server back up Test - - click on mail bookmark to open mail 193

194 Drop The Atlantic Server RDP/VNC to the ATLANTIC server Type in Q on the console Server shuts down 194

195 Launch Your Mail File Launch the Lotus Notes client If you don t know how to do that, you are in trouble! Click on the Mail bookmark The server will try to connect Fails Then will open replica on cluster mate The Status Bar on the client indicates the steps Send yourself an for testing 195

196 On The Server Console On pacific server, you will see Record of failover from a notes client When you send the Server records failure but delivers to replica 196

197 Check the pacific server On the console of the PACIFIC server Type in SH CL Note pacific knows that Atlantic is unavailable Check the cluster queue Type in Sh stat replica.cluster* List of databases waiting to cluster replicate 197

198 Final Checklist - And One You Should Repeat Regularly 198

199 Using the Cluster Analysis Tool What is it for? Cluster analysis will assist in finding issues with your cluster environment mismatched ACLS missing databases mismatched server settings Have Full Access Administration enabled In the admin client go to the Server tab then the Analysis tab Go to the Cluster menu option on the right 199

200 Using the Cluster Analysis Tool Check the items to scan for Server details ACLs not matching in server replicas Databases with replication disabled. Replication selection formulas not matching Databases that don't exist on all servers in a cluster 200

201 Choose typical settings Select everything Except Web Navigator Not really used anymore Note results database This will create a cluster analysis database locally Based off cluster analysis template Which is local to admin client Choose Overwrite this database Click on OK You will see status bar animation NOTE - on large sites, this can take hours 201

202 Cluster analysis database opens Select the By Test view Collapse the view by clicking on or hold Shift key and press - Review the results Many are irrelevant Some are vital e.g. Replicas exist within Cluster Are any applications/mail files failing? Resolve the issues and re-run the test Make the cluster analysis part of your scheduled maintenance routine Close the database when finished 202

203 How To Monitor And Tune Your New Cluster Environment 203

204 Work Queues And Clustering Cluster tasks can get very busy Heavily used mail servers Heavily used application servers On extremely busy servers Domino Clustering can lag behind Cluster replication occurs one database at a time per cluster replicator task all changes to that database are accrued and applied in a batch Increasing number of clrepl tasks will reduce queues at the expense of memory and disk writes it s an option but may not be necessary 204

205 Work Queues And Clustering Work queue depth how many databases are queuing to be replicated by the cluster task sh st replica.cluster.workqueuedepth* - Server Console Command Replica.Cluster.WorkQueueDepth = 0 - how many databases are queuing now Replica.Cluster.WorkQueueDepth.Avg = 2 - how many queue in average Replica.Cluster.WorkQueueDepth.Max = 7 - how many have been in the queue at max 3 statistics found Your work queue depth should be stable, not Increasing 205

206 Work Queues And Clustering Seconds On Queue how long a database sits on the cluster replicator queue before its data is transmitted off and out of memory sh st replica.cluster.secondsonque* - Server Console Command Replica.Cluster.SecondsOnQueue = 6 - Current wait time for data to be replicated Replica.Cluster.SecondsOnQueue.Avg = 7 - average wait time for data to be replicated Replica.Cluster.SecondsOnQueue.Max = 15 - maximum wait time for data to be replicated 3 statistics found Your seconds on queue should remain in single digits if you want your cluster to be transparent to the users. This means that additions / deletions / updates would take only a few seconds to appear on cluster mates. Seconds on queue will occasionally peak, this isn t a big concern but the current wait time and average wait time should be fairly stable 206

207 Work Queues And Clustering If your cluster statistics are showing poor performance or cluster delays then you can consider increasing the number of cluster replicators (CLRepl) to manage the load To add multiple cluster replicators you need to modify the notes.ini file From the server console type the command Set Config Cluster_Replicators=X where X represents the number of replicators you need Each additional cluster replicator is an additional Domino task that consumes processor and memory Too many tasks can also cause conflicts in the queue Adding additional cluster replicators should be carefully considered and applied in stages 207

208 Displaying cluster work queues Select your Atlantic/Svr/Demo server Choose the server from the bookmarks bar Select the Server tab on the admin client Select the Server Console option in the sub-menu Choose the Live option 208

209 Getting Work Queue Depth Console Stats Enter sh st replica.cluster.workqueuedepth* in console box Click on send Output 209

210 Getting Seconds On Queue Console Stats Enter sh st replica.cluster.secondsonqueue* in console box Click on send Output 210

211 Adding Cluster Stats To The Monitor In the Administration Client Select the Administrative server Click on the Server Tab Click on Monitoring Click on the Start button Monitoring connections are from your client, make sure you have a connection or route to all servers you want to monitor or the server won t report 211

212 Adding Cluster Stats To The Monitor Right click on the Statistics Select Monitor New Statistic The following dialog box appears Scroll down to Replica, then expand Cluster Expand WorkQueueDepth Select Avg Click on OK 212

213 Adding Cluster Stats To The Monitor The statistic is now visible on Monitor Repeat previous steps for every stat you want visible 213

214 Cluster Traffic Cluster traffic is significant By default, cluster traffic will use your public network connection. This leads to replication data between servers fighting for bandwidth with client access activity One solution which separates cluster traffic from client traffic Have multiple network cards on cluster servers With different IP address on each network cards Create additional ports for domino servers on each server / cluster mate Map a port to a particular IP address Therefore you are mapping a port to a network card Force cluster replication over the new port/network card If the servers are co-located you could have a high speed fibre connection handling replication 214

215 Setting Up Private Traffic Cluster Replication Edit the notes.ini file to add new values We can do this from the server console If we want to create a new port for handling cluster traffic called ClusterTCPIP We can add the new port settings but they won t take effect until we restart the server 215

216 Setting Up Private Traffic Cluster Replication Ensure both network cards are enabled are have ip addresses bound to them Add entries into local hosts file on the server so that each ip address has its own hostname You should be able to ping both hostnames from the server you are working on You should be able to ping both ip addresses from the cluster mate server 216

217 Setting Up Private Traffic Cluster Replication We have 2 network cards on our server now ip address: ip address: By default Domino will grab all ip addresses for port 1352 and will bind tasks such as HTTP to the first network card We can change that From the server console type Set Config TCPIP_TCPIPAddress=0, :1352 Set Config ClusterTCPIP_TCPIPAddress=0, :1352 Now we have told Domino that there are two network cards and that it should bind each to its own port TCPIP and ClusterTCPIP 217

218 Setting Up Private Traffic Cluster Replication Now we have to tell Domino that both ports should be used From the server console we update notes.ini Set Config Ports=TCPIP,ClusterTCPIP We now have to tell Domino that the ClusterTCPIP port should be used for Cluster traffic From the server console we update notes.ini Set Config Server_Cluster_Default_Port=ClusterTCPIP 218

219 Setting Up Private Traffic Cluster Replication Now we have to update the server document In the admin client go to the Configuration tab Go to the All Server Documents view or menu item Open the server document that you want to update using the button 219

220 Setting Up Private Traffic Cluster Replication Go to the Ports page In Notes Network Ports we need to add the new port details Port order is very important here because Domino tasks will always use the first port listed by default Port - ClusterTCPIP which is the name we just added into notes.ini 220

221 Setting Up Private Traffic Cluster Replication Protocol is non editable and will be kept updated by the server Notes Network - should be different for this port and shared across cluster mates Net Addresses - could also be the hostname that resolves to this 2nd ip Enabled - make sure you enable the port 221

222 Setting Up Private Traffic Cluster Replication Now we just need to restart the server From the server console type Restart Server Once the server comes back up we can confirm the port is active by issuing the following server console command Sh Port ClusterTCPIP We can also confirm that it is being used by the cluster replicator by issuing the following server console command Sh Cl or Show Cluster Cluster name: OCEANS, Server name: atlantic/svr/demo Server cluster probe timeout: 1 minute(s) Server cluster probe count: Server cluster default port: ClusterTCPIP 222

223 Setting Up Private Traffic Cluster Replication When we set a private network to route cluster replication traffic over we are telling Domino not to send cluster traffic over any other port If the cluster port fails on the server, the cluster replication activity will back up as it can t route through any other port Often no-one will notice that the port has failed because no users are active on it Best Practice is to include an auxiliary port which tells Domino what to try if the cluster port fails From the Domino Server console issue the following command Set Config Server_Cluster_Auxiliary_Ports=TCPIP this tells Domino to try the TCPIP port if the CLUSTERTCPIP port isn t responding You can also use Server_Cluster_Auxiliary_Ports=* to mean use any port 223

224 Resilient Port Configuration Single points of failure in the private port concept the network cards the private network itself we can use auxiliary ports to work around this and it does give us the advantage of separating the traffic and sending the cluster traffic over a faster network Alternatively for simpler configuration use teamed NICs on newer servers Ports aren t everything Use resilient switches Use resilient networks 224

225 Fun With Server Clustering You can force all users to fail to another cluster server Useful for Disaster Recovery testing Useful for server maintenance work Useful for migrations On the server console type the following command Set Config Server_Restricted= 1 Prevent any new sessions opened on this server until restart Administrators bypass this restriction if using the Domino Administrator client 2 Prevent any new sessions opened on this server until setting changed Administrators bypass this restriction if using the Domino Administrator client Setting is persistent even through server reboots 0 Feature disabled 225

226 Fun With Server Clustering On the server console type Drop All this disconnects all connected users, including you. Don t panic if you type the command and your Administrator client now says Server not responding, that s what should happen Just connect to the server again Be aware that server_restricted applies not just to clients but also to other servers. Whilst a server is sat in restricted mode No mail will route to it Nothing will replicate with it No cluster traffic will be sent to it When coming out of server_restricted, you want to immediately force a full replication to catch up on missed data 226

227 Restricting A Domino Server Let us test Set Server_Restricted=1 on Atlantic/Svr/Demo (Paul s home server) Type Drop All on the server console Paul will now be failed over to Pacific/Svr/Demo to open his mail I will still be able to access and work on Atlantic via Domino Administrator If Paul tries to open Atlantic/Svr/Demo directly, he is denied access 227

228 Preventing Domino Server Being Overloaded Domino can fail new sessions based on the following Hitting maximum number of sessions specified Hitting low availability (the low setting is set by administrator) Maximum sessions Server_Max_users Server_Max_Sessions these are very hammer to crack a nut type settings. Do you rarely want to restrict server access to specific exact numbers of users? Let me tell you a sad story about clusters, an evil admin and max users gone bad Availability index How much resource the server has available to it 228

229 Let s Talk SAI.. Server Availability Index (SAI) is core to Domino cluster logic. The server monitors itself, checking the response times of requests continually As the response times start decreasing so does the SAI SAI is meant to represent the % of server capacity that is available SAI starts at meaning, I m fully available and ready to take on all comers SAI falls to 0 - meaning, go away I can t handle any more traffic The SAI is relative to the actual capacity of the server 100% SAI on one server won t give the same response times as 100% SAI on another more powerful server 229

230 Preventing Domino Server Being Overloaded You can set a notes.ini setting of Server_Availability_Threshold=X where X represents the lowest acceptable value of availability you want the server to reach before it starts failing users over to a cluster mate But the thing about SAI calculation It kind of, sort of, works and sometimes doesn t The biggest problem appears to be with powerful hardware where the SAI miscalculates its own availability showing itself as being on low availability when the server is barely utilised There are lots of reasons for this, mostly due to the server trying to record when it s busiest as a tracking point and base everything off that 230

231 Preventing Domino Server Being Overloaded What you need to know It gets better with and onwards. Earlier Domino version SAI calculations are less reliable If you have brand shiny new hardware which Domino insists is maxed out - that may not be true You can override Domino s calculation of available resources by modifying a notes.ini setting called Server_Transinfo_Range HERE BE TIGERS!!! We will not be going into any more detail on changing Server_Transinfo_Range because we sincerely hope none of you will be doing it If you want to know more read the Admin help Then read it again Then make a coffee, sit down and ask yourself if you really want to do this 231

232 Disabling database cluster replication You may not want to cluster replicate a specific database temporarily, even though it is present on multiple servers By default Domino will cluster replicate, and make available for failover, any server sharing a replica ID between source and destination If it s an application where evil developers have hardcoded server names and it simply won t work on another server, you don t want it clustered because you don t want users failing over and trying to use it However you may want a replica stored on a cluster mate and kept up to date as a live back up If a single database is taking up too much cluster replication activity disable it for clustering Disabling / Enabling cluster replication doesn t affect standard replication so if a database doesn t need cluster performance take it out of cluster replication You can disable cluster replication for a individual database or group of databases via the Domino Administrator client 232

233 Disabling database cluster replication In the Domino Admin client select server Atlantic/Svr/Demo Select the Files tab Select the file you want to mark unavailable for clustering and choose cluster from the right hand menu (or right click) 233

234 Disabling database cluster replication You have five options on this dialog, all are radio buttons so only one can be selected at a time Out of Service - do not allow client access to this database or fail clients over to it. Existing clients can remain in it until they finish their session but can t reconnect In Service - allow clients back into this database or allow clients to fail over into it from another server Pending Delete - as Out of Service but Pending Delete cannot be marked In Service again 234

235 Disabling database cluster replication You have five options on this dialog, all are radio buttons so only one can be selected at a time Enable Cluster Replication - allow the cluster replicator task to continue sending changes to and from this database. Disable Cluster Replication - halt the cluster replicator from sending or receiving changes to this database. No changes will be held in memory for it. The scheduled replicator task ignores this setting 235