Sizing Lotus Domino before Software upgrades Saqib N. Syed BMC Software, Inc.

Transcription

1 Sizing Lotus Domino before Software upgrades Saqib N. Syed BMC Software, Inc. Lotus Domino has been the leading server side software for Groupware applications for a few years now. So far, an investigation of the architectural details of Domino from a performance perspective has not appeared. In this paper, after a brief discussion of the architectural details of Domino and 5.0, we compare the performance of these two versions of software for different applications by running appropriate benchmarks. Some recommendations for the software upgrade of Domino from to 5.0 are provided based on the results of these benchmarks. 1. Introduction In the last decade or so, software upgrades have become very common and a nightmare to most of the capacity planners and systems administrators. If the software upgrade is major and involves upgrading to a software with lots of new functionality then the situation gets worse. The capacity planners and administrators now have to decide whether to continue with the current hardware setup or install a new scalable hardware to host the new version of software that can perform a lot more functions than the previous one. Yes, hardware is getting cheap these days and that is a good reason to just buy new more powerful boxes than upgrade the existing one. The Systems administrators and Capacity planners need help here or a methodology to follow to make the right decisions to cut costs in the long run. The software vendors provide some help by coming out with some numbers by working with the hardware vendors. But these numbers are again very subjective and very specific to pre-configured boxes sold by the hardware vendors. I would recommend doing some kind of due diligence by testing the performance of the new software version on the same hardware as the previous one is running on. But that would involve bringing down the production server running the current version of software just to run this experiment. So how do I do my due diligence? The answer to that is simple. Use hardware lying around in the testing/acceptance lab. You may also have to supplement this throw away hardware from the lab with some new components. The intent is to keep the costs down BUT certainly not to use something obsolete. Now, install both versions of the software, the current version and the version that you are planning to upgrade to, on similar boxes even if the boxes that you could put together are not powerful. Run benchmarks on these two versions of software running on a similar hardware configuration and this should give some ballpark figures of what performance degradation/enhancement that you can expect when you upgrade your software. If the performance on these boxes have to be extrapolated to more powerful boxes, then software that can do predictive What-ifs for changes in hardware configuration is necessary. I can cite many instances of software upgrades that have been a long process and have really been a major concern not just from security and operational perspective but also from a performance perspective. One such example is Microsoft Exchange 2000, which requires the use of Windows Currently there is no easy upgrade path from Windows NT to Windows This is a major software upgrade. There are significant enhancements in Windows 2000 and more functionality that causes concerns about picking the right boxes for deployment and still satisfying the IT manager by staying within the budget. There have been tools in the market for some time now that can enable capacity planners to do this kind of analysis before they actually upgrade the software. I have tried to use just the basic tools to cut the cost of the benchmarking effort discussed in this paper. I hope that this paper, apart from the recommendations and tips for software upgrade of Notes Server ( Lotus Domino or Domino ), will also provide an insight into a cost-effective way of going through this deployment phase for major software upgrades. In this paper, we will discuss a simple methodology to help the user through this major software upgrade using Lotus Domino as an example. The Workload Characterization section talks about Notes processes and Tasks that run on Lotus Domino and their functions. The section on Benchmarking provides some details of the benchmarking tools used for this effort and the hardware setup. The Interpretation of

2 Results section details the benchmarks and some of the results. We conclude with some tips for the users going through the upgrade from Domino to Workload Characterization Let us discuss briefly some of the architectural details of Domino. These details are important from the workload characterization perspective. It is necessary to get the workload characterization right before we start our benchmarking efforts. This benchmarking effort is for upgrade tips and certainly there have been studies done before where benchmarking efforts were undertaken to figure out the workloads. In this paper, we will not get into the details of figuring out the workloads BUT focus on coming out with some useful software upgrade tips. Based on some of the studies done before, we figured out the core set of processes and Notes tasks that run on Domino. Let us understand the functions of the processes and possible ways of characterizing them. 2.1 The Core set of processes (tasks) for Domino and their functions Following are the core set of processes for Domino that has to exist for the Server to function. There is also a notion of "tasks" in the Notes world. One could also say that the following processes represent some of the key servers tasks in Notes. 1. Nserver.exe 2. Nreplica.exe 3. Nhttp.exe 4. Nrouter.exe 5. Ncallconn.exe 6. Nupdate.exe 7. NagntMgr.exe 1. Nserver: This is the primary process in Notes that reads the Public Address Book, authenticates the users, fetches location records for other servers in the Notes Named Network, assigns work to other subtasks, accesses databases, processes in coming mail, and in general acts as a coordinator of most Notes activity. At least this process needs to be running in order for the server to be UP. The number of threads in this process often indicates the number of active Notes Client sessions. 2. Nreplica: Domino is amongst the most popular Replication Servers in the market. Most of the deployments of Notes use Domino heavily for this purpose. Hence, in most of the customer environments, this process uses the most amount of CPU per transaction on a Notes server. This process is responsible for mainly scheduling replication activity. This process will have as many threads as the number of simultaneous replications in process. The Nreplica process is invoked by the Nserver process based on the replication schedule, as configured by the Notes Administrator. 3. Nhttp: This process is the task that accepts http requests. It is invoked only if the http task is initiated in the Notes.ini file. All Web activity is processed via this process. Web security issues, such as user authentication for web users is handled by the Nserver process on behalf of the Nhttp process. The number of threads in this process represents the number of concurrent Web requests this number can be compared with the Domino.threads.active performance counter. 4. Nrouter: This process handles Notes Mail routing. It polls the mail.box file and routes mail to remote servers based on mail configuration records. This process is more active when the mail needs to be routed to remote users. The mail in general is inserted into the appropriate server mail.box file by the Nserver process and is then delivered to the user mailbox by the nrouter process. 5. Ncalconn: This process represents all calendaring activities. It is the task that processing calendaring functions such as meeting scheduling, reminders, etc. The process interacts with the nsched process that acts as the scheduler for various Notes tasks.

3 6. Nupdate: This process is responsible to update databases. All writes to a database are processed by this task. 7. NagntMgr: This process represents the Notes agent that is responsible for interactions between servers and replication tasks. Other processes: Nsched, NSmtp, Npop3 and others, Workload Characterization Most of the Workload Characterization tools in the market have a limitation when it comes to representing workloads with interdependencies. Some of the tools allow one level of nesting to represent dependent transactions BUT being able to represent multiple levels of nesting and also being able to apportion the work done by each of these transactions when there is a transitive relationship would be a challenge. Assuming that one level of dependency can be represented, the best way to represent the inter-dependency of the server process with other tasks is to make processes "nupdate.exe", "nagntmgr.exe" and "nserver.exe" dependent transactions of the other specific tasks. The distribution of work will be dependent on the independent transactions and they should be based on the Notes transaction counts as opposed to business transaction counts. We found that this base set of processes were the same for both versions of Domino discussed in this paper, though some of the algorithms in these processes were optimized in the case of R5. We used the same workload characterization for both versions for our benchmarking effort. 3. Benchmarking Assuming that we have got the workload characterization right for both Domino and Domino 5.0, we went ahead with our benchmarking efforts. The first dilemma we faced with benchmarking was to find a cheap way of running these benchmarks. We collected some hardware from the lab that was not being used and built a set of identical machines to install both the versions of Domino. We needed 4 machines, 2 each for different versions of Domino. One was System Under Test (SUT) and the other one was the destination machine. The destination machine for Domino was 300MHz as opposed to 350MHz for Domino 5.0 and since there was not much load generated on the destination machine, we thought that this difference is negligible [see Table 1]. We will see from some of the graphs in the next section that this difference did not really matter. All the machines were running Windows NT 4.0 SP5. Name Domino Ver Role CPU RAM Disk CAPS-DOMINO SUT PII/ MB 10G IDE CAPS-DOMINO Dest PII/ MB 10G IDE CAPS-R5-1 R5 SUT PII/ MB 10G IDE CAPS-R5-2 R5 Dest PII/ MB 10G IDE DABRISTO 4.6.4, R5 Parent PIII/ MB 9G SCSI SSYED 4.6.4, R5 Child1 PII/ MB 6G IDE KIN , R5 Child2 PII/ MB 8G IDE KIN , R5 Child3 PII/ MB 2x4G IDE Table 1: The hardware acquired to run benchmarks for Domino and 5.0 Now the second dilemma was to find the right benchmarking tool to drive the SUT. Generally there are a bunch of tools in the market that help you generate loads for stress testing. Most of them are expensive.

4 Notesbench, a consortium of hardware vendors and some performance analysis companies, has built a tool that can create loads to stress test certain functions of a Domino Server. These loads are different for Domino and 5.0 because these different versions of Servers demand different clients. We will address this issue in the next few sections. For this effort we used the client/server architecture of Domino. Notes clients were installed on the machines that were identified as the "Notesbench" Children. The "Notesbench" parent where the benchmark definitions are installed talks to the children and the Notes Clients that are installed on these Children load the SUT based on the benchmark definition provided by the parent node. For our hardware setup we used 3 children [see Table 1 and Fig 1]. One could use as many children as you wish and the more children you use the more accurate the benchmarks are to the real world. The 4 Domino server boxes were dedicated to running either Domino or R5. The Parent and Child1-3 machines had 2 different versions of Notes installed (4.6.4 and 5.0 clients) in separate directories, so as to allow easy setup of NotesBench runs against Domino and R5 [Fig 1]. The following was the hardware setup for the benchmarks. Fig 1: The hardware setup for the Notesbench runs For all our benchmarking efforts, we used only two pieces of software - Notesbench to load the SUT and an analysis tool called PATROL Performance Manager from BMC Software to collect performance data, analyze the data and do predictive What-Ifs. 4. Interpretation of the Results Notesbench provides two types of benchmarks, Mail ("Mail") and Mail+Shared Database ("MailDb"). The "Mail" benchmark is the general messaging benchmark that characterizes the work done by the Domino Server when accepting messages from the client and delivering it to the destination server. The MailDb benchmark loads the server with both the messaging activity as well as some shared database activity (document sharing). These benchmarks differ for different versions of Domino. We distinguish between the

5 benchmarks for different version by prefixing R5 for the benchmarks that were specifically run for R5. Some of the results that are shown in the graphs and tables below mention R5 Mail and R5 MailDb indicating that these benchmarks are specifically designed for R5 version. There were a number of benchmarks that were run varying the number of users and changing the hardware configuration but we made sure that these configurations were consistent for both versions. Since the purpose of this benchmarking effort was distinguish between the performance of Domino and 5.0 and come out with some useful software upgrade tips, our benchmarks were focussed towards comparing the CPU, memory and Disk requirements for these 2 versions of Domino. Let us go ahead and discuss and some of the results now. 4.1 Populating the seed data Setting up the environment before the benchmark was run was a time consuming task. The Mailboxes had to be created for a number of users and seed data had to be populated in each of those mailboxes. We ensured that the seed data was identical for all the setups. The creation of mailboxes and the population of the seed data was resource intensive and this can be seen in the "MailInit" benchmark shown in Fig 3 in Section R4.6.4 Mail v/s MailDb Notes provides two types of benchmarks for Domino One is the traditional Mail and other one is the MailDb which characterizes both the messaging and document sharing activity. From the definition, the latter contains more database activity. We first tried to identify the differences between these two benchmarks before we compared the two different versions of the Domino Server. We noticed a higher paging activity on MailDb and also the CPU usage was more variant. Fig 2: R4.6.4 Mail and MailDb benchmarks run on Domino 4.6.4

6 4.3 R5 benchmarks We ran similar benchmarks on R5 to what is described in Section 4.2. We noticed that the paging activity had increased for both the Mail and MailDb benchmarks when compared to R Does this mean that more memory is a requirement for R5 no matter what purpose you are using it for? We also noticed that there wasn t much difference in the paging activity for Mail and MailDb benchmarks for R5. Can this be a result of better memory management algorithms on R5 for document sharing activities? We shall investigate more on this issue in the final version of this paper. We also ran the R4.6.4 Mail benchmark on R5 to show that there is definitely less activity on R5 if the previous version of Notes client is used. The reason for that being there is less functionality in Notes Client for R4.6.4 as compared to the Notes Client for R5. Trying to compare the activity of Domino R5 v/s R4.6.4 by running R4.6.4 benchmarks on both would not be considered right for this reason. The graphs below (see Fig 3) show the CPU and paging activity for the "MailInit" benchmark that characterizes the work done for creation of mailboxes and also population of the seed data along with the results of the benchmarks for R5 Mail, R5 MailDb and R4.6.4 Mail on R5. Fig 3: Benchmarks run on Domino ver 5.0

7 4.4 Summary of the results The table below shows the summary of the benchmarks that were run (see Table 2). The response time measurements that we got from the Notesbench runs for each of the experiments below are deliberately NOT published in this paper. There are some studies posted on the Notesbench Consortium web site that talk about response time measurements for some of the loads on pre-configured boxes from hardware vendors. We found that the CPU, I/O, Paging and Memory measurements were sufficient for our study, where the intent was to understand the performance of the two versions of Domino server from a resource utilization perspective. The use of Predictive What-ifs helped us understand some of the memory requirements for R5 for increasing number of users. Benchmark Domino CPU IO Page Mem Mail 600 users, 256mb memory45% % Maildb 600 users, 256mb memory45% % Benchmark Domino R5 CPU IO Page Mem R5mail 600 users, 256mb memory35% % R5maildb 600 users, 256mb memory40% % Others benchmark runs for Domino R5 CPU IO Page Mem R5mail 600 users, 512mb memory40% % R5mail 900 users, 512mb memory43% % R5maildb 600 users, 512mb memory45% % R5maildb 900 users, 512mb memory45% % Mail 600 users, 256mb memory20% % Maildb N/A Table 2: Summary of the results We found that R5 scaled very well for increasing number of users as opposed to R Also the less variation in the CPU/Transaction for R5 was due to the fact that R5 locks more memory upfront. So, if the memory requirement for R4.6.4 and R5 is compared, the paging activity on R5 is more than on R4.6.4 but the paging on R5 goes down dramatically with addition of memory. Just to show the scaling that has been achieved in R5, an increase in the Paging I/O of 25 pages/sec for an increase in the number of users from 600 to 900 is small. The memory utilization also remained constant at nearly 75% (for experiments with 512MB physical memory) for varying number of users. This kind of stable behavior is more conducive to production environments than bursty behavior. We know that stable behavior generally results in predictable response time and this may be one of the reasons to upgrade to R5. But the general rule of thumb should be to begin with more memory and may be even machines with 64bit addressing so that most of the database if not the entire database can be cached into memory leading to better I/O performance. If a 64bit addressing machine were not available, another tip for the Notes administrators would be to use the fastest possible system disk if the paging file resides on it. The read and the browse and many other document sharing activities revolve around reading the document more than writing to it. So the disk on which the database actually resides is accessed fewer times than the paging disk itself. If the R4.6.4 box that is currently running the production software is NOT geared up well for memory scaling, then it is time to think about using new hardware for R5. The current version of Domino that is shipped is R5 and still there are plenty of users yet to migrate to R5 from R Lotus is announcing Rnext sometime next year. At that point the users who already migrated

8 to R5 from R4.6.4 will have to go through another major software upgrade. The other users will try to migrate to Rnext directly from R Some may wait for some more time before they migrate. We hope run similar benchmarks to come up with tips and recommendations for migration to Rnext, once the GA version of this software is available. 5. Conclusion We have tried to discuss a cost-effective way of benchmarking Domino for software upgrades. Benchmarking before deployment and software upgrades is getting to be less popular due to time and resources that the effort consumes. In this paper, we have shown that we can still try and get an understanding of the performance of the software that you are planning to upgrade to by running fewer benchmarks and then using some software for predictive What-ifs for extrapolation. This methodology of using cheap hardware for benchmarking NOT only cut costs BUT ALSO at the same time helps create a proactive and comprehensive strategy to meet SLA by understanding the performance of the system before software upgrade. A relative analysis between the two software versions has to be performed before tweaking the old strategy to a new one for the upgraded software. Here is a bulleted list of some of the findings of our benchmarks. R5 consumes less CPU but requires more memory than R4.6.4 Mail DB workload was found to cause more paging than Mail workload R5 Mail & Mail DB workloads were more resource intensive But the R5 server performed well with some memory upgrades R5 scaled well for increasing number of users when compared to R4.6.4 These can be regarded as tips and recommendations when upgrading from Domino to 5.0. A similar approach could be adopted when upgrading from Domino 5.0 to 6.0 or even when upgrading from Domino to 6.0. At this point, we cannot provide any tips and recommendations for upgrades to 6.0. There is a possibility that we will provide these tips for upgrades to Domino 6.0 if there is a demand for it. 6. References [NOTES 1] Notesbench Consortium Web Site - [NOTES 2] Notes Administrator Manual [POPE 99] The use of Life Expectancy to Manage Notes Domino Storage, William G. Pope, CMG 1999, Reno, NV. [LARRY 99] Successful multi-tier benchmarking and Load Testing, Larry E. Watson, CMG 1999, Reno, NV. BMC Software, the BMC Software logos and all other BMC Software product or service names are registered trademarks or trademarks of BMC Software, Inc. All other trademarks or registered trademarks belong to their respective companies BMC Software, Inc. All rights reserved.