Exchange 2003 Deployment Considerations for Small and Medium Business Dell White Paper By Sumankumar Singh, Abhijit Chattopadhyay & Bharath Vasudevan Dec 08 2005-1 -
1. EXECUTIVE SUMMARY...3 2. INTRODUCTION...4 3. HARDWARE SELECTION...5 4. EXCHANGE LAYOUT AND TEST METHODOLOGY...6 A. HARDWARE...6 B. TEST TOOLS...7 C. STORAGE CONFIGURATION...8 D. NETWORK CONFIGURATION...10 E. TEST SET UP...11 5. SIZING TEST RESULTS AND ANALYSIS...12 A. PROCESSOR PERFORMANCE:...12 B. MEMORY PERFORMANCE:...14 C. NETWORK PERFORMANCE:...16 D. STORAGE PERFORMANCE:...17 E. OTHER CONSIDERATIONS:...19 6. CONCLUSION...20 7. RESOURCES AND REFERENCES...21-2 -
Section 1 1.Executive Summary Microsoft Exchange is one of the most popular and widely used email applications. This paper examines the performance & scalability for different numbers of users on a Dell PowerEdge 2800 running Microsoft Exchange 2003 Server. The numbers of users have been chosen keeping in mind the requirements of small & medium business customers. Standard tools like LoadSim & JetStress were used to simulate the Exchange environment. An extensive study of the performance data reveals that the discussed Exchange setup can support up to 900 Exchange users, if it is used as a dedicated Exchange server. - 3 -
Section 2 Section 2 2.Introduction Today, Small and Medium Businesses (SMB) rely more on messaging solutions than ever before. Email has become a mission critical application. Whether in the office or on the road, communicating internally or closing business with customers, users depend upon email to get their jobs done. The performance and availability of the messaging infrastructure is very important. Therefore, it is critical to consider hardware sizing and mailbox design before the initial deployment or re-deployment after hardware or software upgrades are performed. Taking these steps will ensure due-diligence in preventing potential data loss during or after the deployment. Choosing the right platform for your Microsoft Exchange deployment can sometimes be complex and confusing. In this paper, we discuss how to arrive at the right platform for deploying Microsoft Exchange 2003 in small environments and why it may be an ideal vehicle for messaging solutions for the Small to Medium Business (SMB) segment. To size for larger Exchange environments refer to the Dell Exchange Advisor tool. For more information about Dell Exchange Advisor please visit www.dell.com/exchange. In this paper we compare and discuss the performance results for different numbers of Exchange mailboxes and make design recommendations to get the best performance out of your messaging solution. Using the results, we have analyzed the potential bottlenecks that may be encountered in an Exchange deployment and tried to answer some of the most frequently asked questions like: How many mailboxes can be hosted on one server? How much memory is needed for the Exchange server? My server processor utilization is very low. Why then is my email access slow? Is my storage performance too slow? - 4 -
Section 3 3. Hardware Selection Dell s product portfolio ranges from the single socket to the quad-socket servers that support single core or dual-core processors with various memory configurations. This section will help navigate through Dell s server offerings and help identify the ideal candidate for deploying Exchange 2003 in business with up to 900 employees. There are different types of load that can be imposed by clients. This workload can consume a combination of memory, processor, I/O and network resources on the server. A brief analysis of Microsoft Exchange shows that typical client workloads tend to stress the memory and I/O subsystems, while exercising the processor and network subsystems to a lesser extent. Single socket servers may not allow future scalability, as they are limited in their expansion capabilities. Microsoft Exchange 2003 is a 32 bit application which can only address 4GB of virtual memory. There are database solutions such as SQL and Oracle that have versions which are compatible with 64-bit Editions of Microsoft Windows operating systems and can utilize much larger amounts of RAM but the current version of Exchange Server 2003 does not support 64-bit versions of Windows Operating Systems. Given the 4GB memory limit, and the lighter processor workload, the processing power of a quadsocket server may not be required for hosting Exchange Server alone. An ideal server candidate needs to support 4GB of memory and have scalable IO options while providing the processor scalability and network bandwidth required to host Microsoft Exchange 2003. Dell answers all of these requirements with dualsocket PowerEdge Servers. Dell s 8 th generation PowerEdge dual-socket servers are available in both tower and rack optimized models. An all-purpose server which meets all of the performance characteristics, while providing the greatest amount of internal storage is the Dell PowerEdge 2800. For better future IO scalability external storage is recommended. However, this paper focuses on internal server storage only, which is typical in smaller Exchange environments. - 5 -
Section 4 4.Exchange Layout and Test Methodology Sizing and performance tests were done at Dell Enterprise Solutions Engineering Labs. The purpose of the test were to determine the Processor, Memory, Network and Disk Utilization while hosting between 600-1050 Exchange mailboxes. In the following section, the hardware and software components of the test bed are listed and the test tools and the setup procedures are explained. A. Hardware Microsoft Exchange Server: Server Dell PowerEdge 2800 CPU 2 x 2.8 GHz Intel Xeon Processor with 1 MB L2 Cache Memory NIC RAID Controller Internal Disks Operating System Messaging App 4 GB DDR2 ECC 2 dual port Intel 8254NXX Gigabit Adapter PERC 4e/Di 10x73 GB 15K RPM SCSI Microsoft Windows Server 2003 EE SP1 Microsoft Exchange 2003 EE SP1 Table 1. Microsoft Exchange Server Configuration Other Hardware: Since Microsoft Exchange 2003 is tightly coupled with Active Directory and heavily utilizes the domain controller for authentication purposes, it is important to have a reliable network interface to this server. The domain controller / global directory server must also be adequately resourced such that the increased workload resulting from the Exchange deployment will not create a bottleneck. PE 1650: (Domain Controller / Global Directory Server) Processor : Intel Pentium 3 Memory : 1 GB DDR Hard drives: 1 x 73 GB 15K RPM SCSI Network: 2 x Gigabit Ethernet OS: Microsoft Windows Server 2003 EE SP1-6 -
The PowerEdge 750 is used as the client machine. The test tools used generate the load of multiple clients using a single machine. Using a bank of clients, we are able to simulate the load generated by 600 1050 clients. PE 750: (Exchange Clients) Processor : Intel Pentium 4 Memory : 512 MB PC 133-SDRAM Hard drives: 2 x 18 GB 15K RPM SCSI OS: Microsoft Windows XP Professional SP2 All network components used are Gigabit Ethernet. Using an Ethernet or Fast Ethernet infrastructure may result in different outcomes. Network Switch Dell PowerConnect 5224 The hardware used for Domain Controller and simulated clients are for testing only and is not a recommendation of any sort. B. Test Tools Microsoft Exchange 2003 Load Simulator (LoadSim) and Exchange 2003 JetStress were the two tools used to simulate load on the Exchange server. Through the use of the client machines, LoadSim was used to simulate the workload load of MAPI clients using the MAPI Messaging Benchmark 3 (MMB3). It was used to provide statistics for how well an Exchange server would perform when dealing with a given number of simulated users. The MMB3 benchmark is modeled after a typical corporate e-mail usage profile. The higher the MMB3 score, the more theoretical users the server can support. Bear in mind, however, the MMB3 score does not correlate directly with how many real world users the server can support. Since it is indeed a benchmark, the MMB3 benchmark must be consistent, repeatable, and possess the ability to scale to produce a standard metric across different servers and configurations. Given that each real world Exchange Server environment may have different requirements, workload sizes, and dynamic peaks and valleys in messaging traffic, the MMB3 score serves only as a relative indication of platform comparison across different configurations of servers and storage. JetStress, on the other hand, helps verify disk performance by simulating Exchange disk Input/Output (I/O) workload. JetStress simulates the Exchange database and Transaction log file loads produced by a specific number of simulated users without the use of client machines. Since Exchange is an I/O intensive application, it is important to verify that the disk subsystem is adequately sized for your user count and profile. - 7 -
C. Storage Configuration The PowerEdge 2800 has a primary backplane capable of holding eight SCSI hard drives. This primary backplane can be configured as a single SCSI bus with eight drives, or split with four drives each on two independent SCSI busses. With the additional of an optional media bay, the overall internal drive capacity grows to ten. The possible internal hard drive configurations for the PE2800 are: Primary Backplane only: 1x8 for a total of 8 SCSI drives Primary Split Backplane only: 1x4 and 1x4 for a total of 8 SCSI drives Primary Backplane and Media Bay: 1x8 + 1x2 for a total of 10 SCSI drives The RAID controller used in all of our configurations is the internal PERC 4e/Di. This is a two channel, PCI-express RAID controller operating at U320 speeds. RAID Controller PERC4e/Di LUNs OS: One 2-disk RAID 1 Logs: One 2-disk RAID 1 Mailboxes: One 6-disk RAID 1+0 Table 2. PowerEdge 2800 Internal Storage Configuration Some guidelines to improve storage performance: Split transaction logs and database onto separate disk spindles with fault tolerant partitions. Transaction logs spindles should be dedicated for logs only. Allocate one RAID array for each storage group. Allocate sufficient disk spindles for mailbox server. Consider using faster speed disks. Based on our Exchange LUN requirements (10 internal SCSI drives), the PowerEdge 2800 internal hard drive configuration that best suits our needs is the Primary Backplane and Media Bay. For our tests a total of three Virtual Disks / RAID groups were created: OS Virtual Disk: Created from physical disks 0 and 1, configured as RAID 1. This was used for the Operating System. RAID 1 is used for fault tolerance. Logs Virtual Disk: Created from Physical disks 2 and 3, also configured as RAID 1. This was used for Exchange Transaction Logs. Mailbox Virtual Disk: Created from Physical disks 4, 5, 6, 7, 8, and 9, configured as RAID 1 + 0. This was used for Exchange Information stores. This is represented in the figure below: - 8 -
Figure 1. PowerEdge 2800 Internal Storage Configuration for Exchange - 9 -
D. Network Configuration The test bed consisted of the Dell PowerEdge 2800 Exchange server connected to the Domain Controller (PE 1650) and Exchange clients (PE 750) using a Dell PowerConnect Switch 5224. The network interfaces in all of the components (clients, servers, and domain controller) were Gigabit Ethernet. 172.31.10.11 172.31.10.12 172.31.10.13 172.31.10.14 Exchange Clients (PE 750) 172.31.10.10 PC 5224 Domain Controller (PE 1650) 172.31.10.15 Exchange Server (PE 2800) Figure 2. Network Configuration - 10 -
E. Test Set up Creating the Test Bed 1. Domain Controller a. Microsoft Windows 2003 Enterprise Edition SP1 was installed on the PE1650 b. This machine was made a domain controller c. DNS, DHCP, WINS, and other Active Directory services were configured 2. Clients a. Microsoft Windows XP Professional SP2 was installed on all of the PE750s b. Each of these machines were joined to the domain 3. Exchange Server a. Microsoft Windows 2003 Enterprise Edition SP1 was installed on the PowerEdge2800 b. This machine was joined to the domain c. Microsoft Exchange 2003 was then installed and configured on the PowerEdge 2800 d. Following Microsoft Best Practices, the Boot.ini on PowerEdge 2800 was modified to include /3GB, /userva=3030 and /pae flags. e. The machine was then upgraded to Microsoft Exchange 2003 SP 1. Installing LoadSim LoadSim was installed and configured on each of the four clients (PE 750) to simulate load based on MMB3 clients. Each run consisted of Active Directory User Creation and Mailbox Database initialization followed by 8 hrs of simulated mail stress. Multiple performance counters including Processor utilization, memory utilization, network utilization and various disk measurements were captured during the run. Installing JetStress Exchange 2003 JetStress was installed on the PE 2800. The average corporate user has a Mailbox size of 50 MB and generated load which correlates to 0.67 IO s per second (IOPS). These were the settings used in each run of JetStress. Each test run consisted of Exchange mailbox initialization followed by 8 hrs of mail stress. During the test runs, performance counters like Average Disk Queue length, Transfers / sec, Processor utilization etc were monitored and recorded for analysis. - 11 -
Section 5 5.Sizing Test Results and Analysis In this section we discuss the results obtained from tests described in section 3. Some analysis of the obtained results is also presented. Of the various factors impacting the performance of an exchange server, we have studied the effect of increasing the number of mailboxes on Processor, Memory, Network utilization and disk response times. Using these results, we have analyzed the potential bottlenecks that may be encountered in an Exchange deployment. The tests were conducted for simulated Exchange users in the range of 600 1050. Results from tests are presented and analysis is provided to help IT Managers and Administrators make informed decisions about their Exchange infrastructure design and implementation. A. Processor Performance: To observe the effect of increasing the number of mailboxes on Processor Utilization LoadSim was installed on each of the clients (PowerEdge 750) to simulate Exchange workload. The test was run several times for different user counts. The results obtained for Processor measurement counters are shown in the graphs below: System Average Processor Utilization Utilization % 10 9 8 7 6 5 4 3 2 1 0 600 MMB3 750 MMB3 900 MMB3 1050 MMB3 System Processor Time (%) Processor Privileged time (%) Processor user time (%) Exchange Users Figure 3. System Average Processor Utilization - 12 -
System Maximum Processor Utilization Utilization (%) 30 25 20 15 10 5 System Processor Time (%) Processor Privileged time (%) Processor user time (%) 0 600 MMB3 750 MMB3 900 MMB3 1050 MMB3 Exchange Users Figure 4. System Maximum Processor Utilization Observations: Average Processor utilization ranges from 5 to 10% for 600 1050 users Maximum Processor utilization ranges from 17 to 27 % for 600 1050 users Processor Privileged time and User time follow the same trend as Processor Utilization Analysis: From figures 3 and 4, we notice that the Average Processor Utilization only reaches about 10% for 1050 MMB3 users. Typically, if the processor utilization is consistently greater than 75% it may be considered a bottleneck. In this case, an average processor utilization of 10% shows that the PowerEdge 2800 has ample computing power available to handle 1050 or more mailboxes. We do not expect Processing power to become a bottle neck for this size of Exchange deployments. The spare computing power available may be utilized by other applications such as Anti-virus, Anti-Spam, backup etc. - 13 -
B. Memory Performance: The tests used to study memory utilization were similar to the Processor Utilization tests. Memory Utilization 14 12 10 8 6 Available GB Cache Faults/sec 4 2 0 600 Users 750 Users 900 Users 1050 Users Exchange MMB3 Users Figure 5. Memory Utilization Observations: Available memory slightly reduces with increasing mailboxes Cache Faults/sec slightly increases with increasing mailboxes Analysis: With the rapid increase in the speeds of processor, memory, system bus etc, disk has become the slowest component and this makes disk seek a very expensive operation as far as performance is concerned. As Exchange 2003 is an I/O and memory intensive application, try to reduce the cache faults as much as possible to minimize data seek from disks, in order to optimize performance. But, being a 32-bit application 4 GB of RAM is the maximum amount of memory that an Exchange Server computer can efficiently use. We recommend taking full advantage of that by installing the maximum 4 GB of physical RAM. Exchange Memory Optimization: By default, 32-bit Windows operating system allocates 2 GB of virtual address space for Kernel mode (Operating system) and 2 GB for User mode (applications). But 2 GB of virtual address space may not be enough for memory intensive applications like Exchange. To solve this problem and make more memory space available for user mode, set the /3GB switch in the boot.ini file. Also, set the /USERVA=3030 parameter in the Boot.ini file to allocate (3072 3030=42) 42MB of memory to the kernel for dynamically allocating memory back to the kernel mode in case of need. - 14 -
The /3GB switch makes 3 GB of memory space available for User mode and therefore more memory is available for Exchange services. But having said that, it is important to understand that all client activity generated by a MAPI client causes updates to the Exchange Jet database which produces random I/O activity. And, installing the recommended 4GB of RAM does not mean that will solve all memory issues and that most I/O s will be cached. This may only ensure that the Exchange components do not starve of memory. For optimized performance, allocate appropriate value of virtual memory to different Exchange components as well as the OS. If the server is not a dedicated Exchange server and hosts other applications like antivirus, anti spam, backup etc. additional RAM may be required.* From Figure 5, we notice that increasing the number of users does not drastically increase the cache faults/sec; neither does it reduce the available memory. These indicate that the Exchange components have sufficient memory and the deployment may be in good health. *Microsoft Windows 2000 AS and Windows Server 2003 Enterprise Edition (32 bit) can support more than 4GB of RAM by using Physical Address Extension. For more information refer to: http://support.microsoft.com/?kbid=283037 CPU and memory scalability for Exchange Server 2003 and for Exchange 2000 Server Physical Address Extension - PAE Memory and Windows - 15 -
C. Network Performance: Network performance for varying number of users is shown in Figure 6: Network Utilization 350 300 250 KB 200 150 100 50 KBytes Sent/sec KBytes Received/sec 0 600 Users 750 Users 900 Users 1050 Users Exchange MMB3 Users Figure 6. Network Utilization Observations: Increase in the number of mailboxes increases network traffic Both the incoming and outgoing network traffic increases proportionally with the increase in number of mailboxes Analysis: From the magnitude of network traffic shown in the graph above, it appears that a full duplex 100 Mbps LAN connection may be sufficient. But note that this does not include backup or any other application traffic. If you plan to use network for backup and restore or if you have other applications generating high network traffic, consider using Gigabit Ethernet. Exchange uses Active Directory as its directory service. Every Exchange mailbox has to be an Active Directory user. Exchange servers and messaging clients access Active Directory in situations such as when logging on to the network and connecting to a mailbox, or accessing server-based address lists. These activities create high network traffic between the servers. It is therefore important to have enough network bandwidth available between Exchange and the Active Directory server. - 16 -
D. Storage Performance: To observe the effect of increasing the number of mailboxes on Storage Performance JetStress was used to simulate Exchange I/O workload. The test was run several times for different mailbox counts with 50 MB mailbox size and 0.67 IOPS per user. The results obtained are shown in the graph below: Storage Performance Disk Queue Length Read Latency (ms) Disk Transfers / sec (100's) 20 18 16 14 12 10 8 6 4 2 0 600 Mailboxes 750 Mailboxes 900 Mailboxes 1050 Mailboxes Figure 7. Storage Performance Observations: Read latency, disk queue length and disk transfers per sec rise steeply for greater than 900 mailboxes Read latency increase is proportional to increase in disk queue length For 900 mailboxes and less, the read latency and disk queue length almost remain constant Analysis: Exchange is an I/O intensive application and all client activity causes updates to the Exchange database which produces I/O operations to disk. Your disk subsystem should be capable to meet these demands. It is therefore important to size your disks for performance and not just for capacity of the mailboxes. Causes of high disk latency: For good performance, we recommend that your average read or write latency should not exceed 20 ms and the average disk queue length per spindle should be less than 2. A disk queue length of less than 2 per spindle suggests that while one I/O operation is being processed by the disk another I/O is waiting in the queue. A value greater than 2 per spindle suggests that I/O requests are coming in at a higher rate than they can be - 17 -
processed by the disk. This may ultimately result in higher disk latencies and therefore is not recommended. It is advisable to invest in faster rather than higher capacity drives. Faster drives can process more I/O s per second with low latency. If you are considering using SCSI drives, we recommend using 15K RPM U320 drives. To further improve the performance of your disks consider using DiskPar to verify that your disk tracks are sectoraligned. In Figure 7, we notice that the disk queue length, read latency and disk transfers per sec remain relatively consistent up to 900 mailboxes, even though the workload increases. However, these values increase sharply for 1050 mailboxes. For e.g. the disk queue length for 1050 mailboxes increases to 10. As the database RAID group in this case comprises of 6 spindles, the disk queue length per spindle is 1.67, (10/6 =1.67) which is close to the maximum recommended limit of 2. Also, the read latency increases to 17 ms, which is approaching the limit of 20 ms. We notice that the disk performance parameters are approaching their maximum recommended limit for 1050 mailboxes. To allow for spikes and periods of heavy load, we recommend no more than 900 mailboxes on the Exchange server. To accommodate for future growth, consider further reducing the number of hosted mailboxes. For e.g. to accommodate for 25% future growth, have no more than (900 25%=675) 675 mailboxes on the Exchange server.* From the above discussion, we see that available disk bandwidth may become a potential bottleneck with increased number of mailboxes. To overcome this limitation and host larger numbers of mailboxes consider moving to external storage. There are several external storage solutions available including Direct-Attached Storage (DAS) or Storage Area Networks (SAN). *The above storage recommendations are for Exchange users, each generating 0.67 IOPS. Your performance may vary depending on the nature of your Exchange users. For more information about Exchange refer to the following article: http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/optimizestorage.ms px - 18 -
E. Other Considerations: High Availability Clustering: The basic goal of High availability (HA) clustering is to make sure that the physical server hosting an application is not a single point-of-failure by providing the ability for that application to be restarted on one of multiple servers in the cluster. If the server running the application fails, another designated server takes over the responsibility of running that application. Dell s high availability (HA) cluster solutions build upon the Microsoft Cluster Server (MSCS), are designed and tested to help make sure that no single point of failure (SPOF) exists. High availability clustering using MSCS requires shared storage, as every node in the cluster needs access to the Exchange data. Therefore, to implement this, consider moving to external storage. There are several external storage solutions available including Direct- Attached Storage (DAS) or Storage Area Networks (SAN). For more information on Dell supported Exchange HA Clusters refer to: Exchange 2003 Scale-Out Performance on a Dell PowerEdge High Availability Cluster Exchange Front End Servers: Microsoft Exchange Server 2003 supports an Exchange architecture consisting of front-end and back-end servers. The front-end server accepts requests from clients and proxies them to the appropriate back-end server for processing. This architecture is usually recommended when there are multiple back-end servers in the Exchange environment. However you may use it in single back-end server environments also, as it provides the following benefits: Single Namespace: Single namespace for all users Same URL for OWA, POP3,IMAP4 clients Server name does not change even if mailbox moved or new servers added Offload Processing Front-end servers can manage all encryption and decryption processing Strengthen Security Front-end server is single point of access either on or behind a firewall Additional layer of security for mailboxes Need not open RPC ports from Perimeter network to internal network Scalability Ability to increase or decrease the number of front-end or back-end servers without disrupting users - 19 -
Section 6 6.Conclusion Small and Medium Business can range in size from 50-2000 employees. From Dell s product portfolio, a dual-socket PowerEdge 2800 delivers a self-contained means for deploying messaging solutions. The result of our validation shows that if used as a dedicated Exchange server, the internal drives of a PowerEdge 2800 can host up to 900 mailboxes. Often, people erroneously make the claim that adding processing power is the answer for all load-related problems. Bear in mind that the PowerEdge 2800 has enough processing power, memory and network capacity to support much greater than 900 mailboxes, but the available disk bandwidth using the on-board hard drives becomes a limiting scaling factor. To overcome this limitation and provide the ability to host larger numbers of mailboxes on a single server, consider moving to external SCSI storage, or hosting the Exchange mailboxes on a Fibre-Channel SAN. The guidelines in this white-paper allow you to identify the bottlenecks in your deployments and take the appropriate actions to correct them. - 20 -
Section 7 7.Resources and References For More information please refer to: Dell: http://www.dell.com/exchange http://www1.us.dell.com/content/topics/global.aspx/solutions/en/exchange_solution?c=us& cs=555&l=en&s=biz&~tab=3 http://www.dell.com/downloads/global/solutions/e2k3_3_1_cluster_fe400_perfscale.doc Microsoft: http://www.microsoft.com/technet/prodtechnol/exchange/2003/library/optimizestorage.ms px http://support.microsoft.com/?kbid=283037 http://support.microsoft.com/default.aspx?scid=kb;en-us;827281 http://www.microsoft.com/downloads/details.aspx?familyid=e64666fc-42b7-48a1-ab85-3c8327d77b70&displaylang=en THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Dell Inc. 2005. All rights reserved. Dell, PowerEdge and the Dell logo are trademarks of Dell Inc. Other trademarks and trade names are the property of their respective owners and Dell disclaims proprietary interest in the marks and names of others. - 21 -