Technical white paper HP recommended configuration for Microsoft Exchange Server 2010 and HP ProLiant SL4540 Gen8 Servers (3 node) Building blocks for 1500 mailboxes with 3-copy high availability design Table of contents Executive summary 2 Introduction 2 HP SL4500 Series 3 Exchange Server 2010 4 Solution design criteria 4 Database availability group (DAG) design 5 HP Sizer for Exchange 6 Recommended configurations 7 1500 mailbox building block 7 1600 mailbox building block using JBOD 12 Action items 14 For more information 14
Executive summary This recommended configuration is for Microsoft Exchange Server 2010 and HP ProLiant SL4540 Gen8 Servers. It is designed around a building block approach with a multi-copy high availability design using a database availability group (DAG) and can be used to scale to large numbers of users. The DAG provides high availability with multiple copies in a primary datacenter location, and an optional secondary datacenter or disaster recovery (DR) location. This design is based on a specific number of servers per DAG which determines the number of users that can be hosted on a single server in a failover scenario. The number of servers per DAG is a flexible design decision that customers can make, and the considerations are detailed in this document. The HP SL4500 Series is an excellent choice for running Microsoft Exchange Server 2010 and includes both the HP ProLiant SL4540 Gen8 Server and the HP ProLiant SL4545 G7 Server. The HP SL4500 Series is available in one of three configurations (each of these configurations is available in a 4.3U form factor chassis): 1 node configuration one compute node with up to two small form factor (SFF) 2.5" hard disk drives (HDD) and up to sixty (60) large form factor (LFF) 3.5" hard disk drives (HDD). 2 node configuration two compute nodes, each with up to two small form factor (SFF) 2.5" hard disk drives (HDD) and up to twenty-five (25) large form factor (LFF) 3.5" hard disk drives (HDD). 3 node configuration three compute nodes, each with up to two small form factor (SFF) 2.5" hard disk drives (HDD) and up to fifteen (15) large form factor (LFF) 3.5" hard disk drives (HDD). This recommended configuration document will focus on the 3 node configuration of HP ProLiant SL4540 Gen8. It consists of up to three compute nodes and a total of forty-five (45) large form factor (LFF) 3.5" hard disk drives (HDD) in the chassis. The HP ProLiant SL4540 is a dual socket Gen8 server, with a choice of five different Intel Xeon processors, up to 192GB of memory and one PCIe slot for expansion per node. Every compute node also has its own dedicated networking ports. To help customers decide between each of these configurations, a separate document for each configuration covers the details, with recommendations of user counts and mailbox sizing. The decision point may vary between different customers and environments, but in general the 1 node configuration (with up to 60 LFF 3.5 hard drives) can provide larger numbers of users per server with larger mailboxes since it has more disks available. Another decision that will be discussed in this document is how the number of servers per DAG impacts the number of users per server. This recommended configuration is designed around a building block approach with a multi-copy high availability design using a database availability group (DAG). The number of servers per DAG is a flexible design decision that customers can make (up to sixteen servers per DAG based on an Exchange Server limit). This recommended configuration uses a total of five multi-role servers, with three in the primary location and two in the secondary location. The primary datacenter is designed for a normal load of 1500 users per server under normal operations and 2250 users per server under failover of one server per DAG. The secondary datacenter is designed for 2250 users per server under failover. Complete server specifications are given in this document. Also shown is an alternate design using disks without RAID1 protection, with two Exchange copies in both the primary datacenter and the secondary datacenter, which would make it possible to use identical server configurations (CPU, memory and database layout). Since the database volumes are not protected by RAID, a single disk failure will require failover to the second Exchange copy. This design supports slightly more users and larger mailboxes due to this different architecture, but does require an additional mailbox server per DAG in the secondary or DR location. Target audience: The information contained in this white paper is intended for IT decision makers, IT architects and project managers involved in planning and deploying Microsoft Exchange Server solutions. For more information on Exchange terminology and best practices go to hp.com/solutions/activeanswers/exchange. Introduction This recommended configuration is for Microsoft Exchange Server 2010 and is designed to scale to large numbers of users. It uses the HP ProLiant SL4540 Gen8 Server as the building block for a multi-role server combining the mailbox role with the hub transport and client access server (CAS) roles. Each server supports a specified number of users based on a specific send/receive profile, with very large mailboxes, as defined in this document. This configuration uses a database availability group (DAG) for high availability, with multiple copies of the Exchange databases. There are two database copies in the primary location and one in the secondary location (for a total of 3 copies) for site failover protection. An alternate design using two copies without RAID1 in the secondary location is also presented. 2
The DAG consists of servers in the primary datacenter or location, and in a secondary datacenter or disaster recovery location. The servers are properly sized for normal operations and failover to either another server in the primary location (first option) or to the secondary location. The second database copy is within the primary datacenter or location, and allows the failover or loss of that single database or the entire server. The third database copy is within the secondary datacenter or disaster recovery location, and allows the failover or loss of the primary site. Note: Configuration of the secondary datacenter or disaster recovery location and managing name resolution is a complex topic beyond the scope of this document. The reader is advised to follow the guidance Understanding High Availability and Site Resilience at http://technet.microsoft.com/enus/library/dd638137.aspx. HP SL4500 Series The HP SL4500 Series is an excellent choice for running Microsoft Exchange Server 2010 and includes both the HP ProLiant SL4540 Gen8 Server and the HP ProLiant SL4545 G7 Server. The HP SL4500 Series is available in one of three configurations (each of these configurations is available in a 4.3U form factor chassis): 1 node configuration one compute node with up to two small form factor (SFF) 2.5" hard disk drives (HDD) and up to sixty (60) large form factor (LFF) 3.5" hard disk drives (HDD). 2 node configuration two compute nodes, each with up to two small form factor (SFF) 2.5" hard disk drives (HDD) and up to twenty-five (25) large form factor (LFF) 3.5" hard disk drives (HDD). 3 node configuration three compute nodes, each with up to two small form factor (SFF) 2.5" hard disk drives (HDD) and up to fifteen (15) large form factor (LFF) 3.5" hard disk drives (HDD). This recommended configuration document will focus on the 3 node configuration of HP ProLiant SL4540 Gen8. It consists of up to three compute nodes and a total of forty-five (45) large form factor (LFF) 3.5" hard disk drives (HDD) in the chassis. Each server has exclusive access to fifteen (15) large form factor (LFF) 3.5" hard disk drives (HDD) in the storage section. The HP ProLiant SL4540 is a dual socket Gen8 server, with a choice of five different Intel Xeon processors, up to 192GB of memory and one PCIe slot for expansion per node. Every compute node also has its own dedicated networking ports. The HP SL4500 can be chosen in a configuration that best matches a customer s Exchange design, thus eliminating the need to purchase storage enclosures that may not fit the right building block for specific user counts and mailbox sizes. In addition, the HP SL4500 can provide a very dense server and storage infrastructure, thanks to the innovative design. The net result is a reduction in rack space, fewer cables, and savings in overall infrastructure costs. Comparisons with Exchange solutions from other vendors are very favorable, with the HP SL4500 typically cutting required rack space in half. Each SL4500 Gen8 enclosure takes only 4.3U of rack space. The HP ProLiant SL4540 Gen8 server is shown below. Figure 1. The HP ProLiant SL4540 Gen8 server (3 node configuration) 3
To help customers decide between each of these configurations, a separate document for each configuration covers the details, with recommendations of user counts and mailbox sizing. The decision on which configuration to use will vary between different customers and environments, but in general the 1 node configuration of the SL4540 Gen8 server can provide larger numbers of users per server with larger mailboxes since it has more disks available. The other configurations (2 node and 3 node) can support comparable numbers of users per server but with smaller mailboxes since there are fewer disks per server. All of the configurations can be designed and deployed so that Exchange database copies are paired across enclosures, meaning that failure of an entire SL4500 would minimize the impact to availability. Another decision point that will be discussed later is how the number of servers per DAG impacts the number of users per server. The HP ProLiant SL4540 Gen8 Server offers the advancements and benefits of the Gen8 platform: With HP SmartMemory, memory performance improvements and enhanced functionality listed for Gen8 in the HP SmartMemory QuickSpecs at: http://h18000.www1.hp.com/products/quickspecs/14225_div/14225_div.html. HP Smart Array RAID controllers. An HP Dynamic Smart Array B120i SATA RAID Controller is located on the server node and controls the two small form factor (SFF/2.5") hard drives, and an HP Smart Array P420i controller is located in the I/O Module, at the back of the SL4540 enclosure and controls the large form factor (LFF/ 3.5") hard drives in the chassis. HP Agentless Management: With HP ilo Management Engine in every HP ProLiant Gen8 server, the base hardware monitoring and alerting capability is built into the system (running on the HP ilo chipset) and starts working the moment that a power cord and an Ethernet cable is connected to the server. HP Intelligent Provisioning: includes everything needed for system setup and deployment. Firmware, drivers, and tools to get systems online are embedded on a NAND flash chip on the server motherboard (thus SmartStart CDs or firmware DVDs are not necessarily needed). For more information see http://h17007.www1.hp.com/us/en/whatsnew/proliantgen8/index.aspx. Exchange Server 2010 These recommended configurations leverage the Exchange 2010 high availability features and apply best practices to the hardware design, architecture and configuration. The recommended configuration has been designed around an Exchange 2010 Database Availability Group (DAG) with three copies of each database. For further high availability Exchange 2010 uses a client access server (CAS) array, and an external hardware or software load balancer must be utilized. To ensure fault tolerance and to distribute the client requests between CAS servers in the CAS array, a hardware or software based load balancer is required. Microsoft offers a qualification program for load balancers supporting Exchange. The following link lists the qualified load balancer solutions: http://technet.microsoft.com/enus/exchange/gg176682.aspx. Solution design criteria This solution is designed based on the following criteria or assumptions: Each user has a send/receive profile of 200 messages per day, with an estimated IOPS requirement of 0.2 IOPS (testing should include a 20% safety factor over the base IOPS). Each user has a calculated maximum of primary mailbox space calculated using 2TB drives When 1TB, 2TB, or 3TB MDL SAS disks are used, the average mailbox capacity can be adjusted proportionally Mailbox space can be allocated between personal archive and primary mailbox. The difference and managing by policy is explained at: http://technet.microsoft.com/en-us/library/dd979795.aspx 4
Each Database Availability Group has three copies of each database Two copies in primary site One copy in disaster recovery site Copies are distributed according to the Microsoft Exchange 2010 Mailbox Server Role Requirements Calculator (v19.9) so that failure of one server evenly distributes the load within the DAG. o The Microsoft Exchange 2010 Mailbox Server Role Requirements Calculator (v19.9) is available at: http://gallery.technet.microsoft.com/office/exchange-2010-mailbox-server-role-. Servers are sized for multi-role, which combines the mailbox role (MBX), hub transport and client access server (CAS) role The Exchange 2010 Mailbox Server Role Requirements Calculator will alert the user if estimated CPU consumption exceeds 40% for mailbox role functionality. This is a standard threshold for the multi-role server. The configuration in this document is designed within this threshold. The hub transport role uses disk space for the transport queue and log files. These files will be placed by default, during Exchange setup, on the drives located in the server. It is up to the user if they wish to partition the boot drives so that there is no risk of hub queues filling up the operating system drive. Roles may be distributed, using other platforms such as HP ProLiant DL380e, DL380p or similar servers. The HP Sizer allows the end-user to select multi-role or distributed roles. See http://technet.microsoft.com/enus/library/ee712771.aspx for more information on role sizing. For fault tolerance and load balancing with the CAS role, a separate software or hardware load balancer must be used. This is beyond the scope of this paper but more info can be found at: http://technet.microsoft.com/enus/library/dd638137.aspx. Database availability group (DAG) design This recommended configuration is designed around a building block approach with a multi-copy high availability design using a database availability group (DAG). The number of servers per DAG is a flexible design decision that customers can make (up to sixteen servers per DAG based on an Exchange Server limit). Sizing an Exchange Server in the past was an exercise in determining how many users a specific hardware configuration could support. Now, failover must be taken into account and servers must be sized for the worst-case scenario or what is an acceptable failover scenario. With fewer servers per DAG the failover of a single server results in the remaining servers mounting more databases and thus higher user load, so adding servers to the DAG can net additional users per server in the basic building block size. To illustrate, two servers in a DAG can run at approximately half the maximum load, but three servers can run closer to twothirds the maximum load if the failure of one server then spreads the remaining user load to both of the other servers in the DAG. All of the configurations can be designed and deployed so that Exchange database copies are paired across enclosures, meaning that failure of an entire SL4500 would minimize the impact to availability. Rather than pair database copies between two servers in the same enclosure, resilience can be increased by pairing across enclosures. However, note that the loss of any two servers in the primary datacenter would not provide ample database copies to keep all databases online. For example, only 20 out of 24 database copies may be available in the primary but the remaining databases could be mounted in the secondary datacenter. This can be demonstrated using the Distribution tab of the Microsoft Exchange 2010 Mailbox Server Role Requirements Calculator (v19.9). 5
HP Sizer for Exchange The HP Sizer for Exchange is a, downloadable application allowing the end-user to customize input and receive an exact specification for servers and storage for Exchange. Additionally, HP s Exchange Sizer can be used to explore custom solutions or to make slight changes to this set of recommended configurations. The recommended configurations presented here can be customized using the HP Sizer for Exchange when the HP SL4500 Series is added to the Sizer in an upcoming release. The HP Sizer for Exchange can be found at hp.com/solutions/microsoft/exchange2010/sizer. The HP Sizer for Exchange enables generating Bills of Materials for various Exchange configurations, allowing the enduser to customize for their Exchange deployment. It takes into account many factors such as email client usage and mailbox size. New to version 2.11 is the ability to read in the Microsoft Exchange Mailbox Server Role Requirements Calculator (spreadsheet) as input, making it even easier to use the HP Sizer. Figure 2. HP Sizer for Exchange Server 6
Recommended configurations 1500 mailbox building block The configuration below shows a design using both RAID1 protection and multiple Exchange copies. Solution summary The following table provides an overview of the HP recommended configuration for Microsoft Exchange Server 2010 and HP ProLiant SL4540 Gen8 Server for the 1500 mailbox building block. Configuration Number of users (mailboxes) per server Normal operations Failover of one server Disaster recovery (secondary) site Average user send/receive profile (and IOPS estimate) Maximum mailbox size Database Availability Group (DAG) configuration Total servers Servers in primary / secondary locations Copies in primary / secondary locations Total users per DAG Detail 1500 2250 2250 200 messages per day (0.2 IOPS) Up to 2GB 5 server DAG 3 primary / 2 secondary 2 copies primary / 1 secondary 4,500 users per DAG Database volumes 7 RAID1 pairs per server 1 spare disk or Recovery LUN per server The figure below shows the details of the Database Availability Group (DAG) and mailbox server configuration. Note that the secondary locations only needs two (2) servers and thus can use either the 3 node configuration, which would provide an extra server for other functions or the 2 node configuration, which would provide additional storage. Figure 3. Details of Database Availability Group (DAG) and mailbox server configuration 7
Solution details In order to accommodate large numbers of users, multiple Database Availability Groups can be used. Each DAG would consist of: 5 multi-role servers, 3 in primary and 2 in DR/secondary. Primary datacenter is designed for normal load of 1500 users per server under normal operations Primary datacenter is designed for 2250 per server under failover (1 server per DAG) Secondary datacenter is designed with two-thirds of the servers, thus 2250 users per server under failover Each server hosts 7 databases (one volume per each, hosted on a RAID1 pair) Each mailbox database and its logs are hosted on a RAID1 pair of midline (MDL) SAS disks Each RAID1 pair of MDL SAS disks supports 450 mailboxes based on both capacity and IOPS Each mailbox database is sized for up to 1241 GB Server specifications are as follows: Primary datacenter: Dual CPU Intel Xeon E5-2430L (6 core) which provides the estimated megacycles and CPU Utilization per server under normal operating scenarios shown below. Secondary datacenter: Dual CPU E5-2430L (6 core) which provides the estimated megacycles and CPU Utilization per server under the complete site or datacenter failover operating scenarios shown below. Server Configuration Primary Datacenter Server (Single Failure) Secondary Datacenter DR Server Recommended RAM Configuration 48 GB 48 GB Server Total Available Adjusted Megacycles 67549 67549 Mailbox Role CPU Megacycle Requirements 11340 11340 Mailbox Role CPU Utilization 17% 17% The table below shows the disk volume requirements: Volume Configuration Recommended Number of Databases / Volume 1 Recommended Number of Database+Log Volumes / Server 7 Number of Restore Volume / Server 1 Total Recommended Exchange Volumes / Server 8 8
Networking At least one I/O module is configured in the HP ProLiant SL4540 chassis: the HP ProLiant SL454x 1G FIO Module Kit supports two (2) embedded 1Gb NICs. The HP SL454x 10G FIO Input Output Module Kit supports two (2) embedded 1Gb NICs and two (2) embedded 10Gb NICs. The 10Gb I/O module is specified as an option in the bill of materials shown later. Networking configuration for an Exchange mailbox role is typically: MAPI or client network Replication network (Optional) management network (Optional) backup network Figure 4. HP ProLiant SL4540 Gen8 Server with rear view of networking ports 9
Bill of materials The following bill of materials lists the major server and storage hardware components needed for the recommended configurations, however this is not an exhaustive listing of all the necessary components needed to build the complete solutions. For complete configuration details, please contact your HP Reseller or HP Sales Representative. The first bill of materials is for the primary datacenter mailbox server and the second is for the secondary datacenter mailbox server. Table 1. Bill of materials Primary datacenter Mailbox servers, 1500 users Part Number Description Quantity 663600-B23 HP SL454x 3 Node Chassis 1 664644-B23 HP SL4540 Gen8 Tray 3x Node 3 664647-B21 HP 1G I/O Module FIO Kit 3 (if not 10G below) 664648-B21 HP 10G I/O Module 3 692276-B21 HP RAID Mez Ctrllr p420i FIO Kit 3 668943-B21 HP 12in Super Cap for Smart Array 3 631679-B21 HP 1GB FBWC for P-Series Smart Array 3 512327-B21 HP 750W CS Gold Ht Plg Pwr Supply Kit 3 681254-B21 HP 4.3U Rail Kit 1 681677-B21 HP 13U FIO Rack for 3X4.3U Adapter Kit See note* 686169-L21 HP SL4540 Gen8 E5-2430L FIO Kit 3 686169-B21 HP SL4540 Gen8 E5-2430L Kit 3 647897-B21 HP 8GB 2Rx4 PC3L-10600R-9 Kit 18 682632-B21 HP Svr Storage Mezz to PCIe Board Kit 3 647594-B21 HP Ethernet 1Gb 4-port 331T Adapter 3 652757-B21 HP 2TB 6G SAS 7.2K 3.5in SC MDL HDD 45 655710-B21 HP 1TB 6G SATA 7.2k 2.5in SC MDL HDD 6 (Server) *Select this density option for racking in groups of 3 SL4540/SL4545 units in a 13U form factor. 10
Table 2. Bill of materials Secondary datacenter Mailbox server Part Number Description Quantity 663600-B23 HP SL454x 3 Node Chassis 1 664644-B23 HP SL4540 Gen8 Tray 3x Node 2 664647-B21 HP 1G I/O Module FIO Kit 2 (if not 10G below) 664648-B21 HP 10G I/O Module 2 692276-B21 HP RAID Mez Ctrllr p420i FIO Kit 2 668943-B21 HP 12in Super Cap for Smart Array 2 631679-B21 HP 1GB FBWC for P-Series Smart Array 2 512327-B21 HP 750W CS Gold Ht Plg Pwr Supply Kit 2 681254-B21 HP 4.3U Rail Kit 1 681677-B21 HP 13U FIO Rack for 3X4.3U Adapter Kit See note* 686169-L21 HP SL4540 Gen8 E5-2430L FIO Kit 2 686169-B21 HP SL4540 Gen8 E5-2430L Kit 2 647897-B21 HP 8GB 2Rx4 PC3L-10600R-9 Kit 12 682632-B21 HP Svr Storage Mezz to PCIe Board Kit 2 647594-B21 HP Ethernet 1Gb 4-port 331T Adapter 2 652757-B21 HP 2TB 6G SAS 7.2K 3.5in SC MDL HDD 20 655710-B21 HP 1TB 6G SATA 7.2k 2.5in SC MDL HDD 4 (Server) *Select this density option for racking in groups of 3 SL4540/SL4545 units in a 13U form factor. 11
1600 mailbox building block using JBOD The configuration below shows an alternate design using disks without RAID1 protection. With two Exchange copies in both the primary datacenter and the secondary datacenter, it is possible to use identical server configurations (CPU, memory and database layout, as shown below). Since the database volumes are not protected by RAID, a single disk failure will require failover to the second Exchange copy, staying within the same datacenter or location, as set by the Exchange activation preference. This design supports more users and larger mailboxes due to this different architecture, but does require an additional mailbox server per DAG in the secondary or DR location. See the following guidance from Microsoft on when it is possible to use JBOD instead of RAID protection: http://technet.microsoft.com/en-us/library/ee832790.aspx. Using JBOD is only advised with a total of three or more copies, with a minimum of two at each location. Solution summary The following table provides an overview of the HP recommended configuration for Microsoft Exchange Server 2010 and HP ProLiant SL4540 Gen8 Server for the mailbox building block using JBOD. Configuration Number of users (mailboxes) per server Normal operations Failover of one server Disaster recovery (secondary) site Average user send/receive profile (and IOPS estimate) Maximum mailbox size Database Availability Group (DAG) configuration Total servers Servers in primary / secondary locations Copies in primary / secondary locations Total users per DAG Detail 1600 2400 1600 200 messages per day (0.2 IOPS) Up to 4.5GB 6 server DAG 3 primary / 3 secondary 2 copies primary / 2 secondary 4,800 users per DAG Database volumes 14 drives per server (no RAID) 1 spare disk (or Recovery LUN per server) Solution details The figure below shows the mailbox building block using JBOD. Note that the configuration of each location is identical with three (3) servers and thus can use the full the 3 node configuration. 12
Figure 5. Details of Database Availability Group (DAG) and mailbox server configuration Server specifications are as follows: Primary datacenter: Dual CPU E5-2430L (6 core) which provides the estimated megacycles and CPU Utilization per server under normal operating scenarios shown below. Secondary datacenter: Dual CPU E5-2430L (6 core) which provides the estimated megacycles and CPU Utilization per server under the complete site or datacenter failover operating scenarios shown below. Server Configuration Primary Datacenter Server (Single Failure) Secondary Datacenter DR Server Recommended RAM Configuration 48 GB 48 GB Server Total Available Adjusted Megacycles 67549 67549 Mailbox Role CPU Megacycle Requirements 13486 13486 Mailbox Role CPU Utilization 20% 20% The table below shows the disk volume requirements: Volume Configuration Recommended Number of Databases / Volume 1 Recommended Number of Database+Log Volumes / Server 14 Number of Restore Volume / Server 1 Total Recommended Exchange Volumes / Server 15 13
Action items As a matter of best practice for all Exchange deployments, HP recommends implementing a proof-of-concept using a test environment that matches as closely as possible the planned production environment. In this way, appropriate performance and scalability characterizations can be obtained. For help with a proof-of-concept, contact an HP Services representative (hp.com/large/contact/enterprise/index.html) or your HP partner. For more information To read more about HP solutions for Exchange, including best practices and additional testing of HP server and storage products with Exchange, please see: hp.com/solutions/exchange For additional information on HP Sizer for Microsoft Exchange Server, please see: hp.com/solutions/microsoft/exchange2010/sizer For general information on Exchange sizing and best practices, there are several white papers and planning tools that are free to download from: hp.com/solutions/activeanswers/exchange For more information on ProLiant Gen8, please see http://h17007.www1.hp.com/us/en/whatsnew/proliantgen8/index.aspx. To help us improve our documents, please provide feedback at hp.com/solutions/feedback. Get connected hp.com/go/getconnected Current HP driver, support, and security alerts delivered directly to your desktop Copyright 2012, 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. Microsoft is a U.S. registered trademark of Microsoft Corporation. Intel and Xeon are trademarks of Intel Corporation in the U.S. and other countries. 14 4AA4-4492ENW, Created November 2012; Updated March 2013, Rev. 1