This document contains information about the EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server.

Transcription

1 ESRP Storage Program EMC Symmetrix DMX SRDF/A (60000 Users) Storage Solution for Microsoft Exchange Server Replicated Storage Rev 1.0 January 16, 2007 This document contains information about the EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server. The contents of this document are: Disclaimer... 3 Features... 3 Solution description... 4 Targeted customer profile... 8 Tested deployment... 9 Streaming backup Replication Best practices Replication Testing and performance results

2 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Reliability Performance Streaming backup performance Conclusion Contact Information Appendix A: Test Reports Stress testing Performance testing Backup EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

3 Overview Overview This document provides information on the EMC Symmetrix DMX SRDF/A storage solution, based on the Microsoft Exchange Solution Reviewed Program (ESRP) - Storage program 1. For any questions or comments regarding the contents of this document, see the Contact Information section. Disclaimer This document has been produced independently of Microsoft Corporation. Microsoft Corporation expressly disclaims responsibility for, and makes no warranty express or implied with respect to the accuracy of the contents of this document. The information contained in this document represents the current view of EMC on the issues discussed as of the date of publication. Due to changing market conditions, it should not be interpreted as a commitment on the part of EMC. Also, EMC cannot guarantee the accuracy of any information presented after the date of publication. Features This document describes an approach that can be used to configure Exchange solutions around EMC s high-end DMX storage systems. The EMC Symmetrix DMX is a high-performance and highly available storage system. Its multi-bay design can be configured with up to 2,400 high-performance Fibre Channel disk drives, providing a maximum raw capacity of over one petabyte. The performance results and best practices discussed in this document provide tested guidelines for configuring the DMX for a high-performance Exchange environment. For this solution, a DMX storage system was used and configured for 60,000 Exchange users, at 1 IOPS per user. The DMX was configured to replicate the Exchange data to an identical DMX using EMC SRDF/A 1. The ESRP - Storage program was developed by Microsoft Corporation to provide a common storage testing framework for vendors to provide information on its storage solutions for Microsoft Exchange Server software. For more details on the Microsoft ESRP - Storage program, please go to EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 3

4 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Solution description asynchronous replication functionality. SRDF/A provides for a long-distance replication solution that has minimal impact on host performance and provides for a consistent, restartable image at the target site at all times. In configuring this Exchange solution, a concept was used that, for the purposes of this document, is referred to as a building-block approach. The building-block approach defines the number of spindles required for a certain number of users per Exchange Server. In this case, 32 Symmetrix spindles will support 4000 users per Exchange server. To scale the solution, additional server/spindle building blocks are added to increase the number of users in blocks of By using this approach, configuration maintenance and growth of an enterprise Exchange environment is much simpler. Solution description Sizing and configuring storage for use with Microsoft Exchange Server can be a complicated process driven by many variables and factors, which vary from organization to organization. One method effectively used to simplify the sizing and configuration of large amounts of EMC Symmetrix/DMX storage for use with Microsoft Exchange Server is to define a unit of measure or work. Such a unit of measure or work in this case one server with 32 storage spindles meets all of the Microsoft Exchange Server recommended metrics for excellent reliability, scalability, and performance. This unit of work or building block needs to be scalable. In that way, an organization can simply take this unit of work and multiply it by some factor until the desired amount of Microsoft Exchange Server work (i.e., Microsoft MAPI Outlook users) has been properly met or configured to satisfy the Microsoft Exchange Server recommended performance metrics. If this unit of work is properly configured, it will match the Microsoft Exchange Server recommendations for a healthy-performing system, from both a disk and an end-user perspective. 4 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

5 Solution description Table 1 describes the characteristics of each building block. Table 1 Building block characteristics Number of users 4,000 Number of Exchange Servers 1 IOPS per user 1.0 Mailbox size Number of disks required for logs and database Disk type 100 MB GB 15K RAID type 1/0 Number of storage groups per server 4 Using the performance characteristics of the Symmetrix DMX architecture and the I/O capability of the 146-GB disk drives, 32 spindles are required to provide the necessary performance to match the I/O requirements of 4000, 1.0 IOPS users with a 2:1 read/write ratio. Each disk in the group of 32 was split into logical volumes called hypervolumes (referred to by Microsoft as slices, which is the term used in Table 5 and Table 6). Hypervolumes are created in a mirrored configuration across the 32 physical spindles, resulting in an effective usable storage allocation of 16 drives. A drive may be thought of as a logical device comprised of two physical spindles. Hypervolumes from different drives are then combined into aggregate volumes called Symmetrix metavolumes, using a striping implementation within the storage system to ensure distribution of workload across the drives. These striped metavolumes are then presented to the Exchange server as LUNs for database and log storage. Each metavolume is comprised of a number of hypervolumes depending on the amount of storage needed, which differs for database and log LUNs. The size of the hypervolumes, and therefore the number of hypervolumes per spindle, varies. EMC has determined that sharing the database and log volumes on the same spindles does not adversely affect performance. While mixing data and log volumes on the same physical spindles can provide acceptable performance, data and log volumes from the same storage group should not be co-located for recovery reasons. This is further documented in the Best practices section, under Core storage. This design was then duplicated fourteen times resulting in 15 building-blocks of 4000 users to get to the required 60,000 total number of EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 5

6 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Solution description users. See Figure 1 for a view of the DMX and Figure 2 for a view of the disk layout Figure 1 EMC Symmetrix DMX EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

7 Solution description Storage Bay 2A Pod 4 Storage Bay 1A Pod 4 DMX System Bay Storage Bay 1B Pod 4 DAE 13 DAE 14 DAE 15 DAE 16 DAE 13 DAE 14 DAE 15 DAE 16 DAE 13 DAE 14 DAE 15 DAE 16 Fan 1 Fan 2 Fan 3 SPS SPS SPS 8 9 A B C D E F 4B 4B 4B SPS 4A Pod 3 Pod 3 Pod 3 DAE 9 DAE 10 DAE 11 DAE 12 DAE 9 DAE 10 DAE 11 DAE 12 DAE 9 DAE 10 DAE 11 DAE 12 SPS 3B SPS 3A Pod 2 Pod 2 PS 1 PS 2 PS 3 PS 4 PS 5 PS 6 PS 7 PS 8 Pod 2 DAE 5 DAE 6 DAE 7 DAE 8 DAE 5 DAE 6 DAE 7 DAE 8 DAE 5 DAE 6 DAE 7 DAE 8 KVM SPS SPS UPS SPS 2B 2B Server 2B SPS 4A SPS 4B SPS 2A SPS 4A SPS 3B SPS 3A SPS 2A DA 1 DA 2 FA 3-8 ports FA FA 4-8 ports FA DA5 DA 6 FA 7-8 ports FA SPS 3A SPS 2A Pod 1 Pod 1 Pod 1 DAE 1 DAE 2 DAE 3 DAE 4 DAE 1 DAE 2 DAE 3 DAE 4 SPS 1A SPS 1B DAE 1 DAE 2 DAE 3 DAE 4 FA 8-8 ports FA M0-64GB M1-64GB M2-64GB M3-64GB M4-64GB M5-64GB Air Intake M6-64GB M7-64GB FA 9-8 ports FA FA 10-8 ports FA DA 11 SPS 3B SPS 2B DA 12 FA 13-8 ports FA FA 14-8 ports FA DA 15 DA 16 SPS 4A SPS 3B SPS 3A SPS 2A SPS 1B SPS 1A SPS 1B SPS 1A Empty Space SPS 1B SPS 1A EXCN1111 -Mdb ECCN1112-Log 146GB - 15K EXCN1112-Mdb EXCN1113-Log EXCN1113-Mdb EXCN1114-Log 146GB - 15K EXCN1114-Mdb EXCN1115-Log EXCN1115-Mdb EXCN1121-Log 146GB - 15K EXCN1121-Mdb EXCN1122-Log EXCN1122-Mdb EXCN1123-Log 146GB - 15K EXCN1123-Mdb EXCN1124-Log EXCN1124-Mdb EXCN1125-Log 146GB - 15K EXCN1125-Mdb EXCN1131-Log EXCN1131-Mdb EXCN1132-Log 146GB - 15K EXCN1132-Mdb EXCN1133-Log EXCN1133-Mdb EXCN1134-Log 146GB - 15K EXCN1134-Mdb EXCN1135-Log EXCN1135-Mdb EXCN1111-Log 146GB - 15K Hot Spare Clones - 300GB - 10K 146GB - 15K 146GB - 15K 146GB - 15K 146GB - 15K 146GB - 15K 146GB - 15K 146GB - 15K GEN Figure 2 DMX disk layout The ESRP-Storage program focuses on storage solution testing to address performance and reliability issues with storage design. However, storage is not the only factor to consider when designing a scalable Exchange solution. Other factors that affect the server scalability are: Server processor utilization Server physical and virtual memory limitations Resource requirements for other applications Directory and network service latencies Network infrastructure limitations EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 7

8 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Targeted customer profile Replication and recovery requirements Client usage profiles Due to such variables, the number of mailboxes hosted per server, as part of the tested configuration, may not necessarily be viable for some customer deployments. For more information on identifying and addressing performance bottlenecks in an Exchange system, please refer to Microsoft's Troubleshooting Microsoft Exchange Server Performance, available at: Targeted customer profile This solution is intended for Enterprise Exchange customers who would like to consolidate their Exchange users on a high-performance, highly available storage platform. This configuration is designed to support 60,000 Exchange MAPI users, with the following assumptions: 15 Exchange servers 4,000 users per server 1 IOPS per user 100 MB mailbox 4 Exchange storage groups per server 5 databases per storage group 8 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

9 Tested deployment Tested deployment The following tables summarize the tested environment. Simulated Exchange configuration Table 2 lists the simulated Exchange configuration details. Table 2 Simulated Exchange configuration Item Description Number of Exchange mailboxes simulated 60,000 Number of hosts 15 Number of mailboxes/host 4,000 Number of storage groups/host 4 Number of mailbox stores/storage group 5 Number of mailboxes/mailbox store 200 Number of mailbox store LUNs/storage group 1 Simulated profile: I/O's per second per mailbox (IOPS) Database LUN size Log LUN size Backup LUN size/storage group GB 33 GB N/A Total database size for performance testing GB Percentage storage capacity used by Exchange database 70 percent EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 9

10 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Tested deployment Hardware Table 3 lists the hardware used in the environment. Table 3 Hardware Item Storage type (SAN, DAS, iscsi, NAS) Description SAN Storage model and OS/firmware revision 2 EMC DMX Enginuity 5771 DMX WHQL Listing Storage cache Number of storage controllers Number of storage ports Maximum bandwidth of storage connectivity to host Switch type/model/firmware revision 512 GB 24 front end (FAs), 8 SRDF (RAs), 32 back end (DAs) 48 ports 4 GB (2x2 GB per host) Two Cisco MDS 9509 switches SAN-OS 2.1 (2e) HBA model and firmware Qlogic QLA 2340 FW Number of HBAs/host 2 Host server type Total number of disks tested in solution Maximum number of spindles that can be hosted in the storage Dell 6850, 4x3 GHz CPU, 4-GB RAM 480 (=32x15 servers) EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

11 Tested deployment Software Table 4 lists the software used in the environment. Table 4 Software Item Description HBA driver STOR Miniport HBA QueueTarget Setting 0 HBA QueueDepth Setting 128 Multipathing EMC PowerPath Host OS Windows 2003 Enterprise Edition SP1 ESE.dll file version Replication solution name/version SRDF/A Disk configuration (mailbox store disks) Table 5 lists the disk configuration (mailbox store disks) for the environment. Table 5 Disk configuration (mailbox store) Item Disk type, speed and firmware revision Raw capacity per disk (GB) Description FC, 15K, FW: D30A 133 GB Number of physical disks in test 480 (=32 x 15) Total raw storage capacity (GB) GB (133 GB x 480) Disk slice size (GB) Number of slices per LUN or number of disks per LUN 16 GB 16 RAID level RAID 1/0 Total formatted capacity GB (=16 x 16 x 4 x 15) EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 11

12 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Streaming backup Disk configuration (transactional log disks) Table 6 lists the disk configuration (transactional log disks) for the environment. Table 6 Disk configuration Item Disk type, speed and firmware revision Raw capacity per disk (GB) Description 15K FC, FW: D30A 133 GB Number of physical disks in test 480 (=32 x 15) Total raw storage capacity (GB) GB (133 GB x 480) Disk slice size (GB) Number of slices per LUN or number of disks per LUN 8.42 GB 4 RAID level RAID 1/0 Total formatted capacity 2021 GB (=8.42 x 4 x 4 x 15) Streaming backup N/A 12 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

13 Replication Replication Table 7 Replication Item Replication mechanism Description Asynchronous Number of links 4 Simulated link distance Link type 1500 miles 1 GB FCIP Best practices Microsoft Exchange Server is a disk-intensive application. It is characterized as a very bursty (mostly 4-KB random access) read/write operation to the database files, with a sequential, synchronous (mostly 512 byte) write operation to the transaction logs. It is this random, bursty workload, with periods of high peaks, that makes designing a well-performing storage solution with Microsoft Exchange Server a challenge. Different corporate environments have different user and storage requirements, so storage design cannot be simply based on generalizations. Based on the testing run using the ESRP framework, EMC recommends following these Best Practices to provide optimal storage performance with Exchange solutions. Core storage 1. Use diskpart (in Windows 2003 SP1) to align all Microsoft-Exchange-related disks, using an alignment value of 64 (128 hidden sectors). This aligns all of the Exchange-related NTFS partitions on a 64-KB boundary. 2. Format all Microsoft-Exchange-related NTFS partitions using a 64-KB Allocation Unit (AU) cluster size. While this cluster size has been shown to have no effect on normal Microsoft Exchange database operations (transaction activity), studies have shown that a 64-KB cluster size increases performance with certain Microsoft Exchange and NTFS-related operations, such as Exchange backups and Exchange check-summing activities associated with Microsoft Exchange VSS-related operations. EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 13

14 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Best practices 3. Isolate the Microsoft Exchange Database workload from other I/O-intensive applications or workloads. This ensures the highest levels of performance for Microsoft Exchange and makes troubleshooting easier in the event of a disk-related Microsoft Exchange performance problem. 4. In the past it has been a best practice to isolate the sequential, synchronous transaction log I/O from the asynchronous database I/O for performance, as well as for disaster/recovery purposes. Assuming the environment has been properly sized from a total I/O server workload perspective, and a proper RAID-protection level has been deployed (RAID 1/0), it is acceptable to share log I/O with database I/O on the same physical spindles, while maintaining the highest levels of protection and performance. Note: When utilizing synchronous multi-site replication, data and log LUNs from the same storage group may be located on common spindles. When using asynchronous replication or when a remote replica is not implemented, to improve recoverability, data and log volumes from the same SG should not be located on the same spindles. 5. Size and configure the environment for spindle performance as a primary consideration, with spindle or storage capacity a secondary issue. In other words, size for performance first and make the capacity requirements work. 6. Striped metavolumes should be used for all Microsoft Exchange Server implementations, using RAID 1/0 configurations. Striped metavolumes provide full utilization of the back-end hypervolumes. Concatenated metavolumes, on the other hand, limit I/O capacity to a single hypervolume at any given time. 7. Most performance-related issues that arise are usually associated with an undersized environment. Sizing-related performance issues are usually a result of an inadequate number of physical spindles, which are unable to support the Microsoft Exchange I/O workload on the DMX back end, or an inadequate number of DMX front-end adapters (FAs). The number of host front-end adapters is also critical in getting optimal performance. Care should be taken to design a system that has the proper number of spindles and FAs for each environment, as detailed in this document. 8. The EMC Symmetrix/DMX includes some very powerful native I/O tracing and performance-analysis tools. Additionally, EMC Corporation provides sophisticated performance-analysis software-related tools, which are also often used in root-cause analysis in conjunction with the 14 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

15 Best practices native EMC Symmetrix/DMX tracing facilities. In the event a performance problem cannot be resolved using common performance analysis, EMC Corporation strongly recommends that a case be opened with EMC Customer Service, so that the appropriate Customer Support resources may be engaged. Replication 1. EMC recommends a minimum of 2 RDF adaptors on each storage system to support redundancy and fault tolerance. To determine the actual number of adaptors required for a given configuration, EMC uses internal tools that analyze link bandwidth, distance (latency) between enclosures, user count, and workload. The results of this analysis will determine the number and type of RDF adaptors required. 2. In an asynchronous replication environment such as with SRDF/A, the log and database replication activity do not need to be separated. The logs and database volumes are contained in one RDF group and transferred in discrete delta sets. The nature of SRDF/A guarantees a constant, restartable set of database and log files at the remote site at all times. 3. EMC uses tools to analyze the I/O load generated by the Exchange system, as well as the distance and type of connection between the enclosures, to determine the required bandwidth for the connection. Experienced network and communications resources must be employed to ensure that the replication link provides the required throughput for the system. 4. SAN analysis has shown that optimal performance is obtained when zones between enclosures consist of one pair (source and target) of ports. Use multiple, single-pair zones to accommodate multiple replication links. All SRDF zones should only contain RF ports. Use a maximum of 10 RF adaptors per switch to avoid lengthy link recovery times when RSCNs occur. Note: For further information on replication best practices, please see the EMC article, Usage of Exchange 2003 with SRDF/A in Disaster Recovery Configurations, which can be found on EMC Powerlink. Also, for more information from Microsoft on multi-site data replication support for Exchange 2003 and Exchange 2000, refer to this Microsoft Help and Support website. EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 15

16 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results Testing and performance results This section provides a high-level summary of the test data from ESRP, as well as links to the detailed reports located in Appendix A: Test Reports, which are generated by the ESRP testing framework. Reliability One of the tests in the framework is designed to test reliability over a 24-hour period. The goal of this test is to verify that the storage can handle high I/O load for a long period of time. Following the stress test, both log and database files are analyzed for integrity to ensure there is no database/log corruption. To see the stress test results, see the section Stress testing on page 33. The following list provides an overview of the results of those tests: There were no errors reported in the event log for the storage reliability testing. There were no errors reported for the database and log checksum. There were no errors reported for the streaming backup test. There were no errors reported on a checksum of the databases on the remote storage database. Performance To see the Jetstress performance results (2-hour performance test), see the section Performance testing on page 130. The purpose of performance testing is to exercise the storage with maximum sustainable Exchange-type I/O for two hours.the test is used to show how long it takes for the storage to respond to an I/O under load. The data included in the following tables is a sample taken from each of the attached host/s. It is the average of all the logical disks in the two-hour test duration. EXCN1111 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.75 Average database disk read I/O per second IOPS 16 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

17 Testing and performance results Average database disk write I/O per second IOPS Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host Average log disk write I/O per second per host IOPS 2901 IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. (322.7 IOPs * 4 SGs) IOPs (62.39 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1112 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host Average log disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) IOPs (62.24 IOPs * 4 SGs) Max database page fault stalls per second 0 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 17

18 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results EXCN1113 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.50 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (62.79 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN114 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) 18 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

19 Testing and performance results Average database disk write I/O per second per host IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (62.51 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1121 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (63.14 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1122 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.50 Average database disk read I/O per second Average database disk write I/O per second IOPS IOPS EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 19

20 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (62.04 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1123 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (62.33 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1124 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

21 Testing and performance results Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (62.25 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1131 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (67.64 IOPs * 4 SGs) Max database page fault stalls per second 0 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 21

22 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results EXCN1132 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (67.76 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1133 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) 22 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

23 Testing and performance results Average database disk write I/O per second per host IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (67.05 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1134 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (65.80 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1141 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second IOPS IOPS EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 23

24 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (66.88 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1142 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) 1.00 Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (67.94 IOPs * 4 SGs) Max database page fault stalls per second 0 EXCN1143 Average of the database disks read latency (ms) Average of the database disks write latency (ms) Average of the log disks write latency (ms) EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

25 Testing and performance results Average database disk read I/O per second Average database disk write I/O per second Average log disk write I/O per second Average database disk read I/O per second per host Average database disk write I/O per second per host IOPS IOPS IOPS IOPs per sec. ( IOPS * 4 SGs) IOPs per sec. ( IOPs * 4 SGs) Average log disk write I/O per second per host IOPs (66.64 IOPs * 4 SGs) Max database page fault stalls per second 0 Streaming backup performance This test runs through four phases: 1. Performance log generating: This phase runs similarly to the performance test run; however, circular logging is disabled. The test stops after 100 log files are created in each storage group. 2. Soft recovery: After phase one, the checkpoint files are deleted, and the application will mount the databases. This will force Jet to go through each log file. Disk performance is measured while each log file is replayed. 3. Database checksum: This phase is performed to simulate the integrity check (i.e., eseutil /k) against all the databases. Disk performance is measured. 4. Log checksum: This phase is to simulate the integrity check (i.e., eseutil /k) against all log files to ensure no corruption. Database read-only performance To see the complete backup-to-disk database read-only complete performance results, see the section: Backup on page 235. This test is used to measure the maximum rate at which databases could be recovered. The following tables show the average rate for a single database file: EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 25

26 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results EXCN1111 MB read/sec per storage group 60 MB read/sec per host 240 File size/seconds taken MB/sec ( MB/2923 EXCN1112 MB read/sec per storage group 61 MB read/sec per host 245 File size/seconds taken MB/sec ( MB/ 3034 EXCN1113 MB read/sec per storage group 59 MB read/sec per host 235 File size/seconds taken MB/sec ( MB/3004 EXCN1114 MB read/sec per storage group 53 MB read/sec per host 212 File size/seconds taken MB/sec ( MB/3293 EXCN1121 MB read/sec per storage group 53 MB read/sec per host 213 File size/seconds taken MB/sec ( MB/ EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

27 Testing and performance results EXCN1122 MB read/sec per storage group 59 MB read/sec per host 235 File size/seconds taken MB/sec ( MB/2980 EXCN1123 MB read/sec per storage group 55 MB read/sec per host 220 File size/seconds taken MB/sec ( MB/3225 EXCN1124 MB read/sec per storage group 65 MB read/sec per host 262 File size/seconds taken MB/sec ( MB/2749 EXCN1131 MB read/sec per storage group 61 MB read/sec per host 244 File size/seconds taken MB/sec ( MB/2907 EXCN1132 MB read/sec per storage group 54 MB read/sec per host 218 File size/seconds taken MB/sec ( MB/3203 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 27

28 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results EXCN1133 MB read/sec per storage group 64 MB read/sec per host 255 File size/seconds taken MB/sec ( MB/2808 EXCN1134 MB read/sec per storage group 58 MB read/sec per host 232 File size/seconds taken MB/sec ( MB/3065 EXCN1141 MB read/sec per storage group 54 MB read/sec per host 217 File size/seconds taken MB/sec ( MB/3202 EXCN1142 MB read/sec per storage group 53 MB read/sec per host 213 File size/seconds taken MB/sec ( MB/3257 Log read-only performance This test is used to measure the maximum rate at which the log files can be played against the databases. The following tables show the average rate for 100 log files played in a single storage group. Each log file is 5 MB. EXCN1111 Log files played/sec 1.94 Average log disks read bytes/sec EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

29 Testing and performance results EXCN1112 Log files played/sec 2.07 Average log disks read bytes/sec EXCN1113 Log files played/sec 1.84 Average log disks read bytes/sec EXCN1114 Log files played/sec 1.95 Average log disks read bytes/sec EXCN1121 Log files played/sec 2.06 Average log disks read bytes/sec EXCN1122 Log files played/sec 1.87 Average log disks read bytes/sec EXCN1123 Log files played/sec 1.96 Average log disks read bytes/sec EXCN1124 Log files played/sec 1.99 Average log disks read bytes/sec EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 29

30 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Testing and performance results EXCN1131 Log files played/sec 1.91 Average log disks read bytes/sec EXCN1132 Log files played/sec 1.97 Average log disks read bytes/sec EXCN1133 Log files played/sec 1.91 Average log disks read bytes/sec EXCN1134 Log files played/sec 2.05 Average log disks read bytes/sec EXCN1141 Log files played/sec 1.95 Average log disks read bytes/sec EXCN1142 Log files played/sec 1.94 Average log disks read bytes/sec EXCN1143 Log files played/sec 2.05 Average log disks read bytes/sec EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

31 Conclusion Note: Log recovery performance for all servers was calculated based on the average performance of the first three storage groups. The performance of the fourth storage group is skewed based on the database checksum process commencing on the other nodes in the environment. Eseutil is a very I/O intensive operation and should only be conducted outside of normal business hours to minimize the impact on the production environment. Conclusion This document highlights the performance results and configuration best practices for a 15-server, 60,000-user Microsoft Exchange configuration on the EMC DMX storage system. As this document illustrates, EMC s Symmetrix DMX high-performance and highly available storage system is more than capable of supporting enterprise-level Exchange deployments of up to 60,000 users at 1 IOPs per user. A careful analysis of each environment must be performed in order to understand the specific requirements of the architecture and to adapt a solution that best fits those needs. The information included in this document helps provide the tools necessary to configure the EMC DMX storage system to support a high-performance Exchange configuration. This document is developed by EMC, and reviewed by Microsoft Exchange Product team. The test results/data presented in this document are based on the tests introduced in the ESRP test framework. The customer should not quote the data directly for his/her pre-deployment verification. It is still necessary to go through the exercises to validate the storage design for a specific customer environment. The ESRP program is not designed to be a benchmarking program; tests are not designed to get the maximum throughput for a given solution. Rather, it is focused on producing recommendations from vendors for the Exchange application.therefore, the data presented in this document should not be used for direct comparisons among the solutions. EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 31

32 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Contact Information Contact Information EMC recommends that you consult with EMC Professional Services to assist with the design and deployment of a similar solution. For information regarding this or any other EMC solution, contact: United States: (800) (SVC-4EMC) Canada: (800) (543-4SVC) Worldwide: (508) For additional information on EMC products and services available to customers and partners, please refer to: or or 32 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

33 Appendix A: Test Reports Appendix A: Test Reports Stress testing EXCN1111 Database checksum statistics The following section provides a quick overview of the checksum statistics for EXCN1111. Database Pages seen Bad checksums Correctable checksums Wrong page numbers MB read/sec (File size/seconds taken) I:Jetstress.edb MB/sec (29997 MB/579 I:Jetstress1.edb MB/sec (30022 MB/551 I:Jetstress2.edb MB/sec (30011 MB/562 I:Jetstress3.edb MB/sec (30016 MB/498 I:Jetstress4.edb MB/sec (30050 MB/555 I:Jetstress5.edb MB/sec (30045 MB/535 J:Jetstress.edb MB/sec (30223 MB/582 J:Jetstress1.edb MB/sec (30197 MB/552 J:Jetstress2.edb MB/sec (30233 MB/565 J:Jetstress3.edb MB/sec (30240 MB/592 J:Jetstress4.edb MB/sec (30230 MB/556 J:Jetstress5.edb MB/sec (30217 MB/498 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 33

34 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Stress testing K:Jetstress.edb MB/sec (30177 MB/581 K:Jetstress1.edb MB/sec (30191 MB/566 K:Jetstress2.edb MB/sec (30203 MB/566 K:Jetstress3.edb MB/sec (30197 MB/589 K:Jetstress4.edb MB/sec (30158 MB/538 K:Jetstress5.edb MB/sec (30189 MB/502 L:Jetstress.edb MB/sec (29890 MB/577 L:Jetstress1.edb MB/sec (29881 MB/563 L:Jetstress2.edb MB/sec (29890 MB/556 L:Jetstress3.edb MB/sec (29881 MB/585 L:Jetstress4.edb MB/sec (29882 MB/534 L:Jetstress5.edb MB/sec (29874 MB/520 (sum) MB/sec ( MB/3328 The following table shows a quick overview of database and log performance counter sample data: Storage Volume sec/read sec/write Disk Reads/sec Disk Writes/sec Total Seconds I: J: K: EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

35 Stress testing L: The following table shows a quick overview of processor and memory performance counter sample data: Counter Average Minimum Maximum % Processor Time Available MBytes Free System Page Table Entries Pages/Sec Pool Nonpaged Bytes Pool Paged Bytes Performance log C:\Program Files\Jetstress\DatabaseChecksum_2006_12_7_ blg is saved. 12/6/2006 5:01:39 PM -- Environment validation results: Detected JetstressUI version: Detected operating system: Microsoft Windows Server 2003 ( ) Service Pack 1 Detected ESE.dll version: Detected ESEPERF.dll version: /6/2006 5:01:39 PM -- Validating input parameters, it may take a few minutes... 12/6/2006 5:01:39 PM -- Attaching databases... (it may take a few minutes if the databases are in dirty shutdown state) 12/6/2006 5:01:54 PM -- Loading performance counters... 12/6/2006 5:01:54 PM -- Instance6300.1: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Instance6300.2: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Instance6300.3: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Instance6300.4: IO parameters are thread (10), EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 35

36 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Stress testing 12/6/2006 5:01:54 PM -- Start Jetstress test... 12/6/2006 5:01:55 PM -- Starting Stress test run... 12/6/2006 5:01:56 PM -- Performance logging started. 12/6/2006 5:01:56 PM -- Performance data will be saved to C:\Program Files\Jetstress\Stress_2006_12_6_ blg. 12/7/2006 5:01:20 PM -- Adding new data to the performance log file... 12/7/2006 5:01:57 PM -- Performance logging stopped. 12/7/2006 5:01:57 PM -- Stopping Jetstress... 12/7/2006 5:02:31 PM -- Creating test report... 12/7/2006 5:02:47 PM -- Volume L: has for sec/read. 12/7/2006 5:02:47 PM -- Volume K: has for sec/read. 12/7/2006 5:02:47 PM -- Volume J: has for sec/read. 12/7/2006 5:02:47 PM -- Volume I: has for sec/read. 12/7/2006 5:02:47 PM -- Volume E: has for sec/write. 12/7/2006 5:02:47 PM -- Volume E: has for sec/read. 12/7/2006 5:02:47 PM -- Volume F: has for sec/write. 12/7/2006 5:02:47 PM -- Volume F: has for sec/read. 12/7/2006 5:02:47 PM -- Volume G: has for sec/write. 12/7/2006 5:02:47 PM -- Volume G: has for sec/read. 12/7/2006 5:02:47 PM -- Volume H: has for sec/write. 12/7/2006 5:02:47 PM -- Volume H: has for sec/read. 12/7/2006 5:02:47 PM -- Test has 0 Max Database Page Fault Stalls/sec. 12/7/2006 5:02:47 PM -- Test has 0 Database Page Fault Stalls/sec samples higher than 0. 12/7/2006 5:02:48 PM -- Performance logging started. 12/7/2006 5:02:48 PM -- Performance data will be saved to C:\Program Files\Jetstress\DatabaseChecksum_2006_12_7_ blg. 12/7/2006 5:02:48 PM -- Checksum validation may take a while depending on the file sizes. 12/7/2006 5:58:35 PM -- Database checksum in progress: Storage Group #1 (100%), Storage Group #2 (100%), Storage Group #3 (100%), and Storage Group #4 (100%). 12/7/2006 5:58:35 PM -- Performance logging stopped. 12/7/2006 5:58:35 PM -- Checksum is completed, please open C:\Program Files\Jetstress\DatabaseChecksum_2006_12_7_ html for checksum result. EXCN1111 Stress statistics The following section provides a quick overview of the stress statistics for EXCN1111. The purpose of this test is to verify the storage configuration which the customer is planning to deploy. The test run has been successful in terms of the disk latency, but refer to the table below to determine if the actual disk IO 36 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

37 Stress testing throughput requirement has been met. If not, you will need to increase the thread count to increase the load, provided that the disk latency will not exceed the threshold. Also, please check the status pane in the Jetstress window, to make sure that no errors were logged during the database checksum validation. Planned disk subsystem profile: Total test database size Production data size Total number of databases Expected I/O Achieved I/O GB (based on the attached database) (n/a) 24 (4 storage(s) * 6 database(s)) (4000 mailboxes of 1.00 IOPS) JetStress test parameter summary: Instance threadcount logbuffersize opinsert opreplace opdelete lazycommit Disk subsystem performance summary: Volume sec/read sec/write Disk Reads/sec Disk Writes/sec Bytes/Write Data (L:) (n/a) Data (K:) (n/a) Data (J:) (n/a) Data (I:) (n/a) Log (E:) Log (F:) Log (G:) Log (H:) EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 37

38 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server Stress testing Processor/memory performance summary: Counter Average Minimum Maximum % Processor Time Available MBytes Free System Page Table Entries Pages/Sec Pool Nonpaged Bytes Pool Paged Bytes Database Page Fault Stalls/sec Performance log C:\Program Files\Jetstress\Stress_2006_12_6_ blg is saved. 12/6/2006 5:01:39 PM -- Environment validation results: Detected JetstressUI version: Detected operating system: Microsoft Windows Server 2003 ( ) Service Pack 1 Detected ESE.dll version: Detected ESEPERF.dll version: /6/2006 5:01:39 PM -- Validating input parameters, it may take a few minutes... 12/6/2006 5:01:39 PM -- Attaching databases... (it may take a few minutes if the databases are in dirty shutdown state) 12/6/2006 5:01:54 PM -- Loading performance counters... 12/6/2006 5:01:54 PM -- Instance6300.1: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Instance6300.2: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Instance6300.3: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Instance6300.4: IO parameters are thread (10), 12/6/2006 5:01:54 PM -- Start Jetstress test... 12/6/2006 5:01:55 PM -- Starting Stress test run... 12/6/2006 5:01:56 PM -- Performance logging started. 38 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server

39 Stress testing 12/6/2006 5:01:56 PM -- Performance data will be saved to C:\Program Files\Jetstress\Stress_2006_12_6_ blg. 12/7/2006 5:01:20 PM -- Adding new data to the performance log file... 12/7/2006 5:01:57 PM -- Performance logging stopped. 12/7/2006 5:01:57 PM -- Stopping Jetstress... 12/7/2006 5:02:31 PM -- Creating test report... 12/7/2006 5:02:47 PM -- Volume L: has for sec/read. 12/7/2006 5:02:47 PM -- Volume K: has for sec/read. 12/7/2006 5:02:47 PM -- Volume J: has for sec/read. 12/7/2006 5:02:47 PM -- Volume I: has for sec/read. 12/7/2006 5:02:47 PM -- Volume E: has for sec/write. 12/7/2006 5:02:47 PM -- Volume E: has for sec/read. 12/7/2006 5:02:47 PM -- Volume F: has for sec/write. 12/7/2006 5:02:47 PM -- Volume F: has for sec/read. 12/7/2006 5:02:47 PM -- Volume G: has for sec/write. 12/7/2006 5:02:47 PM -- Volume G: has for sec/read. 12/7/2006 5:02:47 PM -- Volume H: has for sec/write. 12/7/2006 5:02:47 PM -- Volume H: has for sec/read. 12/7/2006 5:02:47 PM -- Test has 0 Max Database Page Fault Stalls/sec. 12/7/2006 5:02:47 PM -- Test has 0 Database Page Fault Stalls/sec samples higher than 0. EXCN1112 Database checksum statistics The following section provides a quick overview of the checksum statistics for EXCN1112. Database Pages seen Bad checksums Correctable checksums Wrong page numbers MB read/sec (File size/seconds taken) I:Jetstress.edb MB/sec (30023 MB/599 I:Jetstress1.edb MB/sec (30011 MB/656 I:Jetstress2.edb MB/sec (29999 MB/626 I:Jetstress3.edb MB/sec (30000 MB/571 I:Jetstress4.edb MB/sec (30012 MB/629 I:Jetstress5.edb MB/sec (30010 MB/522 EMC DMX SRDF/A Storage Solution for Microsoft Exchange Server 39