Building a Dynamic and Flexible Exchange Architecture. Scott Drummonds Group Manager, Performance Marketing 16 October, 2008

Transcription

1 Building a Dynamic and Flexible Exchange Architecture Scott Drummonds Group Manager, Performance Marketing 16 October, 2008

2 Why Run Exchange on VMware Support Microsoft SVVP Support and Licensing Performance Cost Management Exchange Performs Well on VMware IO is not an issue Scales out better on shared hardware Reduce infrastructure costs Consolidate Exchange server roles Eliminate need for dedicated standby servers Dynamic Management Streamline testing & troubleshooting with snapshots and clones Provision in minutes Scale dynamically and right-size infrastructure Achieve high availability without complexity of clustering

3 Double Capacity of Exchange 2007 Infrastructure Native VMware Infrastructure 8 VM 16 core 128 GB 16 core 128 GB 8K mailboxes 16K mailboxes

4 Microsoft Support Making Significant Progress Licensing Allows VMotion > Reassign licenses between servers as frequently as needed > Covers 41 server apps including > Exchange 2007 > SQL Server 2008 > SharePoint 2007 > Dynamics CRM 4.0 Support for Applications > Microsoft SVVP Program to validate virtualization products > Technical support for Windows and 31 apps on SVVP validated software > Exchange 2007 > SQL Server 2008 > SharePoint 2007 > Dynamics CRM 4.0 VMware Validated in SVVP > 9/3 announcement

5 Simpler, Cost-Efficient Availability for Exchange 2007 Edge Native Hub Client VMware Infrastructure Edge Hub Client Mailbox MS LCR VMware HA & VMotion Mailbox Network Load Balancing MS CCR Mailbox Simple Eliminate complexity of MS clustering One HA solution for all Exchange roles Cost-effective Windows and Exchange Standard Edition No dedicated standby servers Reliable VMware HA for unplanned downtime VMotion for planned downtime LCR for data replication

6 Improve Testing Efficiency with Snapshots and Clones Test 4 Move changes into production Exchange Exchange Production BlackBerry Server BlackBerry DB 3 Run more tests faster 2 Exact copy of production 1 Archive for Fast Roll-back > Faster testing > More accurate testing on exact production copy > Lower cost testing infrastructure

7 Scale Exchange Dynamically Scale up Re-size Virtual Machines VMotion to more powerful host Scale out Provision new instances

8 VMware Exchange Deployment

9 Business and User Characteristics Business Type: Software Company Typically 1 million internal / 100k external / 40k spam messages per day Primary datacenter in Palo Alto No Business Continuity or Disaster Recovery capabilities Single Active Directory site Production Mailboxes: ~9000 Mailbox profiles Unlimited mailbox sizes for users (largest ~54GB) Total mailbox data: 10TB Average mailbox size: ~1GB Average IOP rating per user: 2.25

10 Exchange 2003 Architecture System-Wide Active Directory Windows Server 2003 Enterprise deployed in a single site Exchange Server 2003 Enterprise Edition No site resiliency or disaster recovery in place 1 GigE network connection Palo Alto Datacenter 2 Microsoft Single Copy Clusters (SCC) 14 physical hosts 9 Exchange Virtual Servers for 9000 mailboxes 2 load balanced Front-End multiple protocol servers (IMAP, OWA, POP3, etc.) 2 Public Folder Server replicas Fibre channel SAN (EMC CLARiiON storage) Streaming backups performed on active mailbox server stores

11 Exchange 2003 Architecture

12 Exchange 2007 Architecture 3 physical locations (Multi-Site configuration) Primary datacenters in Palo Alto and Santa Clara Fail-over between primary datacenters Second Palo Alto site for file share witness 1 Gig connection between facilities Mailboxes (9,000 and rapidly growing) Extra-large or non-quota d mailboxes (>2 GB): ~2000 Standard mailboxes (1-2 GB): ~7000 Mailbox profiles 1 and 2 GB mailbox quotas for regular mailboxes 3, 4 and 5 GB mailbox quotas for extra-large mailboxes Average 1,200 mailboxes per VM No mailbox sizes greater than 5GB

13 Exchange 2007 Architecture System-Wide Active Directory Windows Server 2003 R2 x64 Enterprise Edition (Multi-Site configuration) Exchange 2007 Enterprise Edition Clustered Continuous Replication (CCR) over stretched VLAN used for database replication 1 GB connection between sites (Bandwidth is tunable depending on requirements) 11 application-aware (2 node) High Availability VMs using CCR 9000 mailboxes transitioned to the Exchange Server 2007 environment Project completion date: 30-June-2008 Entire Exchange infrastructure is virtualized using VMware Infrastructure 3 Palo Alto Datacenter Santa Clara Datacenter 17 Exchange Server VMs 14 Exchange Server VMs 6 Hub/CAS 3 Hub/CAS (proxy and transport only) 11 Mailbox VMs 11 Mailbox VMs Fiber Channel SAN Fiber Channel SAN 2 EMC Networker storage node VMs 2 EMC Networker storage node VMs 2 Windows 2003 Std 32 bit DC/GC VMs

14 Exchange 2007 Architecture

15 Addressing General Concerns Performance Exchange 2007 x64 bit architecture allows you to trade I/O for memory. 8K block size and write coalescing allows larger but fewer writes to storage. Enhancements to VMware Infrastructure 3 support 32/64 bit clustering and improved storage layer Availability VMware VMotion, HA and DRS allow dynamic balancing of the Exchange configuration across multiple ESX hosts and also facilitates planned maintenance. Flexibility in use of Clustered Continuous Replication (CCR) to allow immediate failover and reverse replication without the complexities of scripting. Two copies of the Exchange data are available for immediate recovery Management and Maintainability VirtualCenter provides a single pane of glass for managing and maintaining ESX as well as the Exchange VMs. Templates allow for quick provisioning

16 Addressing General Concerns Backups Flexible backup options using array-based, in-guest, streaming, or VMware Consolidated Backups. VCB now supports the VSS framework. Scaling VMware Infrastructure 3 allows us to run multiple Exchange VMs on the same ESX hosts More efficiently utilize the physical resources of your Big Iron servers Overcome the 8 core/32gb Microsoft maximum recommended Exchange configuration Resize VMs with just a reboot (add more memory, cpu, network, storage) Recoverability VMware HA restarts VMs on a different ESX host in a VMware Cluster when a failure is detected No single point of failure

17 Business Requirements Multi-Site Resiliency Leverage VMware Infrastructure 3 features to create a robust, reliable and mobile virtualized messaging infrastructure capable of withstanding datacenter/site failure High Availability Leverage application-aware high availability/replication technology for immediate failover of messaging systems with no single point of failure and minimal impact to connected clients Improved performance Take advantage of Exchange 2007 architecture enhancements to provide better I/O and application performance for an improved end-user experience Scalability Utilize VMware Infrastructure 3 to provide more efficient hardware utilization as well as provide the ability to overcome Microsoft s maximum recommended configuration of 8 cores/32gb of memory for the Exchange application

18 Business Requirements Immediate Disaster Recovery Leverage the VMware feature set ensure fast recovery from ESX host outages Infrastructure Upgrades Leverage VMware to free up critical space in the datacenter, cut power and cooling requirements. Provide redundant power, network capabilities Quota Implementation Leverage quotas to reduce backup window schedules, offline defragmentation due to constant mailbox movement Maintain Exchange performance though controlled mailbox growth Flexible Backup Options Leverage VMware feature set to give the IT department flexible backup options VMware Consolidated Backups (Now supports VSS framework U2) In Guest backups (streaming) performed using backup application of your choice Array based snapshots Passive node backups allow you to be less concerned with backup windows

19 How We Did It Built a proof-of-concept environment Deployed on a test hardware platform Identified POC participants (Core Team, IT, Friends of IT and Other Business users) Followed PSO Virtual Infrastructure Methodology (VIM) Developed a PSO SOW - VIM Assessment (Assess, Plan, Build, Manage) Created a pilot user base Create a mixed community of users (POC consisted of Approximately 100 users) Conducted a rigorous UAT Accepted or rejected based on the testing Design a scalable infrastructure (Based on business/technical requirements and information collected from existing environment) Pilot program IT Training Performance Measurement, # of VM per Core, per ESX server

20 VM Sizing: Mailbox Server! " # $ % &'! "!#$%$ #! #$%$ #! & ##$%$ # ( '$$!!( '$$!! )*&(+,-,. ( ) *&+/,-,$ $0$12 #) $*+3) 4*!$- ) *&+5,-,2*$12*#$6$$$!5$%%%+3) 4*!$- ) *&+/,78 9 :) (;-,) "*+3) 4*!$- ) *&+/,78 9 :) (;-,) "*+3) 4*!$- < +*) $! *$!-

21 VM Sizing: Peripheral Roles &) *'! " # $ % &'! "!#$%$ # ( '$$!!( '$$!! )*&(+,-,. ) *&+5,-,2*$12*#$7$$$!5$%%%+3) 4*!$- ) *&+/,-,) "*12 #) $*+3) 4*!$- & + $ + #! " # $ % &' 9! "!#$%$ # ( '$$!!( )*&+,-,.

22 Exchange 2003 Storage Profile Configured for very heavy profiles All Outlook users configured for online mailbox access Large IMAP population Mailbox ranging from 1GB - > 10GB in size 2500 Online Mobile users (RIM, Good and ActiveSync) Technologies Read intensive configuration (70/30) Average user profile for Exchange Server 2003 was ~ IOPS per mailbox

23 Exchange 2007 Storage Profile Configured for extremely heavy profiles Used Exchange calculator to calculate required storage IOPS/mailbox Exchange 2007 reduces IO by ~50 75% Exchange 2007 calculator yielded 1.15 IOPS/mailbox Added additional.6 IOPS to 1.15 IOP number for a total 1.75 IOPS Profile used messages sent/received (50/300) This is unheard of!! 5MB cache per user mailbox

24 Storage Design

25 Availability and Recovery Exchange 2003 environment utilized Single Copy Clustering Single point of failure was storage Only allowed streaming backups on the Active mailbox stores Difficulty in meeting backup windows Difficult recovery processes and procedures for clusters and data Memory fragmentation and performance problems as we scaled Required frequent reboots of hardware.

26 New Availability and Recovery Strategy VI3 Design Considerations VMware VMotion VMware DRS VMware HA Allows Exchange VMs to be migrated to other ESX Server hosts during host patching and maintenance Uninterrupted GuestOS and Host patching of Exchange infrastructure Allows even distribution of the various Exchange workloads across the VMware Infrastructure Allows automatic restart of Exchange VMs on surviving HA cluster nodes in the event of a host outage.

27 New Availability and Recovery Strategy MS Exchange Design Considerations Application-Aware High Availability (CCR) Maintain duplicate copies of the data Flexible backup options (backup from passive node of cluster) in order to meet tight production schedule windows. New Backup strategy uses Virtual Tape libraries (VTL) to perform staggered (weekly full, daily differential) backups De-Duplication performed at the VTL enables 30 day onsite retention Multi-Site Business Continuity and Disaster Recoverability Uninterrupted GuestOS and Host patching of Exchange infrastructure

28 Lessons Learned Know your existing environment Characterize your workloads (normal and especially peak) Define a Proof Of Concept Proof of concept is a chance to make sure the application meets or exceeds your Business Requirements (include users from all business units in POC) Test Proof of Concept using appropriate test toolset Exchange toolset consists of (Jetstress, LoadGen and ESP) Define attainable objectives when designing your new environment Don t bite off more than you can chew

29 Lessons Learned Testing is critical (Measure Twice Cut Once) Get close to your application vendors Continuously monitor performance in your virtualized environment Don t skimp on hardware!! Virtualized environments are highly dependent on the Core Four (CPU, RAM, Storage and Network) Keep growth and capacity requirements in mind when designing your environment Know and characterize your workloads before tackling any virtualization project

30 Summary We re a satisfied user of our own products and professional services We are virtualizing business critical apps To maintain high availability To provide us better DR solutions To save money Virtualization has changed how we operate We have a strategic view of virtualization

31 Additional Information For more information visit us at the following websites: VMware Exchange Solutions and Performance Studies

32 Backup

33 Backup: University of Plymouth Overview

34 University of Plymouth Overview Located in South West of UK Over 30,000 students 3,000 Staff Annual income of $300m 4 Centres of Excellence in Teaching and Learning

35 University of Plymouth IT Infrastructure PlymSERVE Virtualized server environment Consolidation of 280 Physical Servers to 30 2 Data Centers Use of VMotion, DRS and HA to provide highly resilient environment Projected savings: $90k per annum on energy (75% reduction on non-virtualized) 177 tons of CO 2 94% saving in floor space (32 racks to 2)

36 University of Plymouth IT Infrastructure PlymSTORE Virtualized Storage Infrastructure Based around FalconStor IPStor Appliance (Resilient) 2 HP EVA 8000 SANs 2 IBM DS4800 SANs 2 SATA Arrays Synchronous Mirroring of data between Data Centers

37 UoP Exchange Implementation 4 E2K7 Malibox Servers (4 vcpu, 16GB RAM) Each Server has 17 Stores Approx 12,500 (1,500 2,000 MAPI) users per server 1 E2K7 CAS Server (2 vcpu, 2GB) Approx 5-6K peak concurrent OWA connections 3 E2K7 Hub Transport Servers (2 vcpu, 2GB) Up to 500K SMTP Messages handled per week 2 E2K3 Back End Database Servers (2 vcpu, 4GB RAM) Providing Public Folder support for legacy Intranet Portal

38 UoP Exchange 2007 Topology SMTP Internet OWA Client Access Server MAPI Clients Hub Transport Mailbox

39 UoP Virtualized Exchange VMware ESX 3.02 HA Cluster 8 x HP BL685c Blades (4x2.8GHz DC CPU, 32GB RAM) Storage provided on mirrored HP EVA8000 through FalconStor IPstor appliance Split across 2 Data Centers Anti-affinity rules set to prevent Mailbox servers from co-existing on same physical host HA Cluster also supports other services as well as Exchange (HR System, SQL databases, Filestore)

40 UoP Exchange 2007 Topology HA Cluster Filestore Hub Transport SQL Mailbox Mailbox Hub Transport Mailbox HR DB Hub Transport Mailbox Filestore Data Centre 1 IPStor Data Centre 2 EVA8000 EVA 8000

41 Exchange and VMware Benefits High Availability Hardware Failure Mailbox Servers restart in <5 minutes CAS and Hub Transport restart in 2 minutes VMotion allows maintenance to happen without interruption to service Reduced Complexity No O/S-level or Application-level Cluster Service required Simpler configuration Easier Troubleshooting (not that we ve had to troubleshoot much!) Clustering requires comparable hardware VMware doesn t!

42 Exchange and VMware Benefits (contd) Hardware Independence VMs are hardware independent Could upgrade/switch hardware without requiring new drivers Scalability Possible to scale each VM to 64GB RAM (with ESX 3.5) Can start with smaller RAM then increase until sweet spot found Enhanced Testing and Rollback Use of Snapshot and Clone High user satisfaction 80% reduction in user complaints for Exchange service

43 Virtualised Exchange Performance Real World E2K7 Mailbox CPU Memory Average Max Average Max 620MHz 10GHz 2GB 7GB E2K7 CAS 620MHz 4.6GHz 700MB 1.1GB E2K7 Edge Transport E2K3 Public Folders 180MHz 970MHZ 200MB 600MB 470MHz 5GHz 300MB 1GB Data gathered from live production environment: Mailbox and CAS Servers over 11 months, Edge Transport over 4 months and E2K3 over one month

44 Summary You can run Exchange 2007 with 50,000 mailboxes under VMware and achieve high user satisfaction Take notice of Microsoft s sizing recommendations In our experience there are a number of reasons why it is better to run Exchange under VMware Configuration is simpler VMotion and HA provide superior levels of HA Improved testing and rollback Enhanced Management