Data Protection at Cloud Scale A reference architecture for VMware Data Protection using CommVault Simpana IntelliSnap and Dell Compellent Storage
Contents Introduction... 4 Environment... 4 Scenarios... 7 Summary... 14
Acknowledgements This white paper was originally produced by the Dell Compellent Technical Solutions Group. Author: Kris Piepho
Introduction This document is intended to provide technical personnel (architects, engineers and consultants) with data to assist with planning, design and deployment of a Multi-Tiered data protection solution for a large virtual environment utilizing Dell storage and CommVault Simpana 9 software. This document is based on internal testing and validation performed and findings gathered in Dell Compellent s lab environment. This document specifically focuses on hardware and software configurations of the VMware environment, CommVault Simpana software, and Dell storage. This document serves as a reference architecture to implement and execute CommVault and Dell s Solving Forward by Design architecture. When deploying this architecture, results will vary for each organization. It is strongly recommended that organizations thoroughly test and pilot the reference architecture to determine appropriate settings for their environment. Executive Summary As an organization grows and expands the amount of data stored and processed grows along with it. This leads to increasing demands for storage and data protection. Protecting and managing this data can be a complex, expensive, and slow operation. A solution is needed that manages the data across the most cost effective platforms available on the market today and can protect them in a matter of minutes, not hours or days. To help reduce the cost and complexity of data protection, Dell and CommVault have defined a tiered storage reference architecture called Solving Forward by Design. This reference architecture combines the storage optimization of the Dell Fluid Data architecture with the data protection of CommVault Simpana software to offer comprehensive life-cycle management and retention of data. Essentially, Solving Forward by Design extends the Dell Fluid by Design approach beyond the Dell Compellent storage tier into protection and archiving tiers that leverage complementary Dell storage platforms. Solving Forward by Design provides a unified strategy that brings together physical and virtual server protection with backup and archiving. It is designed to scale, adapt, and react to the changing IT environment. To validate this design, Dell and CommVault tested this solution and were able to create 1000 consistent and persistent Dell Compellent Snapshot based Recovery Copies in a matter of 14 minutes. Environment Below is detailed information on the hardware and software configurations used in this reference architecture. Make, model, software versions, service packs, and firmware versions are included as they are an important part of any configuration in a production environment. Hardware The test environment was comprised of three clustered VMware ESXi 5.0 Servers hosting 1000 virtual machines (VMs). The virtual machines were stored on a Dell Compellent Storage Center SAN. A Dell DL2200 served as the CommVault CommServe, Media Agent, Virtual Server Agent (VSA) and Disk Library. The archival system was comprised of a Dell DX6000 Object Storage platform with 2 attached Dell DX6012S storage nodes. The entire infrastructure was comprised of the following components: Dell Compellent Storage Center SAN Series 40 Controllers o 6 x 2.5 200GB SSD drives o 12 x 2.5 300GB SAS 15k drives o 30 x 2.5 1TB SAS 7.2k drives o 2 SAS Enclosures o Firmware Version 5.5.6 8Gb Fibre Channel Switch 1Gb Ethernet Switch
vcenter Virtual Server o 8 Processors o 16GB RAM o 60GB C: Drive 3 x Dell r720xd Servers running ESXi 5.0 for Virtual machines o Dual 8 Core Intel Xeon E5-2690 2.90 GHz processors o 256GB RAM o Q-Logic QLE2562 8Gb Dual Port Fibre Channel HBA Dell DL2200 Powered by CommVault o Dual Quad Core Intel Xeon E5620 2.40 GHz processors o 24GB RAM o Q-Logic QLE2562 8Gb Dual Port Fibre Channel HBA o 12 x 3.5 1TB SAS 7.2k drives Dell DX6000 for Archival o 6 x 3.5 250GB SATA 7.2k drives o 2 x Dell 6012S Storage Array nodes for DX6000 24 x 3.5 250GB SATA 7.2k drives (12 drives in each node)
Figure 1: Infrastructure Rack Layout Connectivity SAN connections from the VMware hosts and DL2200 were over an 8Gb Fibre Channel fabric. A 1 Gb Ethernet network was utilized for host management and guest connections. Connection to the DX6012S storage nodes was on a private 1 GB network. For detailed information on how to properly cable Storage Center, please refer to the Dell Compellent Storage Center Connectivity Guide located on Dell Compellent Knowledge Center.
Software The following table lists the software utilized in the tested environment. Table 1: Software Software Version/Service Pack Dell Compellent Storage Center 5.5.6 VMware vsphere (ESXi Server, vcenter and View) 5.0 CommVault Simpana software 9.0 SP 6b Windows Server 2008 R2 (vcenter and DL2200 Server OS) SP1 Windows 7 Ultimate x64 (Guest OS) SP1 Red Hat Enterprise Linux (DX6000 OS) 5.5 Virtual Machine Configuration The tested virtual machine platform utilized was 64-bit Windows 7 Ultimate. All available updates were applied. Each virtual machine was configured with one vcpu and 512MB of memory. VMware View 5.0 was used to create all virtual machines. Storage Configuration Prior to the creation of the virtual machines, 100 Storage Center volumes @ 500GB/each were created and presented to the vsphere cluster as datastores. Virtual machines were evenly distributed across the datastores, which each datastore hosting 10 virtual machines. Figure 2: Datastore Guest Configuration Scenarios The Solving Forward by Design architecture utilizes three tiers to protect (backup) and archive data. Dell Compellent tested the following processes to leverage the three tiered model: 1. Storage Tier - IntelliSnap backups of the Dell Compellent volumes hosting the virtual machines. 2. Protection Tier Full backup copy of the snapshot contents to the DL Backup to Disk Appliance. 3. Archive Tier - Archival copy to the DX Object Storage Platform. Deduplication was enabled via the Storage Policy, and each subclient was configured to use deduplication as part of the backup process.
IntelliSnap Snapshot Protection The first step in protecting data in a virtualized environment is to create snapshots of the Dell Compellent volumes using CommVault s IntelliSnap technology. Using IntelliSnap, Simpana software will quiesce and then create VMware snapshots of all VMs on a datastore. When this process is complete, a snapshot of the Dell Compellent volume is taken. Thereafter, all VMware snapshots are released. This process will continue until all VMs and datastores are successfully protected. Configuration The Virtual Server Agent allows for the association of a single VM to a subclient. This configuration works well in smaller environments, but for an environment with 1000 VMs, associating entire datastores (and all VMs housed within) is an easier solution. Each datastore to be backed up should be associated with a subclient. A subclient can be associated with more than one datastore, but a datastore (and its associated virtual machines) can only be associated with one subclient. For example, if subclient 1 is associated with datastore 1(10 VMs) and datastore 2 (10 VMs) all 20 VMs will be quiesced and snapped as part of the backup process for subclient 1. Figure 3: Simpana DataStore Configuration To achieve the best backup performance all subclients should be backed up simultaneously. Simpana software facilitates this by processing each subclient as its own job, and running multiple jobs in parallel. Typically, running multiple jobs at the same time will improve backup performance, but running too many jobs at once can cause system resource contention. As more jobs run, CPU, memory, and disk resources are consumed on the DL2200, vcenter and ESXi host servers. Each environment will have a different point of diminishing returns. It is possible to overwhelm the environment and cause temporary delays as jobs resume. The maximum amount of backup jobs to run simultaneously is highly dependent on the hardware configuration of the infrastructure servers, and may require extensive testing to determine exact values. In the test environment, the configuration that allowed IntelliSnap backups to complete the in the shortest amount of time and without error was 50 subclients with 2 datastores associated to each. Execution To facilitate a single-click backup of all datastores and VMs, a schedule policy was created that contained all backup subclients configured within the Virtual Server Agent. Additionally, a backup of all subclients can be initiated from the Virtual Server Agent.
Backups to the DL Backup to Disk Appliance Powered by CommVault Backup copies to the DL Backup to Disk Appliance involve Simpana software mounting snapshots created using IntelliSnap technology to an ESXi proxy server using VADP. Once the snapshot is mounted the data is copied from the snapshot to the DL2200. When all data has been copied the snapshot is un-mounted. Simpana software will mount only one snapshot at a time to a proxy server. By utilizing all available ESXi servers as proxy mounts, total backup time was decreased. Please note that using an ESXi server as a Proxy host could negatively impact performance on the host. It is not recommended to use a production server as a Proxy server. Backup throughput is determined from the Data Readers setting on the subclient (how fast data can be read from the snapshot) and the number of Device Streams set on the Storage Policy (how fast data can be written to the backup device). For more detailed information on these settings, please see section 3.5.2 in the Lessons Learned section of this document. Configuration Each subclient was configured to use 20 Data Readers, and the ESXi Proxy server setting was configured so that every 3 subclients would use a different Proxy during backup operations. Figure 4: Subclient Properties Data Reader Setting Figure 5: Subclient Properties ESX Proxy Server Setting
Execution The backup process was manually initiated from the Storage Policy. The number of simultaneous jobs was set to three, to match the number of Proxy servers in the environment. It should be noted that running IntelliSnap backups and subsequent backups to the DL Appliance can be configured automatically in a Schedule Policy. In the Advanced Backup Options window select Create Backup Copy immediately. Figure 6: Advanced Backup Options DX6000 Archival Copy The primary purpose of the Archival Tier is to move infrequently accessed data from costly primary storage to the costeffective DX Object Storage Platform. In addition, the removal of the primary data improves backup (and application) performance as it lowers the data to be protected. Configuration The DX6000 Object Storage Platform was configured using the recommended settings as detailed in the Getting Started and Administration guides located on the Dell Support Website. In order for Simpana software to use the DX6000 as an archival target, the DX6000 needs to be added as a Cloud Storage Library in Library and Drive Configuration. Choose Dell DX Object Storage Platform for the Server Type, and enter the CSN IP address in the Service Host field.
Figure 7: Add Cloud Storage Within the Storage Policy, an auxiliary copy must be configured to use the DX Cloud Storage Library as the default library. Figure 8: Storage Policy Copies Execution The archival process was manually initiated from the Storage Policy by selecting Run Auxiliary Copy, and selecting the Storage Policy copy that points to the DX Cloud Storage Library. The archival process can take quite a bit of time depending on the amount of data being copied and the network connectivity allocated.
Results Table 2: Testing Results Job Type IntelliSnap Full Backup Copy Description Create consistent and persistent Dell Compellent Snapshots based Recovery Copies Copy of the selected snapshot contents created using VADP to the DL Backup to Disk Appliance Data Size # of VMs @ 20 GB each Time Average Processing Rate 20 TB 1000 14 mins - 20 TB 1000 19 hrs 1 TB/hr Lessons Learned Configuring Simpana software for optimum performance takes time as configuration parameters vary by system and environment. Thoroughly testing and documenting settings is highly recommended before implementing any backup solution into your environment. Listed below are several key points and best practices to follow when leveraging the Solving Forward by Design architecture to protect and backup a VMware Environment. IntelliSnap Optimal performance using the IntelliSnap feature is achieved by finding the balance between the number of subclients and the number of datastores assigned to each subclient. o The more subclients configured, the more simultaneous jobs will run during a backup. o If the number of simultaneous jobs is overwhelming the DL2200, try using less subclients and assigning more than one datastore per subclient. IntelliSnap backup performance is heavily hardware dependent: o Simpana software is a multi-threaded application and can use all available CPU on the DL2200. o When running the CommServe, Media Agent and VSA on the same server (such as the DL2200) a minimum of 24GB RAM is recommended. o The vcenter server needs to have enough CPU and memory resources to handle requests to quiesce and create snapshots of 1000 VMs. A minimum of 4 CPUs and 16GB RAM is recommended for a vcenter server in a VMware environment hosting 1000 VMs. Backing up to the DL Appliance As a general rule, the number of Device Streams set in the properties of the Storage Policy should be equal to the number of physical drives in the disk library you are using for backups. o The number of Device Streams determines how fast Simpana can write data to the backup target. o Adjust the number of Device Streams accordingly to disk I/O on the backup target. The number of Data Readers set in the properties of the subclient regulates the amount of data that can be read from snapshots when mounted to an ESXi proxy server. o The number of Data Readers needs to be set on each subclient individually. o Total backup throughput is a combination of how fast data can be read from the snapshot (number of Data Readers) and how fast that data can be written to disk (Device Streams). Backup throughput can be greatly increased by finding the correct correlation between these two values. o Setting the number of Data Readers too high can negatively affect performance. Disk I/O on the ESXi proxy host increases as the number of Data Readers is increased. The ESXi proxy host is set individually on each subclient. o In environments with multiple ESXi proxy hosts, staggering subclients to use different proxy hosts and adjusting the number of simultaneous jobs to run during a backup can decrease total backup time significantly. Example: The test environment had 3 available ESXi proxy servers. Subclient 1 was configured to
use Proxy Server 1, subclient 2 was configured to use Proxy Server 2, and subclient 3 was configured to use Proxy Server 3. All 50 subclients were configured in this way. With the subclients correctly configured, increasing the number of simultaneous jobs to match the number of proxy servers will decrease total backup time because a proxy server can only mount one snapshot at a time. Each job will mount to a different proxy server, increasing read times and decreasing overall backup time. During backups to disk, snapshots are individually mounted to the proxy ESXi host(s). This process forces HBA and VMFS rescans on the Proxy. This process can take quite a bit of time on slower systems and can greatly affect the performance of VMs running on the Proxy. It is recommended that any ESXi server utilized as a proxy should not be a production server. Depending on the amount of data on each VM, deduplication can add significant time to the backup process. Archiving data to the DX Object Storage Deduplication of data should occur before copying data to the DX Object Storage Platform. The DX storage nodes communicate with the host server on a private network. Although a separate switch is highly recommended, a private network can be configured on the primary switch by setting the switch ports in a different VLAN than that of the public network.
Summary CommVault and Dell s Solving Forward by Design architecture provides a flexible, persistent data management platform that scales up and out for virtualized data centers. Leveraging CommVault Simpana software and Dell Storage hardware, organizations can manage and protect VMware environments by cutting capacity and time requirements for protecting (backup) and archival operations. The tiered architecture facilitated by the Dell Compellent Storage Center SAN, Dell PowerVault DL Backup to Disk Appliance Powered by CommVault and Dell DX Object Storage Platform is the ideal platform to house and grow your organizations virtual data center. Additional Resources Dell Storage Solutions www.dellstorage.com Dell Compellent http://www.compellent.com/ Dell PowerVault DL Backup to Disk Appliance powered by CommVault http://www.dell.com/us/enterprise/p/powervaultdl2300-commvault/pd Dell DX Object Storage Platform http://www.dell.com/us/business/p/powervault-dx6000/pd CommVault http://www.commvault.com/simpana-software VMware http://www.vmware.com For more information about Simpana software modules and solutions, and for up-to-date system requirements, please visit www.commvault.com www.commvault.com 888.746.3849 info@commvault.com CommVault Worldwide Headquarters 2 Crescent Place Oceanport, NJ 07757 Phone: 888.746.3849 Fax: 732.870.4525 CommVault Regional Offices: United States Europe Middle East & Africa Asia-Pacific Latin America & Caribbean Canada India Oceania 1999-2013 CommVault Systems, Inc. All rights reserved. CommVault, CommVault and logo, the CV logo, CommVault Systems, Solving Forward, SIM, Singular Information Management, Simpana, CommVault Galaxy, Unified Data Management, QiNetix, Quick Recovery, QR, CommNet, GridStor, Vault Tracker, InnerVault, QuickSnap, QSnap, Recovery Director, CommServe, CommCell, IntelliSnap, ROMS, Simpana OnePass, and CommValue, are trademarks or registered trademarks of CommVault Systems, Inc. All other third party brands, products, service names, trademarks, or registered service marks are the property of and used to identify the products or services of their respective owners. All specifications are subject to change without notice.