Backup and Recovery Best Practices With Tintri VMstore

Backup and Recovery Best Practices With Tintri VMstore Backup and Recovery Best Practices with Tintri VMstore TECHNICAL BEST PRACTICES PAPER, Revision 1.0, April 10, 2014

Contents Contents Introduction 1 Intended Audience...1 Consolidated list of practices...2 Data Protection with Tintri SnapVM and ReplicateVM 3 Image-Level Backup 6 VADP Transports...7 HotAdd Transport... 7 NBD/NBDSSL Transport... 8 SAN Transport... 9 Backup Consistency...9 Crash-consistent Backup... 9 VM-consistent Backup... 10 Application-consistent Backup... 10 Guest-Level Backup 11 Data Recovery 12 Summary 13 Backup and Recovery Best Practices Page i

Introduction Introduction Tintri s VMstore systems are built from the ground up for virtualized environments. The common practice of sizing LUNs and partitioning storage array LUNs for VMware datastores with traditional storage arrays is non-existent on Tintri VMstore. Deploying storage for your virtual environment has never been simpler. Tintri VMstore delivers extreme performance and Virtual Machine (VM) density, and a wide variety of powerful features, which are seamlessly integrated with VMware vsphere and vcenter. Examples include snapshots, clones, instant bottleneck visualization, and automatic virtual disk alignment. Tintri VMstore extends and simplifies the management of VMs through an intrinsic VMawareness that starts from the top of the computing stack, all the way down into the flash (SSD) and disk (HDD) drives. However, a datacenter administrator s responsibilities do not end with the deployment of virtual desktops or Microsoft Exchange Database Availability Groups (DAG) servers in a virtualized environment. Business continuity can still be impacted in a disaster if a backup and recovery solution for protecting a virtual infrastructure is not implemented. Two high-level approaches to data protection are discussed in this paper: Data protection using Tintri per-vm data management functionality: SnapVM and ReplicateVM. Data protection using third-party backup application software. In this method, the following VM backup choices are discussed: Image-level backup and Guest-level backup. This paper also focuses on VMware s VADP framework, introduced in vsphere v4.0, enabling backup software to perform off-host backups of VMs. Image-level backup leverages VADP to allow backup processing to be done outside of the vsphere hosts. Intended Audience This Tintri Best Practice Guide for Backup and Recovery will assist individuals who are responsible for the design, deployment, and DR of VMs deployed on Tintri VMstore systems. The target audience of this document are individuals who are looking for an understanding of available backup and recovery solutions that are common with major backup software applications used in virtualized environments using VMware vsphere. This document will focus on the data protection capabilities available with Tintri systems and VMware vstorage API for Data Protection (VADP) backup and recovery, the advantages and disadvantages of different backup solutions and how these backup methods impact both the backup and the restore workflows. Each backup application software provider also has additional features for backup and recovery of VMs. The detailed features of these backup application providers are not addressed in this generic best practice guide. Backup And Recovery Best Practices Page 1

Introduction Consolidated list of practices The table below includes the recommended practices in this document. Click the text in the Recommendation column or see the section later in the document that corresponds to each recommendation for additional information. Tags Data Protection with Tintri SnapVM and ReplicateVM Data Protection with Tintri SnapVM and ReplicateVM Data Protection with Tintri SnapVM and ReplicateVM Image-Level Backup HotAdd Transport HotAdd Transport NBD/NBDSSL Transport VM-consistent Backup Application-consistent Backup Data Recovery Recommendation Do: Use SnapVM and ReplicateVM to ensure your VMs are protected across Tintri VMstores. Do: Select VM-consistent snapshot policy for specific VMs that require VM consistency. Appropriate and up-to-date VMware Tools must be installed on the VMs. Do NOT: Select VM-consistent snapshot policy in the system default schedule. VM-consistent snapshots require coordination with vsphere hosts and could have adverse impact on the virtualized infrastructure. Do: Use the incremental and synthetic backup functionality if your backup application supports such functionality. Do: Deploy the backup proxy host on a volume capable of large block size, in order to backup very large virtual disks. Do: Review your backup application administrator s guide for backing up independent disks with any of the transport modes. Do: Review your backup application administrator guide for guidance with NBD/NBDSSL transport. Do: Follow your backup application vendor s best practices for installing necessary tools in guest VMs. Do: Follow your backup application vendor s best practices for installing necessary tools in guest VMs for creating application consistent snapshots for backups. Do: It is recommended to use NBD/NBDSSL transport for thin-provisioned disk restores. Backup And Recovery Best Practices Page 2

Data Protection with Tintri SnapVM and ReplicateVM Data Protection with Tintri SnapVM and ReplicateVM A Tintri VMstore is presented as a single VMware datastore with per-vm capabilities. Tintri VMstore snapshot and cloning features apply directly to VMs. Individual VMs can be protected with Tintri s SnapVM and ReplicateVM, configured from the Tintri VMStore User Interface (UI) or the Tintri vsphere Web Client Plugin. The combined use of Tintri s SnapVM and ReplicateVM to a secondary Tintri VMstore is a data protection and recovery solution that is available with Tintri VMstore. To protect VMs, select the VMs to be protected from the Virtual Machines tab in the Tintri VMstore UI. Right-click and select Protect. Figure 1-1: Tintri VMstore Protect feature. In the Protect pop-up window, the VM name or number of VMs being protected by the snapshot schedule is displayed. In this example, we have two VMs, from the same clone, selected to use the same Protect configuration. The default snapshot schedule is based on a daily snapshot that starts at 2:30 AM with a retention period of seven days for local and remote snapshots. In general, the system default would be sufficient to meet most data protection requirements. However, if there are additional data protection requirements for specific VMs, click more >> to display other additional snapshot schedules. In this way, you can tailor SnapVM and ReplicateVM to protect the VMs on your Tintri VMstores. Figure 1-2: Protect two VMs. Data protection policies can be customized on a per-vm level. In addition, the local and remote Backup and Recovery Best Practices Page 3

Data Protection with Tintri SnapVM and ReplicateVM retention period within a Protect Window can be independently configured. In Figure 1-2: Protect two VMs. on page 3, the local snapshot has a retention period of two days while on the remote Tintri VMstore and the remote snapshot has a retention period of three days. This allows VMs to be protected with a different retention period on a destination VMstore. Do: Use SnapVM and ReplicateVM to ensure your VMs are protected across Tintri VMstores. Do: Select VM-consistent snapshot policy for specific VMs that require VM consistency. Appropriate and up-to-date VMware Tools must be installed on the VMs. Do NOT: Select VM-consistent snapshot policy in the system default schedule. VM-consistent snapshots require coordination with vsphere hosts and could have adverse impact on the virtualized infrastructure. The following are supported replication configurations with Tintri VMstores: Figure 1-3: Supported Tintri replication configuration with two Tintri VMstores. Using two Tintri VMstores, source VMs on Tintri VMstore A can be protected with SnapVM and ReplicateVM to Tintri VMstore B. In addition, source VMs on Tintri VMStore B can be replicated to Tintri VMstore A for data protection. Figure 1-4: Supported many-to-one replication with multiple Tintri VMstores. Backup and Recovery Best Practices Page 4

Data Protection with Tintri SnapVM and ReplicateVM Tintri replication also supports many-to-one replication at an individual VM level. In Figure 1-4: Supported many-to-one replication with multiple Tintri VMstores. on page 4, VMs from VMstore A and VMstore B are replicated to VMstore C. In addition, VMs on VMstore C are replicated to VMstore B for data protection. This protects your VMs across Tintri VMstores. Figure 1-5: Supported Tintri replication configuration with three Tintri VMstores. ReplicateVM only sends changed blocks between snapshots after deduplication and compression to dramatically reduce the amount of data sent over WAN by up to 95 percent. For detailed information on the Tintri VMstore SnapVM, CloneVM, and ReplicateVM technology, refer to Managing VM Data with Tintri technical white paper. Note: Cascaded replication is not supported. For example, if (VM1, VMStore A) is replicated to VMStore B, it cannot be used as a replication source on VMstore B to replicate to VMstore C. It is an unsupported configuration. Backup and Recovery Best Practices Page 5

Image-Level Backup Image-Level Backup Image-level backup allows a backup application to leverage VMware s VADP framework to backup VMs without any backup agents installed in the VMs. A VMware backup proxy host is required to fulfill backup offload functionality. A VMware backup proxy host can be a VM with a backup agent installed, a virtual backup appliance or a physical backup proxy host. Using most backup applications, the physical backup proxy host is a backup server (in this document, a backup server is a generic reference to a CommVault Simpana Media Agent, Veeam Backup Server, EMC NetWorker Storage Node or Symantec NetBackup Media Server). Figure 2-1: Image-level backup. An image-level backup is also able to take advantage of the Changed Block Tracking (CBT) feature to perform incremental backups. Disk block changes are tracked in the virtualization layer. In CBT, the VMkernel returns only the changed data blocks since the last backup snapshot was taken reducing the amount of data backed up. CBT Notes: In vsphere 5.x, Storage vmotion resets CBT. In general, file-based restores will affect change tracking, and so does the restore of a VM. VMware recommends enabling CBT before taking the first snapshot. This is due to the fact that the backup of a lazy-zeroed thick-provisioned disk with CBT disabled converts lazy-zero sectors to actual zeros. During a data recovery operation, a full eager-zeroed thick-provisioned disk is restored. Some backup applications can also take advantage of CBT and incremental backups. Using synthetic backup functionality, these backup applications are able to create a full image for restore with incremental backups. Some backup applications are able to support incremental forever with VMware backups. Not all backup applications support incremental forever backups with VMware. It is recommended to review the administration guide of each backup application provider to confirm support and terminology used for incremental forever backups. Do: Use the incremental and synthetic backup functionality if your backup application supports such functionality. Backup and Recovery Best Practices Page 6

Image-Level Backup VADP Transports At a VM level, VADP allows the backup process to be off-loaded from the ESX/ESXi host. All enterprise backup applications support more than one access method with VADP for VM level backups. Backup applications connect to a vcenter Server using one of the following transport modes: HotAdd transport NBD/NBDSSL transport SAN transport HotAdd Transport In HotAdd transport mode, the backup proxy host with an installed backup agent is deployed as a VM. A virtual backup appliance such as VMware VDPA can also be used to backup a Tintri VMstore using HotAdd transport. HotAdd transport does not support backing up independent disks. HotAdd transport mode easily moves VMs within a virtual environment; the SCSI HotAdd functionality on the ESX/ESXi server can attach VMDKs of a VM (attached as linked clones) to be backed up to the virtual backup proxy host. The backup proxy host must have access to the datastore of the VM being backed up. Additionally, the backup proxy host and the VM being backed up must be under the same datacenter. A failed backup with HotAdd could leave redo logs that are not properly cleaned up. In such scenarios, virtual disks that were left attached to the backup proxy host must be manually removed before the next backup attempt. Figure 2-2: HotAdd Transport. Do: Deploy the backup proxy host on a volume capable of large block size, in order to backup very large virtual disks. Do: Review your backup application administrator s guide for backing up independent disks with any of the transport modes. Important Note: A backup proxy host VM can have a maximum of four SCSI controllers added. Using a single SCSI controller, a maximum of 15 disks can be attached to the backup proxy host VM. Review your backup application s administrator guide for additional information on supported virtual backup appliances. Backup and Recovery Best Practices Page 7

Image-Level Backup It is recommended to use HotAdd with Tintri VMstore when VADP use with backup applications are supported. Some backup applications, such as NetBackup, allow you to send backup data to a backend secondary storage that is attached to a physical backup server. As another example, VMware VDPA can deploy backup secondary storage on a Tintri VMstore. This essentially maximizes the efficiency in your backups. In this case, backup I/O only travels across the network connection between the vsphere host and the Tintri VMStore over the storage network. In addition, the amount of I/O is minimal when Change Block Tracking (CBT) is enabled. Note: Be aware that snapshot removal can pause a VM for a long time. It is recommended to schedule backups during a backup window when a VM is the least active. For detailed information, refer to the following VMware KB. NBD/NBDSSL Transport Using Network Block Device (NBD) or Network Block Device Secure Socket Layer (NBDSSL) transport, ESX/ESXi host reads VM data from primary storage and sends it over the network to the backup proxy host. Using NBD/NBDSSL transport, the backup server (i.e. a CommVault Simpana Media Agent, Veeam Backup Server or NetBackup Media Server) can also fulfill the role of a VMware backup proxy host. Storage is treated as a block device and backups of large virtual disks takes time for backup. Given that the ESX/ESXi host must have network access to the backup proxy host, additional network traffic is generated as data is backed up over the network. NBDSSL transport is provided on the wire security when data encryption is a requirement. In comparison to NBD transport, NBDSSL performance is lower due to additional resource requirements related to encryption. Using NBD/NBDSSL, VMware utilizes the Network File Copy (NFC) protocol to read VMDKs. Figure 2-3: NBD/NBDSSL Transport. It is important to ensure that the maximum number of VMware NFC connection limits is not exceeded by the backup application. Exceeding the maximum NFS connection limits for NBD/ NBDSSL could cause backups to hang or fail Do: Review your backup application administrator guide for guidance with NBD/NBDSSL transport. Backup and Recovery Best Practices Page 8

Image-Level Backup SAN Transport The physical backup proxy host is connected to an array-based LUN storage via SAN. This bypasses the vsphere host (ESX/ESXi host) for transferring backup data. Using VADP, the backup proxy host obtains the layout of the datastore from the vcenter Server. The backup proxy host then reads the VM data directly from the SAN storage array. Using supported Windows 2008 and Windows 2012 backup proxies, VMware recommends the SAN policy be set to onlineall. Using array-based LUN storage, consult with your provider to determine the supported HBAs, SAN switches, and O/S platforms as well as specific driver and firmware versions for deploying your LUN datastores. Figure 2-4: Unsupported Tintri VMstore port-channel configuration. Important Note: For array-based datastore LUN storage, do not attempt to initialize any of the discovered VMware datastore LUNs on the backup proxy host. Windows initialization on an array-based datastore LUN will destroy your VMware data. The pains associated with deploying array-based LUN as datastores does not apply to Tintri Systems, as it does not support SAN transport. Backup Consistency Using image-level backups, backup consistency is an integral part of the solution. There are three types of backup consistencies that applies to any ESX/ESXi datastore: Crash-consistent backup VM-consistent backup Application-consistent backup Crash-consistent Backup Crash consistent backup of a VM does not capture contents in memory or process any pending application IO operations. When protecting file systems where applications or databases that reside on a Windows/Linux VM is a requirement, crash consistent backup is not sufficient. In a disaster recovery operation, a restore of a crash consistent Windows/Linux VM would be similar to bringing the system back to the system recovery state. For example, a Linux VM might require running fsck to check the file system (FS) consistency. A Windows or Linux VM hosting applications or databases would require additional steps. For example, restoring a Microsoft Backup and Recovery Best Practices Page 9

Image-Level Backup Exchange DAG server would require that both the DAG servers and the witness servers are online and operational. Additionally, the Exchange databases must also be verified as healthy and mounted. If the Exchange databases are not successfully mounted, additional troubleshooting will be required to bring them to operation. Given that, Exchange recovery process could be a time and resource consuming exercise if only a crash consistent backup is available. In the worst case, the databases could end up corrupted. Utilizing a crash-consistent backup is not recommended when file systems and applications must be protected to ensure business continuity. VM-consistent Backup File System (FS) consistent backup involves quiescing the FS of a VM before a snapshot is taken for backup purposes. Depending on the backup application provider, they may recommend their own VM tools to be installed on Linux VMs. For example, Symantec NetBackup recommends installing their SYMCquiesce utility on Linux VMs to ensure FS consistent backups. In most common cases, Windows VM with VMware tools installed is sufficient to ensure a Windows VM is FS consistent when backed up. However, this does not ensure the software applications are in a consistent state. File-Level or File System consistent backup is not sufficient when software applications must also be protected in a VMware enterprise data center. Do: Follow your backup application vendor s best practices for installing necessary tools in guest VMs. Application-consistent Backup Application-consistent backup or transaction-consistent backup ensures the software application on a VM is quiesced and outstanding IO is flushed to storage before a snapshot for backup is taken. Depending on the backup application provider, agents or utilities may need to be installed on VMs to ensure application consistent backups. For example, backup applications such as Symantec NetBackup or EMC NetWorker would require agents to be installed on a VM for application consistency. As another example, a Veeam Backup Server does not require any persistent agents installed on the VM but rather it uses its patented Application-Aware Image Processing (AAIP) to request Microsoft VSS to create application/transaction consistent snapshots. Regardless of the VADP transport mode selected for backup, VMware tools installed on VMs allows backup applications to perform application-consistent quiescing. Do: Follow your backup application vendor s best practices for installing necessary tools in guest VMs for creating application consistent snapshots for backups. Important Note: Regardless of the backup application provider, be aware that subsequent backup of thin-provisioned disk can be larger than expected even with CBT enabled. This is because applications perform random IO to previously unallocated areas. Performing disk defragmentation could help to reduce the size of backups for thin-provisioned VMs before performing a backup operation. Backup and Recovery Best Practices Page 10

Guest-Level Backup Guest-Level Backup Guest-level backup requires installing a backup agent of the backup application provider inside VMs to perform a traditional backup. This is similar to performing the backup of a physical server. Guest-level backup provides the same familiar methodology used to protect physical servers for protecting virtualized servers and the applications in those VMs. For example, backup application s Microsoft Exchange backup agent can be used to backup and protect a Windows VM with Microsoft Exchange server. The advantage of this backup method is that most enterprise backup application providers also support client-direct backups (i.e. EMC NetWorker 8.x and Symantec NetBackup 7.x). This means that the Exchange administrator or the SQL Server administrator or the Oracle DB administrator can manage his or her own backups by taking control of the entire backup and restore operations. Figure 3-1: Guest-level backup. Backup and Recovery Best Practices Page 11

Data Recovery Data Recovery Backup is only one part of the equation in data protection. Data recovery from backups is an important aspect of a backup solution, as backup solutions are only as useful as the ability to recover data from backups that were previously created. Data that has been backed up has to be recoverable for restores. Using image-level backups, most enterprise backup solutions can perform the following types of restores: Recover an entire VM from a snapshot. Recover an entire VM. Recover a VMDK and configuration files. Restore individual files and directories. Perform Instant recovery. Recovery of an entire VM from a Tintri snapshot is as easy as the virtualization administrator browsing the Tintri VMstore, selecting the VM and adding the VM to the vcenter inventory. SnapVM and ReplicateVM are very useful remote management features that allow VMs to be protected and easily recovered within a VMware infrastructure when data protection is important to business continuity. Recovery of a VM restores an entire VM to the original location or to an alternate location, including alternate ESX/ESXi server, as defined within the restore operation parameters in the backup software. VMDK and configuration file recovery allows a new VM to be created using the restored VMDK files. Being able to restore individual files and directories from an image backup is an advantage and avoids restoring an entire VM just to recover a single file or a directory. Each backup application provides its own mechanism for recovering individual files and directories from an image backup. For example, Veeam provides granular recovery for restoring individual files or a directory. As another example, Symantec NetBackup s support of application-aware backup allows granular recovery of individual mailboxes for Exchange server running on a VMware VM. Instant recovery of a VM provides near instant availability of a VM from a backup. For example, this functionality is supported by backup applications such as CommVault s Simpana Software suite, Veeam Backup Server, Symantec NetBackup and EMC Avamar. Instant recovery or instant VM recovery provides the administrator the functionality to copy files from a backup without restoring an entire VM. Backup applications such as Veeam also utilize instant recovery to provide instant access to a VM for testing patches or for verification of an application such as Microsoft Exchange. All enterprise backup solutions support backing up and restoring VMware VMs using VADP with multiple transport modes. SAN transport is an efficient transport for thick-provisioned disk backups; but because of round trips through the disk manager APIs, AllocateBlock and ClearLazyZero, SAN transport for thin-provisioned disk restoration is slower than NBD/NBDSSL. Do: It is recommended to use NBD/NBDSSL transport for thin-provisioned disk restores. Backup and Recovery Best Practices Page 12

Summary Summary All enterprise backup applications support backing up VMware environments using vsphere Data Protection APIs (VADP). Additionally, backup application vendors have innovated additional features for backing up and restoring large VMware environments. For example, Veeam, Symantec, CommVault and a long list of other backup application providers also provide vsphere Client plugins for integrated backup and restore management in modern virtualized data centers. There are many backup applications that support VMware VADP. Not all of the backup application providers were mentioned in this document. Any backup application that supports VMware VADP with HotAdd, NBD or NBDSSL transport modes is recommended for data protection and disaster recovery of VMs running on Tintri VMstore systems. Refer to VMware VADP for a list of backup application providers that support VADP. This is not a comprehensive list. It is recommended to consult with your backup application provider for more information on VMware VADP integration. Copyright 2014 Tintri, Incorporated. All Rights Reserved. Tintri believes the information in this document is accurate as of its publication date. The information in this publication is provided as is, and is subject to change without notice. Tintri makes no representations or warrantees of any kind with respect to the information in this publication, and specifically disclaims any implied warranties of merchantability or fitness for a particular purpose. Tintri, Tintri VMstore, Tintri Global Center and Tintri Zero Management Storage are trademarks or registered trademarks of Tintri, Inc. All other trademarks or service marks are the property of their respective holders and are hereby acknowledged. Backup and Recovery Best Practices Page 13