Disaster Recovery Solutions With Virtual Infrastructure: Implementation andbest Practices

Size: px

Start display at page:

Download "Disaster Recovery Solutions With Virtual Infrastructure: Implementation andbest Practices"

Kristian Sanders
5 years ago
Views:

1 Disaster Recovery Solutions With Virtual Infrastructure: Implementation andbest Practices Govindarajan Soundararajan, VMware Siva Kodiganti, VMware Lokesh Krishnarajpet, VMware

2 Disaster Recovery Sessions (VMWorld 2005) SLN104-A: Implementation and Solutions SLN104-B: Backup and Recovery SLN104-C: Panel Discussion Other sessions LAB003: Backup and Disaster Recovery

3 Agenda Introduction Disaster Recovery (DR) DR process/methods ESX DR Backup and restore (virtual machines, COS, ESX Server) High availability, SAN, multipathing, and clustering

4 Quiz If your data center is impacted by a disaster: Which data (or servers) would you recover first? Payroll/intranet/support/externa l website/on-line store (This slide was picked from a disaster recovery presentation given by a professor of an university)

5 Introduction A real world example of a disaster impact on a business unit Types of disasters Why DRB? Disaster mitigation Backup and recovery High availability (SAN, multipathing) and clustering DRB process

6 A Real World Example of a Disaster Impact on a Business Unit (This is a general example and is not related to VMware in any way) A popular auctioneer website experienced ~20 hours downtime causing $3 million to $5 million decrease in their quarterly revenue Reason was that they had not applied the most recent critical patch

7 Types of Disasters Natural Tornado, hurricane, flood, earthquake, fire Man-made Accidents, operation error, theft, virus attacks, etc. System and facility failures Hardware failures Software failures Power, A/C failures

8 Why DRB? Anticipate Problems Hardware and Software fail Disasters will happen Accidents happen People make mistakes Be proactive Minimize disruptions Stay in business

9 Disaster Mitigation Disaster mitigation are preventative measures to protect against disaster and/or to minimize impact should one occur. It should include: Monitoring web sites on your operating systems, applications and hardware Keeping your OS and application patches up to date Keeping your BIOS and firmware up-to-date Monitoring security web sites Establishing a DR plan and implementing it

10 Backup and Recovery Planning Preparing for: Day-to-day mishaps Subsequent disaster recovery plans Ensure: Servers are backed up on schedule Recovery works on both source and target hardware Backup and recovery process is documented

11 High Availability Redundant features in hardware and software systems to facilitate business continuity in the face of disaster A backup node will take over should the primary node fail Implementations Clustering Fault-tolerant components Shared storage

12 DRB Process Plan Identify assumptions Determine Service Level Agreements (SLAs) and critical servers with priorities Compare price of disaster recovery setup vs. price of disaster Implement Setup hardware and software Configure servers, applications and scripts Test Test each component for failure and fault tolerance As data environment changes, review and revise plan, implement and test

13 ESX Server Disaster Recovery Backup and restore and best practices VMware backup and restore tools and best practices High availability Clustering

14 Backups Types of backups Incremental Backup of files changed since last backup Best for application data Differential Backup of files changed since last full backup Alternative to Incremental requiring less tapes typically Full backup Backup of all files and directories on a system Best for full recovery after a system failure

15 Backups Hot backup Machine is running at time of backup Examples: Guest OS backup agents VMsnap scripting Cold backup Machine is powered off or suspended at time of backup Examples: Manual virtual machine cloning (copy config, nvram, and disk files) VMware Virtual Center cloning

16 Backup Strategies From Guest OS Allows incremental, differential backup From Host/Service Console Full disk image only, including system files Application data can be archived without shutdown Virtual machine power-off required unless using redo logs Archived system data can only be restored into a running virtual machine Easy to boot a restored virtual disk Must install agent software in each virtual machine May need service console disk space for exported disks Best for application data Best for system image

17 Backup Best Practices Guest OS backups for daily recovery needs such as file or directory restores Virtual machine snapshots (disk images + config files) through service console to remote tape for quick bare metal restore of a virtual machine ESX Server attached tape device for service console backup only (/boot, /, /home and optionally /vmimages)

18 Backup Best Practices Non-ESX Server attached tape devices for guest OS backups, virtual machine snapshots and/or dsk backups from service console Archive backup tapes to offsite storage Perform test restores to validate backups

19 ESX Server Best Practices Set up service console disks on an internal RAID-1 (two mirrored disks) Set up VMFS on SAN storage with redundant SPs, HBAs and FC switches. (Alternative: RAID-5) Backup your service console non-vmfs disks with standard backup agents Treat the service console as a distinguished virtual machine Have your backup server on a dedicated piece of hardware

20 ESX Server Recovery DR plan should ensure these key items are available at recovery location ESX Server disk License keys ESX Server configuration and customization Printout of virtual machine configurations Printout of other configurations Backup media of images.vmdk or.dsk,.redo and.redo.redo files.vmss files Installation media for recovery software

21 Backing up ESX Server Custom configurations Monitoring tools Custom settings and additional software User and system information User and group information in /etc. System configuration information in /etc. ESX Server and configuration files Boot partition /etc/vmware directory Virtual machine configuration files and nvram files Extras: vmkusage, reports, esxtop

22 VMware Backup Tools vmsnap.pl (ESX Server 2.x and higher) Sets up redo logs and creates a hot backup by creating a set of.vmdk disk files from their corresponding.dsk disk file. The resulting.vmdk disks and related configuration files are automatically sent to an archival server or to a hot standby server to restore the virtual machine Add a redo log to dsk file. If more than one dsk then perform for each dsk file Export dsk file to vmdk file format

23 VMware Backup Tools vmsnap.pl Transfer vmdk file(s), virtual machine configuration file, nvram file and log files to an archive server or to a local storage location Add second redo log Commit first redo log followed by commit of second redo log

24 VMware Backup Tools vmsnap.pl Do not rename vmdk files. Since the conversion of one dsk file can result in multiple vmdk files, the first vmdk file stores the names of the additional vmdk files Vmdk files created from ESX Server 2.0 systems are not compatible with restoration on ESX Server systems

25 VMware Backup Tools vmsnap.pl Creation of redo files do not affect running virtual machine performance. The first commit also does not affect running virtual machine performance. The second commit results in a brief pause, typically less than 1 second, of the virtual machine The virtual machine disks should be shrunk before using vmsnap. This is to minimize the total size of the resultant vmdk files. Currently this must be done manually from within the virtual machine using the VMware Tools

26 VMware Backup Tools vmsnap.pl Should the scripting be aborted, it is possible that a REDO and/or a REDO.REDO log may remain for a particular disk. If the -r option of vmsnap does not do the commit correctly you may need to choose to restart the guest virtual machine and accept the commit requests that are presented in a pop-up window. This will ensure that when the machine comes up again there will be no redo logs

27 VMware Restore Tools vmres The vmres script is used to restore a virtual machine snapshot created with the vmsnap script The script will retrieve all snapshot files of the original virtual machine from an archive server. This includes vmx, nvram, log and vmdk files Configuration files are copied into the specified owner s vmware configuration directory The vmdk files are converted to dsk files and stored in the specified VMFS directory The virtual machine configuration file is updated, permissions are set and the virtual machine is registered

28 Backup Best Practices Backup of guest OS using current backup software and scheduling to allow for file level and directory level restores Virtual machine snapshots of guest OS using API scripting for hot backups to speed recovery of a virtual machine in the event of a disaster. Once virtual machine is setup using snapshot, guest OS backups can be used to fully recovery remaining changed data on the system since the snapshot timeframe

29 ESX Server Repair In the event of MBR failure on your ESX Server, the following method can be used to perform a repair to the system: Boot off a Red Hat 7.2 CD #1 When the LILO prompt comes up, enter Linux rescue Proceed through the screens When you get a shell prompt, enter: chroot /mnt/sysimage /sbin/lilo -c -v The system will reboot This will recover your MBR

30 High Availability: SAN Advantages of SANs Ease of moving virtual machines between ESX Servers Performance High availability and redundancy High performance backups and restores and use of snapshots Secure and robust data transfers Dual-redundant SAN fabrics, redundant HBAs and storage ports Metropolitan Area Network (MAN) and Wide Area Network (WAN) support Management Data replication Disaster recovery Dynamic expansion (easily scalable: ESX Server 2.0 support for disk extents)

31 SAN: Multipathing Multipathing Maintain a constant connection between server and storage in case of a failure of HBA, switch, or storage controller mru, fixed policies for path failover Verify status of available paths using: vmkmultipath -q

32 ESX Server and SAN Best Practices Buy SAN equipment that is on the supported hardware list Understand your SAN configuration and equipment operation prior to connecting an ESX Server to it. Make sure your ESX Server administrators and your SAN administrators communicate Test with non-production virtual machines first to validate configuration is correct Ensure that the cabling order is correct on all fabric switches Space reboots of ESX Servers accessing the same LUNs by at least 10 minutes to ensure correct setting of active paths to preferred paths

33 ESX Server and Clustering Clustering Support Clustering is done via clustering software, and is not inherent in the VMware ESX Server Applications running in virtual machines need to be cluster-aware Requires VMFS virtual disks to be on a shared volume Refer to VMware ESX Server Administration Guide for clustering examples

34 ESX Server and Clustering How clustering works: Application fails Backup application identifies failure and takes over Clustering on virtual machines on the same physical server protects application failures Cluster across different physical servers for added redundancy Use of clustering services in virtual machines provides high availability with less hardware (such as machines and network adapters)

35 Clustering Scenarios in ESX Server Cluster in a box Multiple virtual machines on a single physical machine To deal with software crashes or administrative errors Supports heartbeat network without any extra network adapters

Clustering Scenarios in ESX Server Cluster across boxes Consists of virtual machines on multiple physical machines Can deal with the crash of

36 Clustering Scenarios in ESX Server Cluster across boxes Consists of virtual machines on multiple physical machines Can deal with the crash of a physical machine Placed on a SAN with shared LUN storage (disks are shared ) Heartbeat across VMNIC interfaces Shared storage required (SAN)

Clustering Scenarios in ESX Server Cost-effective standby host Provides a standby host for multiple physical machines on one standby box with multiple virtual machines

37 Clustering Scenarios in ESX Server Cost-effective standby host Provides a standby host for multiple physical machines on one standby box with multiple virtual machines Physical machine clustered with a virtual machine Physical machine is primary, virtual machine is backup Heartbeat across VMNIC interfaces Shared storage required (SAN)

38 Clustering Best Practices Do not use undoable mode for disks used in clusters Applications must be stateless or have failover support ESX Server supports SCSI Level 2 reservations. Used by Microsoft Cluster Services and Veritas Cluster Services Use shared mode for VMFS when setting up clustering configurations. Allows controlled simultaneous access to files Have separate LUNs for data and quorum Use separate LUN for each cluster Clustering between virtual machines across ESX Servers Specify scsi1.sharedbus = physical in each virtual machine s configuration file Use physical names for shared VMFSes rather than friendly names scsi1:1.name = vmhba0:1:0:1:quorum.dsk scsi1:2.name = vmhba0:1:0:1:shared.dsk Designate the VMFS as shared using the MUI or: vmkfstools F shared vmhba0:1:0:1 Public net should always be on a bond, not a vmnic, for fault tolerance Heartbeat should never be on a bond for timing reasons (prevents full failovers). This is sometimes called split braining the cluster See VMware white papers on clustering or attend ESX Server: Advanced System Management course for training

39 VMware Documents VMware Backup Software Guide ESX Server VMware Backup Software Compatibility Guide VMware ESX Server Backup Planning Technical Note VMware ESX Server and Clustering White Paper VMware Backup Architecture VMware Backup Planning VMware Disaster Recovery Using Veritas Backup Exec 9 ESX Server 2 and Storage Area Networks ESX Server Administration Guide

40 Web Sites VMware Web Sites (disaster recovery) (White paper on backup) (One of customer setup) Other Web Sites There are numerous websites on the Internet (search on Web Site with disaster recovery or business continuity as keywords)

Disaster Recovery Solutions with Virtual Infrastructure: Backup and Recovery. Paul Nashawaty Session SLN104B

Disaster Recovery Solutions with Virtual Infrastructure: Backup and Recovery Paul Nashawaty Session SLN104B What to Expect from This Session An understanding of how to protect your data in virtualized