Creating the Fastest Possible Backups Using VMware Consolidated Backup. A Design Blueprint

Transcription

1 Creating the Fastest Possible Backups Using VMware Consolidated Backup A Design Blueprint George Winter Technical Product Manager NetBackup Symantec Corporation

2

3 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

4 4 Introducing NetBackup for VMware Flexible Virtual Machine protection methods. VMware VCB Off-Host Backup VMware Incrementals Deduplication Client-Based Backup Backup Server & Storage A ESX Server C A C C D A D D Shared Storage ESX Server ESX Server NetBackup for Backup Server & Storage ESX Server VM or object VMware Granular Recovery ESX Server VM Discovery via Virtual Center Backup Server & Storage Automated Restore via VMware Converter Auto-Discovery and Flexible Recovery D A D ESX Server Deduplication D Backup Server & Storage Datacenter Vault Replication coupled with unique enabling technologies

5 NetBackup for VMware Granular File Restore c.vmdk NetBackup for VMware automatically processes and maps vmdk files during backup. Individual file restores are possible from vmdk backups! Ultimate restore options. Restore single OS file or entire VM (vmdk) from a single backup pass. Direct Virtual Center Integration. NetBackup automated client discovery uses direct Virtual Center integration. No scripts necessary! No writing, maintaining and troubleshooting homegrown scripts Intelligent host remapping. Clients are properly referenced to VM hostname no NetBackup client is required in VM to do this! Find files instantly. NetBackup for VMware maps, indexes and catalogs the individual and VMDK files. Search through 1000 s of backups instantly.

6 NetBackup Supports Fibre, iscsi, NAS Network Based VCB Backups Same backup/restore functionality Granular File Restore NEW! Incremental backups Automatic backup path failover l Supports Fibre & iscsi Fastest possible backup Direct path to Media Server Granular File Restore NEW! Incremental backups Automatic backup path failover

7 What Others Are Saying "NetBackup 6.5 for VMware, we believe, has become the leading full-featured backup product for VMware. Raymond James Financial, Inc. Analyst Report, Oct, 2008 VCB integration is where Veritas NetBackup really differentiates itself from the other products This is a very promising and exciting feature to be sure Ashley D'Costa TechTarget, May,

8 What Others Are Saying All vendors should follow Symantec's lead in offering granular filelevel recovery from single deduplicated VM image-level backups NetBackup is also unique in its support for incremental backups via VCB. Symantec remains the clear leader in x86 VM data protection Chris Wolf, Burton Group The State of VM Backups, Jan,

9 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

10 Basic Backup Server Performance Concepts A backup server is an "I/O Machine" By far, majority of work done is I/O related Performance of disk is key Lots of data in from clients and out to disk or tape A fast CPU is important, but secondary Fast internal bus is key Multiple internal buses desirable Expandability of system also important Arguably cheaper to expand current server than buying an additional server and more software These concepts apply directly to Backup Proxy

11 Other Disk / Array Considerations Q: Disk drives made in which year are the fastest? All of the above 7. None of the above Hint: 8. Can you recommend a different session? Don t consider RPM or Access Time. Instead, consider the relationship of I/O compared to the size of the disk drive.

12 Other Disk / Array Considerations Q: Disks drives made in which year are the fastest? All of the above 7. None of the above 8. Can you recommend a different session?

13 Other Disk / Array Considerations Place data over as many spindles as possible High capacity disks make this less obvious 1993: Number of IOPS per GB in the 1000's IOPS per disk = 50 I/O spread across many disks by necessity 2009: Number of IOPS per GB in the 10's IOPS per disk = 250 Density of data per disk MUCH higher => fewer spindles! I/O spread across fewer disks

14 The "70% Rule" Never exceed 70% of the rated capacity of any component Manufacturer throughput and performance specifications based on theoretical environment - never achieved in real world The 70% rule applies to nearly every component in environment Disk CPU Internal Bus - Network - Memory - Fibre Channel Tape drive is an exception to this rule Response time significantly increases after 70% utilization threshold is exceeded.

15 Throughput vs. Response Time Response Time* (latency ms) As throughput approaches 100%, response decreases to 14% of optimal % 40% 60% 80% 100% Percentage of Maximum Throughput (bandwidth) *Hennessy & Patterson, "Computer Architecture A Quantitative Approach"

16 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

17 Benchmark Location Sun Micro Enterprise Technology Centers Sun Enterprise Technology Centers (ETCs) host large server configurations so software engineers and Sun partners can conduct performance, scalability, and availability testing of software and workloads prior to deployment.

18 Symantec/Sun VMware Benchmark 4+ TB of VM Data Datastore NBU Master (Solaris) Virtual Center Server (W2K3) Largest VM = 100 GB Smallest VM = 20 GB OS = W2K3-8, RHEL (RAID 1+0) ESX1 Separate HBA s on Proxy Datastore (RAID 1+0) ESX2 Backup Proxy (W2K3) 4 Gb Fibre Switch Datastore ESX3 Disk Disk (RAID 1+0) NBU Disk Storage Unit (RAID 1+0) Staging Area (RAID 0) 4 Gb Fibre Datastore (RAID 1+0) ESX4

19 Benchmark Hardware What To Look For Selecting the Backup Proxy hardware Benchmark System Backup Proxy: Sun X4600M2 RAM = 8 GB or more (64 GB) CPU = Modern CPU is fine (2.29 GHz) Keep in mind the Backup Proxy is doing all of the backup-related I/O that would normally be done by each ESX server Purchase this box with I/O in mind lots of expansion slots Multiple and fast internal buses (FSB) CPU, I/O and Memory expandability should be consideration Arguably cheaper to expand current server than buying an additional server and more software

20 Benchmark Hardware What To Look For Backup Proxy Holding Tank (staging) Staging Area: Sun 6140 array RAID 0 across 8 disks 4 GB Cache enabled Single most important performance component on the Backup Proxy Holding tank I/O = simultaneous reads and writes Stripe across as many spindles as possible Does not have to be highly available RAID 0 (never RAID 5!) Create multiple Holding Tanks if possible Never share spindles or file system with other data (fragmentation) Keep under 1 TB if possible

21 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

22 Performance Tuning NBU For VMware - Hardware Before we start, some words of wisdom: The maximum achievable backup performance you can obtain will be dictated by the slowest component in your backup environment.

23 Why And How Do You Baseline? You can t fix something if you don t understand the problem Most common question posed to NetBackup Tech Support related to performance Testing your environment will tell you: Estimate of performance that you can achieve Components that are not working as expected Areas that need reconfiguration

24 Baseline Performance Testing Test Utilities HD_Speed ( Writes are destructive Iometer NetBackup vcbmounter.exe Baseline testing typically performed on pre-production systems Avoid RGE s

25 Three Basic Baseline Tests 1. Read performance of ESX Datastore 2. Read/Write Performance of holding tank 3. NetBackup Storage Unit write performance Datastore VM1 VM2 Disk STU Backup Proxy NBU Disk Storage Unit (RAID 1+0) Datastore 4 Gb Fibre Staging Area (RAID 0) Holding Tank VM3 VM4

26 Baseline Performance ESX Datastore VCB process is read only I/O from the Datastore Datastore VM1 VM2 Disk STU Backup Proxy NBU Disk Storage Unit (RAID 1+0) Datastore 4 Gb Fibre Staging Area (RAID 0) Holding Tank VM3 VM4

27 ESX Datastore Read Test VCB process only reads data from Datastore Use HD_Speed to determine read performance 220 MB/sec per Datastore Both Fibre and Datastore performance are measured Either could cause performance issue Drivers, bad HBA, etc. could also be issues Measured performance running 4 simultaneously: 94 MB/Sec per stream (?)

28 Baseline Performance Holding Tank Holding Tank performance is key Performs simultaneous I/O reads and writes Datastore VM1 VM2 Disk STU Backup Proxy NBU Disk Storage Unit (RAID 1+0) Datastore 4 Gb Fibre Staging Area (RAID 0) Holding Tank VM3 VM4

29 Holding Tank Asynchronous Read and Write Use HD_Speed to determine read & write performance Holding tank (A.K.A. Staging Area) most likely bottleneck Read = 440 MB/sec (220 MB/sec x 2) Write = 170 MB/sec (85 MB/sec x 2) Keep in mind, this measured performance is not synchronous We used two staging areas to improve perf

30 Baseline Performance NBU Storage Unit Backup data sent to NetBackup Storage Unit Data read from Holding Tank, written to Storage Unit Datastore Tape is same VM1 VM2 Disk STU Backup Proxy NBU Disk Storage Unit (RAID 1+0) Datastore 4 Gb Fibre Staging Area (RAID 0) Holding Tank VM3 VM4

31 NetBackup Storage Unit Performance Performance reported by HD_Speed Two Disk Storage Units: 1) Write perf = 130 MB/sec 2) Write perf = 85 MB/sec Two arrays total of 20 spindles Write = 215 MB/sec (combined) This can be deduplication target PureDisk vmdk deduplication enhanced (Q2) Extremely efficient deduplication of vmdk s

32 Baseline Performance Summary These performance numbers are independent of each other Where is the bottleneck? What does this not tell us? Datastore VM1 VM2 215 MB/sec Disk STU Backup Proxy 376 MB/sec Datastore 376 MB/sec 170 MB/sec 4 Gb Fibre Holding Tank VM3 VM4

33 Benchmark Results Single Backup Proxy = VM1 VM2 ESX Server GB / hour Translates into 63.1 MB / Sec This performance includes entire backup process Creating & deleting snapshot All transfers to Holding Tank and NBU Storage Unit

34 Benchmark Results - Lessons Learned Careful planning can ensure and improve performance Design Consideration: During VCB backups, 4 separate I/O operations are occurring: 1. Reading data for ESX storage 2. Writing this data to Holding Tank 3. Reading data from Holding Tank (to backup destination) 4. Writing data to backup destination All 4 operations typically use same internal bus Backup Proxy separate HBA s for each device Likely bottleneck = Holding Tank I/O & internal bus

35 Benchmark Results - Lessons Learned Expand Backup Proxy Capacity Create multiple Holding Tanks (on separate spindles) Use multiple internal buses Each device connected via separate HBA Set simultaneous snapshots only as high as needed Too many causes unnecessary I/O contention Does not improve performance NetBackup cache settings were tested New cache settings improved Holding Tank performance +30% New cache settings included in release (Q2)

36 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

37 Impact of Backups on ESX Server Impact on ESX Server (CPU) Not measureable Impact on ESX Storage Hardly see snapshot creation/deletion This will vary depending on how expensive VM quiesce operation is This is why VCB is used

38 CPU and Memory Load on Backup Proxy This measured during maximum load CPU utilization very low Memory use is just over 2 GB

39 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

40 NetBackup Configuration Tips Policy Creation Control flow of data from ESX Datastore Create single NetBackup Policy per ESX Datastore Stream from multiple Datastores Set jobs per policy only as high as hardware can support Minimizes I/O impact on Datastore Backup Proxy

41 NetBackup Configuration Tips - Incremental B/U s Standard vmdk backups effectively always full backups Creates significant I/O and requires lots of backup storage Sat Sun M Tu W Th F Sat Sun M Tu W Th F Sat Sun

42 NetBackup Configuration Tips - Incremental B/U s NetBackup introduced file level backups Possible because of Granular File Restore capability File level incremental based on vmdk full Extends capability of Backup Proxy to protect 10 TB+ of data Fast only files that are backed up are copied thru Backup Proxy 100 GB incremental (10%) takes 7 minutes (entire process) Sat Sun M Tu W Th F Sat Sun M Tu W Th F Sat Sun

43 Agenda Overview NetBackup for VMware and VCB Backup Hardware & Systems Performance Review Benchmark System Configuration Baseline Performance & Results VCB Performance Advantage NetBackup Configuration Tips What s next? Q&A

44 NetBackup and Beyond planned enhancements include Superior vmdk deduplication with PureDisk Enhanced virtual machine reporting Usability enhancements Additional hypervisor support vstorage API Support Supported (VCB 1.5) with the NetBackup release Enhanced support (vstorage) later this year (Q4) vsphere support a priority NetBackup upgrade path extremely simple

45 For More Information.. Best Practices doc (updated with benchmark lessons learned) Links to technical documentation Lots more.

46 Q & A George Winter Technical Product Manager NetBackup george_winter@symantec.com Symantec Corporation

47 Thank you for coming. Rate your session and watch for the highest scores!