Tivoli Storage Manager for Virtual Environments

Transcription

1 Tivoli Storage Manager for Virtual Environments Eric Stouffer IBM Advanced Technical Skills

2 TSM Advanced Technical Support Team Dave Canan Dave Daun Tom Hepner Eric Stouffer 2

3 Topics Background and overview vstorage API for Data Protection (VADP) Backup method Backup components Installation and configuration Components involved Backup specifics Recovery specifics Scalability Considerations Sizing Deduplication Considerations Performance Considerations 3

4 Background and Overview 4

5 TSM for Virtual Environment Goals Requirements Make backups more efficient to have less impact on production VMs Centralize backup and automatically discover / backup new VMs Allow backup and restore to be offloaded from ESX/ESXi host Allow backup and restore to directly access Datastores over the SAN Support all recovery scenarios (file, volume and image) from a single source backup Support the significant improvements provided with VMware vsphere 4 using the vstorage APIs for Data Protection (VADP) Provide a simple to use GUI that is an extension of VMware s vsphere client interface with similar look and feel characteristics Exploit hardware-base disk snapshots 5

6 TSM for VE Overview 6

7 Components TSM Components: vstorage Backup Server ( backup proxy / Datamover ) TSM Server Data Protection for VMWare vcenter plug-in VADP (VMWare API for Data Protection) VMWare Components: vcenter Server ESX/ESXi hosts 7

8 Backup methods In-guest backup On-host backup Off-host backup 8

9 In-Guest Backup Philosophy: treat virtual machines as physical machines TSM Agents placed in each guest; traditional backup over LAN Guest vcenter Host (ESX, ESXi) Cumulative backup load on host hard to measure Result: Individual VM guest files stored in TSM server for operational recovery LA N VMFS on ly NAS/local files SAN TSM Storage Hierarchy This method is generally supported by TSM for VMware ESX/ESXi, Microsoft Hyper-V, Linux KVM TSM Server 9

10 On-Host Backup Philosophy: treat hypervisor as general purpose OS Optional: manual procedure to backup data from a snapshot; no integrated solution for making data consistent Guest Host (ESX, ESXi) TSM Linux agent placed on ESX host; user selectively backs up data stores at directory level ESX only; VMware moving away from ESX VMDK VMFS NAS/local LAN or S AN SAN Result: Individual files representing virtual disks and control files stored in TSM server vmdk(s) TSM Storage Hierarchy This method is generally supported by TSM for VMware ESX and Microsoft Hyper-V 10 vcenter TSM Server

11 Off-Host Backup Philosophy: exploit access to shared storage from another host Guest vcenter Backup Agent orchestrates snapshots of virtual machines and moves data from snapshot to backup storage Host (ESX, ESXi) VMFS VMDK NAS/local LAN or SAN SAN vmdk(s) TSM Storage Hierarchy Can also exploit hardware-based snapshots as backup versions 11 Result: blocks representing virtual machine stored in server for DR of virtual machine Backup Agent TSM Server

12 VCB replaced by vstorage API Vmware Consolidated Backup (VCB) Made obsolete by Vmware s vstorage API Requires VCB Framework product installed on backup proxy Complete export and copy of virtual machine volume data and VM configuration /metadata Volume data exported and compressed *.vmdk files broken up to 2G files Files are copied to staging directory before TSM read and backup Restores are a two step process requiring Vmware converter vstorage API 12 VCB not required Direct reading of *.vmdk volume data and configuration data Content aware (i.e. used blocks) No staging this is a one step backup or restore process

13 vstorage APIs for Data Protection Discovery and Mapping Based on backup request, TSM agent queries host or vcenter to find virtual guests that are backup candidates, e.g., all guests on an ESX host or all guests in a specific folder Guest vcenter Host (ESX, ESXi) VMFS VMDK NAS/local vsphere SDK SAN TSM Server vstorage Backup Server (TSM datamover agent) 13

14 vstorage APIs for Data Protection Snapshot and Preparation Guest 2. Virtual machines given control for backup event to put data in a consistent state either through Microsoft Volume Shadow Copy Services (VSS), custom scripts or VMware sync driver Host vcenter (ESX, ESXi) VMDK VMFS NAS/local vsphere SDK SAN 1. TSM invokes snapshots of guest machines through vsphere SDK; saves machine configuration information 3. VMware takes software snapshot, new.vmdk created, virtual machine passed backup complete event vstorage Backup Server (TSM datamover agent) TSM Server

15 vstorage APIs for Data Protection Data Movement TSM agent reads data directly from snapshot VMDK and sends data to TSM server Guest Change block tracking feature allows to backup only changed blocks since last backup OR only used blocks on first backup Host vcenter (ESX, ESXi) VMDK VMFS NAS/local Virtual Disk Development Kit (VDDK) SAN Snapshot VMDK accessed via SAN, LAN, or HotAdd transport 15 vstorage Backup Server (TSM datamover agent) TSM Server

16 vstorage APIs for Data Protection Termination Guest vcenter TSM Server contains blocklevel image of guest machine(s) which can be used for operational recovery of individual files or DR of entire machine Host (ESX, ESXi) VMDK VMFS NAS/local TSM Storage SAN Hierarchy Software snapshot removed 16 vstorage Backup Server (TSM datamover agent) TSM Server

17 vstorage APIs for Data Protection Recovery 1. Guest machine is redefined by creating new machine with same properties (vsphere SDK) Guest 2. VMDK files written back to virtual machine (VDDK) vcenter Host (ESX, ESXi) VMDK VMFS 1 NAS/local vsphere SDK SAN 2 VDDK vstorage Backup Server (TSM datamover agent) 17 TSM Server

18 Key Product Differentiators Centrally managed avoid deployment in each VM guest Automatic discovery and backup of virtual machines Non-disruptive snapshots at virtual machine level Scalable by introducing multiple vstorage backup servers Space saving through incremental block-level backups that are Changed Block Tracking (CBT) aware Multiple Recovery Options Granular file-level recovery Near-instant volume recovery (non-os volumes) Full virtual machine recovery Leverage functionality that exists in TSM Compression/deduplication LAN or SAN (Lan-free) paths are available to datamover Single unified solution for both physical and virtual machines 18

19 TSM Data Protection for VMware Components 1. Backup-Archive Client Data Mover Full + Incremental backup (via CBT) - Incremental enabled via license file Full VM restore vstorage Backup Server (VBS) Machine where Backup-Archive client installed Physical or virtual machine Windows or Linux 2. DP for VMware Recovery Agent Individual file recovery for Windows and Linux guests Near-instant volume recovery for Windows and Linux guests vcenter Plug-in TSM extension to VMware vsphere client GUI. Management of full VM backups (including scheduling) / full VM restores operations for multiple B/A client data movers

20 Data Protection for Vmware Plug-in 20

21 TSM B/A Client 21

22 Installation and Configuration 22

23 TSM Client and VADP Support 23

24 Data Protection Plug-in and Recovery Agent 24

25 Configuration of the vsphere client plug-in 25

26 Scheduling Recommended Starting Point Best Practices Use asnode to allow multiple data movers (i.e. dsmc instances) to store VMware backups under common datacenter nodename Set backup domain for each data mover to protect an ESX host Allows automatic discovery of new virtual machines Process one VM at a time for each ESX host Scheduling Recommendations: Use batched full or rotating full Batched full Full VM backups during extended time (e.g., weekend), incrementals during weekdays Easiest approach to configure / minimizes schedule definitions Backup window considerations / does not optimize usage of vstorage backup server Rotating full Full and incremental backups are interleaved each day Distributes backup workload/window throughout the week More complex scheduling Schedule parallel data movers on vstorage Backup Server per sizing guidelines TSM Data Protection for VWware Schedule Recommendations Whitepaper: ronments 26

27 Example naming convention The first level bullet is 28 points The second level bullet is 24 points The third level bullet is 20 points The fourth level bullet is 18 points 27

28 Scheduling Example Each data mover is a proxy agent which moves data from a ESX host and has its own node name 28 Each data mover stores data on behalf of a proxy target which represents a datacenter.

29 Scheduling Configuration vstorage Backup Server Configuration Create separate TSM client options file for each data mover Specify data mover nodename used for authentication and schedule association. Create a CAD-managed TSM schedule service for each data mover TSM Server Configuration Node definitions Register data center target node used to store backup data for VMs Register data mover nodes and grant proxy authority to data center target node Register tsmcli and vcenter nodes Schedule definitions: vcenter plug-in Directly issue TSM admin commands TSM server node topology 29

30 Backup Details 30

31 Data Path Considerations Transport is independent of Backup Path VDDK picks best transport in order of SAN, Hotadd, nbdssl, nbd Can override VMware transport with vmvstortransport Backup-Archive client option Guest Hotadd transport not available for physical vstorage Server SAN Backup Path not available when using virtual vstorage Server Host TSM Storage (ESX, ESXi) Hierarchy VMDK VMFS SAN Transport Backup Path Datastore -> vstorage Server vstorage Server -> TSM Server SAN, Hotadd, SAN nbdssl, (LAN-free), TCP/IP nbd vstorage Backup Server (B/A Client Datamover) 31 TSM Server

32 vstorage Transport Methods NBD Data is transported across the LAN from the ESX/ESXi host to the vstorage Backup Server The network data stream is not encrypted NBD / SSL The backup data flows across the LAN but it is encrypted with SSL between the ESX/ESXi host and the vstorage Backup Server SAN The LUN containing the is zoned or masked to allow both the ESX/ESXi host and the vstorage Backup Server to access the virtual machine files for backup purposes No significant amount of data does has to be sent between the ESX/ESXi host and the vstorage Backup Server HOTADD vstorage Backup Server must be a virtual machine VMDK disks will be attached to the vstorage Backup Server allowing direct access 32

33 DP for VMware Data Representation File Types Guest Disk Storage Pool MBLK0000 OVF: machine definition (1) MBLK0001 MBLK0002 DISK 0 Host JOB1 JOB1 JOB1 MBLK0000.DAT MBLKnnnn.CTL BITMAP.DAT JOB2 (ESX, ESXi) MBLKnnnn JOB2 JOB2 VMDK \vmname.ovf CTL/DAT: 73K control MB data BITMAP.DAT MBLK0000.CTL MBLK0003.CTL \vmname.ovf VMFS BITMAP: master bitmap for image (1) JOB1. JOB1 MBLK0000.DAT MBLKnnnn.DAT JOB2 JOB2 MBLK0000.DAT MBLK0003.DAT JOB2 = incr. backup (fewer CTL/DAT files) SAN TSM Storage Hierarchy 33

34 Tape Configuration TSM Storage Pools Disk Storage Pool = Meta-Data Guest MBLK0000 JOB1 JOB1 JOB1 MBLK0000.CTL MBLKnnnn.CTL BITMAP.DAT MBLK0001 JOB2 MBLK0002 JOB2 MBLK0000.CTL MBLK0003.CTL JOB2 BITMAP.DAT VMCTLMC Sizing: ~1% of data backed up DISK 0 Host (ESX, ESXi) MBLKnnnn Tape Storage Pool = Data \vmname.ovf VMDK VMFS \vmname.ovf SAN JOB1. JOB1 MBLK0000.DAT MBLKnnnn.DAT JOB2 JOB2 MBLK0000.DAT MBLK0003.DAT VMMC TSM Storage Hierarchy 34

35 Tape Configuration Tape Policy (physical tape and VTL) VMMC Placement for data (copygroup DESTINATION) Retention for data and control files (copy group VEREXISTS AND RETAINEXTRA) VMCTLMC Placement for metadata (copygroup DESTINATION) Best Practice Recommendations Disk to Physical Tape Migration Set the MIGDELAY parameter for the disk storage pool to enable the majority of mount requests to be satisfied with snapshots that are on disk (e.g. 10 days) For device class file storage pool consider collocation by file space (potentially allows data from multiple incremental backups for a VM to be saved on same volume (disk file). Migration to tape processes volumes and could have the affect of collocating multiple incremental backups on same physical tape. DR Considerations Backup metadata storage pool (re. VMCTLMC) to separate copy storage pool. For DR, recreate metadata disk storage pool (restore stgpool) before restoring any VM data. 35

36 Recovery Details 36

37 Data Protection for VMware Recovery Agent Mount Quickly mounts a snapshot as a partition or iscsi target device Mounts snapshots without actually restoring the files to guest Instant Restore Restores volume to guest (non-os volume) Enables on demand access to data while the volume is being restored Supports Centralized and Self-Service File Restore Request Centralized Agent installed on central off-host (or VM guest) machine VMware administrator, TSM administrator or help desk personal initiates process Self-Service 37 Agent installed in Windows or Linux VM guest machines User logged into the VM guest machine initiates process

38 Data Protection for VMware: File Recovery Centralized File Recovery from Partition Mount (Windows and Linux) Recovery Process Central Admin (help desk, TSM or VMware admin) 1. Central Admin launches Windows or Linux Data Protection for VMware Restore Agent a. Enters TSM node name for authentication with TSM server b. Select VM, recovery point and disk c. Select partition mount / partition Volume is exposed to off-host system 2. Central admin exposes share to VM guest User 3. Guest user accesses share 4. Guest user copies files (files restored from TSM storage pool) Central Admin VM Guest User vstorage Guest Backup TSM Server Server Host (ESX, ESXi) Agent Deployment Centralized install of Windows / Linux DP for VMware Recovery Agent Linux package requires Windows DP for VMware recovery agent (i.e. access iscsi target exposed from Windows) 38 VMFS VMDK NAS/local SAN

39 Data Protection for VMware: File Recovery Self-Service File Recovery from Partition Mount (Windows and Linux) Recovery Process VM Guest User 1. User launches Windows or Linux Data Protection for VMware Restore Agent to mount partition a. Enters TSM node name for authentication with TSM server b. Selects VM, recovery point and disk c. Selects partition mount / partition Volume is presented to local system 2. User copies files (files restored from TSM storage pool) VM Guest User Guest TSM Server Agent Deployment Host Linux and Windows install DP for VMware agent in-guest for VM user initiated restore In guest Linux package requires offhost Windows DP for VMware recovery agent to expose iscsi target 39 (ESX, ESXi) VMFS VMDK NAS/local SAN

40 Data Protection for VMware: File Recovery Centralized File Recovery from iscsi Mount (Windows and Linux) Recovery Process Central Admin (help desk, TSM or VMware admin) 1. Central Admin launches Windows Data Protection for VMware Restore Agent to expose iscsi target disk a. Enters TSM node name for authentication with TSM server b. Selects VM, recovery point and disk c. Selects mount iscsi target and expose to VM guest iscsi initiator iscsi target is exposed to VM guest VM Guest User 2. Guest user invokes iscsi initiator to access iscsi target 3. Guest user copies files (files restored from TSM storage pool) Central Admin VM Guest User vstorage Guest Backup TSM Server Server Host (ESX, ESXi) Agent Deployment Widows install package (typically installed on vstorage backup server) 40 VMFS VMDK NAS/local SAN

41 Data Protection for VMware: File Recovery Self-Service Instant Restore of Volume (Windows and Linux) Recovery Process VM Guest User 1. User launches Windows or Linux Data Protection for VMware Restore Agent to mount partition a. Enters TSM node name for authentication with TSM server b. Selects VM, recovery point and disk c. Selects Instant Restore Volume is restored to guest with on-demand access VM Guest User Guest TSM Server Agent Deployment Host Linux and Windows install packages in-guest for VM user initiated restore In guest Linux package requires offhost Windows DP for VMware recovery agent to expose iscsi target 41 (ESX, ESXi) VMFS VMDK NAS/local SAN

42 Scalability Considerations 42

43 Data Protection for VMware: File Recovery Self-Service Instant Restore of Volume (Windows and Linux) Recovery Process VM Guest User 1. User launches Windows or Linux Data Protection for VMware Restore Agent to mount partition a. Enters TSM node name for authentication with TSM server b. Selects VM, recovery point and disk c. Selects Instant Restore Volume is restored to guest with on-demand access VM Guest User TSM Server Guest Agent Deployment Host Linux and Windows install packages in-guest for VM user initiated restore In guest Linux package requires off-host Windows DP for VMware recovery agent to expose iscsi target 43 (ESX, ESXi) VMFS VMDK NAS/local SAN

44 Overview of the Sizing Process Choose a Backup Plan 1. Choose a backup plan. 2. Calculate required Aggregate Throughput (total data to backup / backup window) 3. Calculate required Number of Concurrent Backup Processes (Aggregate Throughput / Estimated Per-Process Throughput) 4. Calculate the Number of vstorage Backup Servers (Number of Concurrent Backup Processes / Processor cores per vstorage Backup Server) Note - assume 1 core used for each backup process 5. Check for additional constraints / refine calculation How to Scale TSM for VE 1. Increase the number of data movers (processes on the vstorage Backup Server used to send data to TSM) 2. Increase the number of vstorage Backup Servers 44

45 Example: Batched Fulls, Full backup sizing Number of VM s Average utilized VM disk Total data to backup (full scenario) Backup window (extended window for fulls) GB 1000 x 50 = 50000GB (50 TB) 20 hours Aggregate throughput requirement 50000/20 = 2500 GB/Hour Per-Process throughput estimate* Example: 100GB/hour Number of backup processes required: Specify max processes per backup server Number of backup servers required Number of backup servers after rounding /100GB = /8 =

46 Environment for vstorage Backup Server Sizing Example Total Number of VM s 1000 Number of ESX Hosts 50 Average Number of VM s per Host 20 Average Allocated Size of VM Average Used Size of VM Total Amount of Used Storage (Assume HW Level 7) Average Daily Data Change Rate 100 GB 60 GB 60,000 GB (1000 VMs) * (100 GB/VM) * 0.6 Utilization 10% Backup Window for Full Backups on Weekend 20 hours Backup Window for Incremental Backups on Weekdays 10 hours 46

47 Example: Calculate Aggregate Throughput Requirement for Each Scheduling Method VMware Environment Aggregate Throughput Calculation 1,000 VMs Aggregate Throughput = 20 hosts (Total Data To Backup) / (Backup Window) Guest Aggregate Throughput (batched full each week) Host (ESX, ESXi) VMDK NAS/local SAN ~100 TB s Allocated ~60 TB s Used Hour Backup Window (e.g. Batch Full on Saturday) (60,000 GB) / 20 hours = 3,000 GB/Hour VMFS Backup Window / Scheduling Aggregate Throughput (rotating full 7 day) (8, ,143 GB) / 10 hours = 1,372 GB/Hour 60,000 / 7 60,000 * 6/7 *.1 1/7 of VMs will have full backup each day 6/7 of VMs will have inc backup each day 10 Hour Backup Window ( e.g. Rotating Full)

48 Example: Calculate Required Number of Concurrent Backup Processes Estimated Per-Process Throughput for vstorage Backup Server Aggregate Throughput Datastore and TSM Storage Pool I/O speed and utilization Backup / restore data path: SAN and LAN bandwidth speed and utilization vstorage backup server: Specific hardware: processor and adaptors Number of cores / concurrent backup processes I/O capacity (internal bus structure, NIC / HBA) Physical vs. virtual Restore activity TSM Server activity / throughput available Deduplication and compression Backup Processes (factors affecting throughput) Example Estimates of Per-Process Throughput for vstorage Backup Server Values are examples and not intended to guarantee / reflect actual performance. Benchmarking highly recommended before finalizing configurations. Deduplication? Low Estimate Medium Estimate High Estimate No 65 GB/Hour 120 GB/Hour 170 GB/Hour Yes 40 GB/Hour 60 GB/Hour 85 GB/Hour Total Number of Concurrent Backup Processes = Aggregate Throughput / Per-Process Throughput 48

49 Example: Calculate Number of vstorage Backup Servers Total Number of Processor Cores Required = Total Number of Backup Processes X Estimated Processor Cores per Backup Process Recommended Number of Processor Cores per Backup Process: 1 (potentially up to 1.5 when using client side deduplication and compression) Number of VBS Hosts Required = Total Number of Processor Cores Required / Number of Processor Cores per vstorage Backup Server Recommended Number of Processor Cores per vstorage Backup Server Minimum 4 processor cores Maximum 16 processor cores 49

50 Example: Calculate Number of vstorage Backup Servers Total Concurrent Backup Processes Aggregate Throughput (batched fulls each week) 3,000 GB / HR Aggregate Throughput (rotating fulls 7 day) 1,372 GB / HR Estimated Number of vstorage Backup Servers Hosts Per 1000 VM s *NOTE: Throughput estimate for illustrative example only Per Process Throughput Schedule Method *50 GB/Hour (e.g. client dedupe / LAN) *100 GB/Hour (e.g. no client dedupe / LAN) *150GB/Hour (e.g. SAN / VTL) Batched Full 4 cores # Processes / cores = 60 # vstorage backup servers = 15 # Processes / cores = 30 # vstorage backup servers = 8 # Processes / cores = 20 # vstorage backup servers =5 Rotating Full 4 cores # Processes / cores = 28 # vstorage backup servers =7 # Processes / cores = 14 # vstorage backup servers = 4 # Processes / cores = 10 # vstorage backup servers =3 Batched Full 8 cores # Processes / cores = 60 # vstorage backup servers =8 # Processes / cores = 30 # vstorage backup servers = 4 # Processes / cores = 20 # vstorage backup servers =3 Rotating Full 8 cores # Processes / cores = 28 # vstorage backup servers =4 # Processes / cores = 14 # vstorage backup servers = 2 # Processes / cores = 10 # vstorage backup servers =2 50

51 Check for Constraints and Refine Calculation Constraint - Any condition that makes estimate impractical or infeasible, considering the actual production environment. Example Constraints vstorage backup server throughput - may exceed HBA or NIC adapter port capacity (e.g., insufficient slots). Is TSM server able to accept sustained data rate from all data mover instances from all vstorage backup servers When associating data mover instance with ESX host need to consider: Time to process all VMs for ESX host in relationship to backup window Does total number of parallel processes exceeds number of ESX hosts May require scripted approach to process VM s at finer granularity than ESX host Per ESX server backup window - Check individual ESX servers with large amount of storage Considerations for Restore Sizing methodology focuses on backups Consider restore patterns Are restores mostly expected outside backup window? Sizing based on backup may provide adequate capacity What are restore requirements for DR scenarios 51

52 Summary of Deployment Considerations Centralized File Restore using Partition Mount for Windows and Linux DP for VMware Agent Deployment: Agent installed on central off-host Windows (e.g. host where B/A client installed) or Linux host. Note - Linux deployment requires DP for VMware Windows Agent. Function: Mount exposes local virtual volume and volume shared with VM requiring file Who initiates mount process? VMware administrator, TSM administrator or help desk personal Security: Native NFS / CIFS security to control who access exposed share Platform Support: Compare specific OS requirements for centralized host with supported levels where DP for VMware agent can be installed. If required OS level is not supported, determine whether iscsi target mount can be used Self Service File-Level Restore and Instant Volume Restore (VM guest user initiates request) DP for VMware Agent Deployment: Agent installed in Windows or Linux VM guest machines Function: Mount and instant restore Who initiates mount / IR process? User logged into the VM guest machine Security: VE 6.2 access controls are at node (vs.vm / filespace ) level Platform Support: Compare specific OS requirements for VM guests with supported levels where DP for VMware agent can be installed. If required OS level is not supported, determine whether iscsi target mount can be used Centralized File Restore using iscsi Target Disk 52 DP for VMware Agent Deployment: Agent installed on central off-host (or in-guest) Windows machine (e.g. host where B/A client installed) Function: Windows mount exposes Windows or Linux iscsi target. iscsi initiator in VM guest discovers exposed target Who initiates mount / IR process? VMware administrator, TSM administrator or help desk personal Security: iscsi target exposed to specific iscsi initiator Platform Support: > Supported levels where Windows DP for Agent can be installed > Reference tech note for OS platforms where iscsi initiator can be used

53 Deduplication Considerations Determining storage pool for incoming VMware backup data VMware backup data is an exceptionally good candidate for deduplication due to high amount of duplicate operating system data allocate enough disk (file) based storage to take advantage! Balance how much time the data will exist in the storage pool vs the amount of processing time required to deduplicate the data To maximize deduplication ratios, load balancing datamovers between multiple TSM Servers should adhere to some basic principles: Direct guest VM backups with similar operating systems to the same TSM server where possible 53

54 Performance Considerations Example 1: High activity exchange mail server TSM for Virtual Environments Backup of OS drive using skip independent disk option to exclude exchange data containing drives TDP for Exchange Backup of exchange data through dedicated feature rich product Solution advantages: Bare machine restore can quickly restore guest OS image using TSM for VE Prevents file system acquiescing issues that may result from VSS trying to consolidate a busy exchange server data drive which could result in a failed VMWare snapshot (thus also impacting TSM for VE) 54

55 Performance Considerations Example 2: High activity file server TSM for Virtual Environments Backup of OS drive using skip independent disk option to exclude data drives that perform file serving TSM Client (in-guest backup) Backup the file server drives as if this was a physical machine Solution advantages: Bare machine restore can quickly restore guest OS image using TSM for VE Maximum flexibility and performance for restoring individual files 55

56 Thank You 56