A Close-up Look at Potential Future Enhancements in Tivoli Storage Manager Dave Cannon Tivoli Storage Manager Architect Oxford University TSM Symposium September 2005
Disclaimer This presentation describes potential future enhancements to IBM Tivoli Storage Manager Information in this presentation does not constitute a commitment to deliver the described enhancements or to do so in a particular timeframe IBM reserves the right to change product plans, features, and delivery schedules according to business needs and requirements This presentation uses the following designations regarding availability of potential product enhancements Planned: GA is currently planned for 2005, but subject to change Candidate: Will be a candidate for possible delivery in 2006-2007 2 Potential Future Enhancements in Tivoli Storage Manager
Agenda Tivoli Storage, IBM Software Group Management of data in server storage Optimizing restore of client data Backup set enhancements TSM database Snapshot exploitation IBM TotalStorage N series (Network Appliance) integration 3 Potential Future Enhancements in Tivoli Storage Manager
Simultaneous Migration and Storage Pool Backup 1. Backup 2. Migration 2. Storage pool backup Copy pool 1 Client Server 2. Storage pool backup Data flow Can reduce client backup window as compared to simultaneous write to primary and copy storage pools during backup operation Combines windows for migration and storage pool backup Periodic storage pool backup required to ensure all data copied Copy pool 2 Candidate 4 Potential Future Enhancements in Tivoli Storage Manager
LAN-free Simultaneous Write to Copy Pools LAN Server Control Data flow Primary pool Client/ Storage Agent SAN Copy pool 1 Copy pool 2 LAN-free, simultaneous write to primary pool and copy pool(s) Reduces LAN traffic as compared with LAN-based simultaneous write Duplicates data while reducing time for storage pool backup Periodic storage pool backup required to ensure all data copied Candidate 5 Potential Future Enhancements in Tivoli Storage Manager
Management of Redundant Files Common files (operating system, application) are not duplicated in the TSM storage hierarchy for each node Also known as single instance store Client can transparently restore unique and common files Benefits Reduced bandwidth and/or time required for backup Reduced space requirements in storage pools Especially attractive for disk-only TSM configurations Client 1 filesystem Client 2 filesystem File X File X File X Only one instance of File X is stored on server, but each node shows backup of this file TSM Server Client 3 filesystem File X Candidate 6 Potential Future Enhancements in Tivoli Storage Manager
Management of Redundant Files: Model Node Each node can optionally be assigned to a model node with same operating system and preferably common application/project files (e.g., one model node per project) If file has already been stored by model node, same file is not resent by other nodes but unique database entry is created for each instance Although multiple nodes share a common file, each node Can have unique attributes (policy, insertion date, security) for the common file Can query file information for its own objects, whether stored uniquely or in common Efficient checking for previously stored common files because only the assigned model node is checked Data for model node can be stored on disk or collocated tape for faster restores (avoids need for mounting many tapes) Node Name Cataloged Files for Node MODEL_NODE SMITH JONES Solid boxes indicate files stored in storage hierarchy by this node Striped boxes indicate files which require database entries only (file is stored by model node) JOHNSON 7 Potential Future Enhancements in Tivoli Storage Manager
MOVE NODEDATA for Off-site Volumes TSM server location Offsite location DB 1. Use database to identify files for designated nodes on off-site volumes 3. Update database to reference new location of files 5. Transport tapes on-site Off-site volumes 4. Transport tapes off-site 2. Copy files from on-site location to new copy pool volumes Allows consolidation of data on off-site volumes for selected nodes Supports specification of following options for data to be moved On-site data only Off-site data only All data Candidate 8 Potential Future Enhancements in Tivoli Storage Manager
Agenda Tivoli Storage, IBM Software Group Management of data in server storage Optimizing restore of client data Backup set enhancements TSM database Snapshot exploitation IBM TotalStorage N series (Network Appliance) integration 9 Potential Future Enhancements in Tivoli Storage Manager
Collocation of Active Data Active data in disk pool for fast restore Active and inactive data for point-in-time restore Storage Hierarchy Active-only storage pool allows collocation of active backup versions for fast restore Active backup versions still reside on client machine (have not been modified or deleted) Active-only pools will be sequential-access only (typically sequential-access disk) Possible benefits - Optimized access to active versions - Reduced size of disk pools if only active versions are stored - Reduced data movement in preparation for restore of active data Challenge is to collocate active data while also preserving performance benefits of file aggregation and correspondence to data in copy storage pools Candidate 10 Potential Future Enhancements in Tivoli Storage Manager
Collocation of Active Data: Example 1. Client backs up A0, B0, C0, D0 to active-only pool. Then these files are migrated to next pool. 2. Client backs up B1, E1, which deactivates B0. Then B1, E1 are migrated to next pool. A0 B0 C0 D0 B1 E1 Client Server Client Server Active-only pool A0 B0 C0 D0 Active-only pool A0 B0 C0 D0 B1 E1 Migration Migration Active/inactive pool A0 B0 C0 D0 Active/inactive pool A0 B0 C0 D0 B1 E1 3. Reclamation removes inactive B0 from active-only pool. 4. Client restores active files A0, C0, D0, B1, and E1 from active-only pool. A0 C0 D0 B1 E1 Client Server Client Server Active-only pool A0 C0 B0 C0 D0 D0 B1 B1 E1 E1 Active-only pool A0 C0 B0 C0 D0 D0 B1 B1 E1 E1 Active/inactive pool A0 B0 C0 D0 B1 E1 Active/inactive pool A0 B0 C0 D0 B1 E1 11 Potential Future Enhancements in Tivoli Storage Manager
Transfer from Active/Inactive to Active-Only Pool Copy Data Copies active versions from active/inactive primary pool to activeonly primary pool Transfer is performed incrementally Can be restricted by node and file space Allows population of active-only pool with previously stored data Restore to Active-Only Pool Restores active files From copy pool containing both active and inactive data To active-only primary pool Allows recovery of active-only primary pool from active/inactive copy pool Active data in disk pool for fast restore Copy data Restore Active and inactive data for point-in-time restore Active/inactive copy pool Storage Hierarchy Candidate 12 Potential Future Enhancements in Tivoli Storage Manager
Active-Only Copy Storage Pools Incremental backup from active-only primary pool to active-only copy pool Active data in disk pool for fast restore Active and inactive data for point-in-time restore Backup Restore Storage Hierarchy Active-only copy pool Reduced storage requirement for copy pool Simplified tape management because fewer tapes Improved recovery performance for active data Reclamation of active-only copy pool recovers space occupied by inactive and deleted files Reclamation Inactive files Expired files 13 Potential Future Enhancements in Tivoli Storage Manager Candidate
Restore of Individual Files from File System Image Backup Client Metadata Image Server 3. Server stores image and table of contents (TOC) in storage hierarchy 2. Image and metadata sent to server TOC DB 1. Client creates image and collects file metadata Image Restore Client 1. Client specifies images to examine Server 2. Server loads TOC(s) into TSM database TOC 4. Selected files restored TOC DB 3. Client queries file information and user selects files to restore Image Candidate 14 Potential Future Enhancements in Tivoli Storage Manager
Agenda Tivoli Storage, IBM Software Group Management of data in server storage Optimizing restore of client data Backup set enhancements TSM database Snapshot exploitation IBM TotalStorage N series (Network Appliance) integration 15 Potential Future Enhancements in Tivoli Storage Manager
Generation of Backup Sets to Point in Time GENERATE BACKUPSET nodename PITDATE=mm/dd/yyyy PITTIME=hh:mm:ss Storage Hierarchy TSM Server Backup Set Allows generation of backup set to specified point in time Files must not already be expired or deleted from the server Default is date/time of backup set generation Benefits Increases operational flexibility because backup sets need not be generated before next client backup Allows retroactive generation of synthetic full backup if unanticipated needs arise Candidate 16 Potential Future Enhancements in Tivoli Storage Manager
File Selection During Backup Set Restore Client 1. Client specifies backup sets Server 3. Client queries file information and user selects files to restore 4. Server restores data to client TOC 4. Server restores data to client TOC Backup Set DB 2. Server locates backup set TOCs and loads into database Server storage pool has table of contents (TOC) with backup set contents On demand, server loads TOC into TSM database Client queries TOC and displays directory tree structure and files User expands directories and selects files for restore Simplifies task of restoring selected directories/files from a backup set Candidate 17 Potential Future Enhancements in Tivoli Storage Manager
Generation of Backup Sets for List of Nodes GENERATE BACKUPSET bob,fred,sue Storage Hierarchy TSM Server Distinct backup sets for nodes BOB, FRED, and SUE Allows generation of multiple backup sets with a single command A distinct backup set is generated for each node Backup sets generated with a single command are written sequentially to the same media Improves media utilization Physical security may be important for backup set restore at the client (without the TSM server) Candidate 18 Potential Future Enhancements in Tivoli Storage Manager
Differential Backup Sets Supports full and differential backups sets Full includes all files matching the backup set specification Differential includes all matching files that were backed up from the client since the last full backup set was generated Restore requires base and differential backup sets Benefits Reduces media requirement Faster generation of backup sets Full Backup Set Differential 1 Differential 2 Time 19 Potential Future Enhancements in Tivoli Storage Manager Candidate
Backup Set Support for Application Data Expand backup set support to include Data Protection products Allows Data Protection client to restore from backup set via the TSM server Candidate applications include Microsoft Exchange Microsoft SQL Server Lotus Domino Candidate 20 Potential Future Enhancements in Tivoli Storage Manager
Possible Uses for Enhanced Backup Sets Offsite vaulting of self-describing data Traditional tape rotation (e.g., grandfather/father/son) for SMB or departmental environments Long-term retention of point-in-time data Individual files in backup set do not require entries in TSM database because entire backup set is tracked as a single entity Separate policies for retention of backup set and individual files stored by TSM May be useful in environments with stringent regulatory requirements for data retention 21 Potential Future Enhancements in Tivoli Storage Manager
Agenda Tivoli Storage, IBM Software Group Management of data in server storage Optimizing restore of client data Backup set enhancements TSM database Snapshot exploitation IBM TotalStorage N series (Network Appliance) integration 22 Potential Future Enhancements in Tivoli Storage Manager
DB2 Database Engine for TSM Installation of new TSM server TSM Server Conversion of existing TSM database TSM Server Before conversion After conversion DB2 Database Proprietary Database Conversion DB2 Database DB2 included in install package TSM server unavailable during conversion Candidate 23 Potential Future Enhancements in Tivoli Storage Manager
Why a Proprietary TSM Database Today? Portability to diverse server platforms - Proprietary TSM database portable to any server platform - In early days of ADSM, DB2 had only limited platform support TSM database optimized for storage-management operations - Database schema - Customized, multilevel locking - ARIES technology reduces overhead for logging and recovery Bit vectors stored in database for tracking block allocation in randomaccess storage 24 Potential Future Enhancements in Tivoli Storage Manager
Potential Benefits of DB2 as Database Engine Scalability Database Recovery log Database maintenance Online database reorganization Referential-integrity checking to eliminate or minimize need for database audit Monitoring - Complete SQL function including UNION, INTERSECT, and correlated subqueries not available with proprietary database - Ability to optimize SQL query performance by defining secondary indices Potential ability to support alternate configurations - Distributed databases - Enhanced LAN-free and library sharing 25 Potential Future Enhancements in Tivoli Storage Manager
Design Goals for DB2 Database No extra expense for database product license TSM administration should not become more complex Preserve existing administrative interfaces to the extent possible Provide virtualized tables equivalent to those available via SQL today Automate conversion of existing database entries to DB2 with minimal downtime of the TSM server Conversion time may be comparable to that for database unload/load Test using large customer databases Performance should be similar to that of current proprietary database engine Support backup and recovery of the TSM database Continue to support random-access disk storage pools with equivalent of bitvector implementation in DB2 engine 26 Potential Future Enhancements in Tivoli Storage Manager
Agenda Tivoli Storage, IBM Software Group Management of data in server storage Optimizing restore of client data Backup set enhancements TSM database Snapshot exploitation IBM TotalStorage N series (Network Appliance) integration 27 Potential Future Enhancements in Tivoli Storage Manager
Current Snapshot Support in TSM Windows backup-archive client LVSA for open-file support and image backup from TSM Windows client Snapshot image backup from Linux x86 / Linux IA64 client Microsoft Volume Shadow Copy Service (VSS) support for backup of Windows 2003 system state TSM for Hardware (exploitation of snapshot for backup/restore) ESS: Backup/restore of DB2, Oracle, and mysap data on AIX DS6000/DS8000: Oracle and mysap (with Oracle or DB2) on AIX (Available 9/2005) SVC: Oracle and mysap (with Oracle or DB2) on AIX (Available 9/2005) 28 Potential Future Enhancements in Tivoli Storage Manager
Snapshot Exploitation Topology Application System Application Data Local Snapshot Versions Backup System Snapshot Creation Backup to TSM Server Restore can be performed from Local snapshot versions TSM storage hierarchy Policy-based snapshot management TSM Server Storage Hierarchy 29 Potential Future Enhancements in Tivoli Storage Manager
Benefits of Planned TSM Snapshot Support Support for multiple, persistent snapshots Persistent snapshots retained locally after backup to TSM Allows extremely fast restore directly from the snapshot using snapback to the original location Backup of snapshots to TSM server Transfer is outboard of application server to minimize impact to application (if snapshot provider allows access by another machine) Copies on TSM server provide long-term retention and disaster recovery Policy-based management of local, persistent snapshots Retention policies may be different for local snapshots and copies on TSM server Automatic reuse of local snapshot storage as older snapshot versions expire 30 Potential Future Enhancements in Tivoli Storage Manager
Potential Snapshot Exploitation Additional snapshot providers DS6000/DS8000: DB2 on AIX (Planned GA 4Q2005) SVC: DB2 on AIX (Planned GA 4Q2005) Federated backup/restore of multi-partition DB2 on AIX (Planned GA 4Q2005) Microsoft Volume Shadow Copy Service (VSS) exploitation Data Protection for Exchange (Planned GA 4Q2005) Data Protection for SQL Server (Candidate) File system snapshot exploitation Snapshot backup/restore on AIX JFS2 (Candidate) Snapshot backup/restore on SAN FS (Candidate) 31 Potential Future Enhancements in Tivoli Storage Manager
Agenda Tivoli Storage, IBM Software Group Management of data in server storage Optimizing restore of client data Backup set enhancements TSM database Snapshot exploitation IBM TotalStorage N series (Network Appliance) integration 32 Potential Future Enhancements in Tivoli Storage Manager
IBM-NetApp Agreement IBM to OEM Network Appliance (NetApp) hardware offerings (as IBM TotalStorage N series) IBM will have the right to rebrand/resell NetApp software Promotes integration of TSM with NetApp products TSM will be a preferred backup/recovery solution for NetApp products System Storage Archive Manager (formerly TSM for Data Retention) will be a preferred solution in environments with stringent requirements for retention management Current TSM-NetApp integration to be extended for additional customer value NetApp to position IBM as preferred supplier of tape solutions 33 Potential Future Enhancements in Tivoli Storage Manager
Planned N Series (NetApp) Integration Enable TSM archive client for System Storage Archive Manager VSS support for NetApp SnapDrive Improved support for backup/archive using CIFS/NFS NDMP V4 support Disaster recovery / backend data movement for NDMP-generated data NDMP 3-way configuration NDMP filer-to-server configuration 34 Potential Future Enhancements in Tivoli Storage Manager
System Storage Archive Manager TotalStorage DR550 System Storage Archive Manager Software Application TSM API System Storage Archive Manager IBM DS4000 Archive Client SnapLock Storage (optional) System Storage Archive Manager formerly known as TSM for Data Retention Previously, archive/retrieve could be performed only via the TSM API TSM command-line and GUI client will be able to Archive to and retrieve from Archive Manager Activate event-based retention for specific files Request deletion hold/release for specific files Storage Hierarchy Planned GA 3Q2005 35 Potential Future Enhancements in Tivoli Storage Manager
VSS Support for NetApp SnapDrive Exchange / SQL Server (VSS writer) Windows Host Machine DP for Exchange (VSS requester) TSM Server Microsoft Volume Shadow Copy Service (VSS) NetApp SnapDrive storage virtualization software (VSS provider) Fast, low-impact backup of Exchange / SQL servers Multiple versions of local, persistent snapshots Offloaded backup to TSM server N series/netapp (iscsi/fcp) device Exchange: Planned GA 4Q2005 SQL Server: Candidate 36 Potential Future Enhancements in Tivoli Storage Manager
Backup/Archive of NetApp Using CIFS/NFS Today NAS File Server CIFS/NFS Access TSM Client Could be in same box TSM Server NAS file system Storage hierarchy CIFS/NFS allows TSM client to access data on N series/netapp devices, which do not support third-party backup software Benefits File-level granularity Progressive incremental backup TSM storage hierarchy Limitations File-access overhead File attributes may not be preserved Q-trees not preserved Data flow through TSM client/server Data flow over the network 37 Potential Future Enhancements in Tivoli Storage Manager
Improved CIFS/NFS Support NetApp Windows Data CIFS access Windows Client UNIX/Linux Data NFS access UNIX/Linux Client TSM Server Preserve file attributes across backup/restore or archive/retrieve round trip Homogenous environments only Windows data with Windows backup-archive client UNIX/Linux data with UNIX/Linux backup-archive client Planned GA 1Q2006 38 Potential Future Enhancements in Tivoli Storage Manager
NDMP: Disaster Recovery / Backend Movement TSM Server NAS Device NDMP control (TCP/IP) Data flow (SCSI/FC) Read Write Tape Library Use NDMP tape-to-tape copy for backend movement of data stored on TSM server via NDMP Supported operations Storage pool backup Storage pool or volume restore Move Data Intra-pool for space recovery Inter-pool for migration to new device type DRM support for NDMP data Restore Node will use copy pool, if primary data not accessible Candidate 39 Potential Future Enhancements in Tivoli Storage Manager
NDMP: 3-Way and Filer-to-Server Configurations NAS Device Data TSM Server NDMP control (TCP/IP) Data (TCP/IP) Robotics Control (TCP/IP) Target Device Data Robotics Control (SCSI/FC) NDMP control (TCP/IP) TSM Server NAS Device Data (TCP/IP) Robotics control Data Tape library NAS file system Data Target device could be integrated within tape library Tape library NAS file system Allows centralization of tape resource Exploits libraries with NDMP server Allows centralization of tape resource Full capability of TSM storage hierarchy Candidate 40 Potential Future Enhancements in Tivoli Storage Manager
Summary Tivoli Storage, IBM Software Group This presentation has described a sampling of potential future enhancements to Tivoli Storage Manager Themes have included Efficient management of data in server storage Optimized restore of client data Enhanced backup sets for off-site vaulting and long-term retention of point-in-time data Improved scalability, maintenance, and monitoring of the TSM database Exploitation of snapshots to minimize impact of backups and achieve faster restore Extended integration with N series / Network Appliance devices 41 Potential Future Enhancements in Tivoli Storage Manager