Insights into TSM/HSM for UNIX and Windows

Transcription

1 IBM Software Group Insights into TSM/HSM for UNIX and Windows Oxford University TSM Symposium 2005 Jens-Peter Akelbein IBM Tivoli Storage SW Development 1

2 IBM Software Group Tivoli software IBM TotalStorage Taking steps toward an On Demand storage environment Orchestration Orchestration Storage Storage Infrastructure Infrastructure Management Management Hierarchical Hierarchical Storage Storage Management Management Archive Archive Management Management Recovery Recovery Management Management Storage Storage Virtualization Virtualization Advanced Advanced Copy Copy Services Services Hardware Hardware Infrastructure Infrastructure Disk Disk 2 Tape Tape Storage Storage Networking Networking Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

3 Managing too much data on UNIX, Linux, and Windows The Tivoli Storage Manager product family provides now two HSM products IBM Tivoli Storage Manager for Space Management AIX (JFS, JFS2, GPFS), Linux (GPFS), Solaris (VxFS) HP/UX (JFS) IBM Tivoli Storage Manager HSM for Windows Windows 2000 Server, Windows 2000 Advanced Server, Windows 2003 Server, Windows 2003 Enterprise Server (32 Bit), all NTFS This session intends to provide some answers for questions like What differences exist between using HSM on UNIX and Windows? What technologies and interfaces are used for HSM on UNIX and on Windows? How did HSM on UNIX evolve over the past years? and your questions! 3 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

4 Definition of HSM (the TSM-way) HSM automatically migrates files from on-line to near-line storage (typically from disk to tape), based on policy settings Small stub files are retained on disk, appearing as the original files, thus ensuring transparency of HSM for user applications Stub files contain information pointing to corresponding entry in TSM server DB Migrated files are recalled automatically back to disk as required Note: The TSM server has its own type of HSM for migration of files in its storage pool hierarchy. 4 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

5 Usage patterns of File Servers UNIX File Server Applications like Engineering, Geophysical, Multimedia, Scientific data, Bioinformatics, Medical imaging Type of data varies and might not be present in the file name Maximum size of file up to hundreds of Gigabyte or even Terabyte Windows File Server Applications like office tools (text, spreadsheets, presentations), Multimedia Type of data often known by suffix Maximum size of files up to a few hundred Megabyte or a Gigabyte GUI usage is common Command line usage is common Please see this more as a pattern than a rule 5 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

6 Migration policy on UNIX Primary selection criteria are age and/or size where weight factors allow to set priorities Exclude small files from becoming eligible at all (minmigfilesize) Include/exclude statements allow directory and file selection Set a high threshold to define the maximum allocated file system capacity Set a pre-migration percentage to free up online storage space faster during automigration 6 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

7 UNIX 7 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

8 Migration policy on Windows Configure one or more migration jobs A migration job consists of a path name Include/exclude filters can be added for known file groups or specify file name extensions File size and age Jobs can be executed periodically by starting them from the scheduler and do all this easily in a GUI! 8 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

9 Configuring a Migration Job System administration task Part of the System Administration Client Fast and efficient in a GUI Step 1 - Select the root directory for the migration job 9 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

10 Configuring a Migration Job cont d Step 2 - Include subdirectories Step 3 - Select migration mode 10 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

11 Configuring a Migration Job cont d Step 4 - Include/exclude file groups 11 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

12 Configuring a Migration Job cont d Step 5 -Include/exclude files based on date condition - Absolute or relative - Creation, modification or last access date 12 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

13 Windows IBM Software Group Tivoli software 13 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows

14 Migration policies: UNIX and Windows UNIX HSM Windows HSM Capacity oriented approach (classic HSM) Automigration migrates data when high threshold is reached Fixed amount of data being transferred by a single automigration Avoids out-of-space condition Placement of individual files depends on used capacity File attribute oriented approach (more like ILM) Placement of individual files is specified exactly Allows to define jobs for different file paths (often used for e.g. departments/users) Jobs specified in a XML format Scheduled jobs ensure replacing recalled files Does not avoid out-of-space condition 14 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

15 Interfaces to the TSM server TSM Server UNIX File Server File U File System Stub File U TSM HSM Client HSM Verbs Storage Pool (SPACEMGMT) Unique Object ID TSM Server Windows File Server File System Pathname Qualifier (FS/HL/LL) File W Recommended to set archive retention forever to keep data integrity! Stub File W TSM HSM Client TSM API library different interfaces used internally Storage Pool (Archive) 15 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

16 Backup integration (backup) TSM Server UNIX File Server File U File System Stub U TSM HSM Client TSM Backup Client* know the link! (no network traffic) Storage Pool (SPACEMGMT) Inline copy File U TSM Server Stub W Windows File Server File System TSM HSM Client File W Storage Pool (Backup) Stub W TSM Backup Client Storage Pool (Archive) 16 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

17 Backup integration (restore) TSM Server UNIX File Server File U File System Stub U TSM HSM Client TSM Backup Client* restore ( migstate=yes) Storage Pool (SPACEMGMT) Recreating unique object ID File U TSM Server Stub W Windows File Server Storage Pool File System TSM HSM Client restore File W (Backup) Stub W TSM Backup Client Storage Pool (Archive) 17 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

18 Backup integration (versions) TSM Server UNIX File Server File System TSM HSM Client File U* U copygroup (verexists=2) Stub U* U Windows File Server TSM Backup Client* U*/W* file U/W contains now modified contents and gets migrated again Storage Pool (SPACEMGMT) TSM Server File U File U* Stub W Stub W* Storage Pool File System TSM HSM Client File W* File W (Backup) Stub W* W TSM Backup Client Storage Pool (Archive) Retains older versions 18 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

19 Backup integration: UNIX and Windows Both ways of backup integration allow restoring stubs for faster restore of a file system Older versions of file contents are kept on the TSM server UNIX File Server Windows file Server Restore of stubs and complete files through the same interface Number of retained versions controlled by VEREXISTS option Inline copy can avoid network traffic Restore of stubs through the B/A client Restore of migrated files completely through HSM client Both products are using one transaction for each file. Fits for medium and large files! What is a small file? No easy answer, but LTO tape drives can write not less than a certain amount of data (LTO2: 400KB, LTO3 1,6MB). A single TSM transaction wastes tape space if it does not contain at least such an amount. 19 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

20 Technology used for transparent recall UNIX HSM (DMAPI) Windows HSM (Reparse points) DMAPI is an open standard defined 1997 by The Open Group; official name: Data Storage Management (XDSM) API Goal: Independence of data management applications from underlying file system types Avoidance of kernel code by applications using DMAPI DMAPI in general provided by file system implementation Examples of file systems supporting DMAPI: XFS for SGI IRIX and Linux, GPFS for IBM AIX and Linux, VERITAS VxFS for Sun Solaris, JFS for HP- UX, JFS2 for AIX Stub related to object on TSM server by a unique object ID (renaming the file does not break the pointer) Using reparse points for intercepting file system calls File system filter driver resides in the Windows Kernel; notifies recall application Recall application resides in User Space; runs as Windows Service Stub related to object on the TSM server by a FS/HL/LL specifier stored in the stub (renaming the file does not break the pointer; but old file name must be used for manual restore) 20 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows akelbein@de.ibm.com

21 UNIX HSM scalability Number of files I/O throughput Process parallism MAXCANDIDATES duration of candidates search Reconcile and Automig overlapped Reconcile w/o orphan check MAXMIGRATORS Concurrent auto-migration Reconcile and Automig full concurrency Candidate search as own process (scout) CANDIDATESINTERVAL Daemon controlling parallel FS scans MINMIGFILESIZE MAXCANDPROCS Daemon controlling parallel FS scans Streaming Mode LAN-Free* dsmls memory consumptation Distributed Recall for GPFS clusters LAN-Free** Partial File Recall* Partial File Recall** *for AIX GPFS; **for Linux GPFS 21 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows

22 TSM/HSM availability and usability 5.1 and earlier Usability Platform support AIX JFS initial client Solaris VxFS (3.1) AIX GPFS for SP/2 (4.1) AIX GPFS for pseries HP/UX JFS for PA-RISC Linux GPFS For xseries Non-root user support for command line GUI Admin Console preview AIX JFS2 for pseries HSM for Windows GUI Admin Console* Selective premigration for command line No single point of failure during failover for GPFS clients Easy installation on clusters for GPFS clients *not available for all clients 22 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows

23 References TSM HSM for UNIX on the Web: TSM Manuals for UNIX and Windows on the Web Tivoli Field Guide on HSM (from January 2003): 1.ibm.com/support/entdocview.wss?rs=0&uid=swg DMAPI specification: 23 Oxford TSM Symposium 2005 Insights into TSM/HSM for UNIX and Windows