Using the Mass Storage System HPSS. Mitchell Griffith. Oak Ridge Leadership Computing Facility (OLCF) Presented by:

Transcription

1 Using the Mass Storage System HPSS Presented by: Mitchell Griffith Oak Ridge Leadership Computing Facility (OLCF)

2 2 HPSS (archival storage) What is HPSS? HPSS is software that manages petabytes of data on disk and robotic tape libraries. HPSS provides highly flexible and scalable hierarchical storage management that keeps recently used data on disk and less recently used data on tape. Hierarchical storage system Disk cache Tape backend Core Server, DB2 Metadata, Tape Control Software References install_guide.pdf

3 HPSS Layout OLCF/NCCS HPSS is shared between the OLCF and NICS NICS Typical Downtimes are Tuesday 8AM 11 AM HPSS If you need to move data between centers, contact NICS or OLCF helpline 3

4 4

5 HPSS Terms Storage Class Collection of storage devices Either disk or tape (not combination) Class of Service (COS) Collection of disks and tapes Set the hierarchy levels for the storage classes Top-level is disk, bottom levels are tape 5

6 HPSS Terms Migrate Copy the file from a higher storage class to a lower storage class (migrate from disk to tape) Stage Copy the file from a lower storage class to a higher storage class (stage from tape to disk) Purge Remove the file from the top-level storage class(free disk cache) (does NOT delete the file) 6

7 HPSS Layout COS Single copy COS (lscos) COS Name Min Max 5081 Xsmall 0 131,071 (128K) 6001 Small 131,072 (128K) 16,777,215 (16M) 6002 Medium 16,777,216 (16M) 536,870,911 (512M) 6054 Large_T 536,870,912 (512M) 8,589,934,591 (8G) 6056 X- Large_T 8,589,934,591 (8G) 281,474,976,710,656 (256T) 7

8 HPSS Layout Disk Cache Disk cache is striped(multiple disks, faster device) Upgraded disk cache (660TB in disk cache) (12) disk movers in production Tape (17) production movers Future Work Planning on updating hardware Adding additional movers Faster network backplane 8

9 HPSS Layout (6) SL8500 s 10,000 tape slots per silo 40,000+ tapes in use Tape capacity (500GB, 1TB, 5TB, 8.5TB) 112 tape drives 16 T10K-A 60 T10K-B 36 T10K-C 9

10 HPSS Interface only way to interface with HPSS HSI Similar to ftp Authentication (keytab or combo) Can be interactive or non-interactive Preferred only 1 stream at a time Batch options to allow HPSS to retrieve files more efficiently HTAR Acts like a wrapper to tar for HPSS Faster than tar a file then put to hpss Good for file aggregation Preferred only 1 stream at a time 10

11 HSI For a complete list of commands: hsi_commands/ Most commonly used commands: cd chgrp chmod chown du get In ls mkdir mv put rm rmdir pwd 11

12 HSI get and put example local_file_name : hpss_file_name K:[/home/mitchell]: put 1MB : /home/mitchell/test/1mb put '1MB' : '/home/mitchell/test/1mb' ( bytes, KBS (cos=6001)) K:[/home/mitchell]: get test.file : /home/mitchell/test/1mb get 'test.file' : '/home/mitchell/test/1mb' (2014/02/06 01:00: bytes, KBS ) 12

13 htar examples htar cf backup.tar. HTAR: HTAR SUCCESSFUL htar tvf backup.tar HTAR: drwx mitchell/tsg :26./ HTAR: drwxr-xr-x mitchell/tsg :24./a/ HTAR: -rw-r--r-- mitchell/tsg :23./a/a.file HTAR: -rw-r--r-- mitchell/tsg :24./a/b.file HTAR: -rw-r--r-- mitchell/tsg :24./a/c.file HTAR: -rw-r--r-- mitchell/tsg :24./a/d.file HTAR: drwxr-xr-x mitchell/tsg :24./d/ HTAR: -rw-r--r-- mitchell/tsg :23./d/a.file HTAR: -rw-r--r-- mitchell/tsg :24./d/b.file HTAR: -rw-r--r-- mitchell/tsg :24./d/c.file HTAR: -rw-r--r-- mitchell/tsg :24./d/d.file HTAR: -rw mitchell/tsg :26 /tmp/htar_cf_chk_28882_ HTAR: Listing complete for backup.tar, 9 files 12 total objects HTAR: HTAR SUCCESSFUL 13

14 htar examples hsi get backup.tar tar tvf backup.tar drwx mitchell/tsg :26. drwxr-xr-x mitchell/tsg :24./a -rw-r--r-- mitchell/tsg :23./a/a.file -rw-r--r-- mitchell/tsg :24./a/b.file -rw-r--r-- mitchell/tsg :24./a/c.file -rw-r--r-- mitchell/tsg :24./a/d.file drwxr-xr-x mitchell/tsg :24./d -rw-r--r-- mitchell/tsg :23./d/a.file -rw-r--r-- mitchell/tsg :24./d/b.file -rw-r--r-- mitchell/tsg :24./d/c.file -rw-r--r-- mitchell/tsg :24./d/d.file tar: Removing leading `/' from member names -rw mitchell/tsg :26 /tmp/htar_cf_chk_28882_

15 htar examples htar xf backup.tar HTAR: HTAR SUCCESSFUL ls a d mitchell@krakenpf6:/lustre/scratch/mitchell/test> rm -fr a d mitchell@krakenpf6:/lustre/scratch/mitchell/test> ls -l total 0 mitchell@krakenpf6:/lustre/scratch/mitchell/test> htar xvf backup.tar./d/a.file HTAR: x./d/a.file, bytes, 2049 media blocks HTAR: Extract complete for backup.tar, 1 files. total bytes read: 1,049,088 in seconds ( MB/s ) HTAR: HTAR SUCCESSFUL mitchell@krakenpf6:/lustre/scratch/mitchell/test> ls -l d/a.file -rw-r--r-- 1 mitchell tsg :23 d/a.file mitchell@krakenpf6:/lustre/scratch/mitchell/test> 15

16 HPSS Interfaces Access time can be slow if accessing from tape HSI get command: K:[/home/USER]: get a.pdf Scheduler: retrieving file(s) Waiting on a tape drive (resource contention) HTAR you will not see anything HTAR can give you just a standard tar file hsi get file.tar $ tar xf file.tar 16

17 HPSS Errors (possible tape issue) [/home/user]: get test.tar Scheduler: retrieving file(s) HPSS EIO error, will retry in 10 seconds %] [Done: 0 Failed: 0] HPSS EIO error, will retry in 60 seconds %] [Done: 0 Failed: 0] HPSS EIO error, will retry in 360 seconds %] [Done: 0 Failed: 0] HPSS EIO error, aborting0.00 Xferred: %] [Done: 0 Failed: 0] *** hpss_open: I/O error [-5: HPSS_EIO] /home/user/10_5.5a.tar *** get: Error -1 on transfer. /lustre/user/test.tar from /home/user/test.tar 17

18 Troubleshooting Common Codes HPSS_EPERM (-1) Permission issue, user does not have permissions to perform the task HPSS_ENOSPACE (-28) HPSS does not have enough space (disk or tape) to perform the task HPSS_EIO (-5) IO error could be anything from HPSS to network to lustre file system. HPSS_ECONN (-50) Connection issue 18

19 Authentication Issue > hsi result = , errno = 0g] Unable to authenticate user with HPSS. result = , errno = 9 Unable to setup communication to HPSS... *** HSI: error opening logging Error - authentication/initialization failed Workaround Ø hsi A combo (only if you have a OTP) 19

20 HSI Batch When retrieving multiple files, it is strongly recommend to use the MARKER option Schedule retrievals in an optimal way so as to minimize HPSS tape mounts $ cat hpss.marker! get <<MARKER! b1.tar! K:[/home/mitchell]: in hpss.marker! in hpss.marker! get <<MARKER! get b1.tar' : '/home/mitchell/b1.tar' (2014/02/06 01:52: bytes, KBS )! get b4.tar' : '/home/mitchell/b4.tar' (2014/02/06 01:54: bytes, KBS )! get b2.tar' : '/home/mitchell/b2.tar' (2014/02/06 01:52: bytes, KBS )! get b3.tar' : '/home/mitchell/b3.tar' (2014/02/06 01:52: bytes, KBS )! K:[/home/mitchell]:! b2.tar! b3.tar! b4.tar! MARKER!! 20

21 HPSS Tips HPSS is archival storage Many very small files are bad for HPSS (metadata stuff) Too large of files are bad as well (disk cache fills up) Optimal file size for HPSS is between 2GB and 256GB. Use HTAR to archive several smaller files into one A member file of an HTAR file cannot be > 64GB Submit to the HPSS queue (schedule the transfers) 21

22 HPSS Tips Additional information try the transfer with the debug option hsi d5 get /dev/null : file.tar debug 5; get /dev/null : file.tar When moving data to lustre, be mindful of the lustre striping Rule of thumb for NICS:1 stripe every 50GB. Once data is on scratch use standard transfer methods: nccssystems=dtn 22

23 Questions and Contact Information OLCF: NICS and XSEDE users: NICS and non-xsede users: 23