HPSS for Spectrum Scale

Size: px

Start display at page:

Download "HPSS for Spectrum Scale"

Marjorie York
5 years ago
Views:

1 HPSS for Spectrum Scale Point-in-time backup/restore and automatic space management for Spectrum Scale 1

Spectrum Scale for HPSS overview Not only is HPSS a highly scalable standalone file repository, a single HPSS may also provide disaster recovery and space management services for one or more Spectrum

2 Spectrum Scale for HPSS overview Not only is HPSS a highly scalable standalone file repository, a single HPSS may also provide disaster recovery and space management services for one or more Spectrum Scale file systems. Spectrum Scale customers may now store petabytes of data on a file system with terabytes of high performance disks. HPSS may also be used to backup your Spectrum Scale file system, and when a catastrophic failure occurs, HPSS may be used to restore your cluster and file systems. Space Manage Backup Restore 2

Many-to-one advantage of HPSS for Spectrum Scale A single HPSS may be used to manage one or more Spectrum Scale clusters. ALL of the HPSS storage may be shared by all Spectrum Scale file systems.

3 Many-to-one advantage of HPSS for Spectrum Scale A single HPSS may be used to manage one or more Spectrum Scale clusters. ALL of the HPSS storage may be shared by all Spectrum Scale file systems. One Spectrum Scale file system may leverage all HPSS storage if required. The Max Planck Computing and Data Facility (MPCDF, formerly known as RZG) is space managing and capturing point-in-time backups for six (6) Spectrum Scale file systems with ONE HPSS. 3

4 HPSS software for Spectrum Scale HPSS software directs the space management and disaster recovery services. HPSS I/O Manager (IOM) software manages data movement between HPSS and Spectrum Scale. IOM HPSS software is configured on all Spectrum Scale Quorum nodes. o software is only active on the CCM (Spectrum Scale Cluster Configuration Manager) node. IOM The CCM may failover to any Quorum node o software will follow the CCM. HPSS IOM software may be configured to run on any node with a Spectrum Scale mount point. 4

5 HPSS software for Spectrum Scale There are five processes that comprise the HPSS software: o HPSS ProcessManager o HPSS MountDaemon o HPSS ConfigurationManager o HPSS ScheduleDaemon o HPSS EventDaemon ProcessManager MountDaemon ConfigurationManager ScheduleDaemon EventDaemon There are three processes that comprise the distributed HPSS IO Manager software: o HPSS IOManager manage I/O between HPSS and Spectrum Scale o HPSS IOAgent high speed transfers of individual large files o HPSS ishtar high speed transfer of many small files IOManager IOAgent ishtar IOManager IOAgent ishtar IOM IOManager IOAgent ishtar 5

6 HPSS hardware for Spectrum Scale HPSS IOMs are highly scalable and multiple IOMs may be configured on multiple Spectrum Scale nodes for each file system. Bandwidth requirements help determine the expected server count for a deployment. New Spectrum Scale quorum nodes for HPSS and HPSS IOM software may need to be added to the cluster. New Spectrum Scale quorum nodes for HPSS software may be added. 6

7 Space manage Spectrum Scale with HPSS Periodic ILM policy scans are initiated by the HPSS software to identify groups of files that must be copied to HPSS. The HPSS software distributes the work to the I/O Managers. Spectrum Scale data are copied to HPSS in parallel. Periodic HPSS copies files from Spectrum Scale to HPSS. IOMs send data to HPSS Movers periodically 7

8 Space manage Spectrum Scale with HPSS When Spectrum Scale capacity thresholds are reached (the file system is running out of storage capacity), unused files are purged from Spectrum Scale, but the inode and other attributes are left behind. All file names are visible, and the user may easily identify which files are online and which files are near-line. The HPSS software will automatically recall any near-line file from HPSS back to Spectrum Scale when accessed. The ghi_stage command-line tool may be used to efficiently recalling a large number of files. IOMs receive data from HPSS when files are recalled back to Spectrum Scale. 8

Backups & disaster recovery with HPSS The backup process

management process (discussed above) is the process used by

o Namespace data - the Spectrum Scale name space is captured

o Cluster configuration data - the cluster configuration is

The HPSS disaster recovery processing minimizes data movement

9 Backups & disaster recovery with HPSS The backup process captures the following data: o File data - the space management process (discussed above) is the process used by HPSS to capture the data for each file. o Namespace data - the Spectrum Scale name space is captured using the Spectrum Scale image backup command. o Cluster configuration data - the cluster configuration is saved to HPSS to protect the cluster configuration. The HPSS disaster recovery processing minimizes data movement and is ideal for high performance computing (HPC) environments where the goal is to bring the namespace back online quickly. 9

SC17 - Overview

HPSS @ SC17 - Overview High Performance Storage System The value and benefits of the HPSS service offering http://www.hpss-collaboration.org 1 We are storage industry thought leaders HPSS is a development