Overcoming Obstacles to Petabyte Archives

Size: px

Start display at page:

Download "Overcoming Obstacles to Petabyte Archives"

Tamsyn Newman
5 years ago
Views:

1 Overcoming Obstacles to Petabyte Archives Mike Holland Grau Data Storage, Inc. 609 S. Taylor Ave., Unit E, Louisville CO Phone: FAX: Presented at the THIC Meeting at the National Center for Atmospheric Research Boulder CO June 11-12, 2002 Slide 1

2 A Petabyte of Archive is Easy? 30 cu ft Our system utilizes SONY AIT technology, so 1 PB is only 10,000 tapes or about enough tapes to fill a box 1M on a side In the future, we can comment about the size of the cube Slide 2

3 Components of an Archive Application Server Network User Community Content Catalog Storage Mgmt. Software RAID Library Drives Media Slide 3

4 The Challenges of a PB Archive have less to do with sheer Storage Support for Users Common Access to all of the Data Recall Characteristics Data Acquisition Rates Data Expiration Rates and Reorganization of Media Catalog Size (File System Entries) System Conversions Absorption of Older Archives Transcription to Newer Technologies Data Protection Schemes (Media Availability) System Availability (Fault Resilience) Recovery Slide 4

5 I/O Activity within Archive Number of Recall Users Concurrency Peak workloads Data Acquisition Rates Data Request Queue Time Expectations Data Access Patterns Locality of reference for pre-stage data caching Scheduled recalls staged for faster use Actually, less concerned about I/O than management within archive Slide 5

6 What Does a PetaByte Imply? 10 Years to Fill Archive Sustained Data Rate MB/Sec (24 hrs/day * 365 days/yr) Slide 6

7 What Does a PetaByte Imply? HSM File System Entries (M) Current Industry Design Goal 1 50KB 1MB 100MB 1GB Average File Sizes Slide 7

8 PB Archive Must Be Scalable Throughout Resource Manager File System File System File System Meta Data Server Archive Archive # of files drives absolute requirement for multiple filesystems All agents are connected to the RM and MDS Front End s ( File System ) are connected to Back End s ( Archive ) through IP connections. Requirement for a stable archiving OS

9 So, It s Agreed! We Never Delete Any Files! Present a common view of data Add older systems to a current system Continually migrate data to newer storage media Network IVD - FS Metadata Foreign File System Capture old catalog into new management system Migrate old data in background Slide 9

10 High Availability Resource Manager Meta Data Server File System Archive Resource Manager Meta Data Server Multiple instances of Resource Manager and Meta Data Server may exist within in one system Two or more copies of control information are journaled onto separate disk subsystems

11 Data Protection Schemes To guarantee data security, IVD can store several copies of the data to different media pools, local or remote Pools may be located far away from each other but connected through TCP/IP File System Archive Archive

12 IVD - Policies Each FE can manage several partitions A partition contains a file system and one or more tape pools Each partition has a wide set of migration and truncation policies: Partition, directory or specific file level File access time and file size based Wildcard or file extension based Exclude lists Priority based job execution (migration, recall, reorg) Extended via external filter script for user defined policies Associative file grouping for migrate and recall as a group Slide 12

13 Policy Templates Preconfigured policy sets for the most common use Premigration: Data are copied to the tape soon after they are stored to secondary media but stay on disk. If disk space is needed, older data are removed. Immediate Migration: Data are copied to secondary media soon after they are stored to IVD and removed from disk immediately. Triggered Migration: Certain filesystem events control data migration to secondary media. User is free to define policies of any style, using individual scripts Slide 13

14 The Commercial Challenge How do we fill up a 35TB archive, much less a Petabyte? Changes in the industry regarding management of data More sophisticated data copy and migration More attention to TCO of storage Re-purposing data and data mining Regulated communities are seeing larger, online requirements including disaster recovery Data intensive video community ready to move beyond sneaker-net Closer relationship between backup and archiving Slide 14

15 Archiving at it s Best: Today Windows2000 NT Backup Server UNIX MAC NAS Dual copy Asynchronous Replication of each partition in same unit IVD1 Server IVD2 Server Remote copy Asynchronous replication to second unit via the network via dedicated network connection On campus, across town, across the country IVD1 master IVD2 copy DFS Server IVD2 master IVD1 copy High Availability DFS system Load balancing Disaster Recovery High Availability Single system view Slide 15

16 Archiving later this Year Multiple Servers within an IVD Complex plus multiple Data Movers to ingest data for the Archive Client-1 Client-2 Client-3 IVD-Client ML-1 IVD-Client ML-1 IVD-Client ML-1 LAN IVD-box Server-1 RAID disk CfgDB, RMDB, FSC, Disk Buffer (optional) Disk Buffer (optional) IVD-Server IVD- (FE/BE) CfgD B, R M DB, FSC IVD- (FE/BE) Server-2 RAID disk Disk Buffer (optional) TLU IVD-LAN Slide 16

17 Grau System Capacities IVD-XL Capacity (PB) AIT-3 at 100GB/Cart S-AIT at 500GB/Cart LaserTAPE at 1000GB/Cart 576 MB/S with AIT MB/s with S-AIT 960 MB/s with LaserTAPE AIT-3 S-AIT LOTS Slide 17

18 IVD-XL 1PB of Archive Capacity Slide 18

OPENARCHIVE The Final Destination of Your Data. Quick Start Guide

OPENARCHIVE The Final Destination of Your Data Quick Start Guide Version 3.5 Part number: 0309 OA-QG-E-350 First edition: Februar 2009 Content 1 Content 1 Introduction 3 1.1 About this Quick Start Guide...