Preservation DataStores: An architecture for Preservation Aware Storage

Transcription

1 Preservation DataStores: An architecture for Preservation Aware Storage Dalit Naor, IBM research, Haifa Simona Cohen, Michael Factor, Dalit Naor, Leeat Ramati, Petra Reshef, Shahar Ronen, Julian Satran IBM Labs in Haifa

2 Outline of Talk Long term digital preservation Motivation Approached and standards Why is it a storage problem Storage characteristics Why is preservation aware storage needed? Preservation DataStores Proposed architecture Migration and Media formats Summary 2

3 The challenge of data preservation This document was created about 2000 years ago. One can read the letters with one s bare eyes. Dead Sea Scroll, ~70AD. Media: Copper. Language: Hebrew. This information was created this year. Will the media last for 20 years yet alone 2000? Will it be possible to access, interpret and present the data in 20 years? 50? 100? 3

4 Digital Preservation: Bit vs. Logical Preservation Challenge: preserve large amounts of heterogeneous data for long periods of time (tens if not hundreds of years) Preserve Information and not only Bits Preservation of Information implies continuing Understandability and Usability Preservation of Information is hard and requires vigilance Changes in technologies, users (Designated Community) Storage considerations: Bit Preservation Maintaining physical readability: being able to get the bits off a physical media, e.g., who today can read a 5¼ floppy drive Logical Preservation Maintaining logical readability: assuming one can get the bits, how does one read/understand/interpret them? Store with the data, metadata (representation information) that describes how to interpret the data Ensure the metadata can be interpreted in tens or hundreds of years - a recursive problem 4

5 Industries Need Finance Rule 17a-4 requires brokerdealers to retain account record information for six years. The six-year period begins either at the time the account is closed or when the information is replaced or updated Aerospace Aircraft designs records have to be retained for the lifetime of each aircraft (30+ years) Life insurance policies has to be kept for life of policy plus 6-10 years Petroleum Oil-field data is used over life of field (50+ years) Healthcare Medical records should be preserved for the life of the individual and beyond Pharma X-rays are often stored for periods of 75 years Pharma needs offline electronic data storage for 50 to 100 years or longer OSHA requires employers to keep records of both medical and other employees who are exposed to toxic substances and harmful agents. Employers must maintain these records for 30 years The retention requirement for the [medical] records of minors varied from 20 to 43 years of age Scientific and Cultural Satellite data is kept for ever We would like to keep Libraries and Art data for ever 5

6 Example Data Volumes Earth Observation (EO) Data Development of ESA's EO Historical Data Archive between Total Archive in TerraBytes (TB) This does not consider multiple copies, and high level products! Year AQUA Modis (April 03-today) ENVISAT LR (March 02-today) ENVISAT HR (March 02-today) TERRA Modis (June 01-today) QUICK SCATT (01-today) /PROBA (May 02-today) LANDSAT 7 ETM (April 99-Dec 03) SEA STAR SeaWifs (Apr 98-today) ERS 2 HR (May 95-today) ERS 2 LBR (May 95-today) JERS SAR/OPS VNIR (92-Sep 98) At present (2006) the total volume of archived data and products (from ESA acquired missions) is ~3 PBytes & increases by >> 500 TB/y) ERS 1 HR (Jul 91-Mar 00) ERS 1 LBR (Jul 91-Mar 00) SPOT 1-4 HRV (87-today) MOS 1, 1b MESSR (87-Oct 93) NOAA 9-17 AVHRR (86-today) LANDSAT 5 TM (April 84-today) NIMBUS 7 (Nov 78-May 86), SEASAT (Jun-Oct 78) LANDSAT 2-4 MSS (75-Dec 93) Taken from Europe Space Agency (ESA), presentation by Luigi Fusco 6

7 Characteristics of Preservation Data Access pattern Content (raw) data is Read Only. Most content data is cold rarely accessed during its lifetime New metadata added later on, e.g. new RepInfo Heterogeneity Quantity Type, size and value, some data is more important than other Offline, Too large for on-line storage. Quality Lossless format if it needs to be transformed over time. 7

8 OAIS: Open Archival Information System Reference Model (ISO:14721:2002) Information Model Recursive Representation Information (RepInfo) Functional Model 8

9 RepInfo Example FITS FILE PDF STANDARD PDF s/w FITS STANDARD JAVA VM FITS JAVA s/w Taken from CASPAR presentation by David Giaretta FITS DICTIONARY DICTIONARY SPECIFICATION XML SPECIFICATION UNICODE SPECIFICATION 9

10 OAIS AIP Logical Structure Content Data Object - the raw data that is the focus of the preservation. Representation Information the information required to interpret the raw data to its designated community. Reference globally unique and persistent identifiers for the content information. Provenance the history and the origin of the content information and any changes that may have taken place since it was originated, and who has had custody of it since it was originated. Context documents reason for creation of the content information and relationship to its environment. Fixity a demonstration that the particular content information has not been altered in an undocumented manner. 10

11 Preservation Processes Characteristics Packaging External Data Migration and Transformation Ingest - Digest/Access Loss / Corruption Package metadata together with the data referential data to either internal objects or external Need referral to data residing outside the system External data may evolve over time. Supporting obsolescence of formats, hardware or software. System ingesting data may be different from system accessing data Data likely to be lost / corrupted, especially during migrated and transformation 11

12 Preservation Aware Storage: A New Storage Paradigm Functionality Physically co-locate the Information Object (AIP) Execute data intensive functions at the storage component: fixity computations and validation data transformation Handle provenance events internally Support the loading and execution of external transformations Rational Ensure metadata is never lost when raw data survives Utilize the data locality property E.g. migration and copy occurs at the storage Ideally performed during bit-migration performed close to data 12

13 Preservation Aware Storage: A New Storage Paradigm (Cont.) Functionality Maintain referential integrity Update links during migration Ensure readability of the data by a different system in the future. Support global self-described formats Support media migration Rational Ideally done during migration Interaction with backend storage Interaction with backend storage Support a graceful loss of data 13

14 Migration: Short term to long term archive via media migration AIP Near Line and Off Line Off Line Phase 2 Off Line Phase 3 Archiving System Archiving System Archiving System. Media Migration Media Migration Encapsulation of data and metadata is done within the tape/disk Migration is simple just move the tape to the new system If a tape is damaged or lost, the effect is contained the information in the other tapes is still valid! 14

15 Migration System Migration Constant Migrations Into the Future Generate AIPs Near Line Off Line Phase 1 Off Line Phase 2 Off Line Phase 3 Archiving System Archiving System Archiving System Archiving System. Media Migration Media Migration System Migration The encapsulation of data and metadata is done by the system Migration is more timely and complex because the whole system is involved Some damaged or lost media may make other data useless 15

16 PDS Architecture: A Stacked Architecture for OAIS-based preservation-aware storage AIP Preservation Web Services Preservation DataStore Preservation Engine Layer XAM Layer Object Layer backend Ingest, Access, Administration, Applications Layered approach Based on open standards OAIS, XAM, OSD Generic mappings Logical object physical object 16

17 PDS Architecture in detail AIP Preservation Web Services Preservation DataStore RepInfo Mgr PDI Mgr Migration Mgr Placement Mgr Preservation Engine XAM API XAM Library VIM API XAM to FS WAS CE File System posix I/O backend VIM API XAM to OSD sockets HL HL OSD + Object Store web service Preservation WSDL Ingest, Access, Administration, Applications Security Admin 17

18 Scalability Model 18

19 Storage Interface (High Level) OAIS defines Archival Information package (AIP) to be the basic unit stored in the archival storage The high level API to PDS includes: ingestaip accessaip migrateaip transformaip getpreservationpolic setpreservationpolicy loadstorelet 19

20 Towards Self-Describing Self-Contained Data Format (SD-SCDF) Ideas at early stage Proposed by Simona Cohen Context: SNIA s 100 Year archive Task force Digital Information Layers Application Layer Get the interpretation of the data Example: render a Word 3.0 document Diverse applications and data types Data Layer Get the basic units (records/containers/objects) Example: understand OAIS AIPs Can it be standardized? Bit Layer Get out the bits from the media Example: Linear Tape-Open (LTO) for tapes Maybe depends on media type 20

21 Requirements for the Data Layer Format (1/2) Media agnostic Tape, disk, future media Vendor and Platform agnostic Self-describing Support self-contained data Include means to represent internal links and cross references Performance Need to have good performance even for large data that includes text and binaries Enable parallel reads and writes 21

22 Requirements for the Data Layer Format (2/2) Interoperability Need to be able to migrate data between different systems without loss of data Can be interpreted in the future Extensible Additional information which may be added in the future Vendor specific extensions Cost Free parsers Readable by both humans and machines Ability to do offline inspection Support additional functions on the data compression, encryption, cryptography 22

23 Proposal SD-SCDF is a proposal for an open logical format standard based on marrying the OAIS AIP with XAM. The data format will include OAIS concepts RepInfo, reference, provenance, fixity, context Use XOP for a cluster of AIPs Each AIP is a set of XSets Does a media unit (e.g. volume) contain one XOP package or many XOP packages? Add inter-link and external-link mechanism Include TOC that points to the various AIPs on the media VTL could be a natural translation point and search staging area for off line data (Taken from SNIA SDDF presentation) 23

24 Summary Motivated the problem Analyzed special characteristics of the problem Proposed architecture and problems Reference Towards OAIS-Based Preservation Aware Storage - A White Paper. The Need for Preservation Aware Storage - A Position Paper". ACM SIGOPS Operating Systems Review, Special Issue on File and Storage Systems, Volume 41, Issue 1 (Jan 2007), pp Architecture paper Submitted to MSST 2007 Sites

25 Backup IBM Labs in Haifa

26 Collaborations and Partners CASPAR Cultural, Artistic and Scientific knowledge for Preservation, Access and Retrieval Objectives: Build an OAIS-based framework and architecture Demonstrate its validity with heterogeneous data Status Provided User Requirements and Scenarios (D4101) document to EU Working on overall conceptual model and Architecture Each partner defines its component in more detail and specifies its interface Built a user community web site Presented preservation and CASPAR seminars Received some initial data Contacts with US partners Reagan Moore, SDSC 26

27 Preservation Aware Storage Proposed Functionalities (1/3) Encapsulate and physically co-locate in the storage the raw data and its complex interrelated metadata objects, such as representation information network, provenance, and fixity. This ensures that the metadata needed for interpretation is not separated from the raw data and thus never lost (if the raw data survives). Utilize the locality property and execute data intensive functions such as fixity computations, data validation, data transformation within the storage component. Include the representation information of metadata such as the representation information of fixity and provenance, so that the metadata can be validated and interpreted when migrating to newer systems. 27

28 Preservation Aware Storage Proposed Functionalities (2/3) Handle the provenance events internally. The applications on top of the preservation aware storage should be freed from managing events that can be handled internally in the storage. Moreover, the types of provenance events are richer and also include events related to migration and transformation. Support the loading and execution of external transformations during the migration process. Additionally, it should facilitate on demand triggering of those transformations. Support media migration, as opposed to system migration, in which migration from one system to another can be done by physically detaching the media from one system and attaching it to the new system. 28

29 Preservation Aware Storage Proposed Functionalities (3/3) Maintain referential integrity including updating all the links during the migration process such that they remain valid in the new system. This requires an awareness of certain meta-data fields that represent links, both internally to the system and externally. Ensure readability of the data by a different system in the future. This is done by developing and supporting global self-described formats for disks and tapes. Support a graceful loss of data. Some portions of the data are likely to be lost or become corrupted over time. If some data is lost, a good preservation system must prevent cases where data in the system that is still intact cannot be read or interpreted. This is done by utilizing a self-contained format. 29

30 Ingest AIP Data Flow Within PDS (1/3) The OAIS Ingest entity sends to Preservation DataStore (Archival Storage) a storage request with an AIP. The transfer request may need to indicate the anticipated frequency of utilization The Reference Manager either assigns or validates the given Persistent Globally Unique ID for the AIP. The AIP ID may be based on XAM XUID. The AIP ID also resides in the OAIS Data Management entity or CASPAR Directory Service. The RepInfo Manager validates and fetches some of the RepInfo network of the content data object. The given AIP may sometimes embed the RepInfo and sometimes just have links to the RepInfo which resides in some registry. Since the Preservation DataStore is required to have self-contained objects, the RepInfo Manager copies the RepInfo from the registry. The RepInfo Manager will utilize a configurable policy to decide to what extend the RepInfo network is copied. 30

31 Ingest AIP Data Flow Within PDS (2/3) The Fixity Manager computes the fixity using a hierarchical hash computation tree. The AIP may include hints to how strong the fixity algorithm should be. The computed fixity will be exposed The Provenance Manager generates the provenance structure and adds the initial events. The Context Manager checks the context referential integrity. The Context Manager will check the existence of the hard links but not the soft links. The RepInfo Manager associates the PDI RepInfo with the given AIP. The PDI RepInfo is known to PDS but it needs to be associated with the AIP to satisfy the self-containment requirement. 31

32 Ingest AIP Data Flow Within PDS (3/3) The Placement Manager computes to which cluster to assign the given AIP. Each cluster is self-contained and all its artifacts will reside physically together on the same media unit e.g. tape volume. The Placement Manager has a configuration file that includes input from the user, the media units sizes, the AIP context, the Content Information RepInfo, etc. This configuration file is used to compute the cluster assignment. The Placement Manager updates the cluster object and optionally compress it. It also updates the AIP-to- MediaUnit table. The Placement Manager optionally exports (unmount) the cluster object to the media unit using a self-describing self-contained format. Preservation DataStore sends to the OAIS Ingest entity a storage confirmation event indicating (or verifying) the storage identification information for the AIP. 32