PAWN: A Novel Ingestion Workflow Technology for Digital Preservation. Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall

Transcription

1 PAWN: A Novel Ingestion Workflow Technology for Digital Preservation Mike Smorul, Joseph JaJa, Yang Wang, and Fritz McCall

2 Overall Principles Consistent with the Open Archival Information System (OAIS) model Distributed, secure ingestion Use of web/grid technologies platform independent Minimal client-side requirements Ease of integration with archival storage or data grid systems.

3 Producer Producer Management Interface Management Server Archive Producer data suppliers

4 Producer Provides data to an Archive based on a prior agreement. Consists of a management/metadata server and an ingestion client. Provides initial arrangement, context, and metadata.

5 Archive - receiving Producer 1 Digital Archive Producer 2 Load Balancer Producer n Bitstream Validation Service

6 Archive receiving Receives data from a Producer Validates bitstreams and metadata, and sends acknowledgement to Producer. Arranges into collections and specifies preservation policy. Publishes bitstreams into a digital archive.

7 Archive Long term preservation Implemented using grid technologies. Use the existing prototype NARA/UMD/SDSC site. Automated replication and integrity checking. Enforces access control and preservation policy

8 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet (SIP) creation. 3. Transfer of SIPs to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

9 Submission Agreement Based on data appraisal and record schedule, including format and metadata. Create machine actionable set of rules describing items. Final Submission Agreement is composed of: METS document for application defaults METS Constraint document to limit METS form to submission parameters

10 METS Overview Provides a framework for linking structural organization of objects with metadata. Using XML namespace, metadata from various XML schema can be attached to objects Ie, dublin core, FGDC, etc Extensible for more complex metadata

11 Sample METS Document Metadata Linking <?xml version="1.0" encoding="utf-8" standalone="no"?> <mets xmlns=" xmlns:xlink=" xmlns:xsi=" xsi:schemalocation=" <metshdr> <agent ROLE="CREATOR"> </agent> </metshdr> <filesec> <filegrp> <file ID="5" MIMETYPE="application/octet-stream" SIZE="67624" CREATED=" T15:36:05" CHECKSUM="2CE7D79E40BD6C6A65A6684B6FD3D08C" CHECKSUMTYPE="MD5"> <FLocat LOCTYPE="URL" xlink:type="simple" xlink:href="/nfshomes/toaster/iscsi/gfs-contrib-5.1.tar.gz"/> </file> </filegrp> <filegrp> <file ID="7" MIMETYPE="application/octet-stream" SIZE="2517" CREATED=" T17:06:07" CHECKSUM="767185AA022180E701324C592E1C36E3" CHECKSUMTYPE="MD5"> <FLocat LOCTYPE="URL" xlink:type="simple" xlink:href="/nfshomes/toaster/iscsi/gfs.out"/> </file> </filegrp> </filesec> <structmap> <div ID="3" LABEL="iscsi"> <fptr FILEID="5"/> <fptr FILEID="7"/> </div> </structmap> </mets> Structural Organization

12 Why METS Constraints? METS doesn t provide a way to create machine interpretable rules describing a collection Ie: allow only JPEG files in certain structural areas METS profiles allow for developer interpretable rules, not machine interpretable

13 METS Constraints Allows structural, metadata, and file constraints. Structural Constraints: Restrict child div s and restrict pointers to div, file, and other mets documents File Constraints: Restrict files by mime-type or validation tests Metadata Constraints: Restrict allowed metadata schema.

14 METS Constraints Example <techmd ID="WORD97"> <mdwrap LABEL= MS Word 97 > <arc:valgrp required="yes"> <arc:valtest class="wordextension" required="no"> <arc:valtest class="wordparser" required="yes"> </arc:valgrp> </mdwrap> </techmd> <structmap TYPE="logical" > <div ID="DIV1" LABEL="Toxic Chemical Release Inventory System" > ID="DIV2" ORDER="1" LABEL="Reports for 1997 DMDID="tree97"> </div> <div ID="DIV3" ORDER="2" LABEL="Meeting Notes for 1997" DMDID="tree98"> </div>... </div>... </structmap> <divrule ID="DIV1" FILEALLOW="no" DIVALLOW="NO"></divrule> <divrule ID="DIV2" FILEALLOW="yes" DIVALLOW="yes"></divrule> <divrule ID="DIV3" FILEALLOW="yes" DIVALLOW="NO"> <filegrp> <file ID="GIF98"> <file ID="WORD97"> </filegrp> </divrule>

15 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIPs to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

16 Initialize Ingestion workflow Instantiate Producer management server to track registered objects Establish a working trust relationship with the Archive Issue clients.

17 Create SIP Each client registers objects stored locally with producer management server Register file types, validation tests, etc Client follows rules in Submission Agreement Producer-wide agents can arrange registered object to give a broader context

18 SIP Example OAIS Information packet Content Information Preservation Description Information Physical Object Representation Information Provenance Fixity Reference Context Packaging Information Descriptive Information METS Handles all areas of a SIP except Physical Object and Descriptive Information Descriptive Information can be embedded into METS as 3 rd party XML schema

19 Client Interface

20 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIPs to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

21 Transfer SIP to archive Retrieve previously registered SIP from producer management server Authenticate to archive Update METS document with file location within SIP Transfer METS document describing SIP and container for SIP physical objects Archive acknowledges transfer completion to producer management server

22 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIP to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

23 Validation of SIP transfer Check incoming SIP against constraints documents. Ensure object integrity by verifying checksums/cryptographic digest Validate bitstreams against tests described in METS document Update METS document with validation results and movement of objects on receiving server

24 Ingestion Workflow 1. Negotiate Submission Agreement. 2. Workflow Initialization and Submission Information Packet creation. 3. Transfer of SIP to archive. 4. Validation of SIP transfer 5. Organization of data into collections and transfer into persistent archive.

25 Final transfer to archive Transfer objects to digital archive Update METS document with handle to object in archive Transfer METS document into archive Return accept/reject messages to producer metadata server

26 Component Overview Producer Management Interface Archive Management Interface CRL check Success/Failure notification of ingestion Archive Data Grid METS document registration/retrieval SIP transfer Producer data suppliers Bitstream Validation Service

27 Producer Components Database to track registered objects Certificate Authority management Web service for archive security check Management server supplies web service interfaces to ingestion clients and management operations. Clients are designed to be standalone, with security certificates issued by producer

28 Archive Components Receiving servers validate connecting clients and validate SIPs Validation Services are simple webservice calls. All components are scalable using standard load balancing techniques.

29 Recap Implemented using web technologies Architecture independent OAIS compliant XML based metadata METS based SIPs Add-on constraints describing Submission Agreement

30 Questions?? For more information