Digital Preservation Workshop

Transcription

1 Digital Preservation Workshop 10 November 2010 University of Victoria Peter Van Garderen Artefactual Systems

2 Workshop Agenda 10:00 Introductions 10:15 What is digital preservation? From strategy to implementation 12:00 Lunch 13:30 Free and open source software Archivematica technical overview Archivematica & ICA-AtoM demo/tutorial Preservation planning (time permitting) 15:30 Wrap-up NOTE: open discussion / Q&A encouraged throughout

3 The content in this presentation may be freely re-used under the terms of the Creative Commons Attribution-Non-Commercial-Share Alike 3.0 license. Attribution: Title: Archivematica: Digital Preservation Workshop Creator: Peter Van Garderen, Artefactual Systems Date: 10 November 2010 Artefactual Systems Inc. 2010

4 Peter Van Garderen President / Systems Archivist Jack Bates Software Engineer David Juhasz Software Engineer Austin Trask Systems Engineer Evelyn McLellan Systems Archivist Jesús García Crespo Software Engineer open-source sofware for archives and libraries digital preservation consulting services Joseph Perry Software Engineer

5 Digital Preservation: planning for the long-term accessibility and usability of authentic digital information

6 a.k.a. digital curation digital continuity

7 The Digital Preservation Problem: 1) Fragility of digital storage media 2) Lack or loss of adequate metadata 3) Lack of responsibility and resources 4) The volume of digital information 5) The complexity of digital information 6) Rapid technological change

8 content structure context presentation behaviour digital information intellectual entity object entity representation file bitstream

9 content structure context presentation behaviour digital information intellectual entity object entity representation file bitstream Information Record Archival Material

10 The Digital Preservation Problem: 1) Fragility of digital storage media 2) Lack or loss of adequate metadata 3) Lack of responsibility and resources 4) The volume of digital information 5) The complexity of digital information 6) Rapid technological change

11 contextualize authenticate relate / bind find package now header information error correction storage media bitstream storage device stored conserved protected file system user interface operating system storage driver application software metadata input / output devices future

12 Digital Preservation is Risk Management

13 Risk inability to provide services, manage programs and operate business functions efficiently because of digital information that is not accessible or usable.

14 Risk Poor quality decision-making because digital information that would have been otherwise available has disappeared or can t be trusted to be authentic.

15 Risk Exposure to legal liability because the digital information that serves as evidence of the organization s compliance or accountability in its contractual, governance and administrative obligations has been lost or can t be trusted to be authentic.

16 Risk Heightened risk of non-compliance with laws, regulations and policies due to inaccessible electronic records.

17 Risk The organization acquires a reputation as an irresponsible, incompetent or untrustworthy institution.

18 Risk Lost opportunities to re-use and exploit information in digital form.

19 Risk Unforeseen cost-creep because the ongoing preservation of digital information is overlooked during the calculation of costs for new or modified systems.

20 Risk Corporate memory loss and cultural amnesia as the digital information that documents the governance, administration and culture of an organization or society disappears from servers and systems before steps have been taken to preserve it.

21 The Business Case for Digital Preservation Manage risks Manage information Manage storage

22 The anti-business Case for Digital Preservation don t we already have backup and a business continuity plan? don t we just upgrade the software? storage is cheap we ll just index everything why can t we use the ERDMS/ECM system for this?

23 Staff Desktops ( , docs, files) Business Systems (structured data) capture Legacy Systems & Data Website(s) ERDMS / ECM Scanning / Imaging store capture Individual/Ad Hoc Accessions Digital Archives active documents store transfer inactive documents archival material destroy organize organize access access Staff preserve access External Researchers

24

25 E-Record Creating Environment Category 1: preserve in source system ERP records schedule Data Warehouse DAM?? Shared Drives?? Category 2: transfer to digital archives ERDMS

26 What are the requirements for a digital preservation system?...that depends, what do kind of future were you expecting? now future?

27 Digital Preservation Strategies technology preservation emulation migration normalization

28 Digital Preservation System Core Requirements OCLC/NARA Trustworthy Repositories: Audit & Certification (TRAC) ISO Open Archival information Systems (OAIS) df PREMIS Data Dictionary for Preservation Metadata

29 Open Archival Information System Preservation Planning P R O D U C E R Data Management SIP Ingest AIP Archival Storage Administration MANAGEMENT Access DIP C O N S U M E R

30 Open Archival Information System ISO High level reference model Default language of the digital preservation world Key concepts: Mandatory Responsibilities Functional Entities Information Packages Actors

31 OAIS Solutions Proprietary Safety Deposit Box Rosetta Open Source RODA Archivematica

32 OAIS Solutions vs. Digital Preservation Tools Plato Planets Testbed DROID PRONOM JHOVE Fedora Xena Dioscuri

33 OAIS Solutions vs Digital Preservation Initiatives Planets CASPAR InterPARES NDIIPP

34 Business Case Opportunities ERDMS, ECM implementation Enterprise search implementation Business process/records scheduling analysis Archiving and storage pressure Audits FOI and disclosure/transparency inititiaves Open Data / Open Government initiatives Response to digital preservation strategy Inter-institutional partnerships

35 Project Milestones Strategy Business Case Technical Analysis Proof-of-concept Pilot(s) Production System maintenance System scope

36 Identifying Pilot Projects Scoring Factors Technical difficulty Intrinsic value Pilot Candidates Institutional Repository Obsolesence risk Transparency risk ERDMS Breadth of collection Shared directories Descriptive metadata External media

37 Project Costs Software licenses (proprietary) Software installation Application server Security Storage integration Software customization Legacy/source system migrations System-specific ingest/transfer templates Access system integration Annual maintenance Technical support Release upgrades Staff Hardware Storage

38 Free Beer!

39 They ll never take our freedom 1995 Paramount Pictures & 20th Century Fox See fair use rationale:

40

41 Free Puppy!

42 Users Lead institutions Funding Development All users Bug reports Enhancement requests Code patches Documentation Promotion Foundation or Steering Committee Code Time Money Knowledge Time Money Knowledge Open Source Software Code Knowledge Community Code Time Money Knowledge Service Providers Development Technical Support Hosting Training The open-source eco-system Promotion Governance Coordination Funding Promotion

43 What is Archivematica? Archivematica is a comprehensive digital preservation system. Archivematica uses a micro-services design pattern to provide an integrated suite of free and open-source tools that allows users to process digital objects from ingest to access in compliance with the ISO-OAIS functional model. Archivematica uses METS, PREMIS, Dublin Core and other best practice metadata standards. Archivematica implements media type preservation plans based on an analysis of the significant characteristics of file formats.

44 Where did Archivematica come from? Artefactual Systems City of Vancouver Archives UNESCO Memory of the World International Monetary Fund Archives Rockefeller Archives Center University of British Columbia Library?

45

46 Open Archival Information System Preservation Planning P R O D U C E R Data Management SIP Ingest AIP Archival Storage Administration MANAGEMENT Access DIP C O N S U M E R

47

48

49

50 ISO-OAIS OAIS Use Cases UML Activity Diagrams requirements Digital Archives software system

51 ISO-OAIS OAIS Use Cases UML Activity Diagrams System Workflow Instructions requirements documentation

52 = manual step = automated step Producer places SIP in shared folder on host machine [host]/sendsip/ SIP appears in shared folder In Archivematica /1-receiveSIP/ Archivist copies SIP to SIP review folder /2-reviewSIP/ Archivist reviews SIP /2-reviewSIP/ Archivist adds descriptive metadata /2-reviewSIP/ Archivist moves SIP to quarantine /3-quarantineSIP/ = file directory The producer places a folder of objects in a designated folder on his or her computer. This designated folder has been set up so that it automatically sends its contents to a shared folder in Archivematica. The shared folder in Archivematica is 1-receiveSIP. When you are processing the SIP, leave the original copy in this folder as a backup in case you need to go back and start again. Check the SIP to make sure it conforms to Submission Agreement.. If MD5checksum.txt file is included, right-click and select Verify MD5 Checksum. Otherwise Archivematica will add checksums to the SIP logs directory and verify checksums at various time throughout the ingest process. Open the SIP. Right-click and select Add Dublin Core XML from the drop-down menu. Right-click the dublincore.xml file to open it with Mousepad. Add descriptive information to the appropriate Dublin Core elements and save the file.

53 Agile development method Time-based system releases Feb 2009: Release 0.1-alpha May 2010: Release 0.6-alpha November 2010: Release 0.7-alpha Each iteration leads to updated and improved: Requirements Software Documentation Development resources

54

55

56

57

58

59

60

61

62

63 success output directory watched input directory micro-service error output directory

64

65

66 Archivematica is: A classic Unix pipeline of OAIS micro-services provided by a series of opensource tools and integration code written in Python and Bash. Packaged as a virtual appliance that bundles the Xubuntu operating system and can be run within virtual machines, as a bootable USB or Live DVD, or as a bare metal install on dedicated machines.

67

68 Preservation planning: Media Type Preservation Plans

69

70

71

72

73

74

75

76 Preservation planning: Normalizing file formats

77 Defining normalization What is it? Normalization means converting ingested objects into a small number of pre-selected formats Why do it? Some formats are easier to preserve than others A smaller number of formats means fewer preservation actions required

78 Normalization vs. migration Migration is similar to normalization in that it involves converting ingested objects into preservation-friendly formats Unlike normalization, migration is typically done only when the format is at risk of obsolescence Migration as a strategy means adopting a wait and see approach

79 Disadvantages of normalization It requires more planning up front to implement Re-normalization may be required as better target formats or conversion tools become available

80 Advantages of normalization Taking preservation action on ingest helps define and manage risk Adopting a wait and see approach means putting off an undefined amount of work for an indefinite period of time at an unknown cost Normalization does not preclude the future use of migration or other strategies such as emulation

81 Criteria for choosing formats 1. The format must be non-proprietary There must be no associated licenses or patents or the possibility of there being such licenses or patents in the future

82 Criteria for choosing formats 2. There must be freely available specifications A specification is a document that explains exactly how the format is structured and rendered This specification must be freely available to all and not subject to copyright or other restrictions

83 Criteria for choosing formats 3. The format should be widely endorsed and/or adopted Other established repositories should be using or have endorsed the format Formats that have been approved as international standards are particularly desirable

84 Criteria for choosing formats 4. For images and audio files there should be no compression 5. For video files any compression should be lossless

85 Criteria for choosing formats 6. There should be writing and rendering tools available for the format Idealized standards must be matched by practical tools The tools must reliably meet the requirements of the format specifications and must produce normalized objects that are faithful representations of the original objects

86