Montana State University Library Digital Preservation Procedures Last updated 14 August 2018

Transcription

1 Montana State University Library Digital Preservation Procedures Last updated 14 August 2018 Table of Contents 1. Overview of digital collections Content scope for digital collections Born digital content Preservation masters and access copies... 2 Table 1. Supported formats for digital preservation Preservation masters procedure Masters procedure tasks by role... 3 Archivist/Curator tasks... 3 Digital Operations Manager tasks... 3 App Developer tasks... 4 Systems Administrator tasks Technical metadata File naming conventions [last updated August 2018] Sensitive or nonpublic data Storage and backup Tape backup TACC cloud backup of digital masters Disaster recovery Roles and responsibilities Staffing and training Sustainability and financial planning... 7 Appendix A. Identification of content... 8 Appendix B. Accession record template University accession Manuscript collection accession Appendix C. Digital project proposal forms C.1 Digital collection project proposal overview C.2 Digital application project proposal overview C.3 Proposed project evaluation form C.4. Digital collection content scope form Appendix D. Certificate of gift Appendix E. Digital preservation levels - decision flowchart (draft) Appendix F. National Digital Stewardship Alliance levels of digital preservation Appendix G. OAIS reference model definitions SIP, DIP, and AIP contents at Montana State University

2 1. Overview of digital collections The digital collections at MSU Library (henceforth The Library ) currently comprise digitized archival materials, scholarly publications in ScholarWorks, and audio files made available in the Acoustic Atlas. Please see ScholarWorks Preservation and Migration Policy for detailed information about ScholarWorks content. Please see the Montana State University Library Digital Preservation Policy for general policy guidelines. For detailed content profiles, please see Appendix A. 2. Content scope for digital collections The Library s content scope is outlined in the MSU Library Collection Development Policy. The Library requires an application form to suggest new digital collections (see Appendix C for application form). This form provides a structure for prioritizing new digital projects. 2.1 Born digital content The procedures in this document apply only to digitized content, the specific born-digital content in ScholarWorks and Acoustic Atlas (text files and sound files), and some special borndigital projects such as Angling Oral Histories. As of 2018, born digital content is reviewed on a case-by-case basis, and heterogeneous born digital media donations such as hard drives, floppy discs, and CDs are rarely accepted. These procedures will be reviewed as the Library grows and changes. 3. Preservation masters and access copies For all digitized materials in the Library s collection, the high-quality digital file will be considered the preservation master, and a lower-quality access copy will be created for online access via the Library website. This policy will be reevaluated as the collection grows. Especially for at-risk analog materials, the high-quality digitized file will be considered the master. For example, for VHS tapes, the digitized AVI file is considered the master. Some materials may not be subject to full digital preservation treatment. For example, by agreement with the International Snow Science Workshop (ISSW), the ISSW Proceedings are not preservation masters, they are simply access copies. For born-digital materials, we strongly suggest that donors/creators provide materials in a supported format (see Table 1). If materials are donated in other formats, we cannot guarantee full digital preservation treatment only bit-level preservation. The Library also reserves the right to migrate formats if necessary. A format review and migration for supported formats is conducted every five years. 2

3 Table 1. Supported formats for digital preservation Non-proprietary, openly documented formats are preferred. 1 Type of content Filetype Notes Structured data Moving images Sounds Still images Tabular data Text Web archive Compressed/ archived formats Common proprietary formats XML, JSON, CSV MOV, MPEG, AVI, MP4 WAV, FLAC, AIFF, MP3 TIFF, JPEG 2000, PDF, PNG CSV, TSV XML, PDF/A, HTML WARC TAR, GZIP, ZIP We can also currently support MS Office formats such as XSL, DOC, and PPX Files should only be compressed and/or archived when it is necessary due to large file size or the need to gather files together in a particular directory structure in order for them to be understood. This policy may be re-evaluated in the future Please see Appendix E for more information on levels of digital preservation at MSU Library. 4. Preservation masters procedure Master storage. Master files are stored in the DP13 folder in the local DP drive. In 2018, the Library Systems team doubled the size of DP13. As of April 2018, 21% of the drive is in use. Inventory spreadsheet. An inventory spreadsheet is stored alongside the masters, in the root directory of DP Masters procedure tasks by role Archivist/Curator tasks Accession record. Archivist or curator creates an accession record for all collections, including digital collections. Includes either a unique collection identifier, which is either an accession number for University records or a collection number for manuscript collection records. For accession record templates, please see Appendix B. Digital Operations Manager tasks Collection-level readme. Digital Operations Manager creates a collection-level readme for each digital collection in DP13. The readme includes the title of the collection, date accessioned, date 1 For a full list of preferred formats, please see Library of Congress preferred digital formats. 3

4 digitized, date moved to DP13, scope and content note, whether there are sensitive or restricted files in the collection, and any other special information. Digital collections audit. Audits of digital collections and masters are conducted by the Digital Operations Manager and Special Collections staff. Audits will be repeated every 2 years. The frequency of audits can be adjusted as needed. FITS metadata. See 4.2 Technical metadata. App Developer tasks Database snapshots. See 4.2 Technical metadata. Systems Administrator tasks Compression. Systems Administrators individually compress files in DP13. When supplying high-quality TIFFs to patrons, library employees either unzip the file or send the compressed file to patrons. Read only permissions. Systems Administrator moves masters from changes folder to DP13 and sets them to read only. Long term storage. Systems Administrators copy files from DP13 to TACC cloud storage (see 5.2 TACC cloud backup). Checksums. Fixity of metadata and files is checked with yearly checksums, which are created using MD5 Deep. 4.2 Technical metadata Database snapshots A snapshot of our digital collections SQL database, which contains descriptive metadata, is stored with the masters. There will be one database snapshot for each collection. Objects in database correspond with unique URL of materials. File Information Tool Set (FITS) The File Information Tool Set (FITS) identifies, validates and extracts technical metadata for a wide range of file formats. It acts as a wrapper, invoking and managing the output from several other open source tools. Output from these tools are converted into a common format, compared to one another and consolidated into a single XML output file. 2 FITS files will be created in a top level masters folder within each collection Folder: collectionnumber-technicalmetadata FITS generates a file with the same name +.fits Includes file format verification and other technical metadata such as file name, size, file system last modified date, and md5 checksums

5 4.3 File naming conventions [last updated August 2018] ***These naming conventions apply to future collections. All legacy collections will keep their existing naming schema. MSS Collections: [Col#-Box-Folder-Item] img00000 [Col#-Box-Folder-Image] Accessions: [Accn#-Box-Folder-Item] img00000 [Accn#-Box-Folder-Image] Born Digital: [Col#-Series-Folder-Item] Trout Art: [Col#-Series-Folder-Item] Trout Oral: [use lower-case only] smith-bob [Last name-first name-date of Interview (year-month-day)] apple-jane-bert [If there are 2 people in it (as a couple), it is Last name-2 First names-date of Interview (year-month-day)] ford-betty-regan-nacy [If 2 or more people with different last names for each Last name-first name-last name-first name-date of Interview (year-month-day)] ***Future oral programs: assign a collection number and use born digital naming scheme 4.4 Sensitive or nonpublic data Most digital materials in the Library's collections are freely available, with some exceptions: Electronic theses and dissertations may be embargoed for a limited time. Some materials in Special Collections have restricted use policies. For example, the Aubrey Haynes Papers are restricted to one computer in the reading room. 5

6 5. Storage and backup 5.1 Tape backup All Library data are onsite, and all data are backed up monthly. Each month, one backup copy of the monthly data are sent offsite to a location 0.75 miles from MSU. Any changes to the monthly backup copies are backed up nightly (onsite), and every 2 weeks (offsite). Twice yearly (June and October), tape backups of all Library data are sent to an Iron Mountain storage vault in Cincinnati, OH. 5.2 TACC cloud backup of digital masters Yearly (February), a full copy of all digital masters in DP13 is uploaded to the TACC cloud backup service. All files are compressed, using TAR to maintain permissions. 5.3 Disaster recovery In the unlikely event of complete destruction of the onsite data (i.e. earthquake, fire, flood, human error in file handling), the recovery process is as follows: acquire internet enabled facility purchase and install common hardware (the hardware used at MSU Library is readily available) recover systems and data from offsite backup(s). We estimate that disaster recovery procedure could take up to a week, barring any problems obtaining access to a properly networked facility. This scenario assumes MSU would recover their network domain and DNS, so that hostname changes would be unnecessary. 6. Roles and responsibilities Permissions DP drives are accessible to multiple employees in the following departments: Digital Library Initiatives (DLI); Cataloging, Access, & Technical Services (CATS); and Special Collections & Archival Informatics (SCAI). DP13 is the single storage location for all preservation masters. File permissions are set to read only at the point of transfer to DP13. SCAI employees may access the read only preservation masters for patron reuse and reproduction. These requests generally occur once or twice a month. Responsibilities The MSU Library Digital Preservation Group oversees digital preservation in the Library. The Digital Operations Manager provides staffing oversight. The Systems Administrators oversee storage and backup. For specific tasks in the preservation masters procedure, please see 4.1 Masters procedure tasks by role. 6

7 7. Staffing and training This digital preservation implementation plan will be used in new staff training to support standardized digital preservation practices in the Library. The MSU Digital Preservation Group will meet periodically to review current local practices and evolving digital preservation best practices. 8. Sustainability and financial planning While there are no dedicated funds to digital preservation at MSU Library, Library Administration has agreed to support digital preservation activities. With the exception of offsite backups, most of the tools used to implement digital preservation at MSU Library are free and open source. 7

8 Appendix A. Identification of content Last updated: April 2018 Content type = Main area or content stream Description = high level descriptive information about content type Acquisition = how content types are created and/or acquired by the Library Size = current total data size of content type Complexity level = designation related to variety / range of file formats that are typically included in content type Current management / storage = how / where digital objects within content type are currently stored and managed. Locally vs. vendor / hosted Rights = how rights are captured / transferred to Library Value = designation related to value and preservation commitment for content type Priority = numerical assignment of priority for preservation activities (1=highest priority; 3=lower priority) Content type Digitized content Description Digital content created through digitization of analog materials by Digital Production Unit Created internally by Library staff Size 5 TB Complexity level Medium Primarily consists of image and document file formats Current management / storage Internal Library storage network, preservation masters procedure, backups both in the library and offsite. Rights Value Prio rity Licenses for reuse are obtained by MSU when possible. Users have responsibility to obtain permission for reuse beyond fair use Acquisition Longterm 1 Institutional repository content Open Access scholarly work (including publications and data) created primarily by MSU faculty and graduate students Created externally and acquired by Library staff or submitted through IR platform 500 GB High We have preferred formats, but we accept all file formats Internal Library storage network, backups both in the library and offsite, some preservation actions (provenance metadata and checksums) through DSpace. For OA copies of scholarly papers: Scholarly Communication Librarian assures that we have the right to repost. For ETDs: students agree to SW posting when submitting thesis Longterm 2 8

9 Acoustic Atlas Born-digital sound recordings acquired from researchers, primarily Jeff Rice Created externally and transferre d to the library (Molly Arrandale is project manager) 1 TB Medium all sound files Internal Library storage network, backups backups both in the library and offsite Longterm 1 Born-digital content in SCAI Web Archives content (trial beginning in 2018) Licensed electronic resource (permanent acquisitions) Licensed electronic resource (subscription) Born-digital materials acquired from individuals/ orgs. Archival collections, oral history collections, and special collections materials Web content created by University departments, units, and affiliated entities Electronic resources purchased with perpetual access Electronic resources purchased on a subscription basis Created externally and acquired by SCAI (or in the case of Angling Oral Histories, created by SCAI) Will be created externally, acquired through automatic web harvesting via Archive-It service Purchased from vendors Purchased from vendors 2.5 TB 0 GB High Wide range of file formats including high amount of legacy and obsolete formats High Wide range of file formats and complex relationships between digital objects Low vendors typically have a standard file format, however metadata are often lacking Low most subscriptions are preserved through Portico For materials created externally: only storage, no management or preservation of any kind. For Angling Oral Histories: full preservation External storage through vendor hosted solution (Archive-It) Disaster backup through Portico Portico Transferred via Certificate of Gift agreement with Donor. (See Appendix D) Angling Oral History: created with agreement from interviewee No explicit transfer of rights from creators? This will be reevaluated following the creation of a TRAILS-wide web archiving ethical framework Managed individually through license agreement Managed individually through license agreement Longterm Longterm Nearterm Nearterm

10 Appendix B. Accession record template University accession Office of Origin Personal Name Records Title Date of Accession Data Processed Shelf Location Cubic Feet Number of Boxes Notes Provenance Note Historical Note Content Description Restrictions (if applicable) Inventory Location Accession Number Subjects Manuscript collection accession Main Entry Title Donor Address Beginning Date Ending Date Donation Date Record Date Location Number of Boxes Physical Description Accession File Name Notes Provenance Note Historical Note Content Description Restrictions (if applicable) Descriptors 10

11 Appendix C. Digital project proposal forms C.1 Digital collection project proposal overview Date: Name: New Digital Collection Functionality Enhancement Upgrade Digital Collection Content Upgrade 1. Please provide a brief description of the proposed MSU Digital Collection. 2. Briefly describe how the collection's materials will be used to support the educational and research mission of Montana State University and/or the MSU Library's Digital Initiatives Mission (See: for more information.) 3. Does a copyright exist for the materials or collection? If yes, who owns it? 4. Does a detailed physical inventory of the materials already exist? (For example: number of physical items, format and size of items, physical material types, condition, etc.) If yes, please provide the information in a spreadsheet or text file or document table. 5. Have the materials already been cataloged? If yes, please provide a brief overview that explains the catalog form: Dublin Core, MARC, other. 6. Are there any affiliated scholar, organizations, partners or sponsors associated with the project? If yes, please list them. 11

12 7. Please explain how the materials may be considered unique or rare and how researchers currently access them. (For example: finding-aid, card catalog, website, digitized copy, on-site only, etc.) 8. Please elaborate on any known exceptional requirements associated with this proposal that the MSU Library Digital Initiatives Steering Committee should know about. Please attach any Supporting Proposal Materials that will help the MSU Library Digital Initiatives Steering Committee further understand the scope and value of your proposed digital collection project. Supporting Proposal Material examples: detailed physical inventory, metadata suggestions, information about associated project grants, additional staff and/or support available for the project's metadata and/or digitization processes, etc. Please let us know if you need more information in filling out this form. Our contact information is: Digital Operations Manager Montana State University Library Bozeman, MT

13 C.2 Digital application project proposal overview Date: Name: New Application Functionality Enhancement Upgrade 1. Please provide a brief description of the proposed MSU Digital Application Project. 2. Briefly describe how the application will be used to support the educational and research mission of Montana State University and/or the MSU Library's Digital Initiatives Mission (See: for more information.) 3. Are there any affiliated scholars, organizations, partners or sponsors associated with the application project? If yes, please list them. 4. Please explain how the digital application might be considered unique, and how researchers, students, and the general public will use it. 5. Please elaborate on any known exceptional requirements associated with this application proposal that the MSU Library Digital Initiatives Steering Committee should know about when reviewing the Digital Application Project Proposal. Please attach all materials supporting the Digital Application Project Proposal that will help the MSU Library Digital Initiatives Steering Committee further understand the scope and value of your proposed digital project. Supporting Digital Application Project Proposal Material examples may include but are not limited to: information about the project's associated grant(s), list of leadership and/or organizations supporting the project, lists of technology. 13

14 C.3 Proposed project evaluation form Submitted by: Title of Project Proposal: General Questions 1. Is the proposed project for: Digital Application Digital Collection Both Other 2. Is the project proposal in the scope of MSU educational and research mission? Yes No Maybe If maybe, elaborate on what next steps to take for Steering Committee to make firm Y/N decision. 3. Do copyright or proprietary restrictions exist for either content, application or both Yes No Maybe If yes, can they be overcome to deliver open-source, access-free, creative commons content and/or application code? If maybe, explain next steps for Steering Committee to take to make a firm Y/N decision. 4. Is the project proposal subject to grant deliverables? Yes No Maybe If yes, what is the grant, and elaborate on scheduled deliverables. 5. Is the project a potential revenue (MOU) deliverables? Yes No Maybe 6. there specific scholars, organizations, partners, sponsors or in-house champions associated with this project proposal? Yes No Maybe If yes or maybe, elaborate why these are important to MSU Library Digital Initiatives. 14

15 C.4. Digital collection content scope form Date: Name: 1. Please provide a brief description about the person or collection contents. 2. Describe the approach for choosing samples to be included in the digital collection. Please list the names of the Special Collections and/or Archives collections that will be used. 3. Are there any affiliated scholars, organizations, partners or sponsors associated with the development of this digital collection? If yes, please list them. 4. Are there any specific deadlines and deliverables associated with the processing of this digital collection? If yes, please list them. 5. Are there any unique Standards and/or Controlled Vocabularies to be used with the metadata processing of this digital collection? If yes, please list them. 15

16 6. Please describe the files types to be generated during project. 7. Please list all the files types to be generated during project. 8. Are there any special handling of original materials or unique file formats or digitizing processing associated with this digital collection processing project? 9. Please provide an estimated numbers for: Estimated number of Scans Estimated number resource files to be processed (jpg, pdf, wav, mp3, etc.) Estimated number of finished records 16

17 Appendix D. Certificate of gift I/We of (name) (city, state) convey to the Montana State University Library the following: This is an unrestricted gift that transfers to the Montana State University Library all legal title, copyright and literary property rights insofar as I/we hold them unless exceptions or restrictions are specifically noted below: I/We agree that any materials described above that are determined to be inappropriate to the Special Collections or general library collection shall be disposed of by the library as it sees fit or the items be returned to me/us ifl/we expressly state this below: Signature Date Signature Date Title (Organizations or Businesses) Witness Date The gift described is gratefully accepted by Montana State University Library by Signature Date Printed Name Title 17

18 Appendix E. Digital preservation levels - decision flowchart (draft) Is this a unique collection of enduring value that was not commercially mass produced? - No > do not archive content - Yes > go to next question Do you have permission from the copyright holder (if necessary) to archive this digital collection? - No > do not archive content - Yes > go to next question Are any objects in this collection available in another Trusted Digital Repository? - Yes > which repository?. If collection is in another Trusted Digital Repository, Do not archive content. - No > go to next question Is this collection available in a Trusted Print Repository or is there a hard copy available that will be kept long-term? - No > full preservation - Yes > go to next question Is the hard copy deteriorating or in poor condition or on a near-obsolete format or media? - Yes > full preservation - No > go to next question If the content is born digital, is it in a supported file format? - Yes > discuss with SCAI and DLI whether the content warrants preservation - No > bit-level preservation 18

19 Appendix F. National Digital Stewardship Alliance levels of digital preservation Level 1 (Protect your data) Level 2 (Know your data) Level 3 (Monitor your data) Level 4 (Repair your data Storage and Geographic Location - Two complete copies that are not collocated - For data on heterogeneous media (optical discs, hard drives, etc.) get the content off of the medium and into your storage system - At least three complete copies - At least one copy in a different geographic location. - Document your storage system(s) and storage media and what you need to use them - At least one copy in a geographic location with a different disaster threat - Obsolescence monitoring process for your storage system(s) and media - At least three copies in geographic locations with different disaster threats - Have a comprehensive plan in place that will keep files and metadata on currently accessible media or systems File Fixity and Data Integrity - Check file fixity on ingest if it has been provided with the content - Create fixity info if it wasn't provided with the content - Check fixity on all ingests - Use write-blockers when working with original media - Virus-check high risk content - Check fixity of content at fixed intervals - Maintain logs of fixity info; supply audit on demand - Ability to detect corrupt data - Virus-check all content - Check fixity of all content in response to specific events or activities - Ability to replace/repair corrupted data - Ensure no one person has write access to all copies Information Security - Identify who has read, write, move, and delete authorization to individual files - Restrict who has those authorizations to individual files - Document access restrictions for content - Maintain logs of who performed what actions on files, including deletions and preservation actions - Perform audit of logs Metadata - Inventory of content and its storage location - Ensure backup and non-collocation of inventory - Store administrative metadata - Store transformative metadata and log events - Store standard technical and descriptive metadata - Store standard preservation metadata File Formats - When you can give input into the creation of digital files, encourage use of a limited set of known open formats and codecs - Inventory of file formats in use - Monitor file format obsolescence issues - Perform format migrations, emulation and similar activities as needed 19

20 Appendix G. OAIS reference model definitions These procedures follow the best-practices defined by the OAIS Reference Model. 3 This model defines three information packages: Submission Information Package (SIP), Dissemination Information Package (DIP), and Archival Information Package (AIP) SIP, DIP, and AIP contents at Montana State University SIP physical object to be scanned, or in rare cases a born-digital object. DIP PDF, JPG, or other derivative file format, as well as descriptive metadata [at MSU Library, these metadata are documented in a SQL database]. AIP TIFF or other master file format, collection-level readme, FITS metadata, and SQL database snapshot. 3 Consultative Committee for Space Data Systems (CCSDS). (June 2012). Reference Model for an Open Archival Information System (OAIS): Magenta Book. 20