Preservation Metadata Extraction and Collection : Tools and Techniques. Mat Black National Library of New Zealand Te Puna Matauranga o Aotearoa

Transcription

1 Preservation Metadata Extraction and Collection : Tools and Techniques Mat Black National Library of New Zealand Te Puna Matauranga o Aotearoa

2 How to get what you need to keep what you ve got

3 The stack Fixity generation Virus checking Format identification Format validation Enviromental metadata collection Format specific metadata extraction

4 Fixity Get it early and get it right Common fixity types: Hashing algorithms (MD5, SHA1) Digital signatures File size? Use multiple fixity algorythems. Find out the legal implications.

5 Fixity values for what? File Bitstream Compound (all the files in an object) Metadata The whole lot (files, filename & metadata)

6 Virus checking Virus check datetime Results including false positives and any warnings (word macros etc) The virus checker name and version The virus pattern file name and version The virus engine name and version

7 Format identification File / Bitstream / Complex Methods of format identification File name or extension File type/creator codes (Old Mac s) Magic numbers Brute force file parsing (for all, try throw catch)

8 A sound file opened in an image viewer

9 What file format is this? And the winner is..

10 Subzero by Pain Receptor They describe their music as sounding like falling down the stairs carrying leeches and bottles

11 Sub format identification Embedded Bistreams XML Base64 encoded octet streams Microsoft Structured Storage Archives ZIP, TAR, ARC Encapsulation/Container formats OGG, AVI, MIME CODEC s DV, DivX, Indeo, Cinepak, MS MPEG-4

12 Available tools File extensions (google it) Magic utilities (google it) Jhove DROID Build you own! (Java, PERL, C#, C++) If you have a fixed format list You use a proprietary format.

13 Format validation Types of validation Pattern comparison Parsing Rendering

14 Available tools JHOVE NLNZ Extract tool (sort of) The application used to create the file Anything that opens a file and can throw an error. Parsing tools E.g. XML Parsers, XML Schema, PERL Modules, Java Classes. rendering tools E.g. LibTIFF, ImageMagick, Microsoft Office (wrapped), OpenOffice PERL Modules, Java classes, etc.

15 Things to keep in mind Test it till it breaks. Define your requirements, break them, then define them again. (Repeat if required). Not all tools are created equal. Not all tools obey the rules. Some rules are made to be broken.

16 Environmental metadata Consider the native environment of your content. Is there metadata that you need that only exists in a digital objects native environment? Structure and relationships. File system attributes

17 Format specific metadata extraction aka format characterisation Available metadata will vary depending on the format. You will probably need format specific schemas. The types of metadata that can be extracted: Preservation Descriptive Structural Administrative Rights Technical

18 The big question. Why would I extract the metadata now and store it in a database if I can just come back and extract it again later when I need it?

19 Available tools NLNZ Metadata Extract Tool JHOVE Anything you can wrap LibTIFF, ImageMagick, PERL Modules, Java classes etc Build your own! And make sure you open source it

20 What tools should I use? Use as many tools as you need to. Keep the workflow configurable Preferably by content or format type. Allow for multiple tools to be used. Allow for new tools to be added later. Compare metadata from multiple tools.

21 The workflow 1. Fixity generation 2. Virus checking 3. Format identification 4. Format validation 5. Enviromental metadata extraction 6. Format specific metadata extraction 7. Store in repository

22 Paranoid workflow 1. Fixity generation 2. Virus checking 3. Fixity check 4. Format identification 5. Fixity check 6. Format validation 7. Fixity check 8. Enviromental metadata extraction 9. Fixity check 10. Format specific metadata extraction 11. Fixity check 12. Virus check 13. Store in repository 14. Fixity check 15. Virus check 16. Fixity check

23 Paranoid access flow. Retrieve content from repository Fixity check Virus check Send content to consumer

24 Global Digital Format Registry Format identification components Format validation components Metadata extraction components Format registry At risk content alerts

25 Questions?