Document Metadata: document technical metadata for digital preservation

Size: px

Start display at page:

Download "Document Metadata: document technical metadata for digital preservation"

Amice Stephens
6 years ago
Views:

1 Document Metadata: document technical metadata for digital preservation By Carol C.H. Chou - Florida Digital Archive (FDA) Andrea Goethals - Harvard University Library (HUL) March 24,

2 Table of Contents Introduction... 3 Applicable Formats... 5 Data Dictionary... 6 Document Metadata schema Appendix A: A sample PREMIS document with embedded docmd schema

3 Introduction The Florida Digital Archive (FDA) provides digital preservation for the eleven public universities in Florida. Since it was established in 2005, FDA has ingested over nine millions files with over 19 terabytes of data. There are approximately 50,000 files in document formats such as PDF, Word and OpenDocument Text Format. Most of these documents come from the Electronic Thesis and Dissertation (ETD) collections in those universities. Ensuring all FDA collections remain usable and renderable is one of the critical missions for FDA. Extracting technical metadata from documents is essential as it can aid in characterizing the kinds of documents in our preservation collections; listing document properties that may hinder preservation (encryption, external fonts, etc); and providing requirements in selecting tools/facilities for document transformation including normalization and migration. In addition, Document technical metadata can be used to verify the result of document transformations, ensuring the properties of the original document are preserved and properly transformed to the new document format. There are currently many metadata standards for various format groups. For images, there is MIX (NISO Metadata for Images in XML 1 ). For text-based formats such as plain-text, XML, DTD and HTML, there is TextMD (Technical Metadata for Text 2 ) schema. In terms of audio, there are two emerging standards: AES-X098B (work in progress by the Audio Engineering Society SC Working Group on Digital Library and Archive Systems), and the AMD schema 3 by the Library of Congress. When it comes to document formats such as PDF, Word or OpenDocument Text, it has come to our attention that there is currently no technical metadata standard to follow. Though JHOVE provides a metadata extraction function for PDF and presents the extracted PDF metadata in JHOVE schema, the extracted metadata is massive and is expressed in a page-by-page manner. We hope to develop a document metadata schema which is simpler and may be applied to document formats other than PDF. The document metadata schema may be expressed in XML or database form. We will also develop a style-sheet to convert PDF metadata in the JHOVE schema into the document metadata schema. Andrea Goethals at Harvard University Library (HUL) has also expressed the need for a document metadata schema. HUL is enhancing their preservation repository, the Digital Repository System (DRS), to accept born-digital content, including documents, and sees this document metadata schema as the first step towards preserving documents for the long-term. Together, we are developing this document metadata schema for the 1 See 2 See 3 See 3

4 use of FDA and the DRS. We hope to gather input from the preservation community to enhance the document metadata schema which may also be useful to other trusted repositories as well. 4

5 Applicable Formats This metadata schema was developed based on a small set of popular document formats but should be generally applicable to formats that are: primarily text; but that allow creators a choice of fonts, colors, text size, backgrounds; and that support embedded multimedia (images, sounds, video etc); and that may contain application specific features; and that support page layouts with margins, columns, etc. These formats include but are not limited to: Format MIME Type File Extension Native Applications Microsoft Word application/msword application/ms-word (This second MIME type isn't correct but is sometimes used) doc Microsoft Word and Microsoft Office Word Portable Document Format application/pdf pdf Adobe Writer OpenDocument Text Writer 6.0 Document StarWriter 5.x Document application/vnd.oasis.o pendocument.text application/vnd.sun.xml.writer application/vnd.stardivi sion.writer odt OpenOffice.org2.0 / StarOffice 8 and later sxw OpenOffice.org1.0 / StarOffice6.0 and later sdw StarOffice 5.x StarWriter 4.x Document application/x-starwriter sdw StarOffice 4.x WordPerfect Document application/vndwordperfect wpd WordPerfect and WordPerfect Office Works Text Document application/vnd.ms-wor ks wps Microsoft Works For each metadata element listed in the data dictionary, the document formats are listed that are known to contain either the associated metadata values directly in the file or that could be determined indirectly by parsing the files. 5

6 Data Dictionary The data dictionary describes the semantic meaning and constraints of documentspecific metadata. The document specific metadata are those document properties that are deemed preservation-worthy and pertain to most document formats. Some elements are included because they will aid in evaluating the completeness of the content after transformations (e.g. number of pages). Other elements are included because they will aid in selecting or aggregating documents for risk analysis, preservation or delivery planning (e.g. Features). Please note that general preservation and descriptive metadata may also be extracted from documents. This metadata includes size, encryption, title, author/creator, createdate, copyright, digital signature, protection/permission, etc and may be recorded by using standard preservation schema such as PREMIS and MODS. Semantic unit PageCount Semantic components None Total number of pages in the document Data Constraint Min 1 Mandatory Semantic unit WordCount Semantic components None Total number of words in the document Data Constraint Min 0 This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element however because tools and applications that can determine the number of words in a document do not always use the same algorithm for determining this value. Semantic unit CharacterCount Semantic components None 6

7 Total number of characters in the document Data Constraint Min 0 This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element however because tools and applications that can determine the number of characters in a document do not always use the same algorithm for determining this value. Semantic unit ParagraphCount Semantic components None Total number of paragraphs in the document Data Constraint Min 0 This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element however because tools and applications that can determine the number of paragraphs in a document do not always use the same algorithm for determining this value. Semantic unit LineCount Semantic components None Total number of lines in the document Data Constraint Min 0 This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element however because tools and applications that can determine the number of lines in a document do not always use the same algorithm for determining this value. Semantic unit TableCount 7

8 Semantic components None Total number of tables in the document Data Constraint Min 0 This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element however because tools and applications that can determine the number of tables in a document do not always use the same algorithm for determining this value. Semantic unit GraphicsCount Semantic componentsnone Total number of graphics in the document Data Constraint Min 0 This element is included in this schema because it can be valuable for evaluating the completeness of the content after transformations. Caution must be used with this element however because tools and applications that can determine the number of graphics in a document do not always use the same algorithm for determining this value. Semantic unit Language Semantic componentsnone A language identifier specifying the natural language used in the document Data Constraint String (or some kind of controlled vocabulary like ISO alpha-3 language codes) Cardinality 0 - N Characteristic Content Semantic unit Fonts 8

9 Semantic components FontName isembedded A list of fonts used in the document Data Constraint Container Mandatory Cardinality 1 - N Characteristic Content, Appearance This element allows a repository to store the names of all fonts used in a document. Some repositories may choose to store only the non-embedded fonts. The use of non-embedded fonts may hinder the long term preservation of the documents. For example, a document encoded with a proprietary non-embedded math font may not be migrated due to unavailability of the specific math font. It is recommended that repositories record at least the nonembedded fonts to assist in identifying the documents with potential long-term preservation risks. Semantic unit FontName Semantic componentsnone Name of a font Data Constraint String Characteristic Content, Appearance Semantic unit IsEmbedded Semantic components None An indication of whether or not a font is embedded in a document. Data Constraint Y, N Characteristic Content, Appearance Semantic unit Features 9

10 Semantic componentsnone Additional document features Data Constraint istagged, haslayers, hastransparancy, hasoutline, hasthumbnails, hasattachments, hasforms, hasannotations Cardinality 0 - N Characteristic istagged: structure haslayers: appearance hastransparency: appearance hasoutline: behavior, appearance hasthumbnails: appearance hasattachments: structure, behavior hasforms: content hasannotations: content 10

11 Document Metadata schema An XML schema, DocMD, for describing the document specific metadata. It is currently located at <?xml version="1.0" encoding="utf-8"?> <!-- Editor: Florida Center for Library Automation (FCLA) and Harvard University Library (HUL) Revised: March 17, > <xs:schema xmlns:xs=" xmlns:docmd=" targetnamespace=" elementformdefault="qualified" attributeformdefault="unqualified"> <xs:element name="document"> <xs:sequence> <xs:element name="pagecount" minoccurs="1" maxoccurs="1" > <xs:restriction base="xs:integer"> <xs:element name="wordcount" minoccurs="0" maxoccurs="1" > <xs:restriction base="xs:integer"> <xs:element name="charactercount" minoccurs="0" maxoccurs="1" > <xs:restriction base="xs:integer"> <xs:element name="paragraphcount" minoccurs="0" maxoccurs="1" > <xs:restriction base="xs:integer"> <xs:element name="linecount" minoccurs="0" maxoccurs="1" > <xs:restriction base="xs:integer"> <xs:element name="tablecount" minoccurs="0" maxoccurs="1" > 11

12 <xs:restriction base="xs:integer"> <xs:element name="graphicscount" minoccurs="0" maxoccurs="1" > <xs:restriction base="xs:integer"> <xs:element name="language" minoccurs="0" maxoccurs="unbounded" type="xs:string"/> <xs:element name="font" minoccurs="1" maxoccurs="unbounded"> <xs:attribute name="fontname" type="xs:string"/> <xs:attribute name="isembedded" type="xs:boolean"/> <xs:element name="features" minoccurs="0" maxoccurs="unbounded"> <xs:restriction base="xs:string"> <xs:enumeration value="istagged"/> <xs:enumeration value="hasoutline"/> <xs:enumeration value="hasthumbnails"/> <xs:enumeration value="haslayers"/> <xs:enumeration value="hasforms"/> <xs:enumeration value="hasannotations"/> <xs:enumeration value="hasattachments"/> <xs:enumeration value="usetransparency"/> </xs:sequence> </xs:schema> 12

13 Appendix A: A sample PREMIS document with embedded docmd schema <premis xmlns:xsi=" xmlns="info:lc/xmlns/premis-v2" xsi:schemalocation="info:lc/xmlns/premis-v2 version="2.0"> <object xsi:type="file"> <objectidentifier> <objectidentifiertype>daitss2</objectidentifiertype> <objectidentifiervalue>/users/carol/desktop/work/testdata/pdf/etd.pdf</objectidentifiervalue> </objectidentifier> <objectcharacteristics> <compositionlevel>0</compositionlevel> <size> </size> <format> <formatdesignation> <formatname>pdf</formatname> <formatversion>1.3</formatversion> </formatdesignation> <formatregistry> <formatregistryname>pronom</formatregistryname> <formatregistrykey>fmt/17</formatregistrykey> </formatregistry> </format> <creatingapplication> <creatingapplicationname>thesis (Electronic thesis).doc - Microsoft Word</ creatingapplicationname> <datecreatedbyapplication>tue Apr 24 16:22:40 EDT 2001</dateCreatedByApplication> </creatingapplication> <fixity> <messagedigestalgorithm>md5</messagedigestalgorithm> <messagedigest>6bf12f206a3e70c88cfe2aa5213dd227</messagedigest> </fixity> <objectcharacteristicsextension> <doc xmlns=" <document> <PageCount>123</PageCount> <Feature>hasThumbnails</Feature> </document> </doc> </objectcharacteristicsextension> </objectcharacteristics> </object> </premis> 13

Document Metadata: document technical metadata for digital preservation

Document Metadata: document technical metadata for digital preservation By Carol Chou - Florida Digital Archive (FDA) Andrea Goethals - Harvard Library (HL) March 18, 2009 Rev. November 30, 2012 1 Table