Document. conversion. Internal format. Classify. Document. conversion. Internal format

Transcription

1 Ingesting Documents Document Metadata Both Extract metadata Document conversion Importing Internal format Index Classify Building Full text index Database Associated files Ingesting Documents Document Metadata Structural stuff Document conversion Internal format Both Extract metadata Metadata included Section-level Links to external files Plain text, PDF, PostScript, source code, image formats, HTML, WORD, , TIFF images, Gutenberg text, XML metadata format, BibTex, Refer, OAI, USMARC, BRS, Identify language, extract acronyms, image thumbnails Traverse directory structure, split files, ZIP archives,

2 Plugin Architecture: import.pl Top level directory Traverse directory hierarchy Plugin1 Plugin2 WORD Plugin3 unzip Plugin4 PDF Plugin5 MARC Warn Unprocessed and ignore file accepted file accepted file accepted Extraction language plugin1 Extraction keyphrases plugin2 Internal format file Metadata and Document Parsing: LOM Extended Example: View example collection based on Learning Object Metadata (LOM) Sample record Demonstration of working with sub-fields IEEE LTSC LOM (Learning Technology Standards Committee) <lom> <classification> <general> <lifecycle> <educational> <rights> <technical> Example is representative can do similar things with Refer, BibTeX, ISIS, etc

3 Metadata and Document Parsing: LOM <lom xmlns=" xmlns:xsi=" xsi:schemalocation=" <classification> <keyword> <educational> <langstring> <context> Arts <source> </langstring> <langstring> </keyword> 3D Animation/Maya <purpose> </langstring> <langstring> </source> Discipline </context> </langstring> <language> </purpose> English </classification> </language> </educational> <general> <language> English </language> <title> <langstring> A Teddy Bear Picnic </langstring> </title> <keyword> <langstring> teddy bear </langstring> </keyword> <keyword> <langstring> run away </langstring> </keyword> <coverage> </coverage> <description> <langstring> This film is about a... now he has to get back! </langstring> </description> </general> <lifecycle> <version> </version> <status> <langstring> Finished </langstring> </status> <contribute> <centity> <vcard> Dalvin Chung Trieu </vcard> </centity> <date> <datetime> </datetime> <description> <langstring> Born and raised in B.C. Dalvin grew up having a... video gaming industry. </langstring> </description> </date> <role> <langstring> Director/Creator </langstring> </role> </contribute> </lifecycle> <rights> <description> <langstring> Copyright 2002 Vancouve r Film School </langstring> </description> <copyrightandotherrestrict> <langstring> yes </langstring> </copyrightandotherrestric> </rights> <technical> <size> </size> <duration> 1:38 </duration> <location> </location> <format> video/quicktime </format> </technical></lom> Metadata and Document Parsing: LOM

4 Metadata and Document Parsing: LOM Metadata and Document Parsing: LOM

5 Metadata and Document Parsing: LOM Metadata and Document Parsing: LOM

6 Metadata and Document Parsing: LOM Metadata and Document Parsing: LOM

7 Use as Interchange Centre: export.pl Top level directory Traverse directory hierarchy Plugin1 Plugin2 WORD Plugin3 unzip Plugin4 PDF Plugin5 MARC file accepted file accepted file accepted Canonical document format file Warn Unprocessed and ignore Extraction language plugin1 Extraction keyphrases plugin2 Internal format file MODS

8 MODS Metadata Object Description Schema Motivation: Dublin Core too light for some applications MARC, at the other extreme, is cumbersome, was not really designed with electronic resources in mind, numerous variants to worry about MODS tries to address the conflicting needs of interoperability and precision MODS Metadata Object Description Schema MODS is a contemporary form of MARC library metadata, expressed in XML (The MARC Standards Office helped define MODS) It is fairly easy to convert MARC metadata to MODS though not a straightforward one-to-one mapping >30 years between the inception of MARC and MODS MODS includes some additional concepts MARCXML is a separate standard that gives an exact 1-1 representation of MARC in XML also administered by MARC Standards Office at LoC

9 MODS Metadata Object Description Schema MODS is a MARC21 derivative that: Takes advantage of the XML environment; Gives special support to the cataloguing of electronic resources; Is less detailed than MARC Includes special features that differentiate it from MARC, such as: user-oriented tags, regrouped data elements, linking, and recursion. MODS Metadata Object Description Schema Development history: Version 1.0 (draft): trial use for six months in 2002 Version 2.0 (incorporates changes suggested during review): February 2003 MODS listserv provides an avenue for user input and discussion of solutions. Version 3.0 (based on comments received) issued for review: September 2003 official version released: December

10 MODS: Syntax Overview 19-top level elements for describing objects 64 further sub-elements under these Covers standard bibliographic facets, e.g. titles, names of creators and contributors, subject and classification numbers, Plus provision for physical description, information on access restrictions, genre, and so on. Extension mechanism: can incorporate other XML documents (e.g. related objects) Includes facilities for authority control, and is fully granular in the level of descriptions it provides. MODS: Top-level Elements titleinfo targetaudience typeofresource tableofcontents note location extension name classification language origininfo identifier accesscondition genre subject abstract physicaldescription relateditem recordinfo

11 MODS: Sample <mods> <titleinfo> <title>sound and fury :</title> <subtitle>the making of the punditocracy /</subtitle> </titleinfo> <name type="personal"> <namepart>alterman, Eric</namePart> <role> <text>creator</text> </role> </name> <typeofresource>text</typeofresource> <genre authority="marc">bibliography</genre> <origininfo> <place> <code authority="marc">nyu</code> <text>ithaca, N.Y</text> </place> <publisher>cornell University Press</publisher> <dateissued>c1999</dateissued> <dateissued encoding="marc">1999</dateissued> MODS: Sample <issuance>monographic</issuance> </origininfo> <language authority="iso639-2b">eng</language> <physicaldescription> <form authority="marcform">print</form> <extent>vii, 322 p. ; 23 cm.</extent> </physicaldescription> <note type="statement of responsibility">eric Alterman.</note> <note> Includes bibliographical references (p ) and index. </note> <subject authority="lcsh"> <topic>journalism</topic> <topic>political aspects</topic> <geographic>united States.</geographic> </subject>

12 MODS: Sample <classification authority="lcc">pn4888.p6 A </classification> <classification edition="21" authority="ddc">071/.3</classification> <identifier type="isbn"> (pbk. : acid-free, recycled paper)</identifier> <identifier type="lccn"> </identifier> <recordinfo> <recordcontentsource>dlc</recordcontentsource> <recordcreationdate encoding="marc"> </recordcreationdate> <recordchangedate encoding="iso8601"> </recordchangedate> <recordidentifier> </recordidentifier> </recordinfo> </mods> MODS: Extended Example Extended example: Using MODS records, build a Greenstone digital library collection based on the Library of Congress British Cartoons collection (part of the American Memory project). Details: MODS metadata records exported from LOC s OAI server Greenstone configured to support: Browse by title, author, and subject Search by title, subject and all fields Display summary of MODS record nicely formatted Hyperlink to original resource

13 MODS: Extended Example MODS: Extended Example

14 MODS: Extended Example MODS: Extended Example

15 MODS: Extended Example MODS: Extended Example

16 MODS: Extended Example MODS: Extended Example

17 MODS: Extended Example MODS: Extended Example

18 MODS: Extended Example MODS: Extended Example

19 Practical Digital Library Interoperability Standards MODS: Extended Example MODS: Extended Example

20 MODS: Extended Example MODS mapping into Greenstone using XSLT (specify as argument to OAIPlug): <xsl:template match="mods:topic"> <dc:subject><xsl:value-of select="." /></dc:subject> </xsl:template> <xsl:template match="mods:titleinfo"> <dc:title> <xsl:value-of select="mods:title" /> <xsl:if test="mods:subtitle"> <xsl:text>: </xsl:text> <xsl:value-of select="mods:subtitle" /> </xsl:if> </dc:title> </xsl:template> MODS: Issues MODS rich syntax cause complications when mapping to other representations e.g. it can express recursion structures and relationships For a given domain, these problems can usually be surmounted, with some programming effort solutions need to be more sophisticated than for other representations

21 MODS Resources Home page: Overview: An introduction to the Metadata Object Description Schema (MODS), Sally H. McCallum, Library Hi Tech, Volume 22, Number 1, 2004 pp (available on-line through Emerald Publishing). METS

22 METS Metadata Encoding and Transmission Aim: encode metadata for any digital material electronic texts, still images, digitized video, sound files Specifies a framework that works with other XML standards Coherent overall structure for encoding all metadata descriptive, administrative, structural used to describe digital library objects Metadata either embedded within METS structure or held in external files that METS references Internal identifiers link all the object s components extends the conventional use of the term metadata to encompass document structure and content as well METS Metadata Encoding and Transmission Origins in Making of America II, late 1990s.... to create a DL object standard by encoding descriptive, administrative and structural metadata, along with primary content, inside a DL object Led to an XML DTD that specified data elements and encoding for objects text and still image content files including diaries, journals, photographs, correspondence 2001 workshop to discuss future of the DTD Outcome: produce new version for library community Recast the DTD as an XML schema 2001: first draft of METS for review : successive versions now version

23 METS Metadata Encoding and Transmission METS format has 7 top-level sections: METS Header Descriptive Metadata Administrative Metadata File Section Structural Map Structural Links Behavior Key sections METS: Structural Map Only compulsory section Describes overall structure of the object (its structural metadata) Sample: <structmap TYPE="PHYSICAL"> <div ID="title1.div.1" LABEL="Chapter 1"> <div ID="title.1.div.1.1" LABEL="Section 1"> <fptr FILEID="title1.image.1"/> <fptr FILEID="title1.image.2"/> </div> <div ID="title.1.div.1.2" LABEL="Section 2"> <fptr FILEID="title1.image.3"/> </div> </div> <div ID="title1.div.2" LABEL="Chapter 2"> [Contents of Chapter 2 omitted] </div> </structmap>

24 METS: File Groups Lists the individual files that comprise the digital object Each file is referenced by a file element, e.g. <file ID="FILE003" MIMETYPE="image/jpeg"> <FLocat LOCTYPE="URL"> </FLocat> </file> <FLocat> gives file s physical location as URL ID used elsewhere in the METS file to reference it Different manifestations of the same resource may be grouped by <filegrp> <filegrp> referenced together if necessary METS: Descriptive Metadata Descriptive metadata held in <dmdsec> sections Structural map references <dmdsec> using ID attribute Descriptive metadata can be either be held in external files referenced from within the METS or embedded directly with it <dmdsec ID="dmd001"> <mdref LOCTYPE="URN" MIMETYPE="application/xml" MDTYPE="EAD" LABEL="Berol Collection Finding Aid">urn:x-nyu:fales1735</mdRef> </dmdsec> <dmdsec ID="dmd002"> <mdwrap MIMETYPE="text/xml" MDTYPE="DC" LABEL="Dublin Core Metadata"> <dc:title>alice's Adventures in Wonderland</dc:title> <dc:creator>lewis Carroll</dc:creator> <dc:date>between 1872 and 1890</dc:date> <dc:publisher>mccloughlin Brothers</dc:publisher> <dc:type>text</dc:type> </mdwrap> </dmdsec>

25 METS: Administrative Metadata, Behavior Administrative Metadata Like descriptive, but recorded in <amdsec> elements (embedded or external) technical information on files that comprise an object information on IP rights Can use any metadata scheme METS editorial board has recommendations Behavior How to render object components for the user, e.g. specific software packages to be used particular parameters to be used (Late addition to the METS standard) METS: Sample (abridged) <?xml version="1.0" encoding="utf-8" standalone="no"?> <mets:mets xmlns:mets=" xmlns:xsi=" xmlns:gsdl3=" xmlns:xlink=" xsi:schemalocation=" OBJID="HASH01957a39eda286c59471aa53:2"> <mets:dmdsec ID="DM1" GROUPID="1"> <mets:mdwrap MDTYPE="OTHER" OTHERMDTYPE="gsdl3" ID="gsdl1"> <mets:xmldata> <gsdl3:metadata name="language">en</gsdl3:metadata> <gsdl3:metadata name="encoding">iso_8859_1</gsdl3:metadata> <gsdl3:metadata name="filesize">9312</gsdl3:metadata> <gsdl3:metadata name="title">hey JUDE!</gsdl3:Metadata> <gsdl3:metadata name="fileformat">html</gsdl3:metadata> <gsdl3:metadata name="identifier">hash01957a39eda286c59471aa53</gsdl3:metadata>... </mets:xmldata> </mets:mdwrap> </mets:dmdsec>

26 METS: Sample (abridged)... <mets:filesec> <mets:filegrp ID="FILEGROUP_PRELUDE1"> <mets:file MIMETYPE="text/xml" ID="FILE1"> <mets:flocat LOCTYPE="URL" xlink:href="file:doctxt.xml#xpointer(/section[1]/text())" /> </mets:file> </mets:filegrp> <mets:filegrp ID="default"> <mets:file MIMETYPE="text/xml" ID="default.1"> <mets:flocat LOCTYPE="URL" xlink:href="heyjudeus.htm" /> </mets:file> <mets:file MIMETYPE="image/gif" ID="default.2"> <mets:flocat LOCTYPE="URL" xlink:href="title.gif" /> </mets:file> <mets:file MIMETYPE="image/jpeg" ID="default.3"> <mets:flocat LOCTYPE="URL" xlink:href="70usheyjudebig.jpg" /> </mets:file> </mets:filegrp> </mets:filesec>... METS: Sample (abridged)... <mets:structmap ID="Section" TYPE="Section" LABEL="Section"> <mets:div ID="DS1" TYPE="Section" ORDER="0" ORDERLABEL="1" LABEL="1" DMDID="DM1"> <mets:fptr FILEID="FILEGROUP_PRELUDE1" /> </mets:div> </mets:structmap> <mets:structmap ID="All" TYPE="Whole Document" LABEL="All"> <mets:div ID="DSAll" TYPE="Document" ORDER="0" ORDERLABEL="All" LABEL="Whole Document" DMDID="DM1"> <mets:fptr FILEID="default.1" /> <mets:fptr FILEID="default.2" /> <mets:fptr FILEID="default.3" /> </mets:div> </mets:structmap> </mets:mets>

27 METS: Example 1 METS: Example

28 METS: Example 1 METS: Example

29 METS: Example 1 METS: Example

30 METS: Example 1 METS: Example 1 <metshdr> <admsec> <structmap> <div> <div> <div> <div> <div> <dmdsec> <filesec> <filegrp> <file> <behaviorsec>

31 METS: Example 1 Catalog record (MARC) Scanned images (JPEG) Structural metadata dmdsec MODS filesec URL: page1.jpg URL: page2.jpg structmap div DMDID=1 div FILEID=1 div FILEID=2 METS: Example 1 Structure map div TYPE= score div ORDER= 1 div ORDER= 2 div ORDER= 3 ORDERLABEL= Page 1 LABEL= Nocturne, no.15 Descriptive metadata 15me. & 16me. Nocturno composé par Frederic Chopin File list 2000pix wide JPEGs image 1 image 2 image 3 700pix wide JPEGs image 1 image 2 image

32 METS: Example 1 MODS MARC: MODS: a Chopin, Frédéric, d a 15me. & 16me. nocturno / c [composé par Frederic Chopin]. 300 a 10 p. of music ; c 33 cm. <mods> <titleinfo> <title>15me. & 16me. nocturno</title> </titleinfo> <name type="personal"> <namepart>chopin, Frédéric</namePart> <namepart type="date"> </namepart> </name> <physicaldescription> <extent>10 p. of music ; 33 cm.</extent> </physicaldescription> <note type="statement of responsibility"> [composé par Frederic Chopin]</note> </mods> METS: Example 2 Exporting a collection as METS Use Greenstone export.pl in combination with saveas to generate METS compliant files Tutorial exercise #

33 METS: Example 2 One folder for each document: HASH0101.dir/ HASH0103.dir/... HASH0102.dir/ HASH010c.dir/ Sample document folder (HASH0101.dir): 1500s.gif docmets.xml doctxt.xml Sample docmets.xml (as before): <?xml version="1.0" encoding="utf-8" standalone="no"?> <mets:mets OBJID="HASH fd4a2d9174eaa65ef:2"> <mets:dmdsec ID="DM1" GROUPID="1"> <mets:mdwrap MDTYPE="OTHER" OTHERMDTYPE="gsdl3" ID="gsdl1"> <mets:xmldata> <gsdl3:metadata name="lastmodified"> </gsdl3:metadata> <gsdl3:metadata name="gsdlsourcefilename">ab.html</gsdl3:metadata>... </mets:mets> METS: Issues In reality: Lots of people generate METS Not so many people read it in Difficult to handle all possible METS files e.g. can include any metadata standard with an XML manifestation even ones that haven t been devised yet! Subsets have been institutionalized as METS profiles e.g. Greenstone profile Two systems may apparently use METS but fail to interoperate meaningfully You have to read the fine print

34 METS Resources Home page: Overviews: An introduction to the Metadata Encoding and Transmission Standard (METS), Library Hi Tech, Volume 22, Number 1, 2004 pp (available on-line through Emerald Publishing)