Outline of the course

Outline of the course Introduction to Digital Libraries (15%) Description of Information (30%) Access to Information (30%) User Services (10%) Additional topics (15%) Buliding of a (small) digital library Reference material: Ian Witten, David Bainbridge, David Nichols, How to build a Digital Library, Morgan Kaufmann, 2010, ISBN 978-0-12-374857-7 7 (Second edition) The Web FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-1

Where are we Digital Libraries Description of information Metadata MARC Dublin Core MODS METS TEI EAD... Knowledge Representation FRBR RDF Interoperability FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-2

Interoperability Interoperability is the ability of systems, services and organisations to work together seamlessly toward common or diverse goals. In the technical arena it is supported by open standards for communication between systems and for description of resources and collections, among others. Interoperability is of paramount relevance in the context of resource discovery and access. FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-3

Interoperability in digital libraries (1/2) Interoperability between vendors Different databases and user interfaces Interoperability between different organisations Eg. using different library formats Interoperability between groups of users Eg. Public libraries/academic libraries Eg. libraries in different countries Interoperability between communities Eg. libraries, publishers, archives, museums Interoperability across time Preservation FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-4

Interoperability in digital libraries (2/2) Traditional libraries interoperability Union catalogs (OPACs) Interlibrary loan Digital libraries interoperability Documents (digital objects, resources) Metadata Conceptual models Protocols Z39.50 OAI-PMH Queries Z39.50 queries (Type 1, Z39.58, CQL) SRU, SRW SPARQL FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-5

Z39.50 "Information Retrieval (Z39.50); Application Service Definition and Protocol Specification, ANSI/NISO Z39.50-1995" Developed by NISO (National Information Standards Organization), the standards development organization serving libraries, publishing and dinformation services NISO was (is) the Z39 Committee of ANSI (American National Standards Institute), and Z39.50 was the 50th standard defined by NISO Current version (Version 3) was adopted in 1995, superceding earlier versions adopted in 1992 and 1988 (1984 version was rejected) Another revision, initiated in 2001, is still work in progress Z39.50 was heavily influenced by OSI, and was an application layer protocol that needed a full-duplex reliable OSI connection In Version 3 it runs over TCP/IP It is a wide ranging protocol for information retrieval between a client and a database server, which attempts to standardize shared semantic knowledge FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-6

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-7

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-8

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-9

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-10

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-11

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-12

Z39.50 architectural model (1/2) A server houses one or more databases containing records. Associated with each database are a set of access points (indexes) that can be used for searching how to segment logical data into relations and how to name the columns in the relations are hidden (server-specific) Z39.50 includes a set of registries that provide each (application) domain with a an agreed-upon structure and attributes (query syntax, attribute fields, content retrieval format, etc.) A search (sent from the client - origin to the server - target) produces a set of records, called a "result set", that are maintained on the server The client thas also functions for search management t( (e.g. request progress reports for an active search, authorize the server to continue a resource intensive search, abort an active search) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-13

Z39.50 architectural model (2/2) Records from the result set can be retrieved by the client, which has many options for controlling the contents and format of the records that are returned (e.g. sorting a result set, selecting a subset of the result set, using the result set for a new search) The client has available also a general mechanism called "extended services" to invoke services on the server, which can survive past the end of the session (e.g. saving result sets across sessions, queuing result sets for print or electronic mail processing at the server, registering queries that would be executed periodically on the server) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-14

Z39.50 functionality FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-15

Initialization facility Init service: establishes Z-association Oi Origini Init request Version, (id/password), option flags, message sizes, implementation information Target Init response Result, version, option flags, message sizes, implementation information FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-16

Search facility Search service Origin Search request Search type, query, databases, result set limits for small, medium, large Target Search response Number of records found, number of records attached, status information, (records) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-17

Retrieval facility Present service Origin Present request Number of records, starting point, result set Target Present response Number of returned records, status, s (records) Segment service Allows a Present response that is larger than max size to be split in segments FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-18

Sort facility Sort service Origin Sort request result set to sort, sorted result set, sort directives Target Sort response status FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-19

Browse facility Scan service Origin Scan request database, term list, starting point, number of terms, (step size) Target Scan response status number of elements (elements) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-20

Delete service Result-set-delete facility Origin Delete request list of result sets to delete Target Delete response status FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-21

Access-control service Access control facility Origin Request Target Access control response Security-challenge Access control request Security-challenge-response Response FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-22

Accounting Resource control facility Resource-control service Trigger-resource-control service Resource-report report service Complex functionality to control and report resource usage Mostly used for fee based operation FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-23

Termination facility Close service Terminates a Z-association FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-24

Explain facility Explain service Gives access to information about the Z39.50 target Databases Access points Query languages Element sets... This information is maintained by the server in a specific data base, and therefore can be accessed using the Search and Retrieve facilities of Z39.50 The idea is that a (smart) client, when accessing a (unknown) data base, could be able to find its access points, its element sets and other info by querying the Explain data base FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-25

Extended Service facility Extended Services service Persistent Result Set Extended Service Persistent Query Extended Service Periodic Query Schedule Extended Service Item Order Extended Service Database Update Extended Service Export Specification Extended Service Task package Used to create, modify or delete an Extended Service Request FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-26

Queries Query types Type-0: proprietary between 2 parties Type-1: RPN (Reverse Polish Notation) Type-2: ISO 8777 Type-100: Z39.58 Type-101: Extended RPN (v 2) Type 102: Ranked List query FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-27

Consists of Type-1 Query Reverse Polish Notation One or more operands linked (RPN style) with Boolean operators (AND, OR, AND_NOT) Every operand is a search expression consisting of 7 parts Example of query (operand)(operand)operator ( Mark Twain, 1:1003, 2:3, 3:1, 4:1, 5:100, 6:1) ( Clemence Clemence, Samuel, 1:1003, 2:3, 3:3, 4:101, 5:100, 6:2) AND_NOT - RPN + 2 * (3 + 5) * (7 2) 5 7 5 3 5 + 7 2 * 3 8 8 40 FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-28

Operands in Type-1 queries 0. Term What you are looking for 1.Use Attributes Which abstract access point to use (e.g. title, author) 2.Relation Attributes Relation between the term and the data in the access point (e.g. less than, equals, phonetic equals) 3.Position Attributes Where in the access point should the term be? (e.g. first in field, first in subfield) 4.Structure Attributes How is the query term to be treated? (e.g. as phrase, as words, as date, as normalised name) 5.Truncation Attributes Should truncation be applied on the match? (e.g. left truncation, right and left truncation, no truncation) 6.Completeness Attributes What is the term to be matched against? (e.g. part of subfield, whole subfield, whole field) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-29

From Z39.50 to SRW/U Need for a generic Information Retrieval capability more suited to the Web Architecture Motivation to create an easy to implement protocol with (more or less) the power of Z39.50 Use existing off the shelf solutions where possible Re-evaluate Z39.50, a good idea at the time Avoid library-centric perspective Solution: SRU Search/Retrieve via URL SRW Search/Retrieve via Web Service (SRW is now called SRU over SOAP) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-30

Simple SRU query http://sru.miketaylor.org.uk/sru.pl? version=1.1& operation=searchretrieve& query=dinosaur& startrecord=1& maximumrecords=1& recordschema=dc FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-31

SRU response in XML (1/2) <?xml version="1.0"?> <zs:searchretrieveresponse R xmlns:zs='http://www.loc.gov/zing/srw/'> <zs:version>1.1</zs:version> <zs:numberofrecords>29</zs:numberofrecords> <zs:records>... details in a moment... </zs:records> </zs:searchretrieveresponse> FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-32

SRU response in XML (2/2) <zs:record> <zs:recordschema>info:srw/schema/1/dc-v1 v1.1</zs:recordschema> 1</zs:recordSchema> <zs:recordpacking>xml</zs:recordpacking> <zs:recordposition>1</zs:recordposition> <zs:recorddata> <srw_dc:dc xmlns:srw_dc= dc="info:srw/schema/1/dc-schema" xmlns="http://purl.org/dc/elements/1.1/"> <title>fossils</title> <creator>lappi, Megan.</creator> <type>text</type> <publisher>new York, NY: Weigl Publishers</publisher> <date>2005</date> <language>en</language> <description>studying d i fossils -- Fossil facts -- Gone forever -- A fossil is born -- From bone to stone -- Insects in amber -- Dinosaur footprints</description> <identifier> http://www.loc.gov/catdir/toc/ecip0415/2004004136.html </identifier> <identifier>urn:isbn:1590362136</identifier> </srw_dc:dc> </zs:recorddata> </zs:record> FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-33

Contextual Query Language CQL (formerly known as Common Query Language) is the query language used in SRU The conceptual model of CQL is the same as Z39.50 The server has one or more databases, containing records The databases can be searched through access points, or indexes The language defines a number of defaults to make simple queries really simple Ar the same time it defines a number of Indexes, Relations, Relation Modifiers, Booleans and Boolean Modifiers to increase the expressing power of the language FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-34

CQL search clause subject any/relevant "fish frog" index relation Relation modifier Search term Subject to context qualification FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-35

Learning curves for query languages learn Ef ffort to SQL Google CQL Expressive Power FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-36

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-37

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-38

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-39

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-40

FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-41

Z39.50 and OAI-PMH It is interesting to see another protocol for resource discovery, namely the Open Archive Initiative (OAI-PMH) protocol Historical separation from Z39.50 OAI-PMH appears about 15 years after Z39.50 Cultural separation from Z39.50 Z39.50 originated in the traditional library community OAI-PMH originated i in the Web Community Conceptual separation from Z39.50 Z39.50 based on solid (but heavy and bulky) foundations OAI-PMH based on simple and pragmatic ideas FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-42

OAI Open Archives Initiative The roots of OAI lie in the development e e of eprint archives (i.e. Institutional Repositories) such as arxiv, CogPrints, NACA (NASA), RePEc, NDLTD, NCSTRL, etc. Each repository offered a web interface for deposit of articles and for end-user searches It was difficult for end-users to work across archives without having to learn multiple different interfaces Initial experiments for single search interface to all archives Universal Pre-print Service (UPS) renamed OAI at the Santa Fe Convention (1999) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-43

Searching versus Harvesting Two possible approaches for single search interface to all archives cross searching multiple archives based on protocol like Z39.50 (possibly lighter) harvesting metadata into one or more central services Problems with cross searching Not scalable (overall performance determined by slowest server) Problems of deciding which servers to target (collection descritpions not consistent) Differences in interfaces and query languages Problems in the ranked merging of results (different types and size of targets can skew results) Browse interface very difficult to build Decision was to go with harvesting FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-44

OAI - PMH OAI Protocol for Metadata Harvesting Data providers make metadata available for harvesting Service Providers harvest metadata Metadata can be centrally collected or aggregated Data Providers Are creators and keepers of the metadata for objects (repositories) and (possibly but not necessarily) archives of resources Handle deposit and publishing Service Providers Are harvesters of metadata for the purpose of providing a service such as a search interface, peer-review system, etc. FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-45

OAI PMH overview Harvesting based on OAI-PMH Aggregator Searching based on Z39.50 or SRW Service providers FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-46

OAI disclaimer The OAI use of the term archive in fact implies very little of what we normally associate with archives No preservation aspect is implied whatsoever (not what the protocol is about at all) No appraisal or provenance of eprints or digital objects is implied by this descriptive term The term simply pyrefers to a collection of digital objects (full text, learning objects, etc.) which might (only might) also have been harvested along with the metadata Archive is a term within the OAI PMH which serves to distinguish a collection of digital objects (the archive ) from the collected metadata associated with these objects, described as repositories In OAI repositories expose metadata about eprints (strictly there are no metadata archives) In OAI archives hold eprints (strictly there are no eprint repositories) FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-47

Conceptual model of OAI data resource item = identifier all available metadata about David item Dublin Core metadata MARC metadata SPECTRUM metadata records FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-48

OAI PMH records A record contains the metadata of a resource in a specific format It has three parts header (mandatory) identifier datestamp metadata (mandatory) XML encoded metadata with root tag, namespace repositories i must support Dublin Core may support other formats about (optional) rights statements provenance statements FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-49

OAI-PMH Protocol Overview Protocol based on HTTP Request arguments as GET or POST parameters Six request types (verbs) Responses are encoded in XML syntax Supports any metadata format (Dublin Core mandatory for each data provider) Support selective harvesting logical set hierarchy (data providers) date stamps (last change of metadata set) Flow control (token to retrieve subsequent records) Error messages FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-50

OAI PMH verbs Identify description of an archive ListMetadataFormats retrieve available metadata t formats from archive ListSets retrieve set structure of a repository ListIdentifiers abbreviated form of ListRecords, retrieving only headers ListRecords harvest records from a repository GetRecord retrieve individual metadata record from a repository FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-51

Overview of OAI - PMH FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-52

OAI PMH request Requests must be submitted using the GET or POST methods of HTTP Repositories must support both methods At least one key=value pair: verb=[requesttype] Additional key=value pairs depend on request type Example for GET request http://archive.org/oai? verb=listrecords& metadataprefix=oai_dc FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-53

OAI PMH response Formatted as HTTP responses Content type must be text/xml HTTP compression optional in OAI-PMH XML declaration (<?xml version="1.0" encoding="utf-8"?>) Root element named OAI-PMH with three attributes (xmlns, xmlns:xsi, xsi:schemalocation) Three child elements ResponseDate (UTC datetime) Request (copy of the request that generated the response) a) error (in case of an error or exception condition) b) element with the name of the OAI-PMH request FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-54

OAI PMH example http://edoc.hu-berlin.de/oai-2.0? verb=listidentifiers& from=2002-01-06& until=2002-01-08& 01 08& metadataprefix=oai_dc& set=doctypes:dissertations ListIdentifiers returns the record headers FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-55

Response to ListIdentifiers (1/2) <?xml version="1.0" encoding="utf-8"?> <OAI-PMH xmlns="http://www.openarchives.org/oai/2.0/" hi /OAI/2 xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.openarchives.org/oai/2.0/ http://www.openarchives.org/oai/2.0/oai-pmh.xsd org/oai/2 xsd"> <responsedate>2002-10-22t17:49:49+01:00</responsedate> <request verb="listidentifiers" from="2002-01-03" until="2002-01-08" metadataprefix="oai oai_dc dc" set="doctypes:dissertations"> http://edoc.hu-berlin.de/oai-2.0</request> <ListIdentifiers>... details in a moment </ListIdentifiers> </OAI-PMH> FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-56

Response to ListIdentifiers (2/2) <ListIdentifiers> <header> <identifier>oai:huberlin.de:3000819</identifier> <datestamp>2002-01-08</datestamp> <setspec>doctypes</setspec> y <setspec>doctypes:dissertations</setspec> <setspec>dnb</setspec> <setspec>dnb:dnb33</setspec> </header> <header> <identifier>oai:huberlin.de:3000831</identifier> <datestamp>2002-01-07</datestamp> p <setspec>doctypes</setspec> <setspec>doctypes:dissertations</setspec> <setspec>dnb</setspec> <setspec>dnb:dnb27</setspec> </header> </ListIdentifiers> FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-57

Where are we Digital Libraries Discovery of information Describing Information Metadata MARC Dublin Core MODS METS TEI EAD... Knowledge Representation FRBR RDF Interoperability Queries Z39.50 queries Common Command Language (CCL ISO 8777 or Z39.58) Protocols Z39.50 SRU/SRW OAI-PMH FUB 2012-2013 Vittore Casarosa Digital Libraries Part 6-58