nof-digitise Technical Standards and Guidelines

Transcription

1 nof-digitise Technical Standards and Guidelines Version One: June 2000 This document has been developed on behalf of the New Opportunities Fund (NOF) by the UK Office for Library and Information Networking (UKOLN), University of Bath, in association with Resource: The Council for Museums, Archives & Libraries. Acknowledgements This document was prepared by UKOLN staff and edited by Sarah Ormes and Andy Powell. UKOLN would like to thank Bruce Royan, Sandy Buchanan and Mark Bide for their contributions. This document draws on Scientific, Industrial and Cultural Heritage: a shared approach, a research framework for digital libraries, museums and archives written by Lorcan Dempsey for the European Commission s Information Society Directorate General. Scientific, Industrial and Cultural Heritage: a shared approach < The full text of the latest version of this document is available online from: < technicalstandards.html> UKOLN is funded by Resource: The Council for Museums, Archives & Libraries, the Joint Information Systems Committee (JISC) of the Higher and Further Education Funding Councils, as well as by project funding from the JISC and the European Union. UKOLN also receives support from the University of Bath where it is based.

2

3 Table of Contents 1. Introduction The Life Cycle Approach Requirements and Guidelines Vocabulary Other Standards Developments Updates to this Document Creation Requirements File Formats Character Encoding Guidance Data Capture Metadata Preservation Management Requirements Intellectual Property Rights Identification Performance Indicators Security E-Commerce Guidance Preservation Metadata Collection Development Guidance Access Requirements Access To Resources Authenticity Guidance Metadata and Resource Discovery Watermarking and Fingerprinting User Authentication Re-use Guidance Learning Resource Creation...15

4

5 1. Introduction The nof-digitise programme will support the creation of a significant body of digital resources. The promise is of a rich digital fabric of learning materials, cultural artefacts, and civic resources which offer real support for life-long learning in a new cultural network space. This document recommends some agreed approaches to how this resource is created and made available. It incorporates many of the examples of best practise developed by the 15 million UK Electronic Libraries Programme (elib). Without the implementation of agreed approaches throughout nofdigitise the promise of networked information for learning and cultural enrichment will not be realised. Important areas for consideration are: Preservation. It is important to secure the long-term future of materials, so that the benefit of the investment is maximised, and the cultural record is maintained in its historical continuity and media diversity. Interoperability. It is important to be able to share content seamlessly between projects and between projects and users, to be able easily to use content without specialist tools, to be able to discover and interact with content in consistent ways, to be able to manage it effectively. Security. In a network age it is important that the identity of users, content, and projects is established; that intellectual property rights and privacy are protected; that the integrity and authenticity of resources can be established. Accessibility. It is important that materials are as accessible as possible and are made publicly available using open standards and non-proprietary formats. If this material is to be a true national resource it must be accessible to all citizens including those with disabilities. Failure to address these areas will have serious consequences, as the different parties waste their resources: Users - the citizen, the learner, the child. They will waste time and effort as they cannot readily find or use what is most appropriate to their needs, because it is not described, or is formatted in a particular way, or requires specialist tools to exploit, or was never captured in a usable form. Information providers and managers. Their investment may be redundant and wasted as their resources fail to release their value in use, as 1

6 their products reach a part only of the relevant audience, as they invest in non-standard or outmoded practices. Funding agencies. They have to pay for redundant, fragmented effort, for multiple learning curves and reinvention of wheels, for less than optimal projects and techniques, for content that fails to reach users or markets. Creators, authors. Their legacy to the future may be lost The Life Cycle Approach A life-cycle approach has been used in the discussion of standards as decisions taken at any stage in a resource s life may have ramifications for cost-effective use at other stages. The stages in this life cycle are: Creation. The actual creation of an individual digital resource. Management. The digital resource will need to be managed in some way if it is to be accessible and meaningful. Collection Development. The digital resource will typically be placed in a collection with other digital resources. This collection will need to be managed. It should also be capable of growing. Access. Materials need to be made available on the network in accessible, usable, secure, and responsible ways. Repackaging. A digital resource should be able to be used in more than one way. For example, a digitised local history photograph could be created initially to be used as part of an online exhibition but then repackaged to be used as part of a teaching resource. The use of a life cycle approach means that a number of technical standards are revisited through out the document. Metadata, for example, is relevant to several of the stages of the life cycle and is discussed separately at each of these stages Requirements and Guidelines This document outlines technical requirements which projects must meet and also offers guidance on developing technical issues. Requirements are standards which are widely accepted and already in current use. All projects must implement any standards that are identified as requirements. This document also offers guidance. The standards covered in the guidance sections are still in development. It is probable that these standards will become widely accepted during the lifetime of the programme but they cannot be identified as a requirement with full confidence yet. Projects should 2

7 maintain and demonstrate awareness of these standards and their potential applications Vocabulary The words must, should and may when printed in bold text have precise meanings in the context of this document. Must: This word indicates an absolute technical requirement with which all projects must comply. Should: This word indicates that there may be valid reasons not to treat this point of guidance as an absolute requirement, but the full implications should be understood and the case carefully weighed before it is disregarded. Should has been used in conjunction with technical standards that are likely to become widely implemented during the lifetime of the project but currently are still gaining widespread use. May: This word indicates that the topic deserves attention, but projects are not bound by New Opportunities Fund advice. May has therefore been used to refer to standards that are currently still being developed. This vocabulary is based on terminology used in Internet Engineering Task Force (IETF) documentation. IETF < Other Standards Developments This document builds on existing published technical standards developed for e-government services. Additional standards for use in e-government services are currently being prepared and it is probable that they will be very influential. The nof-digitise standards may be realigned at some point to conform with the updated e-government standards, however, it is not expected that this will have a major impact on project applications in the short term. Projects should maintain awareness of ongoing standards developments for e- government services. Projects should also maintain an awareness of the University for Industry (UFI) guidelines on standards. Information Age Government Champions < UFI Qualified Suppliers Content Development Standards < 3

8 1.5. Updates to this Document This document will be developed over the lifetime of the programme in order to reflect the most recent developments in standards. The latest version of this document will be available from: < technicalstandards.html> 2. Creation This section covers the actual creation of digital resources by nof-digitise funded projects. Technical standards decisions made at this stage will have a fundamental impact on the manageability, accessibility and viability of the resources created Requirements File Formats Projects must use open standard file formats when creating digital records in order to maximise access to resources. (Please note that file formats for the delivery of digital records to users are outlined in ). The use of open file formats will help with interoperability, ensuring that resources are reusable and can be created and modified by a variety of applications. It will also help to avoid dependency on a particular supplier. Text based content must be created and managed in a structured format suitable for delivery as HTML. In most cases storing text-based content as HTML, XHTML or XML, either in plain files or within a database of some kind, will be the most appropriate option. Where HTML is chosen as the storage format, the latest version recommended by the World Wide Web Consortium (W3C), currently HTML 4, or the latest version of XHTML, currently 1.0, must be used. HTML and XML documents should be validated against a published Document Type Definition (DTD). Images, video, sound and music should initially be created and stored using appropriate open or de facto standard formats as outlined below. Images must be created and stored using the TIFF, PNG or SPIFF formats. (SPIFF is the official format for storing JPEG images ). Video must be created and stored using MPG, Microsoft AVI, ASF or QuickTime formats. Sound must be created and stored using MP3, RealAudio, Microsoft WAV or Sun AU formats. 4

9 W3C HTML < Character Encoding Projects must store and deliver text-based content in a way that is compatible with the Unicode UTF-8 encoding form. Unicode is a standard for representing characters, letters and numbers. Most modern office software products support UTF-8 by default. If appropriate, the UTF-16 encoding form may be used in preference. The Unicode Standard - A Technical Introduction < principles.html> 2.2. Guidance Data Capture Standards for initial data capture are still developing and it is difficult to identify specific technical requirements at this time. However, projects should demonstrate that they have considered the implications of the following three issues: the selection of materials for digitisation, the physical preparation of materials for digitisation, the digitisation process. The JISC Image Digitisation Initiative (JIDI), the Arts and Humanities Data Service (AHDS) and the Technical Advisory Service for Images (TASI) all provide further guidance on this topic. A Feasibility Study for the JISC Image Digitisation Initiative (JIDI) < jidi_fs.html> AHDS: Guides to Good Practice in the Creation and Use of Digital Resources < AHDS < TASI < 5

10 Metadata Metadata is structured data about data. Typically, the term refers to any data used to aid the identification, description, location or use of networked electronic resources. Any digital object will benefit from the addition of metadata, as it will ensure that it can be more easily identified, understood and utilised. The use of metadata will be needed through the whole lifecycle of a digital resource and is consequently referred to throughout this document. Initial guidance on the types of metadata that may need to be recorded during the digitisation process has been produced by the RLG Working Group on Preservation Issues of Metadata and also by projects like the Making of America II Testbed Project. RLG Working Group on Preservation Issues of Metadata < Making of America II Testbed Project < Preservation Preservation concerns apply both to the information object being digitised and to the surrogate digital object when it has been created. Wherever possible, those responsible for the project must weigh-up the risks of exposing original material to any digitisation process, especially where the items are unique, valuable or fragile. More guidance on this topic can be found in Guidelines for digital imaging. Preservation issues should be considered an integral part of the digital creation process. Preservation will depend upon documenting all of the technological procedures that led to the creation of an object, and this - in many cases - can only be done at the point of creation. Projects should consider the value in creating a fully documented high-quality digital master from which all other versions (e.g. compressed versions for accessing via the Web) can be derived. This will help with the periodic migration of data and with the development of new products and resources. Guidelines for digital imaging: papers < 3. Management In the same way that physical collections are managed, digital collections will also need to be managed. Issues that will need to be addressed include preservation of the data, copyright issues, security and e-commerce. Again, 6

11 standards decisions made at this stage will have long term implications for the accessibility and usability of the data Requirements Intellectual Property Rights The contract that any funded project will be expected to sign with the New Opportunities Fund will hold the Fund harmless in any dispute over Intellectual Property Rights (IPR). Projects must therefore only digitise material of which they own the IPR or have clear permission to use from the rights holder, or for which it can be shown that all reasonable steps to establish IPR have been taken. Care is particularly advisable in the circumstances below: Published material. Publishers are unlikely to give permission to digitise incopyright material unless this is of some advantage to them. Older material may be out of copyright but the project is responsible for confirming this. Copyright restrictions cease 70 years after the death of the material s creator. In-house productions. The rights in any work undertaken by an institution s staff, as part of their normal duties, remains the property of that institution. In some academic institutions these rights may not have been asserted, and authors may have assigned them to external publishers. Unpaid volunteers retain the copyright of their work unless they sign away their rights. Institutions commissioning work. This work, for example photography, will normally have secured reproduction rights, but this may not have extended to digitisation unless specifically stated in the agreement. Projects will only have copyright on digitised material if this is permission is secured. Gifts, bequests and loans. These may have particular conditions attached to them that affect their availability for digitisation. Copyright and the Networked Environment Issue Paper from Networked Services Policy Taskgroup < copyright.html> Creating Digital Resources for the Visual Arts: Standards and Good Practice < contents.html> Identification Digitised resources should be unambiguously identified and uniquely addressable directly from a user s Web browser. It is important, for example, 7

12 that the end user has the capability to directly and reliably cite an individual resource, rather than having to link to the Web site of a whole project. Projects may also want to ensure that logical sets within the resources they are providing are uniquely and persistently addressable. Projects should make use of the Uniform Resource Identifier (URI) for this purpose. For the moment, the only widely deployed form of URI is the Uniform Resource Locator (URL). Projects may wish to consider assigning a Digital Object Identifier to resources where appropriate. Uniform Resource Identifiers < Digital Object Identifier < Performance Indicators Performance indicators can be used to provide objective measures of the usage of a Web service. The most popular performance indicator makes use of Web server log files. Analysis of server log files can provide valuable information on the growth of a service and usage patterns, although reports need to be interpreted carefully. Web projects must maintain server log files and must use them to provide regular reports which must be made publicly available Security The machines used to deliver projects must be operated in as secure a manner as possible. The advice in operating system manuals concerning security must be followed. All known security patches must be applied. Machines should be configured to run only the minimum number of network services. Machines should be placed behind a firewall if possible, with access to the Internet only on those ports that are required for the project being delivered. Projects should be managed in accordance with the Information Security Management guidelines laid out in BS7799: Part 1. The management and use of any personal information must conform to the Data Protection Act Where sensitive information is being passed from a client to a server across the network, projects must use Secure Sockets Layer (SSL) to encrypt the data. This includes the transfer of usernames and passwords, credit card details and other personal information. Note that the use of SSL also provides the end-user with an increased level of confidence in the authenticity of the service. 8

13 Information Security Management (BS7799) < Data Protection Act 1998 < Introduction to SSL < /sslin/index.htm> E-Commerce Material digitised or content created with nof-digitise funding must be made available for educational purposes, free at the point of use. The New Opportunities Fund recognises that there is a continuing cost in making this material available and the project may therefore charge provider institutions for such a delivery service, or may deposit materials with delivery services which themselves make such charges. Even if the project decides to absorb the costs of delivery of nof-digitise funded content from its own funding, it will have been created with public money and its IPR must be protected. Whereas public domain delivery on the Web of some resources (e.g. thumbnail images) will be encouraged, the New Opportunities Fund will expect the more valuable resources to be made available only to users and through institutions that have signed a licence restricting them to non-profit, non-proliferation, educational use. The New Opportunities Fund will require projects to use some form of protection to ensure valuable resources neither leak into the public domain nor are appropriated for unauthorised commercial use Guidance Preservation Once a digital surrogate has been created, its own preservation then becomes an issue. There is (as yet) no standardised way of ensuring the long-term preservation of any digital information, but all projects should be aware of, and demonstrate that they are aware of the need for preserving their digital resources Metadata Digitised resources need to be described at a sufficient level of detail to support their management and access. At a minimum, metadata should conform to developing e-government and UfI metadata standards and should be capable of supporting the delivery of Dublin Core (DC) descriptions of all project resources. 9

14 In the network environment, every transaction that involves intellectual property is by its nature a rights transaction. The expression of these Terms of Availability or Business Rules is dependent on rights metadata data which identifies unambiguously and securely the intellectual property itself, the specific rights which are being granted (for example to read, to print, to copy, to modify) and the users or potential users. There is no standardised way (of which we are aware) of expressing rights management data. The <indecs> project has created the necessary foundation (in the form of a metadata dictionary for the relatively static area of descriptive data) to enable the essential work on (dynamic) rights description to take place. It is currently expected (June 2000) that the rights elements of the <indecs> metadata dictionary will be published within the next 12 months. Projects should therefore maintain data about the rights that they hold and acquire in an internally consistent form, so that they can be converted as efficiently as possible into a standard format when one becomes available. The type of information that is likely to be required includes: The identification of the resource itself. The name of the person or organisation granting the rights. The precise right or rights which are being granted (including, for example, whether modification is permitted) and any specific exclusions. The period of time for which rights are granted. The user group or groups permitted to use the resource. Any obligations (including but not limited to financial obligations) that users of the resource may incur. Dublin Core < DC-dot < <indecs> < 10

15 4. Collection Development 4.1. Guidance The material digitised by each nof-digitise-funded project should be developed as a digital collection which will continue to grow and be added to beyond the lifetime of the nof-digitise project. Digital collections depend for their usefulness on critical mass. This in turn depends on a strategic approach to collection development, which encompasses a view of user needs and the wider availability of resources. All projects should demonstrate awareness of these issues. 5. Access It is expected that end-user access will be through the People s Network and the National Grid for Learning and consequently will operate using Internet Protocols. All projects will need to follow similar standards if they are to be accessible across the whole of this network Requirements Access To Resources Projects must be accessible using a Web browser. This will normally be achieved using HTML and the HTTP protocol. If other protocols are used (e.g. Z39.50) gateways must be available to provide access by a Web browser. Projects must be accessible by a variety of browsers, hardware systems, automated programs and end-users. All materials must be available 24 hours a day, 7 days a week and must be accessible to all. Projects must implement W3C Web Accessibility Initiative Recommendations and so ensure a high degree of accessibility for people with disabilities. Web services must be accessible to a wide range of browsers and hardware devices (e.g. Personal Digital Assistants (PDAs) as well as PCs). Web services must be usable by browsers that support W3C recommendations such as HTML, Cascading Style Sheets (CSS) and the Document Object Model (DOM). Projects that make use of proprietary file formats and browser plug-in technologies must ensure that they are still usable on browsers that do not have the plug-ins. The appearance of a Web site should be controlled by use of style sheets in line with W3C architecture and accessibility recommendations. The latest version of CSS recommended by W3C (currently CSS 2) should be used, 11

16 although, due to incomplete support by browsers, not all features defined in CSS 2 may be usable. Text-based content must be delivered as HTML or XHTML in most cases, though the use of other XML DTDs may sometimes be appropriate. Where HTML is chosen as the delivery format, the latest version recommended by W3C (currently HTML 4) or the latest version of XHTML (currently 1.0) must be used. HTML and XML documents must be validated against a published DTD. In some cases, delivery in proprietary formats such as PDF, RTF or Microsoft Word may be appropriate. Images, video and sound must be delivered using appropriate open or de facto standard formats as outlined below. Images must be provided on the Web as GIF (for line-drawings) or JPEG (for photographs) formats. PNG may be an alternative format to GIF. Video must be delivered using MPG, Microsoft AVI, ASF or Quicktime formats. Sound must be delivered using MP3, RealAudio, Microsoft WAV or Sun AU formats. W3C WAI recommendations < Authenticity Project specific domain names should be registered in the Domain Name System (DNS). The domain name forms part of the project branding and will help end-users identify the authenticity of the content being delivered. Domain names should therefore be clearly branded with either the name of the project or the organisation delivering the project. In some situations it may be appropriate to secure the network connection between the client and the server using Secure Sockets Layer (SSL) to give end-users increased confidence that they are exchanging information with the correct project Web site Guidance Metadata and Resource Discovery The body of resources created through nof-digitise funding will be significant, with the corpus as a whole potentially far more valuable than the individual holdings of each funded project. All projects should be capable of delivering summary descriptions of all the resources that they make available. Item-level descriptions should be based on the Dublin Core and should be in line with 12

17 developing e-government and UfI metadata standards. In order to facilitate searching across this corpus, and to enable integration of nof-digitise funded collections into the wider body of digital collections available in the UK, some consideration should also be given to the creation of collection level descriptions. The purpose of collection description is to: allow users to discover, locate and access collections of interest, allow users to perform searches across multiple collections in a controlled way, allow software to perform such tasks on behalf of users based on known user preferences. There are no agreed standards for collection level description, but nof-digitise funded projects should take into account work under progress within the UK library and HE sectors such as Research Support Libraries Programme (RSLP) and the National Preservation Office. Projects may also need to display awareness of Z39.50, a network protocol that allows searching of (usually remote) heterogeneous databases and retrieval of data, via one user interface. It is most often used for retrieving bibliographic records, although there are also some non-bibliographic implementations. Z39.50 Maintenance Agency < Z39.50 for All < RSLP Collection Description Metadata work at UKOLN < National Preservation Office < Watermarking and Fingerprinting Projects should give consideration to watermarking and fingerprinting the digital material they produce. Watermarking is the embedding of a permanent mark within a file that can subsequently be used to prove image origination or image copyright. This is normally achieved by integrating the watermark with the image data in such a way that it is virtually impossible to remove. Watermarks can be visible, 13

18 invisible or a combination of both. In all cases the watermark is introduced in such a way that there is minimum distortion of the original image. Watermarks must be able to withstand the image being cropped, rotated, compressed or transformed. Digimarc Watermarking software < Signum Technologies, Suresign Watermarking software < As well as watermarking images before they are distributed, images can be fingerprinted dynamically at delivery time i.e. as the image is downloaded from a Web site. When this is done, other information such as username, date, time, IP address etc. can be encoded as part of the watermark. This makes each instance of download unique and traceable through a transaction database enabling tracking of who is downloading images. This dynamic watermarking and fingerprinting is used by SCRAN. Similar techniques can be used in audio and video media. SCRAN < User Authentication Some projects may wish to limit access to their resources to authenticated users only. User authentication is an important tool for ensuring that only legitimate users can access the project s online resources. If projects choose to implement user authentication it should be based on a username and password combination. In the case of Web-based projects, HTTP Basic Authentication must be used to pass the username/password combination from the browser to the server. In some cases IP-based authentication (comparing the IP address of the client against a list of known IP addresses) may be an appropriate alternative to usernames and passwords. However, the use of this authentication method is strongly discouraged. Projects may choose to make use of third party authentication services, for example the UK Higher Education Athens service, to manage usernames and passwords on their behalf, if appropriate. Hypertext Transfer Protocol HTTP/1.0 < Athens Access Management < 14

19 6. Re-use Users will want to repackage and re-purpose material that has been developed for the nof-digitise programme. In order to facilitate this re-use the implementation of standards will be important Guidance Learning Resource Creation Projects should consider the potential re-use of the resources they create, and recognise that end users or third parties may wish to extract elements of a given resource and repackage them with parts of other resources from nofdigitise and elsewhere. An important area in which this is likely to happen is the educational sector, where learners and teachers are both liable to draw heavily upon nof-digitise resources throughout the learning process. In the wider global educational community, a number of initiatives are underway to create tools for managing educational resources. Some of this effort is concentrating upon the description of content such as that which might arise from nof-digitise. Projects may wish to stay abreast of work on the Instructional Management System, the work of the Dublin Core Metadata Initiative s Education working group, as well as the plethora of UK initiatives from various public sector institutions. Work is currently underway to harmonise approaches in the UK, and further advice may be issued in due course. Instructional Management System < 15