The dawning of the Dutch network of Digital Academic REpositories (DARE): a sharing experience

Transcription

1 The dawning of the Dutch network of Digital Academic REpositories (DARE): a sharing experience Title The dawning of the Dutch network of Digital Academic REpositories (DARE): a sharing experience Creator Authors: Feijen, M.; Kuil, A. van der Contributors Domingus, M; Maijers, R. Subject Institutional Repositories, Open Archives Initiative, scientific communication Description The DARE (Digital Academic REpositories) programme has brought together all (thirteen) Dutch Universities and three major academic institutions to create a network of digital repositories of Dutch academic output. This article focusses on sharing DARE best practices and lessons learned in the community-driven process to create a Dutch network of OAI (Open Archives Initiative) data providers. Publisher The SURF Foundation Date Type Article Format Text/richtext; Identifier Language Eng Rights Copyright Stichting SURF. The text of this document may be used freely, without permission of Stichting SURF. With many thanks to all DARE project managers who are involved in the DARE programme and have given their input to this article. 1

2 2

3 Abstract The SURF-based programme DARE (Digital Academic REpositories) has brought together all (thirteen) Dutch Universities and three major academic institutions to create a network of digital repositories of Dutch academic output. The first year of DARE focused on implementing the basic infrastructure by setting up and linking the repositories. In the spirit of the open access movement, this article will focus on sharing DARE best practices and lessons learned in the community-driven process. This resulted in creating a Dutch network of OAI (Open Archives Initiative) data providers. A demonstrator portal called DAREnet ( has been set up to access this national network s academic output. With the DARE management at SURF as the beating OAI heart, all universities agreed on a few basics: the use of relevant standards. the use of existing systems as tools for creating the OAI institutional repositories one solution does not fit all, and learn by doing. Other common agreements were: the repository s infrastructural requirements the amount of items to be harvestable from the repository, all items within an repository are ultimately freely accessible, and the number of university faculties participating in and contributing to the repository. The above defined the scope of the ambition and tasks for the institutions libraries as a whole. The DARE management created a platform on which knowledge was developed and shared with all DARE key people in the universities. This approach led to rapid results instead of (the illusion) of a complete theoretical framework in which all pros and cons were weighed. Finally, the breaking down of the OAI ambition into two levels, i.e. a data level and a service one, kept issues relatively manageable. SURF stimulates and supports projects on a data level and funds projects on a service level aiming at the delivery of content from the academic community. The intended audience for this article is implementers and managers of new and existing OAI repositories worldwide. 3

4 Content Abstract 3 1. Introduction 5 2. Approach Data and service model Individual responsibility, joint action Preservation 7 3. DAREnet: Purpose Implementation Experiences 7 4. Infrastructure Architecture Evaluation of Technology / Software 8 5. Metadata DARE metadata guidelines Experiences 9 6. The implementation process Success factors Hurdles taken The DARE community Programme management Organisation of the different institutes Content Lessons learned Organisational Harvesting Technical issues Metadata Future plans 13 References 14 4

5 1. Introduction The SURF programme Digital Academic Repositories (DARE) is a joint initiative of Dutch universities to make their academic output digitally accessible. The KB (National Library of the Netherlands), the KNAW (Royal Netherlands Academy of Arts and Sciences) and the NWO (Netherlands Organisation for Scientific Research) also cooperate in this unique programme. DARE is being coordinated by the SURF Foundation. The programme will run from January 2003 until December DARE has several goals: Implementing the basic infrastructure by setting up and linking the repositories; Stimulating the development of services based on the research information made available through the infrastructure; and Initiating and promoting the submission to and use of scientific content from the repositories. The DARE programme has been given financial support by the government, through the NAP (National Action Plan electronic highway) fund of 2 million for the period With this grant the Dutch government is giving a strong boost to the innovation in the provision of academic information in the Netherlands. 2. Approach 2.1 Data and service model Figure 1 shows the data and service model that repositories use internationally. It comprises a) a basic facility (the data level) and b) services (the service level). These services are developed from the basic facility and provide added value for specific end users. - The data level is where the infrastructure is set up and maintained. A scientific institution establishes a repository that stores academic output from that institution and keeps it available for use (or reuse) according to uniform international standards. This includes working papers/preprints, dissertations, research reports, datasets, conference reports, teaching material, graduate essays, multimedia material, etc. including the corresponding metadata. - The service level: basic material from the data level can be used to develop services providing added value for scientists, students, universities, funding agencies and other interested parties. The possibilities are numerous, including developing current or new services (e.g. management information or the updating of resumes; services the institutions themselves wish to offer, either individually or in association with others (e.g. subject portals); or services provided by third parties such as publishers (e.g. e-journals). The supply and development of services can take place at a local level (e.g. individual homepages), at a national level (e.g. national academic output service, e.g. organised by document type or discipline) or even at an international one (e.g. virtual communities). The most appropriate level of co-operation can be decided upon depending on the situation in question. However, it is crucial that institutions retain control of their own information. 5

6 Figure 1 Data-service model The advantage offered by the data-services model is that it also provides a guide to making sound decisions about the level and extent of services that an institution wishes to offer and the costs involved therein. The data level offers every institution a basic facility for the reliable, structured digital storage of its own intellectual property. This is an important factor in the digitisation of formal scientific communication. The DARE programme ensures that the data level is set up as simply as possible. As a result, the basic facility can be offered free of unnecessary extras, at the lowest possible operational cost. This can then also better guarantee its financial sustainability in the long-term. All additions to functionality (and thus work and manpower) aimed at activities above and beyond the basic facility belong to the service level. This makes it possible to ascertain what the corresponding costs are for each service, for whom such services are intended, whether they are worth the effort, and how best to finance them. The DARE Programme stimulates the development of services in such a way that good use is made of available (open) technology for improving their effectiveness and efficiency Individual responsibility, joint action An important aspect of the DARE programme is the combination of individual responsibilities with joint actions. Every university is responsible for its own repository, has its own motivation for its implementation and decides itself which services it wants to offer. Within the DARE community, participants work together and forces are joined for greatest effect, to be able to do the job, share knowledge and experiences and to be able to realize interoperability. There is no single prescribed standard repository solution, as long as the repository complies with the agreed formats. The rule is to do locally what can be done locally and restrict centralized activity to the bare essentials like creating a frame of reference, setting preconditions where and when necessary and working together on shared issues. The DARE programme stimulates the development of different approaches in concrete projects. This way the participants learn by way of doing, a pragmatic way of pioneering, at both data and service levels. 6

7 2.3 Preservation It was considered an a priori requirement that public and other data of sufficient importance stored in the repositories would be automatically preserved for long-term future use. To that end the Koninklijke Bibliotheek (The Royal Library) has been involved from the very start of the project. The Royal Library has already set up a system for the long time preservation of data, which will also be used for the data in the DARE repositories. This is therefore no task for the local repository managers, but one dealt with centrally by the Royal Library. 3. DAREnet: Purpose DAREnet is a demonstration website providing basic information services. It has been developed to check and show the interoperability and the possibilities such a network of institutional repositories can offer for scholarly communication. It has not been set up as a permanent national harvesting service, but merely as a demonstrator. 3.2 Implementation DAREnet uses i-tor 1 for harvesting the metadata of all repositories and for providing services to end users (service level in Figure 1). Some of the DARE participants used i-tor as a data provider for their institutional repository (the data level in Figure 1, see next section). For end users, the standard services include the browsing of the repositories and searching the metadata (on specific or all fields). Full text searching of the digital objects (referenced in the metadata) is also a standard feature. For administrators with the right permissions (who can log in on the site), it is possible to edit the web pages and to manage the harvesting of the various repositories. The i-tor tool was selected for implementing DAREnet because its goal fits the goals of DARE: providing open access while retaining data at the source. It uses open standards and is an open source product, providing a web content management system. It features the collaboration and coupling of existing data sources, including OAI repositories. It was especially this feature which was used particularly intensively in DAREnet. 3.3 Experiences Software development Whilst DAREnet was being implemented, i-tor features for handling Open Archives were also under development. Although this meant occasional bugs, an advantage was that missing features identified by DARE participants and their implementation were part of the I-Tor development process, resulting in a more tailor-made product for DARE. The fact that the software developers were a division of one of the DARE partners was another advantage. Using i-tor The ability to manage the site remotely, not needing technicians, turned out to be a very useful feature of i-tor. Changes could be made swiftly this way. The DAREnet site was implemented in both Dutch and English. Multi-lingual interfaces are a matter of concern in most systems. For the web interface i-tor handles it reasonably well by providing tools to maintain versions of pages for various languages. For the harvesting, however, separate harvesting has to be done for both the Dutch and English version and this unfortunately leads to unnecessary additional work. 1 i-tor: Tools and Technology for Open Repositories 7

8 4. Infrastructure 4.1 Architecture At the start of the DARE programme, a specification document Specifications for a Networked Repository for Dutch Universities served as our compass for the DARE architecture. This document describes DARE as a network of local repositories, with local policies, local repository software, linked together via OAI-PMH (the OAI protocol for harvesting metadata). 4.2 Evaluation of Technology / Software Starting positions for the DARE partners were very different. Some institutions already had a repository in place, some had a repository but it was not OAI-compliant or otherwise sufficient. Others started from scratch. Starting from scratch Those who had to start from the beginning experienced that the choice of software was problematic. It was hard to find up-to-date comparisons of software. One of the universities had a lot of problems with the available software. The lack of good installation manuals and software which had not been tested on various platforms made it impossible for them to implement their own repository on time. By using the repository of another DARE partner, on a temporary basis, they were able to join the effort. Adapt or change Some institutes used their existing systems and made them OAI-compliant. Another university had to make a workaround (with the help of one of the other DARE partners being their data provider) because of problems with their existing system (DigiTool). For this library, problems with crosswalks, xml-files and Dublin Core made it necessary to switch to DSpace. Transferring data from old databases, which had been set up for other purposes, to the new repository was a considerable challenge for some. Systems already in place Three universities were involved in the ARNO-project (Academic Research in the Netherlands Online) 2. This project, also funded by the SURF Foundation, resulted in an OAI-compliant institutional document server that could function as a repository. These universities therefore had their repositories up and running, despite the problems caused by a new release. Another institute already had a lot of experience with their DSpace repository. Five different systems are in use at present: ARNO is being used for five institutional repositories. Three of these institutions have been involved in the ARNO-project. Six universities are using DSpace. Four of them started off with other software but have changed since or are in the process of implementing DSpace. i-tor is being used by three organisations, one being the institute that has developed this software. i- Tor has the advantage of being developed by a DARE partner where tailor-made changes can be made. However, because i-tor is a new product it has insufficient documentation and is still under development. Two institutes use other proprietary software. 2 ) ARNO has been developed by the University of Amsterdam, Tilburg University and the University of Twente, as a result of a project funded by the SURF Foundation. It aims to develop and implement university document servers to make the scientific output of participating institutions available. Read more on Like i-tor and DSpace, ARNO is an open source programme. 8

9 5. Metadata Each element in the repository is described by a set of metadata. The original specification document for DARE states that the OAI-PMH standard should be used within the DARE programme. This affected the standard for metadata. The OAI protocol requires that repositories offer the 15 metadata elements employed in simple Dublin Core (DC). 5.1 DARE metadata guidelines The application of standards always leaves room for local choices and implementations and the OAI world is hardly an exception to this rule. The OAI standard [OAI PMH 2.0] and metadata formats [oai_dc, qdc] led to a short but intensive study of the DARE use of Dublin Core metadata, which, in turn led to a version 1.0 that was acceptable for the DARE community to begin with. This DARE use of Dublin Core metadata, version 1.0 benefited a great deal from the best practice observed at UKOLN, Bath, with E-Prints. In this document, which was endorsed by all DARE partners in October 2003, DARE chose to use simple DC as the mandatory metadata set (because of OAI-PMH), and DARE qualified DC (dare-qdc) as the optional metadata set within DARE. This was because the librarians soon concluded that simple Dublin Core was too simple and limited for DARE and its institutions needs. At the time of writing this article, all DARE partners use simple DC for data exchange. The project managers have spent quite some time discussing where to draw the line between the data and service levels. The basic question here was whether to first define services and then determine the use of metadata or vice versa. Some people argued that Z39.50 experience showed that garbage in produces garbage out. It is difficult to imagine all services possible beforehand and therefore to think of all of the metadata needed in advance. Hence, you either get caught up in endless discussions on services or those on metadata. DARE chose the pragmatic solution to work with simple DC and leave more elaborate metadata exchange until the moment this seems necessary. We also discovered that discussing metadata within an OAI context is in fact a discussion of data mapping and data exchange. Each DARE partner has developed some form of mapping from internal systems and richer metadata to the simple DC. Librarians tend to see DC as a set of cataloguing rules, which it is not, it is for data exchange. In a networked, co-operative setting like DARE, one needs some guidelines that answer the question what content should go into which DC element. So some cataloguing rules can be applied to make sure the same language is spoken. Dublin Core, as is, is not specific enough for that purpose. 5.2 Experiences Once the DARE metadata guidelines were defined, we decided to freeze them for a period of six months. During the following months (October 2003 March 2004) we kept track of issues, questions, problems and experiences with mapping, harvesting and using the metadata. Some issues that arose were: - Non UTF-8 characters in local input: The use of UTF-8 is mandatory in OAI PMH, but some repositories found that users were uploading special characters into their repositories. The DC Checker utility helped here; - The use of dc:identifier (1): Should it contain the url of the object or the url of a local resolving mechanism like a jump-off page? And where should the identifier lead the service provider to in case of one record with multiple related digital objects (bit streams))? - The use of dc:identifier (2): should we use OpenURL, the CNRI handle or DOI? We agreed not to use DOI but a persistent URL to be chosen locally; - How do you harvest the object file itself? OAI-PMH has no provisions for that; - Should we include metadata of printed and digital objects, or purely digital ones? Digital only was decided upon as the ultimate goal (as an intermediate step metadata referring to non- 9

10 digital objects were accepted temporarily, as long as the goal was to get access to the digital object in the end); - Should we use sets, and if so what should be the set contents? - Problems with the correct use of the date stamp; - Challenges with the dare_qdc namespace and XML schema; - Use of resumption tokens (we agreed to use one after every 200 records). During the period when we were building the DARE demonstrator we were forced to create additional guidelines for metadata for the following elements: name (full name and/or initials), source (to link the digital object to the printed version we used journal title, ISSN, volume, pages in dc:source when applicable), the jump-off page in dc:identifier and the direct link in dc:source, etc. Within the context of this article it would be inappropriate to explain all the details behind these issues. Some were solved and some were not. The biggest remaining issue is the identifier/jump-off page issue. DARE has set up an international OAI expert meeting on this subject in May The implementation process Looking back on the implementation period, the DARE community is proud of having achieved its first major milestones: A summary of the evaluation by the programme participants of all institutions involved is given below. 6.1 Success factors DARE as a national initiative, not only made all universities aware of the necessity of an institutional repository but also made it a reality for all universities to set up and have an institutional repository. With the commitment from all parties and the central co-ordination by SURF an operational network of repositories now exists, demonstrated by DAREnet. These facts also mean that a number of new services can be developed upon this Dutch repository infrastructure. DARE has given a Dutch boost to the dissemination and utilisation of OAI technology. A simple standard as defined within the Open Archives Initiative has helped to quickly agree on the way to go about retrieving and exchanging data. In addition, the fear that on presentation of the programme at the deadline that there would be a no show unless all repositories were live, proved to work as a great stimulant. The interoperability of information and the profiling of the different organisations have also been critical success factors. 6.2 Hurdles taken Significant results have been achieved by a group of diverse people and institutions. Although the different cultures and approaches were apparent and sometimes a challenge, they never stood in the way of the common good of the project. Not all institutions started at the same time. Those who followed in a later phase had less time to set up servers, install the software and get content ready for the repository. Copyright issues need a lot of organization and will be a threshold for a lot of academics to join the DARE movement. However, this issue as well as filling the repositories with content will be the major focus for 2004 and beyond. It is therefore important to develop a strategy aimed at scientists and managers being committed to contributing content to the repositories. You could say that the technical implementation was the 'easiest' part; the institutional and organizational embedding is the challenge for the coming period. 10

11 6.3 The DARE community Without a doubt, everyone has experienced the added value of working together. This added value was felt on both practical and technical levels. Partners were also there for one another for moral support, inspiration as well as stimulation. Not all institutions started implementing a repository at the same time. Those who followed in a later phase were able to profit from the experience of others. Some thought it would not have been possible to have implemented their repository at such short notice without the help of DARE partners. Those with an existing repository on the other hand were inspired and stimulated to further enhance the content and management of their operational repositories. The central co-ordination and sharing of knowledge has certainly helped to lead to the achieved results. The community proved to be a lively one, although decision-making on technical details was sometimes perceived to be too slow. To communicate and exchange information and knowledge online, in a private and secure area, an extranet was set up (making use of CommunityZero 3 ). However, this extranet needed a bit of getting used to and the software could have been more user friendly perhaps. Nevertheless, it has certainly added to the community feeling and to the exchange of information and experiences. 6.4 Programme management Due to good project management, local initiatives and the central approach have been brought together in a very good working environment. Some participants say that having been pushed to have the repository up and running has given them the basis for the further development that they needed. Presentations by the DARE program manager to different stakeholders has had a positive effect. The never ending enthusiasm from the programme management at SURF added value as did good and timely information and communication. 6.5 Organisation of the different institutes The different institutes have had different ways of dealing with the organisation around the institutional repositories. Some examples can be found below: One university set up a new unit that supports scientists with electronic publishing. This unit is also responsible for the IR. Another university set up a steering committee and instated project managers for the implementation process, very similar to the national DARE organisation of steering committee and project managers. These people were also responsible for the communication within the university to get the DARE notion across. Commitment from a number of important users has helped the start of the local repository of another university. Although some universities were not quite ready for the implementation of a repository, the DARE initiative has given them a start: For some universities the lack of human resources is the reason that they have only implemented the repository on a very small scale meaning that they have only put their focus on the technical aspects of the IRs and on making content more easily available. Fragmentation within one of the institutes made it impossible to have all sub-institutes participate in the DARE programme. One section was committed to making a mini-repository available but all participants had different ideas on how to do this, especially concerning the issue of having the digital object available. Persuasiveness was needed to convince the participants. However, seeing DAREnet, they now realise the potential of DARE and the benefits of sharing scientific information. As for the submission of content, different organisational approaches are apparent: Most institutions use decentralised ways of collecting content

12 The national Metis system 4 (research information system) is being used in several places for (decentralised) submission. However, this submission is often only done once a year, for the sake of the annual reports, and this slows down the digital object becoming available. One university library has started a project to integrate its library catalogue, documentation database and the repository in order to make the submission process more efficient and to improve on the quality of the metadata. 6.6 Content All DARE participants started off with readily available content from one or more faculties or departments. Most universities have dissertations and research reports available. Today, the content mainly consists of text documents, but some photographs and videos are also included. Although the repositories can contain digital objects of any kind, this is in practice not yet the case. 7. Lessons learned 7.1 Organisational An important success of the DARE programme is combining individual responsibilities with joint actions. Each university is responsible for its own repository, has its own motivation for introduction and decides itself which services it wants to offer. It was important for the DARE participants to know that they could hold onto their own ways of working, but also to learn of other ways of working. By using each others knowledge and expertise the community was able to realise interoperability. The DARE programme stimulates trying out different approaches in concrete projects. In this way the participants learned by way of doing, a pragmatic way of pioneering, at both data and services level Harvesting As soon as a repository was harvestable, testing could begin. It is important to make sufficient time for the testing phase. This involves several cycles of adjustment and retesting. One reason for this is that with such a large number of diverse parties involved, several cycles are needed before the repositories deliver exactly what the harvester needs. In the DARE case, several problems arose due to the incorrect use (mostly misinterpretation) of the Dublin Core standard. In other cases problems were caused by lack of specification (like which character encoding to use, e.g. OAI-PMH requires UTF8 but one repository contained non-utf8 characters in its metadata). Another recommendations would be to set strict specifications which each party has to comply with, including examples. The repositories may revert to creating temporary databases, as was done by some of the DAREnet participants, just to be able to deliver the right format for harvesting purposes. Every time repositories changed their output format, the time-consuming process of harvesting had to be performed again. New software releases of i-tor (bug fixes or new features regarding harvesting) also occasionally made re-harvesting of all repositories necessary. The main bottleneck that made harvesting so time-consuming, was the network connection between the DAREnet harvester and the repositories. This was a matter of trial and error and finally we succeeded in linking all repositories to DAREnet. Full texts 4 This is a research information database system which enables universities, organizational units within universities, research institutes, research groups or individual researchers to register information about their research online and to make this information available worldwide in a multitude of ways. 12

13 Another issue with harvesting turned out to be local resolving mechanisms, like the 'jump off pages' often used in Open Archives. The metadata contains a link to the digital object, but this is not always a link to the object itself but a (jump off) page on the site of the institute. That page in turn contains a link to the digital object. The jump-off page, however, is a more stable url to link to. When, as is the case with DAREnet, the digital object is needed for the purpose of full text searching, you have a problem with linking to a jump-off page. This problem still exists and cannot be solved easily. 7.3 Technical issues The original assumption that the required technology already existed proved to be true. Several technical problems encountered during this project stress the importance of good hardware, backups and software as well as measures regarding support and maintenance. 7.4 Metadata OAI-PMH is not a set of cataloguing rules. It is a lingua franca for data exchange between partners who have agreed to use simple DC. However, OAI-PMH allows for data exchange based on various other formats. This can be any metadata set. OAI-PMH does not protocolise the exchange of the digital object file itself. An additional protocol is needed here as in most cases service providers also want to have the object and not only the metadata. One solution might be to use MPEG21-DIDL (as described by Herbert van de Sompel c.s. in his article in D-Lib Magazine of February 2004, Using MPEG-21 DIP and NISO OpenURL for the Dynamic Dissemination of Complex Digital Objects in the Los Alamos National Laboratory Digital Library. However, his solution has only been implemented at Los Alamos so far.) OAI-PMH is simple enough to get you started. From a librarian s perspective it might be too simple. From a user s perspective (scientific author, student, professor), it might be elaborate enough. At present we have not yet decided whether we need additional qualifiers in DC. This question is still under discussion and will have to be settled based on experiences with actual service development. The unique identification of metadata, the digital object and its creator has proved to be necessary and useful. DARE is therefore working on a so-called digital author identification. 8. Future plans The libraries involvement in the DARE experience has shown a movement within traditional library issues of (online) access [metadata and infrastructure] and (digital) preservation to actively publishing and the possibility of the multiple use of metadata and digital objects. Furthermore, libraries and scientists have found themselves in new, more interactive roles between one another. DARE fundamentally improves open access to the university s hidden web or deep web in which scientific publications were at best downloadable but less easy to find. This shift in opportunities, possibilities and competences is only just being explored and is currently being translated into services that meet the needs of scientists as well as those of the general public. The DARE focus will mainly be on this process in its second year, i.e This also implies an emphasis on the more efficient re-use of metadata and digital objects for the whole process of its creation, access and long term digital preservation and using today s solutions to link the separate systems that manage these sub processes. As far as the metadata is concerned, further development of the DARE use of metadata is needed. Communication and marketing are also important issues for the next year(s) in order to make the academics aware of the possibilities of repositories for academic communication and publishing. The involvement of the scholarly community in the further submission of content and in service development will therefore be a major point of attention. Directions for (re)use 13

14 Certain scientific communities have already long-term experience in sharing their publications in OAI environments such as Los Alamos. One could state that in these early adopter environments the need for quick access to different versions of a paper was a drive for use of the oai protocol. It served a specific need: the rapid dissemination of information and knowledge and feedback which is often only relevant for a short period of time - a process defined as 'science in the making' by Bruno Latour 5. Latour makes a distinction between 'science in action' - the process of 'science becoming fact' and 'science as fact'. One could stress that the new challenge is to create environments, based on the oai protocol, to gain access to science as it has been established to be a fact. In other words: the pre press phase and demand for quick access serves the 'science in the making' aspect, which has already manifested itself in the early oai-adopter environments, serving mainly scientists. The task now is to give the public open access to the static science corpus we now know to be 'scientific establishment', in environments that (re)use data in oai dataproviders. References 1. DARE programme: 2. DAREnet: 3. SURF Foundation: 4. DARE Report Specifications for a Networked Repository for Dutch Universities : 5. DARE report DARE use of Dublin Core metadata : 6. ARNO: 7. i-tor: 8. DSpace: 9. Using MPEG-21 DIP and NISO OpenURL for the Dynamic Dissemination of Complex Digital Objects in the Los Alamos National Laboratory Digital Library, J. Bekaert, L. Balakireva, P. Hochstenbach and H. van de Sompel, D-Lib Magazine, February 2004: 5 [Science in action : how to follow scientists and engineers through society, Open University Press, 1987 ] 14