KEEPING YOUR DATA SAFE - MAINTAINING THE INTEGRITY OF CULTURAL INFORMATION ARC CONFERENCE, TASMANIA. Doug Rogan. 16 November 2006

Transcription

1 KEEPING YOUR DATA SAFE - MAINTAINING THE INTEGRITY OF CULTURAL INFORMATION ARC CONFERENCE, TASMANIA Doug Rogan 16 November 2006 INTRODUCTION Firstly I would like to make clear that this is not a paper specifically about digital preservation, although that aspect is covered. This paper is a more holistic discussion about all aspects of cultural data or information and the various aspects that go into maintaining the integrity of that data/information. As I see it, there are essentially two forms of integrity that need to be considered when looking at how best to manage and care for cultural information. These are: Data Integrity Intellectual Integrity Data integrity I would best describe as being the technical and physical storage and accessibility of data/information. Intellectual Integrity on the other hand, is about maintaining the context,relationships and characteristics that give meaning and significance to the material culture that we all care for. This paper aims to briefly address each of these area s, acknowledging that considerably more depth can be pursued, and to provide some suggestions, practical tips and case studies for further consideration. This paper focuses particularly on digital information (ie information in electronic form) but many of the concepts apply to the analogue (hard-copy) world. However, we need to constantly remember that the reason for maintaining the integrity of cultural information (be it data or intellectual) is so as to provide access, in its myriad of forms. Context

2 I d like to put some of what I m going to discuss in this paper in context. The week prior to the ARC Conference I attended the Digital Futures briefing held by the Collections Council of Australia (CCA) at the National Archives in Canberra (8 November 2006). Some of the key issues raised at the briefing were: Risks, especially o preservation o marginalization if content is not readily available on-line Sustainability Standards (formats, coordinated actions) Conservation skills. A couple of statistics were presented at the Digital Futures briefing which serve to highlight just how important the digital environment has become, and therefore how important it is to maintain the integrity of our cultural information. The statistics presented at the briefing were that: National Archives of Australia (NAA) 84% of access to NAA collections is online or remotely accessed now The NAA s only education delivery instrument is now the Vroom (Virtual Reading Room) system All of these issues are integral to what is discussed in this paper, however, other issues to consider include: Quality of information o maintaining the link with the real world object/item o how much time/effort should be spent on making sure the information is of a certain quality? Perceptions digitised information can be considered more correct than non-digitised non-digitised information is often sidelined and seen as less relevant often simply because it isn t digitised. Accessibility of information need to ensure information is being delivered to the right audience(s) DATA INTEGRITY digital information lasts forever - or 5 years, whichever comes first. Jeff Rothenberg (computer scientist at RAND corporation), Scientific American 1995

3 Many will have heard this quote before, but it encapsulates the data integrity and digital preservation issues that we all face. This section covers some of the main data integrity & digital preservation issues as I see them, and offers some practical tips and suggestions to assist with managing these challenging issues. Back-ups The first line of defence in maintaining data integrity is to ensure a regular back-up system is in place. Whether it uses tapes, hard drives, DVD s, etc it is important to have a sound system &/or strategy in place that is well documented and strictly adhered to. As a part of this strategy it is imperative that trial restores (ie copying a back-up system onto a file server to ensure that the back-up process is working properly) are carried out regularly to ensure that the system is working properly. In managing cultural data it is important to ensure that you have the IT staff on side, although the management of cultural data should never be left to the IT personnel that is what Registrars, Collection Managers, Curators, and other Museum professionals are for. IT staff should be left to look after what they re good at - the IT systems and software. That said, don t trust everything IT staff promise/assure you. My experience is that it is fairly common for a back-up system to fail (or not run properly) at least once a week. Get IT staff to clearly document backup systems, especially how it affects Collection-related systems (ideally this should be part of an organisation-wide disaster plan), and get them to demonstrate the system in operation. Collection Management Systems (CMS) are often proprietary and not well known by many IT professionals, Many IT staff can therefore fail to grasp some of the unique foibles and requirements of these systems. For example, administrative functions (eg indexing) may be done overnight on a CMS, but this may be when overall network back-ups are done as well. It could be that the CMS isn t being backed up properly because of simple scenario s such as this which may not be seen until a restore is required. Collection Management Systems A good, well-run CMS can be the backbone of a cultural institution. However, a poorly designed or poorly managed system can be highly disruptive, waste valuable resources and in many cases can actually lose data. In the past, some organisations have built and designed their own CMS but unless you can afford your own IT department this is not something that I would recommend. There are many advantages with using an off-the shelf CMS (standardisation, support readily available, belonging to a community of users, functions already built-in, not requiring

4 specialist IT skills in-house to develop/maintain the system, etc), with generally the only real disadvantage being the initial capital cost. Many CMS s have functions built in which actually assist with data integrity, the best example of this being an audit trail. In addition, the use of similar systems is improving consistency and coordination within cultural sectors, although not yet across sectors. Digital Preservation The next major issue for data integrity is what is termed digital preservation. This can be broadly described as providing long term access to digital files in their original context. Given the ever-changing nature of the digital world the definition of longterm can be very subjective. However the main point to make is that digital material needs to be made accessible. Digital preservation encompasses a wide range aspects but the main areas for consideration are: Hardware issues are the files stored on CD s, DVD s, tapes, memory sticks, etc. All of these are subject to failure in their own right, plus need a device to operate them, which has hardware issues of its own. The ideal system is to have master files essentially independent of computer hardware. In practice this means files are stored on a computer network (usually on a file server) that is regularly backed up and maintained. Files on a network are easily migrated to any new network infrastructure and are easily accessible. In this day and age the cost of network storage is so cheap that it is far more costeffective to have data stored on file-servers rather than in off-line systems (ie CD s, DVD s, tapes, etc). Software issues - open source versus proprietary. If software needs to be operational for the long term then open source software is preferred over proprietary. That said, in today s world (and for the foreseeable future in my view) there is still a need to use some proprietary systems in the cultural sector these are most commonly the Microsoft Office products, CMS, and graphics programs. The main issue to consider in terms of digital preservation in relation to these proprietary systems is to ensure that the data/information can be made accessible using other software or in different formats (see below). File formats again open source versus proprietary. For the long term preservation of information, it is imperative that open source file formats be

5 used. While it seems unlikely for the foreseeable future, even documents created in Microsoft Word (.doc files) may well be unreadable in years to come. Most files can be converted to an open source format without losing their original formatting and context. The National Archives of Australia with their XENA project, has embarked on a process of using xml formatting to normalise files, to guarantee the long term preservation of files (the original context and format of the files is stored as xml also). See Data migration strategies in relation to the two points above it is important to have a strategy in place that addresses any need to migrate data to new formats so that it remains accessible. Data capture garbage in = garbage out. If the data or information being preserved is incorrect, of poor quality, or out of date then valuable resources can be wasted in its preservation. It is important that good quality data be recorded in the first instance so that it maintains its accessibility and usability into the future. In relation to this, very careful consideration needs to made around what data should be captured. That is should a minimum dataset be created first or should all available information be digitized at the outset; at what resolution do images need to be captured? Essentially no two organisations can do this exactly the same each has different needs, requirements and resources. Metadata (described best as data about data ) descriptive and technical. It is important to have sound metadata around your information so that it can be properly managed. Good CMS s usually provide the descriptive metadata required for documenting and managing collections, and are usually developed along accepted standards (eg Dublin Core). Technical metadata generally relates to the information needed to manage digital files eg image size, camera type, date image taken, etc. It is essential to maintain both the technical and descriptive metadata with any files for their long-term preservation. Much technical metadata is automatically written to file headers when digital files are created. INTELLECTUAL INTEGRITY Preservation of intellectual integrity. As outlined above, this is all about maintaining the context,relationships and characteristics that give meaning and significance.

6 Many of the issues relating to data integrity apply equally to intellectual integrity, although with a different focus. Context and relationships In relation to the management of cultural material, the documentation and long term preservation of the context and relationships of information are managed using some of the following: Collection Management Systems (CMS) Metadata Data capture garbage in/garbage out Other systems holding related information The CMS is usually the primary area/system for recording and maintaining intellectual integrity. Usually, through having a CMS, the issue of what metadata is required is solved in terms of documenting context and relationships with cultural objects. Having the right metadata in place ensures that the organisation will have a good framework to build upon in developing its knowledge base. However, having the right metadata framework in place still doesn t guarantee good data. The old adage of garbage in = garbage out applies to intellectual integrity even more so than for data integrity. Veronica Bullock (CCA), in her paper presented on the previous day at this conference, highlighted some of the risks that organisations face in caring for our material culture. She mentioned the issue of custodial neglect and issues such as physical misplacement, failure to secure legal title, etc. These problems are often caused by having incorrect, inaccurate or out of date data within the CMS. Often such problems are also caused through organisations having numerous systems which don t communicate eg financial, asset management, contacts, etc. Often these systems record peripheral information that relates to collections in some way but is rarely tied back, and is therefore rarely used or accessed by those who actual deal with the collections. It is becoming more and more important for all of this information to be easily available, ideally centralised into a single system. Indeed, this is the essence of what is required as a first step when carrying out a significance assessment 1. Access, Risk-averseness and Copyright 1 (significance) A guide to assessing the significance of cultural heritage objects and collections.

7 It is my assertion that many organisations in the cultural sector are highly risk-averse, particularly in the area of copyright. Collecting institutions should be well aware of their copyright responsibilities and take all reasonable efforts to manage these accordingly. However, these institutions are dealing with generally ever-increasing collections, both physical and digital, and have finite resources to manage these issues. But to me, the risk of not making the information (be it images, text, video, etc) accessible, and easily discoverable, is a far greater risk than the incredibly unlikely scenario of being taken to court for infringing someone s copyright. Taking the most straightforward example of putting images of artworks online, most artists would expect to, and be happy to, have their work represented in an on,line catalogue, if it is part of that organisation s collection. Well written disclaimers and orphaned works policies can help to avoid problems with tricky artists who aren t! Audience(s) In making cultural information accessible, it is important to understand who the users of this information are. Web users are becoming increasingly sophisticated with how they interact with the web and to an ever increasing extent, are becoming the creators of much of the new content/knowledge. This new paradigm in the way the web operates is broadly known as Web2, and is demonstrated in such sites as Wikipedia ( Youtube ( Myspace ( and numerous others. Solutions and Tools Whatever solution(s) is implemented, needs to be sustainable, both in terms of cost (capital and on-going) and in terms of IT interoperability. Does the solution only run on old software, is it upgradeable, etc, This is particularly important in relation to digital preservation Other aspects to consider include: Ease of use training of staff on a system is ongoing and time-consuming Functionality ensure that systems have the ability to do what you require now, and are upgradeable or flexible enough to be able to implement further developments as and when you need them. System resources CMS s in particular can be very system-intensive, especially when there are numerous users doing lots of complex searches/retrievals/reports/etc.

8 Web interoperability if a system can be fully accessible over the internet this provides considerable flexibility for example with institutions that have off-site stores. Integration how easily can the system be integrated with other systems within the organisation? Bear in mind that there are numerous automated solutions available for many of the issues surrounding data integrity, digital preservation, and intellectual integrity. Some of these are outlined below. Systems/Tools in use that can assist with automating many of the issues discussed in this paper: Xena XML Normalising of Archives developed by the National Archives of Australia. Website - DART (and ARCHER) Dataset Acquisition Accessibility and Annotation e- research Technologies essentially a tool that enables/assists with data curation and allows for annotations by users (on an object, image, etc). ARCHER is an extension of DART and is developing more tools (Monash University). Website - DELOS Digital Preservation Testbed to use for testing the feasibility of a preservation approach. Website - CASPAR Cultural, Artistic, Scientific Knowledge for Preservation, Access and Retrieval. Website PLANETS Permanent Long-term Access through Networked Services. Website Automated Migration processes available through the National Library of Australia. Website The Australian Partnership for Sustainable Repositories: AONS - Automated Obsolescence Notification System. Website FIDAS Fieldwork Data Sustainability. Website -

9 OAIS - (Open Archival Information System) Standard using the same language eg Designated User Community (needing to know what the audience wants). Website - Conclusion The main point that I would like to make in conclusion is that cultural organisations need to develop and implement their own strategies to address data and intellectual integrity. These strategies: Should be sustainable Use available tools Integrate systems wherever possible Ensure that access (in all of its guises) to information is the highest priority We need to be aware of the risks in making collections/information accessible, and where practical, implement strategies to minimise these risks. As an industry or sector, there is considerable scope to be less risk averse. We could be embracing our audience(s) considerably more in this area and utilising them as a resource.

10 Case Studies There are many examples that could be used, but presented here are a couple that help illustrate some of the aspects relating to both digital and intellectual integrity that have been discussed in this paper. In particular they highlight the importance and benefits of making cultural information accessible. Case Study 1 - Auckland Art Gallery Many thanks to David Reeves of Auckland Art Gallery for the following information. Auckland Art Gallery had over 13,000 records on their CMS when they made all of their collection information publicly available on the internet. They carried out a basic cleanup especially using authority files and picklists to standardise terminology, names, etc, but simply did not have the resources to check all 13,000 records. So the information went public and the Gallery has been quite open about receiving corrections from the public feedback, even a few embarrassing cases where an artist advised that their work was pictured upside down on the website! The Gallery has been contacted by several people asking why particular images are not showing on the website. When the Gallery responds that they are restricted by copyright and that they have been unable to contact the copyright holder, they often receive the response "I am the copyright holder" and so begins a beautiful

11 relationship! The Gallery has a simple system for accepting the terms and conditions of searching their collection, plus a well worded copyright statement. These can be viewed at Case Study 2 - Old Parliament House Heritage Management Plan The Old Parliament House (OPH) Heritage Management Plan Exposure Draft has recently been released. This document invites public comment on how OPH plans to manage and care for the building, its fittings and furnishings, and general collections, as well as the curtilage. It is planned that this document be developed into an interactive Heritage Management Plan, which will be used internally by OPH staff at first, then eventually made publicly available over the internet.

12 In other words the public will have access to more than just the collection they will be able to research specific information about rooms/areas of the house, events, etc. OPH relies on members of the public (often ex parliamentary staff members) providing them with much of the information about how the house operated, what rooms were used for what, who resided in which areas, etc. This interactive should provide an ideal forum for the exchange of that information.