The Argus Digital Collection: A Digital Archive for Access and Preservation

Transcription

1 Wofford College From the SelectedWorks of Fred Miller November 13, 2003 The Argus Digital Collection: A Digital Archive for Access and Preservation Fredrick Miller, Illinois Wesleyan University Available at:

2 The Argus Digital Collection: A Digital Archive for Access and Preservation Fredrick Miller November 14, 2003 The Argus Digital Collection is Illinois Wesleyan University s first digital library project. This project began as an archival and preservation project, and presented management and technical challenges for the University s library and central Information Technology organizations. This paper reviews how the project came to be, project hurdles, lessons learned, and where we go from here. Academic institutions contemplating similar digital archive projects can benefit from Illinois Wesleyan University s experience with the Argus Digital Collection. What is the Argus Digital Collection? The Argus is the name of Illinois Wesleyan University s student newspaper. It has been in print continuously since 1894, and for the most part published weekly during the academic year. The Argus Digital Collection is the name archive and preservation project. This project seeks to preserve and provide access to issues of the Illinois Wesleyan student newspaper from 1894 through Access to the Argus Digital Collection is available over the web. 1 The project uses Greenstone digital library software to provide indexing and search capabilities to the digitized collection of Argus issues. The project also included the creation of reproductions of the issues on acid free paper, as well as preservation microfilm copies of the collection. Prior to creating the digital collection, access to the Argus collection was through the Tate Archives of Illinois Wesleyan University s Ames Library. The Tate Archives averages about 100 walk-in visits per academic year and 50 phone or requests. The library s archivist estimates that 80% of these inquiries use The Argus collection as a reference source. 2 The Argus Digital Collection project aimed to preserve and improve access to these historical materials. Since going online, the project has been well received by the University s alumni, and we expect that the collection will contribute to scholarship by providing a cultural and historical perspective on higher education and collegiate life in the 20 th century. About Greenstone Greenstone is a suite of software used to serve the digital collection to the Internet. Greenstone is open source software issued under terms of GNU Public License. Greenstone is produced by the New Zealand Digital Library Project and distributed in cooperation with UNESCO (the United Nations Educational, Scientific and Cultural Organization) 3. As on open source project, this digital library software is distributed at no cost. Programming, support and documentation are provided by volunteers. The free nature of the software makes getting started fairly simple, but if you run into trouble using the software, you need to work to find answers. There is no 800 number for Greenstone support. Funding, Timing and Other Challenges Illinois Wesleyan University's Archivist and Special Collections Librarian, Anke Voss- Hubbard, began the project by applying for a grant from the Illinois State Library. The grant was awarded by Illinois Secretary of State and State Librarian Jesse White as part of the Illinois Digital Archives database. The project, titled The Argus: Student Life in the 20 th

3 Century, was presented as a preservation project to include microfilming and digitizing the print collection. The original grant was for $10,000 and commenced on October 1, The grant proposal included price quotes from two vendors 4 for microfilming and creating digital images of the collection. Additional funding for the project came from the Andrew W. Mellon foundation, through a grant for archives and special collections. At the time of the grant award, Illinois Wesleyan University was in the process of completing the construction of a new library building the Ames Library. Unfortunately, shortly after the opening of the new library, a sprinkler in the building s fire protection system failed. This caused water damage to a significant part of the library s collection. Recovery required that the library's archivist, Anke Voss-Hubbard, devote a large part of her time to managing the repair and restoration of the damaged items in the collection. The water disaster also caused a substantial delay to work on the Argus project. Late in the Fall semester of 2002, library staff contacted Illinois Wesleyan s central office of Information Technology (IT) for assistance with a Microsoft Access database. They had created a database with metadata describing newspaper issues in the Argus collection, and needed assistance getting the database accessible on the web. They said that images of the collection were to be served from a server at the Illinois State Library. While much work had already been done to create the digital images and set up the database, the project was about to change course. In my role as Director of Information Technology for Illinois Wesleyan, I suggested a different approach for access to the digital archive. I had recently learned of the Greenstone software from projects at the University of Illinois and the University of Chicago. I asked our university webmaster to download and see how difficult it would be to install and set-up a test of Greenstone on one of our existing Linux servers. The software compiled and installed with little trouble; in just a few days we had a test site running. At this point we were able to do a quick demo of Greenstone for library staff. We could also show sites already using the software. I asked the library to consider using Greenstone for the Argus project. I also offered assistance from the office of Information Technology. This assistance was to include: use of an existing university server, adding additional storage to the server, and assistance setting up and configuring the Greenstone software. The library staff agreed to this approach, and what had been a library project was now an opportunity for library and IT collaboration. Soon after beginning our collaboration, the university webmaster pointed out what he saw as a serious deficiency in the pdf images being created for the project. These pdf documents were images only; no optical character recognition (OCR) had been performed. This made it impossible for the Greenstone software to search the text of the scanned images. The library staff explained that they had decided not to have OCR done on the Argus images. They had a quote from the vendor doing the digitizing that would only guarantee an 85% accuracy for the OCR scanning, and the OCR scanning would add an additional $22,000 cost to the project. The decision not to OCR scan the Argus images made the creation of metadata describing the contents of the images extremely important. IT staff wrote Perl scripts to allow library staff to enter metadata about an Argus issue into a web form. Metadata included date of the issue, keywords, and the URL of the where the pdf image of the Argus was stored on the server. This The Argus Digital Collection Project 2

4 form was then used to create an HTML web page that could be searched by Greenstone to create searchable indices for the collection. This work occurred during the Spring semester of Towards the end of the Spring 2003 semester, library staff determined that a single keyword index was not sufficient for searching the Argus issues. They asked IT to add metadata tags to allow searching by a number of different keyword descriptors including subjects, people, buildings, and organizations. The Perl scripts were modified to include this additional metadata, but we encountered a problem with Greenstone. It was only indexing the first term in each of these metadata elements instead of all the search elements. For example if we had created metadata for an issue that had 10 names defined as searchable, Greenstone would only find the first name. Our webmaster consulted Greenstone documentation and mailing lists for a solution. Unfortunately, the Argus collection was not the only project the IT department had going at this time. A major upgrade to the University s portal software was scheduled for the Summer and this would require much of the time of the IT staff. As Summer approached and we still did not have a solution for the multiple metadata problem, we discussed how this problem might require another delay to the Argus project. At this point, the library staff suggested that we use external resources to help complete the Argus project. They had found a PHD candidate at the University of Illinois Graduate School of Library and Information Science. Xiao Hu had worked with Greenstone before, and was comfortable with Perl programming. Several meetings and a consulting contract later, the Argus Digital Collection project was back on track. Acting as project consultant, Xiao was able to find and fix the metadata bug in one of Greenstone s Perl modules, and also provided assistance with customizing the web site s interface to use the multiple search terms. The Argus Digital Collection went live in September of While most of the images of the Argus collection are now on line, work on entering metadata for search terms continues. What did it cost? In 1999, Steve Puglia wrote an article where he presented the average cost distribution of a digital imaging project as 1/3 for digital conversion, slightly less than 1/3 for metadata creation, and slightly more than 1/3 for other activities such as administration and quality control. 5 This estimate for project costs is within the range of Illinois Wesleyan s experience with the Argus project. Since Illinois Wesleyan outsourced the digitizing, microfilming and preservation copy printing for the project, these costs are the most easily tracked. Microfilming the 26,000 page images and making duplicates cost about $5500. The cost of bi-tonal scanning to 600dpi TIFF files and conversion to 300dpi multi-page pdf files (one file per issue) was the single largest expense. This cost over $30,000. Printing preservation copies on acid free paper is not something that Puglia covered in his costs estimates, but for the Argus project this was an additional $5700 expense. There also were some incidental charges for recording the TIFF images to archival CD and shipping. 6 The Argus Digital Collection Project 3

5 Data entry and creating metadata for indexing was also a large part of the project s expense. Most of the data entry and indexing are being performed by a Meg Miner, visiting faculty librarian, and two student workers. Since no OCR was performed on the image files, this indexing is critical to being able to access content in the Argus collection. In order to keep this part of the project manageable, the library has developed rules for selecting what will be indexed. These include rules such as index: letter to the editor writers names but not subjects of letters names of people mentioned in an article if they are the main focus, not just quoted people who are the major subjects of sports article, not just players mentioned or a coach who is asked about the team 7 The library estimates that with this selective manual indexing they can index an issue in about 1.5 hours, and they estimate needing 3000 hours to index the entire collection 8 Besides administration and quality control, the Argus project had a number of expenses other than digitizing and metadata creation. Although we used an existing Linux server for the project we did buy an additional 120 Gigabyte hard drive for the server for $500. (The Argus Digital Collection is using less than 10% of the space on this drive.) The project also paid for the graduate assistant to customize the Greenstone user interface, fix the bug in the Greenstone metadata Perl module, and provide system documentation. We also spent at least 50 hours of IT staff time to set-up and test the Greenstone system, and write Perl programs for index creation in html format. While the Argus Digital Collection is live on the web, it is not finished. There is still much manual indexing to be done. To date manual indexing only covers the years 1990 through The library has applied for another Illinois State Library grant to help cover the cost of finishing the metadata for indexing. The library is also contemplating additional digital library projects using Greenstone. They have already applied for an NEH Preservation and Assistance Grant to begin digitizing pictures of the library s John Wesley Powell Pottery Collection. They are also considering other digital preservation and access initiatives such as the library s collection of letters from WWII and possibly early college catalogs and alumni journals 9 What would you do differently? The ProQuest Historical Newspapers product is arguably the gold standard of access and preservation projects of this type. ProQuest offers full-text and full image articles of newspaper articles from the New York Times, Wall Street Journal, Washington Post and Christian Science Monitor going back to the 19 th century. 10 While the entire feature set of this for-profit venture is beyond the scope of the Argus Digital Collection, it does provide a good model for archival projects of this type. There are two major differences between the ProQuest Historical Newspaper project and the Argus Digital Collection. An Argus pdf file stores an entire issue while ProQuest archives articles, and the ProQuest collection is full text searchable. Few issues of the Argus go beyond 12 pages, so not having access to individual articles is not a major deficiency for the utility of the Argus Digital Collection. The decision not to OCR scan does present some major limitations to the usability of the collection. As presently implemented, you cannot search all the words in an issue of the Argus only selected index terms. A striking example of the limits of manual indexing is a simple search for the terms computer, technology and network. The use of information technology has had a The Argus Digital Collection Project 4

6 profound impact on student life in the last decade of the twentieth century, and numerous Argus articles have discussed computers, networks and information technology on campus, Yet a search of the manual index of the Argus Digital Collection presents one issue for the term computer on September 8, 1995, and one issue for the term network on October, 3, An example of issues missing from this search strategy are the Argus issues from September 19, 1997, and September 26, These issues have feature stories on the University s central servers, and a letter to the editor from the Director of Information Technology. Even with a manually indexed key word, you can t tell which article in an issue contains the selected search term. For example, if you were to use the People index to search for the name Emily Miller, you will find that she is listed in the Argus issue from November 12, The front page of this issue has a headline Miller Shares Wild Life, but that article is about a Chris Miller, and does not mention Emily. Looking carefully at each page of this issue reveals that Emily Miller s name is listed in a photo caption on page 3. Considering the expectation that it will take 3000 hours of manual indexing to create an incomplete search capability, the $22,000 for OCR scanning does not seem like an unreasonable expense. Another option that should be considered for future digitizing projects is to consider doing the scanning in-house. The library staff already use scanning software for articles put on their electronic reserves systems. In addition to scanning equipment, in-house scanning would require OCR software and the development of workflow processes for efficient and effective scanning. The advantage to in-house scanning is that it has the potential for applications for other digital media processing. Data from the Greenstone project indicate that one part-time person doing OCR scanning for three hours per day should be able to scan and proof 120 to 150 pages per month. Full time staff doing similar scanning and proofing should be able to achieve 500 to 600 pages per month. 11 This is considerably faster than the 1.5 hours estimate for manually indexing of an Argus issue. Scanning in-house could also provide an opportunity for increased quality control. We would seek to improve the vendor guaranteed 85% accuracy rate for OCR scanning. Besides improving usability of searching, having full text of issues would also enable text mining capabilities. The Greenstone software already ships with the KEA keyphrase extractor module. The KEA software can be used to extract key phrases from text or html files. It s algorithm help it guess what a document is about. The KEA software generates a list of key phrases and the probability that the phrase would be chosen by the author as a keyword for the article. KEA can be trained to recognize keywords from a model of keyphrases. 12 This software has the potential for allowing documents not only to be searched, but aid in the creation of key word indices for the documents. Another concern about the Argus Digital Collection implementation is the metadata format. Metadata is data about data for our project it is the web page showing the indexed terms found within the pdf image of each issue. Greenstone is very loose in terms of how you define the metadata you want to use for your digital collection, yet there has been much work done within the library and museum communities to define metadata standards. While the Dublin Core is perhaps the best known of metadata formats, we need to consider what metadata standards would be best for federated searching. Federated searching would allow a single The Argus Digital Collection Project 5

7 interface to search multiple collections. This could be all Illinois Wesleyan digital collections, or even collections accessed from other institutions. We need to consider if we should continue with the flexible Greenstone software, or if we should consider a digital library solution that would be compliant with standards such as the Open Archive Initiative (OAI) metadata harvesting protocols. 13 OAI enables projects like the University of Michigan s OAIster project which provides access to digital collections from over 200 institutions. 14 Finally, we need to be concerned about the usability of the user interface to the Argus Digital Collection. Besides the usability problem created by not having the full text of each issue available for searching, it s not clear how successful the current user interface is for searching. For example, you can search within an index category, but you can t search all categories at the same time. Additional user testing would help determine additional improvements needed for the user interface. Lessons learned Illinois Wesleyan University learned a number of lessons from this project. These lessons included learning about technical systems, interdepartmental cooperation, and what the University will require for doing more of this type of project in the future. Illinois Wesleyan s library and information technology staff learned about the installation and operation of a digital library project using the Greenstone open source digital library software. While we are considering using Greenstone for more projects, we need to consider whether it will be appropriate for future projects. One concern we have is it s ability should we decide that federating searching is an important feature of a digital collection project. For future projects we may want to consider other open source alternatives to Greenstone. These alternative solutions include OAI compliant systems like CDSware, DSpace, Eprints, i-tor, and MyCoR 15. There are also commercial products that can work for digital library and institutional repository functions. One commercial product that library staff have already looked at is OCLC s Olive product. 16 This product includes the ability to see full page layout with individual articles and full text search capabilities. Olive is used for a number of newspaper archives including the University of Missouri s historical newspaper project. 17 Another commercial approach involves using an enterprise content management system solution. We have begun to look at the Digital Asset Management capabilities of systems like Documentum 18 for a wide range of digital media management capabilities. While the scope of this type of product is well beyond the single collection approach of Greenstone, it may be worthwhile from an institution-wide perspective. One very important thing we learned from the Argus project was the need for library and IT collaboration on future digitizing projects. The Argus project could have been completed more efficiently with more and earlier communication between library and IT staff. One approach to improving communications may be in the grant proposal process. Consulting with the Director of Information Technology before submitting a grant involving university technology resources could help with the project s implementation. The Argus Digital Collection Project 6

8 Considering all the time and effort that went into this first digital archive project, the institution needs to consider its commitment to similar future projects. This includes support for the ongoing maintenance for projects initially funded with grant monies. In addition, the library is not the only department interesting in digitizing institutional assets. These include music files for the School of music, art images for the School of Art, digital archives for the university s records and many more. The University needs to consider whether it wants to make a strategic commitment to building what Clifford Lynch calls an institutional repository. Lynch defines such a repository as a set of services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and the community members. 19 The key here is the focus on services. What we learned most from the Argus Digital Collection is that this type of project is much more than digitizing and software. It includes a big commitment to metadata creation, project management, and inter-department cooperation. Conclusions Archives and preservation are part of the work needed for maintaining and providing access to an academic library s collection. The Argus Digital Collection demonstrated one way to use open source digital library software to improve access to an important part of the University s archives. In addition, the project strengthened communication and cooperation between IT and library staffs. The hard work of all those involved with the project: librarians, library staff, IT staff, graduate assistant, and student workers, has created a valuable academic resource and a success story for IT and library collaboration. The Argus Digital Collection project gave us an opportunity to think about digital media and the impact of institutional repositories on teaching and learning at the University. We now have a concrete example to show people when we start to discuss future digital media projects. We also need to reconsider the importance of full-text search capabilities in the Argus Digital Collection, and other digital document collections. Now that we have the Argus Digital Collection online, we need to take what the library and IT have learned from this project and consider the lessons learned in a broader institutional context. The institution needs to consider whether this type of work is strategic to fulfilling its mission as a liberal-arts college. If it is, then the library and IT need to focus more on establishing a core of services that will enable Illinois Wesleyan University to continue to build its institutional repository of digital collections. Fredrick Miller is Director of Information Technology at Illinois Wesleyan University. The Argus Digital Collection Project 7

9 Images from the Argus Digital Collection Project 1. Argus issue metadata maintenance page The Argus Digital Collection Project 8

10 2. Selected issue front door page The Argus Digital Collection Project 9

11 3. Argus Search web page The Argus Digital Collection Project 10

12 1 (2003) The Argus Digital Collection, Illinois Wesleyan University s Ames Library, available 2 Voss-Hubbard, Anke (2003) The Argus Digital Collection Indexing and Search Interface, LSTA Grant Proposal, Illinois Wesleyan University, p (2003) Greenstone Digital Library Software, available 4 (2001) The Argus: Student Life in the 20 th Century, FY2002 Educate and Automate Grant, The Illinois State Library, Contract Number , Exhibit A, pp Puglia, Steven (1999) The Costs of Digital Imaging Projects, RLG DigiNews, v3. n.5. available 6 (2002) Proposal for Scanning and Microfilming Services, Illinois Wesleyan University Ames Library Tate Archives and Special Collections, p (2003) Indexing Term Selection Criteria, Illinois Wesleyan University s Ames Library, available online 8 Voss-Hubbard, p Voss-Hubbard, p (2003) ProQuest Historical Newspapers, ProQuest Information and Learning Company, available online 11 Loots, M., Camarzan, D. Witten, I.H. (2003) From Paper to Collection, Greenstone Digital Library Project, p. 16. available 12 (2002) Kea Automatic Keyphrase Extraction, New Zealand Digital Library, The University of Waikato available 13 (2003) Open Archives Initiative, available 14 (2003) OAIster, University of Michigan Digital Library Production Services, available 15 (2003) A Guide to Institutional Repository Software, Open Society Initiative, available v1.htm 16 (2003) Olive for Historical Newspapers, OCLC Online Computer Library Center, available 17 (2003) Missouri Historical Newspapers Project, University of Missouri, available 18 (2003) Digital Asset Management, Documentum, Inc. available 19 Lynch, Clifford, (2003) Institutional Repositories: Essentail Infrastructure for Scholarship in the Digital Age, ARL, no. 226, Feb, 2003, pp 1 7. The Argus Digital Collection Project 11