Mark Sweeney Library of Congress 101 Independence Ave., SE Washington, DC

Transcription

1 Date submitted: 24/06/2009 The United States National Digital Newspaper Program (NDNP): a distributed national effort to enhance access to America s historic newspapers Mark Sweeney Library of Congress 101 Independence Ave., SE Washington, DC mswe@loc.gov Meeting: 99. ICADS WORLD LIBRARY AND INFORMATION CONGRESS: 75TH IFLA GENERAL CONFERENCE AND COUNCIL August 2009, Milan, Italy ABSTRACT: This paper describes the multi-partner relationships, technical specifications and tools, providing access and preservation elements incorporated into building the U.S. National Digital Newspaper Program (NDNP), a partnership between the National Endowment for the Humanities (NEH) and the Library of Congress (LC). NDNP is a long-term effort to provide permanent access to a national digital collection of newspaper bibliographic information and selected historic newspapers, digitized by NEH awardees in all U.S. states and territories. This program provides the Library of Congress with a testing ground for the development of large-scale distributed, digitization programs and predicting long-term needs for management and preservation of digital assets. The current development phase focuses on creating digitized newspaper page surrogates through a distributed effort, ingesting the resulting digital objects into a system, providing user-friendly access to the data, while implementing a system that is capable of sustaining the content for future use. The U.S. National Digital Newspaper Program (NDNP), a partnership between the National Endowment for the Humanities (NEH) and the Library of Congress (LC), is a long-term effort to provide permanent access to a national digital resource comprised of newspaper bibliographic information and selected historic newspapers, digitized by NEH funded institutions (Awardees) in all U.S. states and territories. This program builds on the legacy of the strategically successful United States Newspaper Program (USNP, ) sponsored by the NEH with LC technical support an excellent example of successful collaboration at both the national and state levels to inventory, catalog, and selectively preserve on microfilm a corpus of at-risk newspaper materials. The newer NDNP not only extends the usefulness of USNP bibliographic and microfilm assets by increasing access to this valuable information, but also provides an opportunity for many institutions to contribute select digitized newspaper content to a freely accessible national newspaper resource. 1

2 Historic newspapers are the primary record of events that chronicle the development of communities. They provide a venue for sharing the facts and opinions of moments in time, significant people, and local perspectives a unique resource for recording and understanding the effects of both singular and united voices on ideas, events, and democratic identity, as well as defining the historic record. In recent decades, under USNP, the preservation of newspapers on microfilm and the establishment of imaging and bibliographic standards has been an important component of archival programs in order to manage and sustain the vast quantity of material representing the historic record effectively. However, even this critical aspect of newspaper librarianship does little to address the use and access needs of text-intensive newsprint. Utilizing this valuable resource, imaged on film or in original paper is a challenge for libraries and users alike, with its cumbersome physical aspects, discolored and brittle paper, and complex organization. Even with the best imaging standards and process, the intellectual content of the newspaper is contained in a complicated layout, with varying visual cues and small type faces, wearying to the eye and the mind. The development of new digitization technologies, text recognition, search engines, etc. enables the NDNP to now provide enhanced access and discovery to this material, as well as the national leadership necessary to establish best practices and standards for the digitization and structure for historic newspaper materials intended for a sustainable electronic resource. Since the U.S. national newspaper collection is dispersed among hundreds of libraries throughout the country, a decentralized selection and digital conversion model was adopted with data aggregation provided by the Library of Congress for access and preservation. The primary goals of the program are long-term provide enhanced access to select newspapers by creating and aggregating millions of geographically-diverse digitized newspapers while also repurposing existing bibliographic and holdings data for over 139,000 titles in a freely-accessible and searchable system. Multi-partner Relationships (NEH/LC/Awardees): Since 2004, the NEH and the LC have collaborated to develop a nationwide program that enhances access to this material through the use of new technologies and information channels, scaled to include representative content from all U.S. states and territories produced over several decades, and to encourage interoperability between digital libraries through shared specifications. A memorandum of understanding between the NEH and the LC clearly delineates the responsibilities of the two agencies in developing the overall national program. While the NEH manages and funds annual award competitions among state-level institutions to select and convert historic newspapers to digital form, the LC focuses its efforts on the program s technical specifications, data management and publicly serving the content. State level institutions, known in the program as Awardees, are responsible for selecting newspapers published in their state according to program guidelines and converting them to valid digital form for central aggregation at the LC. In 2005, NEH awarded $1.9 million among six institutions University of California- Riverside, University of Florida, University of Kentucky, New York Public Library, University of Utah, and the Library of Virginia - to select and convert newspaper holdings representing their state collections. These original awardees were selected for their experience with historic newspapers, digitizing collections, and digital library infrastructures. In the initial phase the program produced a developmental system that stored hundreds of thousands of pages of historic newspapers converted from both the collections of LC and NEH awardees. In March 2007, the NDNP launched its Web service dissemination from that system - Chronicling America at The site was launched with a complete newspaper title directory of data created under USNP 138,000 titles and 900,000 holdings records and over 225,000 full-text searchable pages of converted newspapers, 2

3 published between 1900 and NEH held successive award competitions to gradually increase the scope both in geography and time of the aggregated national collection and to build expertise at the state level in large-scale newspaper digitization. In 2007, NEH renewed awards to five of the six original awardee institutions and made new awards to three institutions the Minnesota Historical Society, University of Nebraska, and the University of North Texas. This round of awards was to convert 800,000 newspaper pages, 100,000 pages per state, published between 1880 and In 2008, NEH made awards to six additional institutions Arizona Department of Libraries, Archives and Public Records, Ohio Historical Society, Pennsylvania State University, State Historical Society of Missouri, University of Hawaii, Manoa, and the Washington State Library. These six institutions are converting 600,000 newspaper pages from their respective states published between 1880 and A 2009 award competition to further expand the program to additional states and larger chronological coverage ( ) is currently underway. Technical Specifications and Tools: In the development and overall management of the program, the Library of Congress provides technical support of the program s primary goal creating open access to the nation s historic newspapers. The Library s role has three parts: to establish technical digitization specifications that permit aggregation, to serve and unify this content through a publicly-available Web site, and to permanently sustain the aggregated content. As LC reviewed the means available to accomplish these more technical objectives, it became clear that the requirements of the goal of sustaining the content would inform many decisions for the other two objectives. The still-evolving NDNP system environment is based on requirements to support four major workflows as identified in the Open Archival Information Systems (OAIS) Reference Model: ingest, archiving, dissemination and preservation management. From the outset, LC recognized the scope of the planned program millions of newspaper pages produced by many different organizations over approximately 20 years (equaling, at least, hundreds of terabytes) and the commitment between publicly-funded agencies to manage these assets required emphasis on the creation of digital assets according to emerging standards and uniform best practices. Well formed data operating in a robust technical infrastructure was seen as the best approach to ensuring cost-effective management of the content over time. The Library s first steps included determining high-level operating principles and functional requirements for the digital asset system and the associated dissemination workflow. In a climate of emerging (and evolving) best practices for digital preservation, LC initiated an explicit development phase to allow for research and assessment of long-term workflow and curation needs, as well as incremental progress toward NDNP goals. The principles applied in making technical choices were intended to support the development of a system that is sustainable in today s best estimation open, modular, certain to change, and able to evolve to meet future uses. In addition, the decisions made were informed by realities of the overall program structure: The content analog versions (microfilm, paper) of historic newspapers - resides primarily in state repositories, rather than at the national library, therefore the program requires distributed production of the digital assets; The funding to apply new technologies to enhance access to this material is finite, therefore, o given the sheer quantity of available material, content included in the program will be selective, rather than the entire corpus available; 3

4 o technical requirements for converted materials should account for potential re-use and reprocessing over time (scan once, use many times) o should provide a model for similar distributed efforts that may eventually interoperate sharing best practices, conversion specifications, and standardizing basic access for historic newspapers; Demonstration of good use of public funds by providing open and perpetual access; In expectation of change, avoid closing off options, by developing a system environment that would be open, expandable, and modular. In order to build an extendable and scalable activity, NDNP considered various requirements for production and management of the digital information created by NEH awardees. First, in order to fulfill LC s role in aggregating and managing the digitized content over the long-term, LC needed to consider five main requirements: convert the content to achieve the highest quality information for discovery and reuse, ensure technical consistency across content created by multiple producers over time, use open and sustainable formats to encourage long-term preservation, develop a data architecture that would allow for both manageability and scalability over time, and develop scalable workflows, processes, and quality management that support the large-scale ingestion of content from multiple producers. Building on its experience with large-scale digitization of historic materials, LC developed a set of technical specifications for content created in NDNP based on best practices. The image specifications TIFF, JPEG2000, and PDF are intended to play specific roles in the NDNP system (TIFF for archiving, JPEG2000 for production and PDF for portability) and conform to current best practices for digital file format sustainability. 1 These practices include wide-ranging adoption in the cultural heritage community, transparency of the digital information itself, and self-documentation within the file format. The image specifications for NDNP, primarily 8-bit grayscale at dpi, attempts to capture the most data possible from newspaper microfilm negatives in order to provide for future reprocessing and reuse at a later date with improved technology. In addition, LC chose a standard XML metadata scheme (Metadata Encoding and Transmission Standard 2 ) for description of the digital objects at the newspaper issue and page level and the ALTO (Analyzed Layout and Text Object) schema extension 3 was chosen for structuring the automatically-recognized machine readable page text (known as optical character recognition or OCR). Metadata requirements were intended to provide a basic level of access to newspaper pages, capturing as much structural and technical information as possible from both film and intellectual content at the point of digital creation. NDNP recognized that a distributed production model would require improved mechanisms for quality assurance of the content as it was created and aggregated, as well as explicit incorporation of metadata intended to assist in long-term management and sustainability of the digital objects. These requirements led to the development of two NDNP-related tools, a microfilm scanner target for objective image quality analysis and a technical validation and quality review software, both used by program participants to assist in capturing technically valid, high-quality images and ensuring that metadata conforms to NDNP technical specifications. 4

5 The NDNP image specifications attempt to capture the most data possible from newspaper microfilm and the program has established technical specifications and workflow components to that end. While original paper issues may substitute for microfilm under certain restrictive situations, microfilm is assumed to play a leading role since most original paper issues from the target time period have significant deterioration or are simply no longer available. The capture of images of a standardized target along with the digitized visual content is a best practice used by many digital library projects to further the goals of producing accurate materials that can be managed in the absence of the original item. Recognizing that no such test target existed at the time for the digitization of microfilm, NDNP worked with Image Science Associates 4 to develop the Preservation Microfilm Scanner Target (PMT), a standardized technical test target on microfilm (see Fig. 1) with associated analysis software, to assist in creating the high quality digital images that the program requires. Figure 1. Preservation Microfilm Scanner Target (PMT), image provided by Image Science Associates. The use of PMT by program participants serves two purposes: to create a benchmark, and to support ongoing quality control of the images. An initial set of scanned target images from a specific capture device can be thought of as a benchmarking tool for anticipated performance of that particular device. Analysis of those images can tell if the capture system, from the optics to the CCD chip to the software, is capable of creating images that meet NDNP specifications. If the initial analysis shows less than expected performance of the system, the results can lead the operator to adjust the equipment to create a better scan. Comparison of benchmark scans from different scanning equipment or different vendors can assist in making choices among them. During the mass production of digitized newspaper pages, each title, each reel and even each page will have different visual characteristics. To develop a quality control plan, the use of objective metrics derived from the PMT can be very helpful. For example, if a digitized page image looks blurry, it may be an artifact of the original printing process, the condition of the paper original at the time of filming, the microfilming process or the digitization process. Determining which variable among these and others is responsible for quality concerns can be challenging. Capture and 5

6 subsequent examination, both visual and through analytical software, of a standardized target image captured at the same time as the newspaper page can provide an objective non-page-specific measure of digitization quality factors at the time of the scan. If analysis of the target image(s) indicates the scanner was performing as anticipated, the visual inconsistencies are likely to be found in the microfilm, with the scan an accurate representation of it. The film target and associated analysis software developed for microfilm scanning contains a variety of elements to ensure consistency with current International Organization for Standardization (ISO) imaging specifications. Following is a description of image quality elements of the PMT and how they can be used to assess scan quality. Generally, image quality can be divided into the categories of tonal reproduction, sharpness, noise and color reproduction. Since microfilm is designed to be monochromatic in its imagery, color reproduction is not a consideration here. In ISO 14524, the responsiveness of the capture device to tones is defined as its Opto- Electronic Conversion Function (OECF). The PMT contains a series of gray boxes, with graduated levels of darkness, which should be distinctly observable in a high quality scanned image. This creates both a visual clue that the target image captured the full range of tones available, and data points that may be analyzed by software to calculate the OECF for the system. Since newspapers contain a wide-range of fonts, type sizes, and visual elements of varying quality, clarity and contrast, the reproduction of the details of an image are essential to capturing the information contained in the original newspaper. It is important not only that enough pixels per inch are captured, but also that the optical system captures enough detail to justify those pixels. In ISO , sharpness is measured in the system s Spatial Frequency Response (SFR). The PMT contains a slant-edge border between white and black areas and eye-readable resolution charts consisting of narrowly spaced lines. This is the input for a software analysis of the system s SFR capability. The process of digitization often produces unintentional noise: artifacts that appear randomly or systematically in the digital image that were not in the original. The PMT measures noise through the use of a series of squares with very small vertical and horizontal lines and a long diagonal line across the entire target. If there is interference created by the distance between pixels on the sensor and these lines, the target image will make that visible by producing a pattern of widely spaced lines. The degree of this fluctuation is displayed by the analysis software, which also provides information on the likely range of acceptable fluctuation. For NDNP production workflows, the analysis of PMT target images may be done as often or as little as awardee project managers deem necessary. The target can be used to monitor scanning performance on a daily basis, provide an opportunity for quality sampling within a large batch of data, or it may be captured with analysis left to some future date, or on an as-needed basis. To date, analysis of scanned targets within NDNP have quantitatively revealed scanning performance in the areas of tonal reproduction, sharpness and noise, enhancing producers ability to monitor aspects of the digitization process. The use of the target has assisted in vendor selection. It 6

7 has shown when scanning performance was inadequate. Just as importantly: when poor newspaper images on the original microfilm caused the scanning process to be questioned, the use of the target has also demonstrated when the scanning performance was working correctly. In order to ensure conformance with other stated NDNP technical metadata specifications, a second tool was needed that could be used by NDNP staff, awardee institutions, and digitization vendors, for validation of technical aspects of NDNP data and quality assurance processing. To support efficiency and scalability, it was clear that the required tool would support a mixture of both automated analysis of technical characteristics where feasible and suitable features for additional human-mediated evaluation. An automated approach could effectively measure technical conformance (e.g. whether a field is populated with the appropriate data type), while a data object viewer would support more subjective human inspection (e.g., whether the field data is correct). To support automated technical conformance, much work had already been done at Harvard University by the creation of the open source JSTOR/Harvard Object Validation Environment (JHOVE) software 5. This software was able to measure and characterize many aspects of the file types (JPEG 2000, PDF, TIFF) that are used in NDNP. Further programming was done at the LC to extend the capabilities of the software, and to validate the XML metadata necessary for the NDNP system 6. This analytical code was wrapped in a graphical user interface that became known as the Digital Viewer and Validator (DVV). The DVV then allowed automated analysis of the objective criteria of the data objects ensuring the right data types were used, and a visual quality review to ensure the metadata was being employed correctly (e.g. the right title was used, the date in the metadata matches the date on the page image). During validation, the DVV verifies approximately 100 characteristics of the data package. If the files meet the specifications, it extracts header data from the various self-documenting file types for transformation into Preservation Metadata Implementation Strategies (PREMIS) and Metadata for Images in XML Schema (MIX) schemas within the associated METS object. In addition, the DVV adds a digital signature to the METS object for each associated file. This digital signature can be checked later, to determine if the file has changed in the interim, whether intentionally, by operator error or bit degradation. Thus the DVV allows the validity of NDNP data files to be monitored throughout their lifecycle. In addition to these tools in use by awardees and vendors, NDNP has also begun use of the BagIt specification and toolset 7, created by LC with partners in the National Digital Information Infrastructure Preservation Program (NDIIPP). Early in the program, NDNP determined that in order to maintain a sustainable cost-effective program, the data produced by NDNP must be managed in such a way as to ensure both its reliability and quality for use. One early lesson was identifying the need to minimize human-interaction (and therefore human error) in the data lifecycle. To that end, when the NDNP data is received at LC, valid and with digital signatures for each file intact, the data delivery is then bagged using the BagIt specification to automate the content's receipt, storage and retrieval. Once bagged, generalized transfer utilities will provide management services for the bag that will enhance 7

8 reliability, findability, and integration with other digital collection material. The tools described above will help ensure the initial NDNP investment in data specifications and quality monitoring will result in a sustainable resource worth the cost of management and maintenance over time. Access System Model: After a period of exploration, LC developed requirements for an access system through extensive use cases and scenario planning. Basic access functionality for the program was defined as the ability of a general user to search and/or alphabetically browse newspaper title directory records, browse through various digitized titles by issue date and logical page order, and to support simple keyword search at the newspaper page level. Automatically recognized machine readable text (OCR) with associated word coordinate data in the ALTO schema provided the basic page structure and keyword searchability needed as well as vocational information that used in a visual interface to highlight search results. Additional structural or descriptive metadata identifying parts of the page was not included in the specification, in order to maximize the available resources and meet the needs of providing basic access to the content. Formal usability testing was conducted on an early prototype incorporating this functionality to confirm general usage assumptions and needs. In the initial site development, the access website itself (the Browser Application) was imagined as a component of the Library's repository, tightly tied to the preservation and management of the NDNP data. At that time, the preserved digital asset lifecycle was achieved through employing discrete system level tools and technical staff s system skills for the processes of ingesting, indexing and dissemination through repository services to the Browser Application. However, as the program and technical infrastructure have grown, LC technical management and access strategies evolved. The data management architecture was re-engineered in order to exploit both maturing website publishing tools and generalizable digital library management principles more effectively, resulting in coupling the website more loosely to the internal inventory, workflow, storage, and transport systems. To that end, in May 2009 NDNP released a revised system architecture and access application, transparent to most individual users, to support a variety of new functionalities and resource needs. The primary goals of the revision were: Improve access performance in order to allow web crawlers and search engines access to Chronicling America content. Providing this access will greatly enhance use of the newspaper collection by putting it in front of millions from whom it was previously hidden. Add standard programmatic interfaces (Application Programming Interfaces, or APIs) with the goal of improving the reach of the site, though this time to mash-ups and hobbyists more than to search engines and crawlers. For example, utilization of the OpenSearch API, which allows users to search the newspaper pages and titles directly from their Web browser, and to subscribe to the results as a feed. In addition, API access also allows enthusiasts to 8

9 use and remix the content itself while taking advantage of the modeling and indexing work already done at the Library. Reduce code complexity resulting from the tight relationship with preservation and administrative services and with an eye toward future renovations of the look and feel of the access interface. The result is a site that has almost twenty times fewer lines of code than the original implementation. Loosen the relationships between access and preservation components of the data management environment in order to increase the services supplied by both, specifically enhancing NDNP data processing capabilities to make the process of acquiring, managing, and providing access to the data consistent, repeatable, auditable, verifiable, and automated. Most importantly, this revision was successfully completed, enhancing access to the site and use of the site by multiple factors, without any changes to the NDNP digital object technical specifications or structures. Open source software projects relied on for the revised access application include Apache Web Server, the Django web publishing framework 8 (originating from the newspaper publishing sector and able to fulfill many of our digital resource use cases with very little additional effort), the JQuery JavaScript Library, MySQL database, Solr search server, and the RDFLib Python library. These open source tools are rapidly gaining adoption in the digital library community for their flexibility, reliability and ease of use, both externally and internally at LC. The current version of the user interface to NDNP data, known as Chronicling America, ( is freely accessible to the public and available from the Library of Congress--Digital Collections Web site. The site currently includes over 1 million newspapers pages from more than 100 titles, published between 1880 and 1922 from 11 states and the District of Columbia. Also included is a directory of newspapers published in the US from 1690 to the present and information on libraries that hold them in both physical and digital form. Since March 2007, the site has provided content to approximately 400,000 visitors with over 6.8 million page views. In addition to basic keyword search functionality, the Web site provides access to citation information for each newspaper page, visual calendars indicating available issues for a given year, files for download and re-use, special print features for printing detailed images of a page, persistent bookmarkable links for all site views, newspaper histories for each digitized title, a weekly RSS feed of newspaper content highlights and program developments, and more. The recent site performance improvements described above promote use of the content in addition to the Chronicling American Web site interface using new research techniques such as data mining, visualization or machine-user methods of discovery. Likely areas for future enhancement to access are incorporation of multilingual page content and associated searchability, additional automated analysis and manipulation of OCR to enhance search specificity and the overall user experience and additional shareability of page data. 9

10 Sustaining the Content: An important component in the fulfillment of LC s role in this program is the development of a system environment that ensures the digital assets acquired from many different sources over a long period of time will be sustainable. The environment must guarantee that when people, process, and technologies change, the digital asset can be (transparently and automatically if possible) migrated from old generations to new. Appropriate repository architecture is an essential component in determining if a digital preservation environment is successful. Development of technical infrastructure and services at the Library to sustain, provide access, and enhance long-term management of NDNP data is an ongoing pursuit. The two major architecture layers in the repository are the preservation or archive layer (managed bit storage) and the data management layer. The key difference between the two layers is the focus on the performance. The preservation layer emphasizes durability or longevity of the preserved digital asset and the data management layer emphasizes input/output (I/O) speed, richness of functionality, and flexibility of data management. Over the short history of the program, actual preservation threats have delineated known and, heretofore, unknown areas of risk to the reliable acquisition, management and preservation of this data 9. Evaluation of the lessons learned, as well as input from other LC projects, has broadly informed the development of additional generalized tools and services that will improve reliability for managing the complex workflows associated with data acquisition, management, access and archiving. One recent improvement to the NDNP toolkit is the NDNP Transfer application, a collection of three tools: a workflow interface, automated bit transport mechanisms, and an automated inventory database. As the data moves through multiple lifecycle stages in the digital library, it requires tracking, auditing, retrieval, and safe storage, as well as correlated knowledge of those processes and more. Initial implementation allowed only technical staff to find, retrieve, and manage archived assets using system level tools, disconnecting curatorial and program management staff from direct access to the digital library collection. To mitigate risks associated with this disconnect, for NDNP and other projects, LC has developed transfer tools that will enable more robust and cost-effective administration of the digital information, as well as scalability 10. Used together, these tools form the "back end" to the Chronicling America website in a way that is decoupled and flexible, while meeting overall productivity and performance goals for the program and its anticipated growth in both content and use. In future, NDNP will continue to enhance these tools beyond basic workflow to include a curation manager that can provide user-friendly functions/features to create, read, update, delete, navigate, monitor, and report on the permanently preserved digital newspaper content. Again, these functional tools will likely be generalizable to other digital collection management activities and increasingly useful over the long-term. 10

11 Supporting Infrastructure for Sustainability: The NEH and LC have made a long-term commitment to the development of this program and its digital assets, including a formal agreement regarding goals of the program, cost-sharing for development and management of the program products and cooperatively guiding the program s development. In order to fulfill its role in providing permanent access to this high-value historic content, LC initiated the development of a supporting infrastructure both programmatic and technical to enable the long-term sustainability of the collection. The infrastructure established at LC also included an internal program management team, made up of stakeholders representing collections interests, digital production (conversion and acquisition), and digital preservation. These stakeholders had hands-on experience in a broad range of LC programs, including newspaper collection development, the American Memory digital historic collections, Ameritechfunded partnerships, information technology and the National Digital Information Infrastructure Preservation Program (NDIIPP). Together, these committee members represented various management groups in the Library and successfully scoped the LC roles and deliverables that fulfill program requirements administering a successful distributed production model, provide a Web interface to acquired data, and develop a system environment to maintain and sustain the digital content. To accomplish any of these goals it was essential that LC also establish a dedicated technical development team, representing various specialties - including preservation architecture and repository development, data modeling, software development, search analysis and UI development - and who were willing to experiment and contribute to the advancement of best practices in digital preservation. This team shared expertise (and in some cases, staff) with other LC repository efforts, including electronic journals, Web archiving, and other digital collection projects, using and generalizing the lessons learned in initial NDNP development to extend the repository efforts to other content types. The technical development group supporting NDNP is involved in not only the creation of a system environment that meets NDNP goals, but also the establishment of a repository development center (hardware, software, and systems) within LC for on-going research into the challenges of preserving all types of digital information. In conclusion, the NDNP provides participants with an opportunity for multi-partner relationships that result in: the creation of select digitized newspaper content, that meets strict technical specifications for aggregation, that meet basic user access needs beyond what is possible with analog versions, and where the data is stored in a system environment that has a high-degree of sustainability. The immediate upfront decisions on the best practices and strategies that would lead to a successful program have been validated. As the program continues to develop and expand, LC will adapt and evolve the tools and systems available for this program. Facing the challenges of building a national digital newspaper collection will inform universal understanding of needs and capabilities for the preservation of all digital information. 11

12 1 Sustainability of Digital Formats Planning for Library of Congress Collections. accessed 19 May Metadata Encoding and Transmission Standard (METS), accessed 19 May Analyzed Layout and Text Object Schema (ALTO), accessed 19 May See Image Science Associates, for more information, accessed 19 May JSTOR/Harvard Object Validation Environment (JHOVE), accessed 19 May For more explanation of the digital object validation strategies implemented for NDNP, see Littman, Justin, "A Technical Approach and Distributed Model for Validation of Digital Objects", D-Lib Magazine, 12:5 (May 2006). accessed 19 May BagIt specification, accessed 19 May Django Web Framework, accessed 19 May See Littman, Justin, "Actualized Preservation Threats: Practical Lessons from Chronicling America, D-Lib Magazine, 13:7/8 (July/August 2007). accessed 20 May See Littman, Justin, A Set of Transfer-Related Services, D-Lib Magazine, 15:1/2 (January/February 2009). accessed 20 May Thanks to David Brunton, Ray Murray, and Deborah Thomas at the Library of Congress for contributing to this article. 12