Oh My, How the Archive has Grown: Library of Congress Challenges and Strategies for Managing Selective Harvesting on a Domain Crawl Scale

Transcription

1 Oh My, How the Archive has Grown: Library of Congress Challenges and Strategies for Managing Selective Harvesting on a Domain Crawl Scale Abbie Grotke WAC, June 16, 2017

2 LC WEB ARCHIVING PROGRAM OVERVIEW

3 LC Web Archiving Started in 2000 Event and Selective crawling Outsourced crawling under contract (IA) Limited crawling in-house Permissions- and notice-based approach Now based on country of publication, and category Digiboard, developed in-house, manages nomination, permissions, seed list creation, quality assurance, collection management Public access at loc.gov/websites/collections

4 Current Staffing/Organization Web Archiving team (5 FTE technical and program management staff) in new Digital Collections Management and Services Division Recommending Officers in custodial divisions (subject experts) develop the collections and select the content No FTEs assigned, interested staff do this as a part of their other recommending duties Collections oversight/coordination by newly formed Web Archiving Collection Development Working Group Staff supporting infrastructure, access, tools development in CIO s office (limited) Cataloging assistance in Acquisitions and Bibliographic Access Division

5 LC collects Library of Congress and other U.S. Government Events and subjects of interest to special collections divisions, including U.S. and International Elections International Government and Organizations Full list:

6 Number of Web Archive Collections

7 Amount of data: PB (as of May 2017)

8 LC WEB ARCHIVING PROCESS

9 9

10 Digiboard (workflow) Tool Improvements: Better collection management

11 Digiboard Tool Improvements: Much better permissions management Curators select Country and Category when nominating and tool sets the appropriate permissions based on these settings Digiboard sends three attempts automatically

12 Digiboard Tool Improvements: Web Archiving Entities multiple seeds covered under same permission response, in multiple collections

13 Digiboard Tool Improvements: Web Archiving Entities allows for better management of seeds changes when collecting over a long period of time

14 Workflow Tool Improvements: Scoping tools better management of scopes (additional URLs crawled related to the seed)

15 We crawl in buckets (mostly) based on frequency: Election: U.S. Election crawls, weekly during Election years (~1200 seeds max) Monthly: includes scheduled monthly, once, twice-yearly, yearly seeds. Runs days (~ seeds) Weekly: crawls run a week (~350 seeds) RSS: 12 hr crawls twice-daily (~900 seeds) Quarterly/Extended: Runs for 4-6 weeks, sometimes longer depending on budget (until we hit our terabyte limit) (~1200 seeds)

16 Crawling Strategies for Better Capture RSS feed crawling to supplement collection of larger, rapidly changing websites (2x daily) Extended crawls for very large sites Crawling in-house if more custom approach needed for particular sites

17 Quality Assurance Process We pay by the terabyte, for compressed WARCs Our QA process is twopart, for each crawl: Inspection of required crawl reports/charts Visual inspection of random 5% of crawled seeds Ongoing visual inspection of sites in all collections happens as staff/resources permit

18 Managing and Storing LC Web Archives IA stores a copy for LC staff to access during the contract period (generally for one year) WARCs transferred over Internet2, using LC s Content Transfer Services tools Copied to long-term and to access storage Indexed for Wayback access (staff only) Indexed for public Wayback access after one-year embargo

19 Internal Content Management and Reporting Tools Web archives are just one digital content type among many being managed and preserved by the Library of Congress we are one of the largest though! Storing it is not a problem, but processing or moving the data is a challenge

20 Providing Researcher Access Some content embargoed Some content restricted to onsite only The rest is publicly available We release Wayback content every 6 months or so Onsite/office access controls based on permissions via OpenWayback and MODS record controls Collections/records released as we have resources to do so A few formal agreements with researchers has allowed sharing of WARCs

21 28 collections now available at loc.gov/websites item-level MODS records, searchable alongside other Library digital items collection-level records in ILS a mix of onsite/offsite (depending on permissions received) Wayback.loc.gov displays content (including uncataloged sites) up through June 1, 2016

22 We re trending on loc.gov!

23 MORE TO COME (CONTENT AND IMPROVEMENTS)

24 Plan for Digital Collecting Over the next five years, the Library intends to follow a strategic framework categorized into six objectives: Strategic Objective 1 Maximize receipt and addition to the Library s collections of selected digital content submitted for copyright purposes Strategic Objective 2 Expand digital collecting via routine modes of acquisitions (primarily purchase, exchange and gift) Strategic Objective 3 Focus on purchased and leased electronic resources Strategic Objective 4 Expand use of web archiving to acquire digital content Strategic Objective 5 Develop and implement an acquisitions program for openly available content Strategic Objective 6 Expand collecting of appropriate datasets and other large units of content Read more:

25 Plan for Digital Collecting Strategic Objective 4 Expand Use of Web Archiving to Acquire Digital Content Expand the use of web archiving as a standard acquisitions tool for digital content, such as openly available materials in various digital formats. Coordinate with NARA, GPO and other Federal agencies to ensure that the government s web sites are harvested, preserved and made available to researchers, with the Library taking on a special responsibility for harvesting Legislative Branch web sites.

26 Automating More Processes The Problem Access to collections backlog + limited cataloging resources Too much content to manually QA with the staff we have Reports about all this data are difficult to create and run Automation Goal Extracted data from Digiboard tool to create minimal record Experimenting with scripts to help identify replay issues Web archives being integrated into broader digital collection reporting tools at LC

27 Improving Researcher Access Digital Scholars Pilot Project Report: 16/DChudnov-MGallinger_LCLabReport.pdf labs.loc.gov launching later this year, which will serve as a sandbox: downloadable data sets (hopefully including web archive data), applications, reports, educational videos and projects from fellowships and more

28 @agrotke THANK YOU!