Openness, Growth, Evolution, and Closure in Archival Information Systems

Transcription

1 Grow Evolve Close Keys to the Digital Future Openness, Growth, Evolution, and Closure in Archival Information Systems Lessons from NARA s Experience September 2008 Kenneth Thibodeau, Director Electronic Records Archives Program National Archives and Records Administration IEEE Symposium on Mass Storage Systems & Technologies

2 Grow Evolve Close Archival Information System Conceptually: an archive, consisting of an organization of people and systems, that has accepted the responsibility to preserve information and make it available for a Designated Community. ISO Reference Model for an Open Archival Information System (OAIS). ISO 14721:2003 Empirically: the National Archives Open Archival Information System, the Electronic Records Archives

3 Grow Close What is the Electronic Evolve Records Archives (ERA)? ERA is the system the National Archives and Records Administration (NARA) is developing to Reengineer and automate the lifecycle management of all types of records of the U.S. Government Preserve and provide sustained access to electronic records of the U.S. Government

4 Grow Evolve Close ERA Development Timeline 9/05 9/06 9/07 9/08 9/09 9/10 9/11 ERA Base System ERA Search & Access System 6/08 Initial Operating Capability (IOC) Enhancement Enhancement Enhancement Enhancement Operation & Maintenance Full Operating Capability

5 Grow Close ERA Base System Evolve Development Focus: Federal Records National Archives IOC Functions (2008): Creation, review and approval of records schedules Requests to transfer records, transfer of physical and legal custody Transfer, inspection, and archival storage of electronic records

6 Grow Close Evolve Initial Users

7 Grow Close ERA Search and Access Evolve Initial Focus: System Development Electronic records of the Executive Office of the President, G W. Bush Presidential Libraries 100 TB Functions: Rapid ingest & indexing Transformation to more accessible form. Archival storage Full content search Basic case management for special requests

8 Grow Evolve Close Future Development Public Access to Any information about records Ordering of copies of records Electronic records stored in the system Long-term preservation of electronic records Ability to use a variety of techniques simultaneously and over time Review and redaction of sensitive content Support for Federal Records Centers Exponential growth in stored data

9 Grow Close Keys to the Digital Evolve Future Openness Growth Evolution Closure Lessons from the ERA experience

10 Grow Evolve Close Openness An Archival Information System needs to be open to New types of electronic records Rising and changing user expectations Creative approaches to meeting the challenges of electronic records and demanding users.

11 Grow Evolve Close Openness An Archival Information System needs to be open to New types of electronic records Rising and changing user expectations Creative approaches to meeting the challenges of electronic records and demanding users.

12 New Types of Records: Geographic Information Systems

13 New Types of Records: Product Data Computer Assisted Design Computer Assisted Engineering Product Analysis and Testing Computer Assisted Manufacture

14 New Types of Records: Critical Infrastructure Data Source: CLindberg

15 New Types of Records: Virtual Reality: Crime Scene Investigation

16 New Types of Records: Medical Tests and Observations

17 Grow Evolve Close Openness An Archival Information System needs to be open to New types of electronic records Rising and changing user expectations Creative approaches to meeting the challenges of electronic records and demanding users.

18 Grow Evolve Close Openness An Archival Information System needs to be open to New types of electronic records Rising and changing user expectations Creative approaches to meeting the challenges of electronic records and demanding users.

19 Rising and Changing User Expectations

20 Grow Evolve Close Openness An Archival Information System needs to be open to New types of electronic records Rising and changing user expectations Creative approaches to meeting the challenges of electronic records and demanding users.

21 Grow Evolve Close Openness An Archival Information System needs to be open to New types of electronic records Rising and changing user expectations Creative approaches to meeting the challenges of electronic records and demanding users.

22 Creative Approaches The conceptual apparatus we bring to bear on The nature of records Requirements for preserving records Requirements for serving users

23 Creative approaches: Partnerships National Science Foundation NIST Army Research Laboratory National Computational Science Alliance Global Grid Forum San Diego Supercomputer Center

24 Grow Close Keys to the Digital Evolve Future Openness Growth Evolution Closure

25 Grow Evolve Close Growth An Archival Information System needs to be able to grow to Process, store and provide access to increasing volumes of electronic records Accommodate increasing numbers of users and frequency of use

26 Grow Increasing Volumes of Digital Information In 2006, the amount of digital information created, captured, and replicated was 281 exabytes or 281 billion gigabytes. This is about 3 million times the information in all the books ever written. By 2011, the digital universe will be 10 times the size it was in Not all information created and transmitted gets stored, but by 2011, almost half of the digital universe will not have a permanent home. The number of electronic information containers files, images, packets, tag contents is growing 50% faster than the number of gigabytes. The information created in 2011 will be contained in more than 20 quadrillion 20 million billion of such containers IDC. The Diverse and Exploding Digital Universe. An Updated Forecast of Worldwide Information Growth Through March 2008

27 Grow Transfers of Digital Files to NARA 16,000 14,000 12,000 10,000 8,000 6,000 4,000 2,

28 Grow Transfers of Digital Files to NARA 250, , , ,000 50, Reagan/ Bush

29 Grow Transfers of Digital Files to NARA 35,000,000 30,000,000 25,000,000 20,000,000 15,000,000 10,000,000 5,000, Reagan/ Bush Clinton

30 Grow Planning for Open-ended Growth Federation System Management Base ERA Instance SAS ERA Instance Future ERA Instance DREN NARANET Internet Public Other Agencies

31 Grow Close Keys to the Digital Evolve Future Openness Growth Evolution Closure

32 Grow Evolve Close Evolution An Archival Information System needs to be able to evolve in response to Changing Information Technology Obsolescence Opportunities Changing business requirements

33 Grow Evolve Close Evolution An Archival Information System needs to be able to evolve in response to Changing Information Technology Obsolescence Opportunities Changing business requirements

34 Evolve Obsolescence of Formats of Electronic Records Strategy: Preservation and Access Levels Common: Retain records in original formats Basic Level: Use original or contemporary software for access Enhanced Level Create new version in current format, or Use new software for access to original format Ideal Level Create version in persistent format, or Create persistent software for management and access

35 Evolve Obsolescence of Formats of Electronic Records ERA System Architecture: Does not prescribe specific preservation solutions Allows a variety of different software tools to be introduced and used for different formats. Enforces archival requirements Preservation Framework Record: Correspondence Attached Photograph Example to structured data converter Proprietary to standard image format converter Header Message Standard form Photograph

36 Grow Evolve Close Evolution An Archival Information System needs to be able to evolve in response to Changing Information Technology Obsolescence Opportunities Changing business requirements

37 Grow Evolve Close Evolution An Archival Information System needs to be able to evolve in response to Changing Information Technology Obsolescence Opportunities Changing business requirements

38 Evolve Preservation Options Specific General Maintain original technology Emulation Dynamic Translator Static Translator Programmable Chips Universal Virtual Computer Virtual Machine Re-engineer Software Viewer Rosetta Stone Translation Typed Object Conversion Format Standardization Version Migration Persistent Object Object Interchange Format Multi- Valent Document Preserve Method Preserve Technology Objects

39 Evolve Changing Information Technology: Service Oriented Architecture ERA Mgt To Greenbelt External Access Gateway (DMZ Access) WebServer VLAN System/Business Application VLAN Ethernet Switch SOC SOC WS Web Server Documentum Server App Server (SBA) Database Server (SBA) Asset Mgnt WS DNS/ Directory Server Strong Auth Server Firewall + IDS Ethernet Switch Ops LAN, Management LAN HW Load Balancer Instance Datastore Ops LAN, Management LAN, Storage Network Mgt Router (w/ipsec) Firewall + IDS ERA Management VLAN Accountability Server Accountability WS ERA Extranet Firewall + IDS Firewall + IDS Extranet Router (w/fw + IDS + IPSec) Firewall + IDS Firewall + IDS Firewall + IDS Firewall + IDS Core Switch (w/fw + IDS + LB) Firewall + IDS Firewall + IDS Mgmt LAN, Storage Network Ethernet Switch Backup Restore Server NARA GOVT Agencies HW Load Balancer Ethernet Switch Ops LAN, Management LAN, Storage Network Firewall + IDS Database Server (T&I) Ingest Server (App Server) Ingest Working Storage Central Data Manager Server FC Switch Ops LAN, Management LAN, Storage Network Archive Disk Storage (Managed Storage) Archive Label Printing WS Printer Enterprise System Mgt Server Help Desk Server Network Mgt Server Security Server (Code Modification) Printer CD ROM DVD Tape Drive Windows Input Workstation Physical Ingest Ethernet Switch Unix Input Workstation Ingest VLAN Printer CD ROM DVD Tape Drive Archive Tape Storage Backup Network Archive Storage VLAN Data Services Server (Mgmt DB Server) Security Server (Vulnerability Scan) Software Deployment Server Derived from: ERA Hardware Block Diagram (Tab: I1R2 U/USBU Detailed Block) Updated 24 Aug Gb Ethernet 2/4Gb Fibre Channel

40 Evolve Service Oriented Architecture As Built WAN To Greenbelt ERA Extranet NARA GOVT Agencies Base ERA Ingest DB Server Base ERA Ingest Server Firewall + IDS Firewall + IDS Physical Ingest Printers Ethernet Switch 1 HW Load Balancer Ethernet Switch 1 DMZ Enclave Ingest Working Storage Extranet Router (w/fw + IDS + IPSec) EOP Ingest Workstations EOP Ingest Servers Ethernet Switch 1 Ingest Security Review WS Firewall + IDS Ethernet Switch Firewall + IDS Firewall + IDS Ethernet Switch 2 HW Load Balancer Ethernet Switch 2 Base ERA Web Server CD ROM DVD Tape Library Separate Dedicated Physical Disks for Base ERA & ERA EOP Physical Ingest Workstations Tape Subsystem Management LAN Ingest Workstation Enclave (Base ERA) Firewall + IDS System/Business Application Enclave Asset Mgt Workstation ECM Server Label Printer Application Servers Database Servers Separate Dedicated Physical Disks for Base ERA & ERA EOP Archive Label Printing Workstation Central Data Management Servers Firewall + IDS Firewall + IDS Managed Storage Base ERA Web/ Application Server Instance Data store Database Server Core Switch Backbone (w/fw + IDS + LB) 1 st Floor EOP, 2 nd Floor Base Fiber Channel Switches Archival Storage Enclave HCAP Storage Nodes FC Switches USP/V EOP ERA Operational Services Enclave Firewall + IDS Firewall + IDS HCAP Search Nodes Ethernet Switches Infrastructure Appl Server Primary Archive Storage LDAP and Radius Server Active Directory Server DNS Server Strong Authentication Server Replica Archive Storage Tape Library Management LAN Management Enclave Mgt Router (w/ipsec) Firewall + IDS Ethernet Switch Management LAN Firewall + IDS SOC Workstation SOC Enclave Log Logic Applicance Storage Mgt Server Mgmt DB Server Enterprise System Mgt Server Help Desk Server Security Server (Code Modification) Security Server (Vulnerability Scan) Software Deployment Server Network Mgt Server Network Compliance Mgt Server Backup Restore Master Server Backup/ Restore Media Server Backup/Restore Media Servers may be separate & dedicated between (Base ERA & EOP ERA) Color = Unchanged from Base Color = Changed from Base Color = New for EOP

41 Evolution An Archival Information System needs to be able to evolve in response to Changing Information Technology Obsolescence Opportunities Changing business requirements

42 Evolution An Archival Information System needs to be able to evolve in response to Changing Information Technology Obsolescence Opportunities Changing business requirements

43 Evolve Evolution of Business Requirements Requirements Technical Solutions

44 Evolve Records Schedule: Current

45 Evolve Create Schedule Item Temporary Records Permanent Records

46 Grow Close Keys to the Digital Evolve Future Openness Growth Evolution Closure

47 Grow Evolve Close Closure An Archival Information System needs to be able to provide closure to ensure Preservation and presentation of authentic records Comprehensive lifecycle management of electronic records Consistency with well-established archival science

48 Close ERA: a Set of Nested Systems Outer system lifecycle management of records of all types Inner Electronic Records System Ingest, preservation, disposition, and access to electronic records Search & Preservation Frameworks Support a variety of different approaches to different needs Archival mini-systems Specific, systematic management for each series or aggregate of electronic records

49 Close Document v. Record A document is a bounded physical representation of a body of information designed with the capacity (and usually intent) to communicate. A document may manifest symbolic, diagrammatic or sensory-representational information.... en.wikipedia.org/wiki/docume nt The information communicated by a document depends on its content and structure. A record is a document made or received in the course of a practical activity as an instrument or a by-product of such activity, and set aside for action or reference. p2_terminology_db.cfm The information communicated by a record depends on its content, structure, and context.

50 Close Document What does this document tell us?

51 Close Record What does this document tell us about the U.S. Government?

52 Close Archival Aggregate as Directed Graph Every record has an archival bond, the set of relationships established by an actor between that record and other records of the actor s activity.

53 Close Preservation Documents can be preserved as individual objects Records can only be preserved as ordered sets. An Archival Information System for records must ensure that Submission Information Packages, Archival Information Packages and Dissemination Information Packages are managed to respect the original order of records.

54 Close Electronic Records Open May be instantiated as subsets of complex ordered sets

55 Close ERA as a Set of Mini-Systems A Lifecycle Management Plan for a Records Aggregate, such as a series, defines a Mini-system; i.e., systematic controls for that aggregate stretching from ingest to dissemination.

56 Grow Evolve Close SUMMARY Openness New types of electronic records Rising and changing user expectations Creative approaches Growth Exponential increase in volumes of data Increasing users and usage Evolution Changing Information Technology Changing business requirements Closure Preservation of electronic records as members of ordered sets.

57 Grow Close Evolve Thank you. For more information: