General Model of E-ARK Services

Size: px

Start display at page:

Download "General Model of E-ARK Services"

Baldwin Lane
6 years ago
Views:

1 General Model of E-ARK Services DLM Forum Members Meeting June 2014, Athens Istvan Alföldi National Archives of Hungary

2 Agenda E-ARK General Model Conceptual framework Used methodology Results (not final)

3 General Model of E-ARK Services The scope of the E-ARK service is to provide a reference implementation, which integrates these currently noninteroperable tools into a replicable and scalable, common seamless workflow, allowing data owners and repositories to flexibly select and use the components most relevant for their specific situations. To achieve this, a set of common interfaces and information package formats will be defined by the E-ARK project and implemented using these tools. (E-ARK DoW Part B Finalised version B1.3)

4 OAIS Reference Model P r o d u c e r Pre-Ingest Preservation Planning Data and Content Management e-document Ingest Archive SIP AIP AIP Archival Storage Access queries result sets orders DIP C o n s u m e r Administration

5 Conceptual Framework Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access

Standards and Format Definitions OAIS, METS, ISAD(G), Moreq2010, PREMIS,

6 Conceptual Framework Standards and Format Definitions Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access Standards and Format Definitions OAIS, METS, ISAD(G), Moreq2010, PREMIS, SIARD, CMIS, Bagit, InterPARES 2, EAD, EAD-CPF, etc. E-ARK SIP SIP E-ARK AIP AIP E-ARK DIP DIP

7 Conceptual Framework Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access

Conceptual Framework Tools and Interfaces Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access Tools and Interfaces (already established) DMS RMS DB ET,

8 Conceptual Framework Tools and Interfaces Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access Tools and Interfaces (already established) DMS RMS DB ET, DBExport, UAM, RODA-in E-ARK SIP E-ARK AIP Archival Repository ESSArch Preservation Platform (EPP), Safety Deposit Box (SDB), RODA, The National Danish Bit Repository (NDBR) CMIS interface Scalable Computation Staging Area Lily E-ARK DIP Content and Records Management System Alfresco SOFIA Apache Hadoop

9 Conceptual Framework Tools and Interfaces Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access Tools and Interfaces (to be developed) SIP Creation Tools DMS RMS DB ET, DBExport, UAM, RODA-in E-ARK SIP SIP-AIP Conversion Component E-ARK AIP Archival Repository ESSArch Preservation Platform (EPP), Safety Deposit Box (SDB), RODA, The National Danish Bit Repository (NDBR) CMIS interface Scalable Computation Staging Area Lily Apache Hadoop AIP-DIP Conversion Comp. E-ARK DIP Search, Access and Display Interface Content and Records Management System Alfresco SOFIA Data Mining Showcase

10 Conceptual Framework Tools and Interfaces Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access Tools and Interfaces (to be developed) SIP Creation Tools DMS RMS DB ET, DBExport, UAM, RODA-in E-ARK SIP SIP-AIP Conversion Component E-ARK AIP Archival Repository ESSArch Preservation Platform (EPP), Safety Deposit Box (SDB), RODA, The National Danish Bit Repository (NDBR) CMIS interface Scalable Computation Staging Area Lily Apache Hadoop AIP-DIP Conversion Comp. E-ARK DIP Search, Access and Display Interface Content and Records Management System Alfresco SOFIA Data Mining Showcase

11 Conceptual Framework Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access

12 Conceptual Framework Processes and Use Cases Process and Use Case Diagrams Create SIP GM-PI-6 GM-PI-7 SIP Creator (WP3) ESSArch RODA UAM SIP Producer Start transfer to archive

13 Conceptual Framework Processes and Use Cases Process Diagrams

14 Conceptual Framework Processes and Use Cases Business Process Modeling Notation (BPMN) Process Flow Participant role Activity

15 Conceptual Framework Processes and Use Cases Business Process Modeling Notation (BPMN) Collapsed view

16 Conceptual Framework Processes and Use Cases Standard Process Diagrams (BPMN) Detail view

17 Conceptual Framework Processes and Use Cases Use Case Diagrams Create SIP GM-PI-6 SIP Creator (WP3) ESSArch RODA UAM SIP Producer Start transfer to archive GM-PI-7 Participant role Activity Used tool Input / Output

18 Conceptual Framework Processes and Use Cases Create SIP GM-PI-6 GM-PI-7 SIP Creator (WP3) ESSArch RODA UAM SIP Producer Start transfer to archive Who does what? In what order? Using what kind of tool? Required input? Produced output?

19 Conceptual Framework Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access

20 Conceptual Framework Pre-Ingest Ingest Archival Storage Preservation Planning Administration Data Management Access Recomended Practices To be added according to the experiences of the Pilots

21 Used Methodology and Schedule Surveys Questionnaire 1 (11 February, 2014) Infrastructure overview Process overview Data overview Questionnaire 2 (3 March, 2014) Detailed process information Detailed tool description General high-level requirements Input data profile survey (30 May, 2014) Draft model and Discussion Draft model sent out (26 May, 2014) Comments in E-ARK technical meeting (12-13 June 2014, Athens) Final model and approval (July 2014)

22 Pre-Ingest Results Processes and Use cases

Results Processes and Use cases Pre-Ingest Define SIP content GM-PI-1 GM-PI-2 Select data (with rules) GM-PI-3 Select data (manual) GM-PI-4 Extract data from DB

23 Results Processes and Use cases Pre-Ingest Define SIP content GM-PI-1 GM-PI-2 Select data (with rules) GM-PI-3 Select data (manual) GM-PI-4 Extract data from DB DBExport ESSArch Alfresco RODA DB DMS /RMS Producer Extract data from DMS/RMS GM-PI-5 Create SIP GM-PI-6 SIP Creator (WP3) ESSArch RODA UAM SIP GM-PI-7 Start transfer to archive

24 Results Processes and Use cases Ingest

25 Results Processes and Use cases Ingest GM-I-1 SIP Upload SIP E-ARK SIP GM-I-2 E-ARK SIP AIP tool AIP Technical stuff Start AIP generation workflow RODA ESSArch PP SDB E-ARK AIP

26 Results Processes and Use cases Data Management

27 Results Processes and Use cases Data Management Select records to export GM-DM-1 Archivee GM-DM-2 Manipulate records Archivist Start export workflow GM-DM-3 CMIS Lily

28 Results Processes and Use cases Access

29 Results Processes and Use cases Access Search and select data GM-A-1 AIP Archivee GM-A-2 E-ARK AIP Request data End user (Consumer) Request access from presentation platform Download DIP GM-A-3 GM-A-8 Presentation / Data Mining Platform

30 General Model Work package overview Work Package - OAIS Process cross reference table E-ARK General Model Work Package WP 1 Project Co-Ordination WP 2 Use Cases and Pilots WP 3 Transfer of Records to Archives WP 4 Archival Records Preservation WP 5 Archival Records Access Services WP 6 Archival Storage, Services, and Integration WP 7 Evaluation & Assessment WP 8 Dissemination and Exploitation Pre-Ingest Ingest Archival Storage Preservation Data Management Access

31 General Model Pilot overview Pilot - OAIS Process cross reference table E-ARK General Model Full-scale Pilot SIP creation of relational databases Pilot 1 (Danish National Archives) SIP creation and ingest of records Pilot 2 (National Archives of Norway) Ingest from government agencies Pilot 3 (National Archives of Estonia) Business archives Pilot 4 (National Archives of Estonia, Estonian Business Archives) Pilot 5 Pilot 6 Pilot 7 Preservation and access to records with geodata (National Archives of Slovenia) Seamless integration between a live document management system and a long-term digital archiving and preservation service (KEEP SOLUTIONS) Access to databases (National Archives of Hungary) Pre-Ingest Ingest Archival Storage Preservation Data Management Access Focus of the pilot Elements also used/tried within the pilot

32 Cross Reference Tables Use case view Use Case View E-ARK General Model Use Case Pilots Work Package Tools Pilot 1 (DNA) Pilot 2 (NAN) Pilot 3 (NAE) Pilot 4 (EBA) Pre-Ingest GM-PI-1 Define SIP content x x x x x x x GM-PI-2 Select data (with rules) x x x x x x DBExport tool GM-PI-3 Select data (manual) x x x x x DBExport tool GM-PI-4 Extract data from DB x x x x ESSArch Tools GM-PI-5 Extract data from DMS/RMS x x x x x ESSArch Tools, Noark, Alfresco, RODA GM-PI-6 Create SIP x x x x x x x x DBExport tool, ESSArch Tools, SIP creation tools, RODA-in, UAM GM-PI-7 Start transfer to archive x x x x x x x GM-PI-8 SIP reception x x x x x x x GM-PI-9 Validate SIP x x x x x x x x SIP to AIP conversion tools GM-PI-10 Enhance SIP??????? x GM-PI-11 Create fond(s) x x x x x x x SDB, EPP, RODA, AIS Pilot 5 (NAS) Pilot 6 (KEEP) Pilot 7 (NAH) WP3 WP4 WP5 WP6

33 Cross Reference Tables Tools view Tools View E-ARK Tool Alfresco ESSArch Tools (ET) DBExport Tool E-ARK SIP creation tools Universal Archiving Module (UAM) General Model Description Alfresco is the most used open source content and records management platform worldwide. While its main use is in content creation, management and access the platform will also be used to pilot the E-ARK records export recommendations and providing crucial source data for the E-ARK service. ET is a stand-alone application that allows producers and preservation organizations to prepare, create, deliver and receive SIPs. DBExport is used to create SIP packages in SIARD format based on the content in relational database systems and provide a first level of integrity and technical checks. SIP creation tools (to be developed in WP3) implement the final Pan- European SIP format based on the Alfrexco platform, ESSArch Tools (ET) suite and the DBExport tool. UAM allows data owners to prepare the transfer themselves import data from records and content management systems; rearrange, classify and further describe the contents of the ransfer; validate the transfer according to the rules established by the National Archives of Estonia and finally create SIP packages to be transferred to the digital repository. Pilot 1 (DNA) Pilot 2 (NAN) Pilot 3 (NAE) Pilots Pilot 4 (EBA) Pilot 5 (NAS) Pilot 6 (KEEP) Pilot 7 (NAH) WP3 Work Package WP4 WP5 WP6

34 Thank you!

E-ARK: Harmonising pan-european archival processes to ensure continuous access to e-government records and information

E-ARK: Harmonising pan-european archival processes to ensure continuous access to e-government records and information Andrew WILSON a, Miguel FERREIRA b, Kuldar AAS c, Anders Bo NIELSEN d, Phillip Mike