IRODS USER GROUP 2014 CAMBRIDGE,MA John Burns. 6/25/14 Archive Analy3cs Solu3ons 1

Transcription

1 IRODS USER GROUP 2014 CAMBRIDGE,MA John Burns 6/25/14 Archive Analy3cs Solu3ons 1

2 Credits Archive Analy3cs Solu3ons is presen3ng an archive system that embodies best prac3ce for long- term, high integrity bit preserva3on. But firstly many thanks for invaluable help and advice from Chris Burnet et al. from netapp. James Coomer from DDN Paul Watry et al. at the University of Liverpool, UK

3 Digital Archiving the two faces thereof Bit Preserva+on Immediate and unrelen3ng axen3on to opera3ons Hardware / IT oriented. Unambiguous do we s3ll have the bits? Func+on Preserva+on Deferrable ( up to a point ) So^ware oriented. Rules, workflows &c Ambiguous what are we preserving, for whom? Different skills, different technologies, different 3me- scales, different impacts, different players in fact just different

4 Alloy is about bit- preserva3on

5 Some fundamentals of bit- preserva3on 1) Archives belong to the organiza+on Contain finished objects Access via Roles No personal users, 2) Integrity. Replica3on, Checksums, signing &c 3) Traceable / tamper resistant Audit logs on storage, signed &c. 4) Proper+es ( aka user- metadata) Finding via metadata ( c.f. search ). 5) Reconstruct just from storage No dependence on so^ware No encryp3on or obfusca3on Self- evident organiza3on on storage 6) Disaster resistant Geographical Dispersion, placement constraints, + Documented, intui3ve structures & naming 7) Vault mentality Limited access, highly controlled, cer3fiable No foreign objects inside the shell. 8) Support for func+onal preserva+on Versions, alternates, dependencies,

6 Alloy An Open Source based solu3on for intelligent storing and manipula3on/management of data for long periods of 3me Alloy is: An archival solu3on that simplifies the crea3on and management of a TRAC* compliant archive by ensuring that good archival prac3ce is readily prac3ced. Alloy integrates with exis3ng IT infrastructures where possible, and enables archive applica3ons via simple standards- based API(s).

7 Architecture LDAP Server Web interface Archival Applica3ons (including ac3ons ) H t t p s H t t p s Alloy API (CDMI) Triggers & no3fica3ons RFC 5424 Server (syslog) Auth n & Auth zn Repl n & Fixity metadata Logs & Audit Collec3ons Objects Manifests Services (Storage & Catalog) Isola3on Layer irods

8 Using Alloy Single Front door RESTful (CDMI) base API for data access and admin Authen3ca3on via LDAP out of the box Browser interface for humans Intui3ve Replica3on & Tiering Policy specifica3on User defined proper3es (metadata) and search on both names and values. IETF RFC (syslog) external trigger/rules system Authoriza3on via roles and ACLs (authen3ca3on is LDAP) Both hierarchical ( c.f. Posix / irods &c ) & OID access.

9 Alloy also provides Policy driven replica3on, with healing/rebalancing. Simple to scale out (appliance). Config- file driven deployment. Slaves manage storage and user access scalable performance. Archive can be reconstructed just from storage state is not stored only in Alloy. Rules can be axached to the IETF RFC 5424 listener e.g. logstash, syslog- NG, rsyslog (or custom). All provenance relevant logs are stored to archive.

10

11

12 Using Alloy scrip3ng and automa3on All interac3on is via hxp(s) / CDMI Command line tool to perform add, retrieve, delete, list, of collec3ons, objects and metadata. e.g. cd, mkdir, ls, get, put, rm, pwd Can also use `curl, or scrip3ng- languages via hxp.

13 Roadmap for the future Func3onality applica3ons on Alloy TRAC wizard facilitate an ISO audit Escrow, reten3on and disposi3on manager Alternate API support ( S3, SWIFT, CIFS(?) etc) Anything else that is widely needed. Under the covers Addi3onal storage layer support ( Ceph, Propr y Object stores as requested) Master- less cluster Improved inter- operability support Alternate authen3ca3on (based on demand )

14 Availability Available under Apache 2.0 license. A turnkey installa3on is available from NetApp

15 Archive Analy3cs Archive Analy3cs (AA), a UK company, creates applica3on- specific irods based applica3ons and providers consultancy, opera3ons and support for the same. AA s products deliver robust archival, content management and analy3c capabili3es with explicit, transparent and audit- able guarantees of compliance to applicable laws, policies and prac3ces and ease of configura3on and use. AA will be the solu3on of choice for cost- sensi3ve, data intensive applica3ons in the higher educa3on, bio- informa3cs, and media fields. AA is a key partner in the irods and related communi3es, collabora3vely contribu3ng to the development of the open- source plarorm.

16 A History of Open Source Collaboration Body Level One Linux Body Level Two RDMA FreeBSD Body Level Three OpenStack iscsi Body Level Four SnapCreator ovirt NFS Body Level Five OSF irods NDMP DCE 2014 NetApp, Inc. All rights reserved. 16

17 Broker of Services Risk and Reward

18 Alloy - Archive as a Service Data Centre 1 Alloy Appliance Client A File Servers Replication Archive Network Shares Alloy Appliance providing replicated storage from an on-site Archive Replication of data (files) and catalog (SQL) from DC1 to DC2 Data Centre 2 Replication of data (files) and catalog (SQL) from DC2 to DC1 Alloy Appliance Client B File Servers Archive Replication Alloy Appliance providing replicated storage from an on-site Archive Network Shares

19 CONTACTS Thank you..