Ceph at the DRI. Peter Tiernan Systems and Storage Engineer Digital Repository of Ireland TCHPC

Transcription

1 Ceph at the DRI Peter Tiernan Systems and Storage Engineer Digital Repository of Ireland TCHPC

2 DRI: The Digital Repository Of Ireland (DRI) is an interactive, national trusted digital repository for contemporary and historical, social and cultural data held by Irish institutions. The DRI follows the Open Archival Information System (OAIS) ISO reference model and The Trusted Repository Audit Checklist (TRAC)

3 OAIS Model: - is concerned with all technical aspects of digital repositories - describes components and services required to develop and maintain archives - is broken down into 'Functional Entities' and 'Work Packages'. - WP8 is responsible for the Archival Storage functional entity.

4 OAIS Model: Source:

5 DRI Storage Requirements: OAIS/TRAC requires the following from storage: - Minimal conditions for performing long-term preservation of digital assets - Long Term Preservation of digital assets, even if the OAIS (repository) itself is not permanent or present.

6 DRI Storage Requirements: - Open Source/Open Standards - Independence - High Availability - Dynamically Configurable - Ease of Interoperability (Interfaces, APIs) - Data Security/Placement (Replication, Erasure coding, Placement, Tiering, Federation) - Self Contained - Commodity Hardware

7 Storage Solutions We Tested:

8 Why we didn't choose HDFS: - Interfaces limited. Not posix compliant due to immutable nature of filesystem. - Performance geared towards large data streams. I/O of many small files is poor. - Single point of failure and bottleneck at its Namenode. - Doesn t provide any federation

9 Why we didn't choose irods: - Default Interfaces limited. No Restful, RBD. - Single point of failure at its icat metadata server - Overlapping functionality with Fedora Commons Why we didn't choose GPFS: - Default Interfaces limited. No Restful, RBD. - Data Replica limit of 2. - Closed source

10 Why we chose Ceph: - We like its distributed, clustered architecture - Provides complete high availability on install - Scales out horizontally to massive levels - Data Security/Placement: Distributed, Replicated - Many interface options - Rich, documented, multi-level APIs - Dynamically configurable - Very good Performance for general use (many small file I/O) - Solid release schedule, new features

11 Findings: HDFS irods Ceph GPFS API Yes Yes Yes Yes Fedora 3.6.x Driver Yes No No No Interface: Posix No Yes Yes Yes Interface: RBD No No Yes No Interface: RESTful Dynamic Configuration High Availability: Data High Availability: Service Max Raw Storage (PetaByte) On-Read Data Checking Yes No Yes No Yes Yes Yes Yes Yes Yes Yes Yes No No Yes Yes >100 N/A > ^14 No Yes No No Max Replicas 512 >2 ~2.1 Billion 2 Federation No Yes No Yes

12 What is Ceph: Source: ceph.com

13 Ceph Daemons: Monitor Daemon (MON): Tracks Cluster membership and state (Cluster Map) Object Storage Daemon (OSD): Stores Data, Checks its own and other OSD states and reports to MON.

14 Ceph Deploy: ceph deploy new NODE1 ceph deploy install NODE1 ceph deploy mon create initial ceph deploy install NODE2 ceph deploy osd prepare NODE2:/var/osd0 ceph deploy osd activate NODE2:/var/osd0

15 Ceph Client: apt get install ceph common rbd create foo size 4096 m <NODE1IP> k ceph.client.keyring rbd map foo pool rbd name client.admin m <NODE1IP> k ceph.client.keyring mkfs.ext4 /dev/rbd/rbd/foo mount /dev/rbd/rbd/foo /mnt/ceph block device

16 Ceph Architecture: Cluster Map OSD OSD OSD OSD OSD OSD File system Disk State MON MON MON

17 Ceph Data Placement (CRUSH): PG1 PG2 PG3 NODE2 OSD OSD OSD NODE1 OSD OSD OSD

18 Ceph Architecture: OSDs OSDs - 1GB RAM per OSD - 1 Core per OSD - Public / Private Networks - Bonded Quad GB NICs - 12 OSDs per Node - 3 MONs OSDs OSDs OSDs MON MON MON Client NW Cluster NW

19 Ceph Features: - Erasure Coding - Federated S3 RadosGWs - Cache Tiering - Cephx authentication layer - Encryption - User Quotas - Calamari Dashboard

20 Ceph Interfaces: RADOS Block Device (RBD): Provides resizeable, thin-provisioned block devices with snapshotting and cloning. Ceph FileSystem (CephFS): provides a POSIX compliant filesystem usable with mount or as a filesytem in user space (FUSE). Rados Gateway (RGW): Provides RESTful APIs that are Amazon S3 compatible.

21 DRI Infrastructure

22 Performance Poor performance with low number of OSDs (6) and replication.

23 Source: Diana Gudu, KIT Performance Source: Diana Gudu, KIT Source: Diana Gudu, KIT Adding OSDs (26) improves replicated performance

24 Questions? DRI: Trinity HPC: Trinity College Dublin:

25 Links: Ceph: HDFS: IRODS: GPFS: Project Hydra: Fedora Commons: Apache SOLR: HAProxy: hadoop.apache.org projecthydra.org lucene.apache.org/solr/ haproxy.1wt.eu