irods Scalable Architecture

Transcription

1 irods Scalable Architecture WORKSHOP 28 TH /29 TH APRIL 2015 Mark van de Sanden

2 Outline irods resources What is irods scalable architecture irods scale up within a zone irods scale up with a federation 28-29th of April, Utrecht, The Netherlands

3 irods Scalable Architecture 28-29th of April, Utrecht, The Netherlands

4 irods s (Storage) is a Software or Hardware system that stores data A resource is a logical mapping of a resource name to a number of physical attributes of a resource 3 classes: - Cache, Archival and Compound Integration with existing Data Services via Mounted collections 28-29th of April, Utrecht, The Netherlands

5 irods Classes Cache Archive Compound Low Latency High Latency POSIX like IO calls 5

6 Compound POSIX like type calls: synctoarch(), stagetocache(), mkdir(), chmod(), rm(), mv() Standard modules for: Unix FS, Windows FS, HPSS, S3, FTP, Database Universal MSS (Mass Storage Service) Driver

7 irods Groups Group Group Low Latency

8 irods Groups Group Cache Cache Cache Low Latency Low Latency POSIX like Low IO Latency C

9 irods Groups Replicate Group Cache Cache Cache Low Latency Low Latency POSIX like Low IO Latency C

10 irods Groups Replicate Group Cache Cache Cache Low Latency Low Latency POSIX like Low IO Latency Z

11 irods Groups Implemented as a delayed rule: irule replicatediskcache \ "*Collection=/vzSARA1% \ *PnfsRescName=eudatPnfs% \ *CacheRescName=eudatCache \ ruleexecout Group Cache Archive Low Latency Migrate Stage High Latency

12 irods Mounted CollecUon User Interface irods Policy Rule engine icat /zone/home/user/collection irods Ireg physfilepath irodspath /mnt/data/collecuon irods mounted collecuon /data/collecuon Unix NFS Mount Mass Storage Backend Systems

13 irods Mounted CollecUon Mount exisung directory as a collecuon into irods IRODS works as an Proxy Pro s Labour extensive way to integrate existing data collections Data can be modified via other means Con s No metadata is stored in the icat database Not aware of users Checksums are calculated on the fly ireg to bridge the gap

14 irods Scalable Architecture User Interface irods Policy Rule engine irods S3 Metadata Catalog (icat) i WIN HPSS MSS UNIX DB DOWS Storage Access Storage OrganizaUon Data DB irods Zone

15 irods Scalable Architecture irods Zone

16 irods Scalable Architecture irods Zone

17 irods Scalable Architecture i Mass Storage Backend Systems irods Zone

18 15 x Data Servers (700TB) 2x HA icat servers (30M Objects) HPSS (6.3PB) 2x Oracle RAC servers Mass Storage Backend Systems irods Zone

19 irods FederaUon iadmin mkzone ZoneB remote irods.zoneb.org:1247 iadmin mkzone ZoneC remote irods.zonec.org:1247 ZoneA /ZoneA/Collection/DataObject /ZoneB/Collection/DataObject /ZoneC/Collection/DataObject ZoneB ZoneC Independent Administrative domains Users can access shared data and metadata in other zones (user#zone) No passwords needed, authenticated to local zone

20 irods FederaUon example / /vzmpi /VZRZGE /VZSARA1 vzmpi vzrzge / /ORFEUS /vzmpi /VZRZGE /VZSARA1 vzsara1 / /ORFEUS /VZSARA1 ORFEUS

21 MPI- TLA LAT so_ware stack DB NFS Mounted CollecUon irods /vzmpi irods /vzrzge MSS HPSS SAM- FS rpc MSS grid_p irods /vzsara dcache

22 PID ORFEUS Seismic system irods irods dcache HPSS NFS share NFS mount KNMI DMF OS: /data/orfeus/data/continuous/... irods: /ORFEUS/eudat/data/continuous/ irods: /vzsara1/eudat/knmi/

23 irods SURFsara

24 QuesUons