irods hands-on tutorial

Transcription

1 irods hands-on tutorial WORKSHOP 28 TH /29 TH APRIL 2015 Christine Staiger

2 Contents The Basics Installation Tools for administration Adding data and metadata Complex storage systems Federations between data grids Resources and Compound resources

3 Complex storage systems

4 Federation between data grids List all existing zones: ils / Zone A acknowledges Zone B and vice-versa iadmin mkzone B remote Host:Port Zone A adds remote user from Zone B: iadmin mkuser user#b rodsuser Example: Alice and Bob: Alice acknowledges Bob s zone: iadmin mkzone bobszone remote :1247 Bob acknowledges Alice s zone: iadmin mkzone aliceszone remote :1247 Alice creates remote user Bob: iadmin mkuser bob#bobszone rodsuser Now Bob can list and alter files in Alice s zone: ils /aliceszone

5 Data replication between grids Bob copies a file from his zone to his account in Alice s zone irsync -R demoresc i:/bobszone/home/bob/put1.txt i:/aliceszone/home/bob#bobszone/put1.txt -R specifies the target resource Bob copies a file from his home directory to Alice s zone irsync R demoresc put3.txt i:/aliceszone/home/bob#bobszone/put3.txt Bob copies a file from Alice s zone to his home directory mkdir test irsync i:/aliceszone/home/bob#bobszone/put3.txt test/put3.txt Bob deletes a file in Alice s zone irm /aliceszone/home/bob#bobszone/put3.txt

6 Remote Resc Install irods, icat disabled Build an irods server [yes]? Make this Server ICAT-Enabled [yes]? no Host running icat-enabled irods server? Resource name [demoresc2]? remoteresc Resource storage area directory [/home/irodsadmin/irods/vault]? /home/irodsadmin/irods/alicesvault The irodsserver will authenticate to the other servers via an existing irods administrator account. Existing irods admin login name [rods]? alice Password [rods]? ******** Enter the zone as defined in your ICAT irods zone name [tempzone]? alicezone

7 Resource Groups Exercise: Resource Groups Create two data resources of type unix file system cache Group them: iadmin atrg groupname resourcename Deposit data in resource group: iput -R groupname file Check with ils -L where the file is located virtually and physically How to automatically replicate data to resources in the same resource group?

8 Microservices irods 3.3 Micro-services are small, well-defined procedures/functions that perform a simple task ( Examples: msicollcreate, msidataobjrepl, msidataobjchksum, To convert any C procedure (createcollection) into a microservice create an interface routine (msicreatecollection)

9 irods Rules Example: rules/replicate.r replicate{ msiwriterodslog("starting replication", *status); msiwriterodslog("source = *source", *status); msidataobjrsync(*source, "IRODS_TO_IRODS", "null", *destination, *rsyncstatus); } INPUT *source= /alicezone/home/alice/put3.txt", *destination= /bobzone/home/alice#alicezone/put3.txt" OUTPUT ruleexecout à Execute irule -vf ~/rules/replicate.r à Check with ils /bobzone/home/alice#alicezone

10 Delayed and periodic jobs Example rules/test.r mytestrule{ delay("<pluset>60s</pluset>"){ msiwriterodslog("hello World.", *status); } } INPUT *A="1" OUTPUT ruleexecout Writes the output to irods/server/log/relog*, also check rodslog* Use iqstat, iqdel and iqmod to list, delete and modify delayed rules. delay( <PLUSET>5m</PLUSET>EF>1h</EF> ) Start job in 5 minutes and restart it every 1 hour Predefined rules: irods/server/config/reconfigs/core.re

11 Compound resources List resource types: iadmin lt resc_type A compound resource consists of cache and a storage resource (s3, MSS universal driver) Storage resource can be located on several different servers Example: Universal Mass Storage (MSS) User writes data to cache, system replicates data to storage and deletes it from cache admin mkresc Cache "unix file system" cache /home/alice/cache iadmin mkresc mssstore MSS universal driver compound

12 Compound resources MSS can be of any hardware type might require certain protocols to put and get data e.g. gridftp The resource type 'MSS universal driver triggers irods to look for a file irods/server/bin/cmd/univmssinterface.sh to perform the put and get functions Template in irods/server/bin/cmd/examples Example file in testrules/univmssinterface.sh

13 Compound resources admin mkresc Cache "unix file system" cache /home/alice/cache iadmin mkresc mssstore MSS universal driver compound /home/alice/tape Iadmin atrg rescgroup Cache Iadmin atrg rescgroup mssstore Resource Group Resource Resource

14 Universal Mass Storage interface Logging: _log() LOGFILE=/home/<user>/univMSSInterface.log ß Change the path Copy between cache and storage: synctoarch(), stagetocache() _log 2 synctoarch "entering synctoarch()=$*" _log 2 synctoarch "executing: /bin/cp -f \"$1\" \"$2\"" /bin/cp -f "$1" "$2 error=$? if [ $error!= 0 ] # copy failure then fi STATUS="FAILURE" _log 2 synctoarch "The status is $error ($STATUS)" return $error

15 Replicate data replicatediskcache { delay("<pluset>30s</pluset><ef>15m</ef>") { }} *Condition = "DATA_RESC_NAME = '*CacheRescName' AND COLL_NAME like '*Collection%'"; msimakegenquery("data_name, COLL_NAME, DATA_SIZE, order(data_create_time)",*condition,*query2); msiexecgenquery(*query2,*list); while ( *ContInxOld > 0 ) { foreach ( *List ) { à Find rule in testrules/sararules.re msigetvalbykey(*list,"data_name",*d); msigetvalbykey(*list,"coll_name",*c); msidataobjrepl("*c/*d","backuprescname=*pnfsrescname++++ } rescname=*cacherescname++++irodsadmin=",*status)}

16 Replicate data Save SaraRules.re in irods/server/config/reconfigs/ Edit server/config/server.config and restart the server reruleset SaraRules,core No space! irule replicatediskcache "*Collection=/yourZone%*PnfsRescName=yourMssStore %*CacheRescName=yourCache" ruleexecout iput -R mssgroup../put1.txt -k ils -l /aliceszone/home/alice: alice 0 Cache :26 & put1.txt Check with iqstat -l when the rule will be executed Some time later: alice@surfsara-cloud:~/irods$ ils -l /alicezone/home/alice: alice 0 Cache :26 & put1.txt alice 1 Tape :36 & put1.txt

17 Purge cache Write a rule to purge the cache Purge cache when data is replicated and if not enough space left in cache Call by: irule purgediskcache Hints: - cast from string to number: int(), double() - sum in a query: "sum(data_size) - msidataobjtrim("*c/*d","*cacherescname","null","1","1",*status); Example in: irods/clients/icommands/test/rules3.0/rulemsidataobjtrim.r - input *Collection="/yourZone", *CacheRescName="Cache", *MaxSizeStr="2 - output ruleexecout Solution hidden in testrules/.solution

18 Thank you!

19 Trouble shooting ERROR: pututil: put error for /aliceszone/home/bob#bobszone/put1.txt, status = status = SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit irods/server/config/irodshost localhost you.rip.xx.xx surfsara-cloud localhost /irodsctl restart