Future plans for irods. John Constable Informatics Support Group

Transcription

1 Future plans for irods John Constable Informatics Support Group

2 About the Institute Funded by Wellcome Trust. 2nd largest research charity in the world. ~700 employees. Large scale genomic research. Sequenced 1/3 of the human genome (largest single contributor). We have active cancer, malaria, pathogen and genomic variation studies. All data is made publicly available. Websites, ftp, direct database, access, programmatic APIs. About Us

3 The Sanger Institute: A Little Background 1997 (yeast genome completed) 2003 (first mouse genome draft Malarial parasite sequence completed) 2008 (start of 1000 genome project) 2001 (First draft of human genome. Sanger upped contribution to 1/3) 2005 (WTGCCC established) 2010 (Completion of 1000 genomes Start or uk10k study) About Us

4 About Us Sequence till 2011

5 Original Design Brief Design Brief Image credit: Ryan Raffa, ryanraffa.com

6 Design Brief Is the data safe?

7 Can the scientists find their data? Design Brief Image credit: searchengineland.com

8 Path of least surprise Design Brief Image credit: betanews.com

9 Minimal Maintenance Design Brief Image credit: failblog.cheezburger.com

10 Current Setup

11 Metadata Heavy Usage Example attribute fields Users query and access data largely from local compute clusters Users access irods locally via the cli Largely provided on creation with Baton API s via pipelines Current Setup attribute: library attribute: total_reads attribute: type attribute: lane attribute: is_paired_read attribute: study_accession_number attribute: library_id attribute: sample_accession_number attribute: sample_public_name attribute: manual_qc attribute: tag attribute: sample_common_name attribute: md5 attribute: tag_index attribute: study_title attribute: study_id attribute: reference attribute: sample attribute: target attribute: sample_id attribute: id_run attribute: study attribute: alignment

12 Current Deployment Replication between data centre rooms (direct, not queued) One resource set as default for incoming objects, migration via cron script as it fills up Checksum via iput strongly encouraged Current Setup

13 Current Logical Design Sanger1 (Portal Zone) Portal provides kerberised access Federation using head zone accounts /seq /humgen green red green red Current Setup

14 Image copyright: flyinglow.ca The Future

15 Future Logical Design Sanger1 (Portal Zone) Orange will be offsite at Infinity /seq /humgen green red Orange? Red green Orange? The Future

16 The Future Why the change?

17 The Future Everyone has an airport, why is that noteworthy?

18 The Future 10g via JANET between sites Tested on AWS first (successfully!) Using Oracle failover (HA tnsnames entry to RAC clusters) Build here and ship to new site ~3PB to transfer Need to have local replicas

19 Image credit: diy.despair.com

20 The spindles spin, but does the data(base)? IRODS upload of genotyping data archive Started :50 Files uploaded per hour mean = 55.6/min 5 pm Friday 9 am Monday Mean files uploaded per minute Experience Hours after start

21 Our block storage deployment

22 Experience Your mileage may vary

23 Experience Wait, WHAT?

24 Databases are good at data, right? Experience

25 Experience Can you spot the optimisation time point?

26 Experience Database Tuning FTW

27 Features we re not using.. yet Image credit: Melissa Penta; mydigitalmind.com

28 More rules based notification E.g. notifying PI s on access to restricted data Pam authentication instead of Kerberos idrop (metadata query non trivial ATM) Features

29 Features we want Features Image credit:

30 Object store plugins/ integration Caching plugin thoughts Streaming files Local cache space Managing local cache Multi site, esp updates Integration with Vendors Replica count? Site/geographical awareness metrics, metrics, metrics! (also reliability, manageability, low cost.. ) Features

31 Features Instrumentation

32 Features More like this, pls

33 Features Oracle MySQL

34 What our users are doing Source: projectcartoon.com

35 Serapis (REST API, python, RabbitMQ, MongoDB) Baton Stuff they haven t told us Users

36 Thank you! Acknowledgements: Dr Pete Clapham John Constable Informatics Support