Welcome to the CyVerse Data Store. Manage and share your data across all CyVerse pla8orms

Size: px

Start display at page:

Download "Welcome to the CyVerse Data Store. Manage and share your data across all CyVerse pla8orms"

Byron Rodgers
5 years ago
Views:

1 Welcome to the CyVerse Data Store Manage and share your data across all CyVerse pla8orms

2 ü Follow Along ü Workshop Packet ü mcbios.readthedocs.org Logis;cs

applied to data sets whose size is beyond the ability of commonly used sojware tools

constantly moving target, currently ranging from a few dozen terabytes (TB) to many

3 Working with Big Data What? Challenges: the scope and scale of life sciences data con;nue to grow Big data - term applied to data sets whose size is beyond the ability of commonly used sojware tools to capture, manage, and process the within a tolerable elapsed ;me Big data - sizes a constantly moving target, currently ranging from a few dozen terabytes (TB) to many petabytes (PB) of data in a single data set Big Data': Big gaps of knowledge in the field of Internet". Interna'onal Journal of Internet Science 7: 1 5.

4 Working with Big Data How? Challenges: changes in data require changes in tools Difficult / slow transfers Expense for storage / backup Difficult to share and publish Analysis Metadata (What Is metadata?) Changes in scale introduce quan;ta;ve and qualita;ve complica;ons

5 Beyond Biological Big Data Challenge: biology encompasses more than sequence data Advanced Imaging Geospa;al Network Biologists work with and require access to diverse data types, and LOTS of it

6 Data Store Overview The Data Store services all CyVerse pla8orms Access your data from mul;ple CyVerse services Automa;c backup (redundant between University of Arizona and University of Texas Default 100 GB alloca;on, > 1 TB alloca;ons available with jus;fica;on

Data Store Overview Benefits Get Science Done Store

evolving Data Commons lets you access important

captures informa;on needed for reproducibility

project s data management plan irods capable of

7 Data Store Overview Benefits Get Science Done Store any type of files related to your research An evolving Data Commons lets you access important datasets Reproducibility Produc;vity Metadata captures informa;on needed for reproducibility Automa;c backup and accessibility support your project s data management plan irods capable of high-speed transfers (100 GB in ~30 min) Share data instantly with collaborators within CyVerse

8 Data Store Overview Mul;ple ways to access for varied skill levels Point-and-click Command line Discovery Environment (DE) Cyberduck icommands

9 Data Store Overview Important points Data Backups Data Transfer Source Des+na+on Copy Method Time (seconds) Arizona Texas Key component of your data management - Worry-free CD My Computer cp 320 Berkeley Server My Computer scp 150 External Drive My Computer cp 36 USB 2.0 Flash My Computer cp 30 Data Store My Computer iget 18 My Computer My Computer cp 15 Closer to op;mum condi;ons: transfers between University of Arizona and UC Berkeley 100 GB: 26m15s, 1 GB 17.5s

10 Data Store Overview Important points hop:// Local connec;ons and ins;tu;onal policies limit data transfer

11 Data Store Overview Demo Data Storage that supports the Life Cycle of Data page 7 Data Upload / Download pages 10, 29 Easiest way to share data with CyVerse page Managing Metadata page 30 By the end of this demo, you should be able to: Navigate the Data Store using: the Discovery Environment and icommands Upload / Download small and large files (see Community Folder) Import files from a URL Share data via a public link and via the Discovery Environment View and manage file metadata Search the Data Store from the Discovery Environment

Bench Scien;st Data Store Overview User perspec;ves and poten;al

videos Shares all his analyses results with his thesis advisor

her students and collaborators will place in a shared folder Uses

Facili;es Developed a script to automate transfer of data to core

12 Bench Scien;st Data Store Overview User perspec;ves and poten;al applica;ons Uploads all of his FASTQ files and 50GB of root growth videos Shares all his analyses results with his thesis advisor Bioinforma;cian Created a metadata template for assembled genomes her students and collaborators will place in a shared folder Uses public links in the supplemental materials of her publica;ons Core Facili;es Developed a script to automate transfer of data to core users Uses a shared folder to make large datasets accessible Welch et al. 2013

13 Data Store Overview Summaries and Tips

14 Summary: Upload and Download in the DE Simple, for small files (~5 files, < 1.8GB) Bulk, for larger files and folders (< 10GB) Import from URL (no size limit) Advantage Covers most upload/download sharing needs Point and Click Disadvantage Some size / speed limita;ons

15 Summary: Faster transfers with Cyberduck Free cross-pla8orm open source file transfer program Drag and drop files and folders Extensively tested with large transfers (60-70 GB) from desktop to Data Store Access public and private data with your CyVerse account login can access data anonymously data that has been shared with anyone without the need of an account

16 Tips for any transfer method Spaces / Special Characters Many sojware packages are sensi;ve to spaces in files names and/or the special characters below Rename uploaded files before using them in an analysis ~ # $ % ^ & * ( ) + = { } [ ] \ : ; " ' < >,? /

17 Summary: Sharing files in the Data Store Two easy ways to share data from the Discovery Environment 1) Discovery Environment Sharing Share files / folders instantly Control access permissions Manage sharing between collaborators 2) Sharing via Public Link No CyVerse account required Limited to individual files URLs are public (less secure, can revoke)

18 Tips When sharing, use this chart to decide appropriate permissions Permission Read Download Metadata Rename Move Delete Read x x Write x x x Own x x x x x x

19 Viewing and Edi;ng Metadata in the DE User metadata stored in Acribute Value Unit (AVU) Template-based metadata

20 Tips Metadata in the DE Can only use one template at a ;me (this will change) Can create custom metadata templates No just for data management, think reproducibility!

21 Searching in the Discovery Environment Basic search bar searches all files and folders where you have permission Advanced search allows searching based on metadata, permissions, and share status Create auto-updated smart folders based on searches

22 Keep asking: ask.iplantcollabora;ve.org Detailed instruc;ons with videos, manuals, documenta;on in Learning Center

Pre-Workshop Training materials to move you from Data to Discovery. Get Science Done. Reproducibly.

Pre-Workshop Training materials to move you from Data to Discovery. Get Science Done. Reproducibly. Pre-Workshop Packet Training materials to move you from Data to Discovery Get Science Done Reproducibly Productively @CyVerseOrg Introduction to CyVerse... 3 What is Cyberinfrastructure?... 3 What to do