Climate Data Management using Globus

Transcription

1 Climate Data Management using Globus Computation Institute Rachana Ananthakrishnan

2 Data Management Challenges Transfers often take longer than expected based on available network capacities Lack of an easy to use interface to some of the high-performance tools Tools [are] too difficult to install and use Time and interruption to other work required to supervise large data transfers Need data transfer tools that are easy to use, wellsupported, and permitted by site and facility cybersecurity organizations No easy mechanism to share selected data with collaborators Credit: Some data points from ESNet reports

3 Reliable, secure, high-performance file transfer and synchronization Fire-and-forget transfers Automatic fault recovery Data Source 2 Globus moves and syncs files Data Destination Seamless security integration Powerful GUI and APIs 1 User initiates transfer request 3 Globus notifies user

4 Transfer Files

5 Transfer Options

6 Command line and scripting Interactive login to command line interface: $ ssh tuecke@cli.globusonline.org Running commands remotely: $ ssh tuecke@cli.globusonline.org <command> $ ssh tuecke@cli.globusonline.org scp r s 3 -D \ nersc#dtn:~/myfile* mylaptop:~/projects/p1 Task ID: 4a3c471e-edef-11df-aa b1 $ _ Using CLI with gsissh: $ gsissh tuecke@cli.globusonline.org <command>

7 Simple, secure sharing off existing storage systems Easily share large data with any user or group No cloud storage required 1 User A selects file(s) to share, selects user or group, and sets permissions 2 Globus tracks shared files; no need to move files to cloud storage! Data Source 3 User B logs in to Globus and accesses shared file

8 Share files/folders

9 Manage permissions

10 Deploying Globus

11 Globus Connect Server Globus Connect Server MyProxy Online CA GridFTP Server Local Storage System (HPC cluster, campus server, ) Local system users Create endpoint in minutes; no complex GridFTP install Enable all users with local accounts to transfer files Native packages: RPMs and DEBs Also available as part of the Globus Toolkit

12 Endpoint Administrator Controls Restricted paths Files and folders that can be used to transfer Enable or disable ability to share from an endpoint Sharing restricted paths Files and folders that can be used to share Whitelist of users who are allowed to share from an endpoint Only allow read permissions on shared endpoints created by users And many more

13 8,000 active endpoints (in the past year)

14 Some relevant endpoints ALCF Ins$tu$on Endpoints Features OLCF PNNL alcf#* olcf#* pic#* LBNL lbnl#* Sharing enabled NERSC nersc#* LANL Globus Connect Personal DTNs and SDNs being deployed ESGF Argonne esg#anl ESGF data node, sharing enabled ESGF PCMDI esg#pcmdi9 ESGF data node XSEDE xsede#* SDSC has sharing enabled

15 Globus Connect Personal 15

16 Amazon S3 Endpoints

17 Using Globus

18 Ann Syrowski (Illinois) moves data across XSEDE, NCSA MSS and PSC, to leverage HPC facilities across the country Weather Research and ForecasMng Model Source: UCAR

19 Dan Kozak (Caltech) replicates 1 PB LIGO astronomy data for resilience

20 Earth System Grid Federation leverages Globus for data download: including the user interfaces 20

21 Using Globus web pages 21

22 OLCF ALCF Mark Taylor (Sandia) securely shares climate data with collaborators using Globus Sharing Argonne Sharing Enabled

23 Globus is moving beyond transfer and sharing to data publication and discovery

24 Globus Data Publication SaaS for publishing large research data Bring your own storage Extensible metadata Publication and curation workflows Public and restricted collections Rich discovery model

25 Publish walk-through Univ. of Chicago Argonne IIT UIUC 1. Publish Data ScienMst 2. Describe Submission 4. Curate Dataset 3. Assemble Dataset (Transfer Data) CollaboraMon Archive Curator

26 Discover walk-through 5. Search Univ. of Chicago Argonne IIT UIUC 1. Publish Data 6. Download 2. Describe Submission ScienMst 4. Curate Dataset 3. Assemble Dataset (Transfer Data) CollaboraMon Archive Curator

27 Publication as Service for ACME

28 Thank you to our sponsors! U.S. DEPARTMENT OF ENERGY