Welcome! Presenters: STFC January 10, 2019

Transcription

1 Welcome! Presenters: Vas Vasiliadis Brendan McCollam STFC January 10, 2019

2 Agenda Morning topics Introduction to the Globus SaaS Service overview & architecture Demo: A researcher s perspective Our sustainability model Break Globus Endpoints Installing and configuring GCS Security mechanisms Deployment best practices Lunch Afternoon topics Introduction to the Globus PaaS Accessing Globus APIs Globus Transfer API walkthrough Automating Research Data Workflows Break Facilitating Collaboration with Globus Data sharing best practices/automation Jupyter + Globus for interactive data science Building the Services Ecosystem Deep dive into Globus Auth Securing your apps and services

3 Introduction to Globus SaaS Vas Vasiliadis STFC January 10, 2019

4 Research data management today (circa 2008) Index? How do we......move, share, describe, discover, reproduce? Facilitate data stewardship

5 Globus: A Brief History of Time Oct Globus Toolkit v1.0.0 Nov Globus Online initial release globus online Nov Sustainability model launched Dec ,000 registered users, 200PB+ moved Jan Globus Toolkit support EOL Jan Globus service >50% sustainable??? Globus service becomes fully self-sustaining

6 Globus bridges data and people within and beyond organizational boundaries

7 Unified access to data across storage tiers Research Computing HPC National Resources Personal Resources Desktop Workstations Public/Private Cloud Mass Storage Instruments 7

8 Sharing with collaborators, community External campus storage Project repositories, replication stores Public repositories Public / private cloud stores

9 Manage data from instruments Analysis store Next-Gen Sequencer High-durability, low-cost store Cryo-EM Advanced Light Source Remote visualization MRI Light Sheet Microscope Personal system

10 Globus: Core functions Compute Facility Instrument Globus transfers files reliably, securely 4 2 Transfer 3 1 Researcher initiates transfer request; or requested automatically by script, science gateway Use a Web browser Access any storage Globus controls access to shared files on existing storage; no need to move files to cloud storage! Share Researcher selects files to share, selects user or group, and sets access permissions 7 Curator reviews and approves; data set published on campus or other system 6 Collaborator logs in to Globus and accesses shared files; no local account required; download via Globus 5 Use an existing identity Personal Computer Researcher assembles data set; describes it using metadata (Dublin core and domainspecific) Publish Publication Repository 6 8 Peers, collaborators search and discover datasets; transfer and share using Globus Discover

11 Endpoints (Collections) Storage abstraction All transfers happen between two endpoints Globus Connect instantiates endpoints Collection ~= Endpoint Test / Demo Endpoints Globus Tutorial Endpoint 1 Globus Tutorial Endpoint 2 ESnet Read-Only endpoints (test files of various sizes) 11

12 Conceptual architecture: Hybrid SaaS Subscriber Control Domain Source Endpoint DATA Channel No data relay or staging via Globus cloud service Destination Endpoint Subscriber owned and administered storage system Globus connector software CONTROL Channel Globus Control Domain Single, globally accessible multi-tenant service

13 The Globus Web App Contextual menu, based on type of endpoint/storage File transfer settings label when using the activity monitor it s nice to see a recognizable name sync - only transfer new or changed files delete files on destination that do not exist on source preserve source file modification times verify file integrity after transfer encrypt transfer (for data channel; control channel always encrypted) Search: Endpoints, Users, Groups

14 Demonstration File transfer Data Sharing Data Publication

15 Globus accounts and identities Globus Account = Primary identity (e.g. vas@globusid.org) + Optional linked identities (e.g. vas@uchicago.edu) Note: Identity Address Linked identities simplify sharing Identities/consents managed like any other resource

16 Groups What can they be used for? Sharing: Access permissions for more than one person Roles: Endpoint management and monitoring Policy based membership, management Group creation and visibility Member invitation and approval Optional membership fields Terms and conditions Roles for delegating management Subgroups inherit policies; may be overridden

17 Demonstration Linked identities Group management

18 Globus for high assurance data management Restricted data handling: PHI, PII, CUI Security controls: NIST , Low Business Associate Agreement (BAA) w/uchicago University of Chicago has a BAA with Amazon Equivalent UK/EU privacy contractual agreements e.g. to cover Data Processor requirements under GDPR

19 High Assurance features Additional authentication assurance Per storage gateway policy on frequency of authentication with specific identity for access to data Ensure that user authenticates with the specific identity that gives them access within session Application instance isolation Authentication context is per application, per session Encryption of user data in transit and Globus data at rest Detailed audit log (on data transfer nodes)

20 Demonstration High Assurance Endpoints

21 client software that makes a storage system accessible via Globus

22 Globus Connect Personal Rapid installation/removal by non-privileged account Zero configuration; auto updating Handles NATs

23 Demonstration Globus Connect Personal installation and configuration

24 on sustainability

25 Globus by the numbers 520 PB moved 70+ petabyte movers 82 billion files processed 1,923 most shared endpoints at a single 125,000 institution users 1,835 active server endpoints 100+ subscribers 21,306 active personal endpoints 569 identity providers 1 PB largest single transfer to date 7,274 active shared endpoints 99.9% availability

26 Thank you to our sponsors... U. S. DEPARTMENT OF ENERGY

27 and THANK YOU, subscribers!

28 Globus sustainability model Standard Subscription Sharing, data publication HTTPS access Console, usage reporting Priority support App integration support High Assurance subscription App instance isolation Additional authentication assurance Audit logging NIST , NIST (+ BAA) Branded Web Site Premium Storage Connectors Alternate Identity Provider (InCommon is standard)

29 Support resources Globus documentation: docs.globus.org Community list: Helpdesk and issue escalation: Customer engagement team Globus professional services team Assist with portal/gateway/app architecture and design Develop custom applications that leverage the Globus platform Advise on customized deployment and integration scenarios