dcache integration into HDF

Transcription

1 dcache integration into HDF Storage service at DESY for Helmholtz Data Federation (HDF) Paul Millar Karlsruhe, This project has received funding from the European Union s Horizon 2020 research and innovation programme under grant agreement No and

2 What is dcache Used by scientists for photon science, astronomy, particle physics, genomic analysis, neutrino experiments, medical, Storage software that allows a site to... move beyond limitations of a single node. scale performance with number of nodes. choose which hardware to buy. Used to provide roughly 50% of the storage needed to find the Higgs particle Widely deployed storage software, used by institutes throughout the world, often with users throughout the world. A collaboration between DESY, Fermilab and NEIC A software development team, with experience in supporting scientific communities for over 15 years. Providing cutting-edge features: pluggable authentication, delegated authorisation, event-driven model / storage events,

3 dcache authn & authz Protocol layer Management layer Storage layer WebDAV door gplazma Frontend door FTP door Namespace pool NFS door Pool manager Page 3

4 dcache authn & authz Protocol layer Management layer Storage layer 1. Identify user gplazma 5. Data Transfer Transfer data WebDAV door 2. Authorisation 4. Load management pool Namespace 3. Resource management Pool manager Page 4

5 What is OpenID-Connect? DISCLAIMER: it's more complicated than shown here Server has a resource that (normally) only owner can access PR OIDC Provider knows who is the user PR Details about the User Owner Client wants access to a protected resource. User Client wants to know who is the user OAuth2 OpenID-Connect Page 5

6 Why use OpenID-Connect? OIDC Provider knows who is the user PR User Client wants to know who is the user Page 6

7 dcache authn & authz: working with OpenID-Connect OpenID-Connect Provider (OP) Sub uid Group gid Client / Web portal WebDAV door gplazma Access Refresh uid, gid dcache Page 7

8 Third-party copy: single file (simplified) DATA Storage #1 Storage #2 Page 8

9 Third-party copy: in practice DATA Storage #1 Storage #2 Transfer service Page 9

10 dcache authn & authz: third-party copy w/ OIDC delegation OpenID-Connect Provider (OP) C OpenID-Connect Delegation T T C C Sub uid Group gid Client / Web portal C WebDAV door gplazma Access Refresh T T uid, gid Pool dcache T WebDAV endpoint Other storage Page 10

11 dcache authn & authz: third-party copy with macaroons OpenID-Connect Provider (OP) C Sub uid Group gid C Client / Web portal Access Refresh C M WebDAV door M uid, gid gplazma pool dcache M M WebDAV endpoint Other storage Page 11

12 dcache integration points What bits are missing How these are being added

13 dcache integration points OpenID-Connect Provider (OP) Site policy: Outside inside identity Accept new users automatically? Given users a home directory? Sub uid Group gid Client / Web portal WebDAV door gplazma Access Refresh uid, gid dcache Page 13

14 The big picture Unity Feudal dcacheview WebDAV door C gplazma Sub uid Group gid Web-FTS / FTS Command-line tools uid, gid dcache Page 14

15 Current status

16 Current status dcache code Support for OIDC in webdav & frontend Support for OIDC sub and group mapping Third-party transfer support (macaroons and OIDC-delegation) OIDC support in dcacheview Deployment Support for Unity in test-bed: prometheus and dcache-xdc dcacheview / prometheus support enabled Initial investigation Feudal oidc-agent Page 16

17 Future work & time-line

18 Future work Development work Support dcacheview login longer than the access-token lifetime: either support refresh-token, or add session support. Work on command-line clients; work with KIT on oidc-agent Work with Jülich on improvements within Unity; e.g., improve client registration, providing more information, Deployment Finish enrollment policy at DESY continue work based on Feudal (collaborating with KIT) Update a production dcache instance at DESY to support HDF November 2018 Closed beta testing (selected people) Early 2019 Open beta testing: users should accept possible unannounced interruption to service. Page 18

19 Possible usage scenarios: leverage dcache advantages Data for computation work Move data to DESY to allow analysis work with low latency access to data. Sharing data within HDF Enable easy sharing data between institutes. Support multiple access protocols, with all of dcache features. Sharing data with people outside of HDF Allow others to upload data; e.g., volunteer-computing / BOINC, IoT, Allow non-hdf users access to data without making it public Make use of other advanced dcache features Multiple protocol support Event driven work-flows / catalogue synchronisation / Third-party transfer support Different Quality-of-Service Page 19

20 Thank you

21 Contact Deutsches Elektronen-Synchrotron Paul Millar IT