Data Transfer on a Container. Nadya Williams University of California San Diego

Size: px

Start display at page:

Download "Data Transfer on a Container. Nadya Williams University of California San Diego"

Bruce Lloyd
5 years ago
Views:

1 Data Transfer on a Container Nadya Williams nwilliams@ucsd.edu University of California San Diego

2 Moving data across WANs scp fdt bbcp curl wget Mb to Gb range One Tb range Two GridFTP server 2

3 Moving data to PRP GridFTP server 3

4 Setup to use GridFTP server 1. GridFTP server on nautilus cluster ps-40g-gridftp.calit2.optiputer.net 2. Add user to GridFTP server 3. Become a user of nautilus cluster 4. Run globus-connect container Access GridFTP server anonymously Copy from GridFTP server to a container Legend: admin user 4

5 Building a Globus Connect Container: yaml files Step 1: create volume apiversion: v1 kind: PersistentVolumeClaim metadata: name: globus-data namespace: npw spec: storageclassname: rook-block accessmodes: - ReadWriteOnce resources: requests: storage: 50Gi Step 2: create service apiversion: v1 kind: Service metadata: name: globus-service namespace: npw labels: k8s-app: globus spec: type: NodePort ports: - port: 2811 targetport: 2811 protocol: TCP selector: k8s-app: globus 5

6 Building a Globus Connect Container: yaml files (cont d) Step 3: create StatefulSet apiversion: apps/v1beta1 kind: StatefulSet metadata: name: globus namespace: npw spec: servicename: globus-service replicas: 1 template: metadata: labels: k8s-app: globus spec: containers: - name: globus-server wrap lines - name: globus-server image: gateway.calit2.optiputer.net:5000/prp-k8s/globus-connect:latest imagepullpolicy: Always args: ["/usr/sbin/globus-gridftp-server", "-c", "/etc/gridftp.conf", "-pidfile", "/var/run/globus-gridftp-server.pid", "-no-detach", "-config-base-path", "/data"] volumemounts: - mountpath: /data name: globus-data volumes: - name: globus-data persistentvolumeclaim: claimname: globus-data nodeselector: nw: 40G 6

7 Create a Globus Connect Container Create service, volume, and container kubectl create f glob-service.yaml n npw kubectl create f glob-volume.yaml n npw kubectl create f glob.yaml n npw kubectl get pods n npw NAME READY STATUS RESTARTS AGE globus-0 1/1 Running 0 18h Connect to your running container kubectl exec it globus-0 n npw 7

8 Demo (on a container) : Step 1 add packages 1. add location for data transfer mkdir /data/gridftp chown gridftp:gridftp /data/gridftp/ 2. install globus CLI tools yum -y --enablerepo=epel install python2-pip pip install --upgrade globus-cli 3. install Globus Connect Personal (Linux) installer cd /data wget globusconnectpersonal-latest.tgz Optional: add a few data transfer tools yum -y install rsync wget openssl-clients 4. switch user and add Globus connect personal su - gridftp cp /data/globusconnectpersonal-latest.tgz. tar xzvf globusconnectpersonal-latest.tgz cd globusconnectpersonal-2.3.5/ 8

9 Demo (on a container) : Step 2 ftp user setup 1. Login using your Globus account globus login --no-local-server Please log into Globus here: code_challenge=pvwox_ps_dhl1g8aps- rn%3aglobope %3Aauth.globus.org%%3Aglobus%3Aauth%3Ascope =S256&access_type=offline Enter the resulting Authorization Code here: afm033pht9guthvlc18feug You have successfully logged in to the Globus CLI as 04...f2b404cb@ucsd.edu 2. Create endpoint globus endpoint create --personal bwep Message: Endpoint created successfully Endpoint ID: 0c d18-11e a6d4e Setup Key: 3bedc3e c-8ea9-0da1c932e4a1 3. Save variables ep=0c d18-11e a6d4e epkey=3bedc3e c-8ea9-0da1c932e4a1 4. Generate a setup key for the endpoint and create endpoint./globusconnectpersonal -setup $epkey Configuration directory: /home/gridftp/.globusonline/lta Contacting relay.globusonline.org:2223 Done! 5. verify endpoint is confgured globus endpoint search --filter-scope my-endpoints ID Owner Display Name c a cb@ucsd.edu bwep 6. Start personal globus connect./globusconnect -start & 9

10 Demo (on a container): Step 3 save setup When container restarts your setup is gone. Save gridftp setup in /data (persistent volume) mkdir /data/gridftp-home cd ~gridftp/ vi ~/.globusonline/lta/config-paths Optional to add more paths to show in Globus Web App cp -p endpoint-info /data/gridftp-home/ cp -p -r.globus* /data/gridftp-home/ After a container restart will need cp /data/globusconnectpersonal-latest.tgz. tar xzvf globusconnectpersonal-latest.tgz cd globusconnectpersonal-2.3.5/ cp p r /data/gridftp-home/.glob* ~ cp /data/gridftp-home/endpoint-info ~ 10

11 Links PRP GitLab repo Globus Connect container How to move data via network Globus Connect Personal for Linux Globus Transfer App 11

12 PRP has support from: US National Science Foundation (NSF) awards: CNS , CNS , CNS , CNS , ACI , & ACI CENIC, PacificWave and StarLight University of California Office of the President CIO UCSD Next Generation Networking initiative Calit2 and Calit2 s Qualcomm Institute DOE ESnet 12

13 Thank you for listening! Questions? 13

PRP Distributed Kubernetes Cluster

PRP Distributed Kubernetes Cluster John Graham (UCSD) Dmitry Mishin (SDSC) Shawfeng Dong (SLAC) Workshop on Applying Advanced AI Workflows in Astronomy and Microscopy UCSC Silicon Valley Campus, Santa