Globus Online and HPSS. KEK, Tsukuba Japan October 16 20, 2017 Guangwei Che

Transcription

1 Globus Online and HPSS KEK, Tsukuba Japan October 16 20, 2017 Guangwei Che

2 Agenda (1) What is Globus and Globus Online? How Globus Online works? Globus DSI module for HPSS Globus Online setup DSI module setup ESnet endpoints use case test Experience with POC Q & A

3 What is Globus (2) Globus =?... Globe Bus? =?... Globe Data Transport Bus? A connected set of services for research data management A powerful, reliable, and secure file transfer service GridFTP is the underlying technology

4 What is Globus (3) GridFTP (Globus toolkit) developed by Ian Foster, Carl Kesselman, and Steve Tuecke around 1998 Based at the Computation Institute at the University of Chicago and Argonne National Laboratory Ian Foster and Steve Tuecke co-lead the Globus project

5 What is Globus Online (4) Globus Online : A simple web interface lets users select the source and destination endpoints and the data movement directions as user s will. Reliable file transfer: Easy fire-and-forget transfers Automatic fault recovery (retry logic) High performance Across multiple security domains

6 QRep Disk What Install Cache is and Globus Utilization Setup Online Test Graph (5) BNL (1)

7 QRep Disk What Install Cache is and Globus Utilization Setup Online Test Graph (6) BNL (1)

8 How Globus Online Works? (7) Globus Online architecture Globus Online Globus connect server OAuth/My proxy, GridFTP, servers Source Endpoint Destination Endpoint

9 QRep How Disk Install Globus Cache and Online Utilization Setup Test Works? Graph in BNL (8) (1)

10 Globus DSI module for HPSS (9) DSI = Data Storage Interface A Bridge between GridFTP and HPSS Translate communications between the GridFTP protocol and the HPSS API Developed at the NCSA (National Center for Supercomputing Applications) by Jason Alt around 2012 DSI first version V 0.1 alpha released in May 2012 Version 2 tested against HPSS7.4.2, released in Sep 2016 Latest version 2.4 released in June 2017

11 Globus DSI module for HPSS (10) Globus, DSI and HPSS architecture Globus Online Globus connect server (GridFTP, OAuth/My proxy servers) HPSS DSI module HPSS client and config. HPSS

12 Globus Connect Server Setup (11) Supported OS platform: CentOS 5, 6, and 7 Debian 7 and 8 Fedora 24 and 25 Red Hat Enterprise Linux 5, 6, and 7 Scientific Linux 5, 6, and 7 SuSE Linux Enterprise Server 12 Ubuntu LTS, and LTS

13 Globus Connect Server Setup (12) Network setup: Internet accessibility (hostname internal and external DNS resolution) Open TCP ports (2811, , 2223, 443, 7521) Download and install Software packages: Major package and latest version: globus-connect-server el7+gt6.noarch globus-gridftp-server el7+gt6.x86_64 $ sudo curl -LOs $ sudo rpm --import Globus $ sudo yum install globus-connect-server-repo-latest.noarch.rpm Endpoint creation : Endpoints are named entities that represent a gridftp or other file transfer service server. Defined in Globus config file: /etc/globus-connect-server.conf

14 Globus Connect Server Setup (13) Globus connect server start/restart: Endpoint is the minimum configuration in the default config file for connect server to start and run Run the cmd to start/restart Globus connect server in order to load/reload the configuration: globus-connect-server-setup Globus connect server configuration: Full configuration all defined in Globus config file: /etc/globus-connect-server.conf User Configuration Endpoint Configuration Security Configuration GridFTP Configuration MyProxy Configuration

15 Globus DSI Module Setup (14) DSI Installation: DSI requires either a client or a full HPSS install Installing HPSS 7.4.3p2 packages on Globus connect server hpss-lib el6.x86_64.rpm hpss-clnt el6.x86_64.rpm hpss-lib-devel el6.x86_64.rpm Install and setup HPSS7.4.3p2 core, gateway and mover server remotely Download DSI package from github: Follow the installation readme file to build and install HPSS DSI

16 Globus DSI Module Setup (15) DSI configuration: Gridftp related configuration Loading DSI module : /etc/gridftp.d/hpss Setup LD_LIBRARY_PATH add the following line to /etc/init.d/globus-gridftp-server: export LD_LIBRARY_PATH="<path to DSI install dir>" e.g. export LD_LIBRARY_PATH=/usr/local/gridftp_hpss_dsi-2.4 HPSS related configuration under /var/hpss/etc Create gridftp.keytab file for user hpssftp Download gridftp.conf file from Github source directory HPSS native config files must be in sync with HPSS core and gateway servers

17 Esnet Use Case Test (16) Endpoints: BNL endpoints: BNL-ESnet-test: HPSS storage FS, Upton, NY SDCC: BNL-RACF disk storage FS, Upton, NY ESnet Data Transfer Nodes (DTNs): RAID6 disk FS, 10 Gbps network, GridFTP read access for Science DMZ ESnet Read-Only Test DTN at ANL: Near chicago, IL ESnet Read-Only Test DTN at LBL: Berkeley, CA ESnet Read-Only Test DTN at CERN: Geneva, Switzerland Test data Single directory /data1/climate-large - 11 files (size range GB) total 244 GB Data moving direction Write into BNL-Esnet-test HPSS endpoint

18 Esnet Use Case Test (17) BNL-Esnet- test endpoint configuration: Red Hat Enterprise Linux Workstation release 7.3 (Maipo) 2 x 10 Gb NIC channel bonding 64 GB RAM Intel(R) Xeon(R) CPU E GHz CPU (28 threads, 14 cores) Mover: NetApp E5560 RAID6, remote to Globus DSI server Globus connect server: GridFtp server: Globus HPSS DSI: 2.4 HPSS release: p2e1 Test metrics Disk IO performance Network throughput Per job transfer speed with file integrity check Per job transfer speed no file integrity check

19 Esnet Use Case Test (18) Disk IO performance: Mover disk: NetApp E5560, RAID6 Stress single lun on mover disk with 100% writing (reading) capacity constantly, writing speed: 713 MB/s.lun, reading speed: 743 MB/s.lun Network throughput: Stress two luns on mover disk with heavy data transfer intensity Max transfer speed: 1.15 GB/s (9.2 Gb/s) Screen shots from nmon tool monitor:

20 QRep Disk Esnet Install Cache Use and Utilization Case Setup Test Graph (19) in BNL (1)

21 QRep Disk Esnet Install Cache Use and Utilization Case Setup Test Graph (20) in BNL (1)

22 QRep Disk Esnet Install Cache Use and Utilization Case Setup Test Graph (21) in BNL (1)

23 Esnet Use Case Test (22) Per job transfer speed with file integrity check Per job transfer speed no file integrity check

24 Esnet Use Case Test (23) Per job transfer speed with file integrity check Network use setting for both Ends: Max / preferred Concurrency: 32/8 Max / preferred parallelism: 16/4 Single user request, run time 4 8 concurrency Overall performance Result: CERN to BNL: 396 MB/s LBL to BNL: 456 MB/s ANL to BNL: 578 MB/s SDCC to BNL: 525 MB/s

25 Esnet Use Case Test (24) Overall transfer performance (with CKSM):

26 Esnet Use Case Test (25) Overall transfer performance (no CKSM): CERN to BNL: LBL to BNL: ANL to BNL: SDCC to BNL: 604 MB/s 755 MB/s 1.11 GB/s 1.04 GB/s

27 Esnet Use Case Test (26) Overall transfer performance comparison CKSM heavy overhead, slow down xfer speed % Endpoints xfer speed (CKSM) MB/s xfer speed (no CKSM) MB/s No-CKSM /CKSM Comment CERN to BNL Write data into BNL HPSS storage LBL to BNL Write data into BNL HPSS storage ANL to BNL Write data into BNL HPSS storage SDCC to BNL Write data into BNL HPSS storage

28 Conclusion (27) Proof of Concept Globus Online/DSI services HPSS POC successful More test for restore needs to be done Improvement required for real production env. Known issues and limitation Open port range for HPSS remote mover out of due to HPSS API limitation. Our test working port range: COS selection based on file size only, no other options CKSM overhead too heavy Scalability issues with large directory structure data transfer unreliable (over 200 folders with more than files of 4 TB data hit CKSM errors, retries decrease xfer speed dramatically, DSI dev is still working on for resolution) Same as other HPSS client interface, stage files back from tapes no optimization module plugged in, files retrieval randomly

29 QRep Disk Install Cache and Q & Utilization Setup A (28) Test Graph in BNL (1) Questions????