Data Staging: Moving large amounts of data around, and moving it close to compute resources

Transcription

1 Data Staging: Moving large amounts of data around, and moving it close to compute resources Digital Preserva-on Advanced Prac--oner Course Glasgow, July 19 th 2013

2 Definition Starting point Moving data around Size/Performances The user Data selection Transfer options Numbers Registered data Outline 2

3 Data movement: staging or replication? PID Safe Replica/on Data preservation and access optimization Data Staging Dynamic replication to HPC workspace for processing Data 3 3

4 Staging 4 4

5 Moving large amounts of data around Data sets can be large both in terms of Numbers of objects Single object size 5

6 All data are gray in the data staging The data types are irrelevant Except when they can affect the size or the number of the files hap:// 6

7 When I should care about data types? When you can compress them Data transfer efficiency is crucial in data staging 7

8 Time to copy 1TB 10 Mb/s network: 300 hrs (12.5 days) 100 Mb/s network: 30 hrs 1 Gb/s network: 3 hrs (are your disks fast enough?) 10 Gb/s network: 20 minutes (need really fast disks and file system) Compare these speeds to: USB 2.0 portable disk 60 MB/sec (480 Mbps) peak 20 MB/sec (160 Mbps) reported on line hours to load 1 Terabyte 8 Parallel I/O and management of large scien-fic data

9 Data Throughput Transfer Times This table available at 9 Parallel I/O and management of large scien-fic data

10 Data staging is Just in Time You do not plan it You can pre-stage sometimes There are techniques to improve the efficiency 10

11 Faster than light TCP tuning: refers to the proper configuration of buffers that correspond to TCP windowing Pipelining (of commands): speeds up lots of tiny files by stuffing multiple commands into each login session back-to-back without waiting for the first command's response 11

12 Parallelizing: on wide-area links, using multiple TCP streams in parallel (even between the same source and destination) can improve aggregate bandwidth over using a single TCP stream 12

13 Striping: data may be striped or interleaved across multiple servers 13

14 Parallelism and TCP tuning are the keys It is much easier to achieve a given performance level with four parallel connections than with one A good TCP tuning can improve drastically performances Latency interaction is critical Wide area data transfers have much higher latency than LAN transfers Many tools and protocols assume a LAN Examples: SCP/SFTP, HPSS mover protocol 14

15 Use the right tool SCP/SFTP: 10 Mb/s FTP: Mb/s assumes TCP buffer autotuning standard Unix file copy tools Parallel stream FTP: Mbps fixed 1 MB TCP window in OpenSSH only 64 KB in OpenSSH versions <

16 But efficiency is not all Easiness of use High reliability Third-party transfer Possibility to control/limit the transfer throughput to avoid engulfing the network Which priority? Different scenarios, different needs 16

17 and moving it close to compute resources 17

18 Who will move the data? Any user, part of a community or citizen scientist. In most part of the cases, a domain expert, such as a biologist, a linguist, a seismologist, not an expert of data transfer software. 18

19 Wizard gap As Ethernet speeds have increased, there has been a widening gap between ability of the novice to fully use bandwidth capabilities, compared to that of network capabilites achieved by users who have had their network tuned by an expert ('Wizard"). [Diagram c/o MaA Mathis, hap:// 19

20 Data selection Data Staging Overview 20

21 How the data sets are selected by the domain experts? Through their local tools, where local means on the community side Through remote services, generic or specific per community 21

22 Move the data 22

23 Eudat Transfer options Globus Online Third party transfer, web portal Globus client server to server to server irods specific client server to to server server HTTP options Work in progress 23

24 GridFTP: third Party Transfer Control channels GridFTP Server Data channels GridFTP Server Site A Site A 24

25 Globus OnLine Service 25

26 EUDAT PID Howto transfer #TB or PB From to HPC which site EUDAT X site Which A,B tool or C to use: GridFTP, Howto Scp, to get irods? access all sites? EUDAT site B GridFTP scp 1Gb/s WAN HPC site X EUDAT 10Gb/s WAN Globus Online irods 10Gb/s OPN EUDAT PID Bittorrent? EUDAT PID EUDAT site C EUDAT site A 26

27 Stage-out Data transport protocols: GridFTP, HTTP. SCP, FTP, RSYNC, OpenDap? WAN Which tool to manage the transfers: irods, Globus Online, FTS, RSYNC, Others.? PID EUDA T LTA EUDAT SP site A WAN Third party transfers Dedicated Network Third party transfers HPC site X HPC space EUDAT User Forum Barcelona, 7-8 March

28 Stage-out/stage-in Workflow framework Stage 1 Stage 2 Stage # PID EUDAT LTA EUDAT SP site A Data Mining cluster EUDAT User Forum Barcelona, 7-8 March

29 Multi-step staging EPCC EUDAT center Community center VPH ENES CLARIN Lifewatch CINECA BSC EPOS 29

30 Local staging targets 1. Single step 30

31 Sample Data Transfer Results Sample Results: RTT = 53 ms, network capacity = 10Gb/s. Tool Throughput scp: 140 Mb/s HPN patched scp: 1.2 Gb/s FTP: 1.4 Gb/s GridFTP, 4 streams 5.4 Gb/s GridFTP, 8 streams 6.6 Gb/s 31

32 irods irods is a data management system, which integrates a rule engine The rules can be triggered automatically based on specific events (for example a data set is moved to a particular directory) Invoked from remote via irods command line client or integrated into applications based on irods java (Jargon) and python (PyRods) API Data Staging Overview 32

33 Metadata! "Managing Large Datasets with irods a Performance Analysis" - Proceedings of the Interna-onal Mul-conference on Computer Science and Informa-on Technology pp

34 haps://openwiki.uninea.no/norstore:irods:workshop

35 Data Staging and registered data PID Data Center Store PID Data Center Store Data Center Store HPC ain of registered data 35

36 Conclusions The way to move data has to be enough flexible to accomodate different transfer protocols, different access mechanisms. Flexibility means also that the transfer tools can be used as they are, with default parameters, for average performances, but also fine tuned by experts for faster transfers. No solution fits all, so different services are provided 36

37 e- IRG Workshop - Dublin - Ireland 37 37

38 Thank you! 38