Optimizing reasonably secure Long Distance Data Transfer. How to transfer Data while being poor

Transcription

1 Manfred Stolle Zuse Institute Berlin Optimizing reasonably secure Long Distance Data Transfer Or How to transfer Data while being poor 1

2 The situation A scientific team gains on a daily basis 250 GB of observation data. In order to be processed and stored the data has to be transmitted over a long distance. Transmission with ftp performance is not faster than 160 GB per day. No influence on the network no root permission no privileges at all! 2

3 Keep it small if you are poor! BIG solutions (probably) exist. Big solutions require big ressources (manpower,hardware, $). I mproposing a small (inexpensive) solution 3

4 Steps Separate the (session and task) control flow from the data flow Secure the control flow Optimize the data flow Keep it simple 4

5 dmscp(2) control and data flow This was done in dmscp(1) See San Diego MSS Conf Client Access by a (dynamic) passphrase Server dmscp2 Data transfer (send or receive) (unprivileged) data port dmscp2 ssh Session control Access passphrase Negotiating transfer parameters Success check sshd port sshd This runs with ftp performance too slow! 5

6 Optimizing the data flow and data objects network streams via 1+ subnets *) Client network buffer Server Switch buffer Switch buffer File Pipe to (from) process HSM disk buffer *) more than 1 subnet not (yet) released HSM File File fragments Pipe to (from) process 6

7 Tuning parameters Number of network streams (-streams s) Sizeof network buffer (-maxbuf m) The total buffer is 2*m*s TCP-Windowsize (-wsz w) 7

8 HSM related features of dmscp2 (user exits = perl or sh scripts ) at certain events e.g. disk full error Examples (in perl) come with the source 8

9 More goodies Optional reading/writing from/to pipes at client and server instead of files, Recursive Copy of a file tree (Metadata via the secure line) Network tuning option (copy from core to core) 9

10 Success check Amount of data on server/client is compared Optional checksumming 10

11 Performance Comparing dmscp2 with dmscp(1)/ftp (1.9 MB/s) and scp (0.6 MB/s) Distance 200 Miles Latency 5.7 ms The line contains 100 Mbit components. 3.5 x faster than dmscp(1)/ftp 11.3 x faster than scp streams MB/sec

12 Performance (HPC involved) Local network (1 Gbit/s): Latency < 1ms hpscp 31 MByte/sec (hpscp is a tuned scp *) ) ~340 km (200 Miles) Bandwidth 1 Gbit/s Latency 3 ms hpscp: 28 MByte/sec streams MByte/sec in 2 subnets 201 **) 226 **) streams MByte/sec (101.1 **) *) **) core to core **) feature going to be released next week 12

13 dmscp2 needs No root permission for the installation Install it simply in your ~/bin directory! No daemon under root (except sshd) At least one unprivileged IP port on the server that is accessible to the client By default a free port is searched on the server in the range The user can define a specific port. 13

14 dmscp2 is free and easy to install! contains This presentation (long version) a dmscp2 users guide The sources 14

15 dmscp2 is meant to bea back end to solve routine big data transfer problems, build user front ends for typical data transfer tasks. It is not really easy to use (too many features), an all purpose user front end for anybody (scp/sftp has its advantages!) 15

16 Don t you mess with my data! 16

17 Manfred Stolle Zuse Institute Berlin How to transfer Data while beeing poor Reasonably secure fast data transfer via the Internet

18 Do you know the latest Computer virus? It s called Caperucita Roja It sawfull! 2

19 Actually it doesn t exist, but I can sell you a very performant toolbox against it 3

20 This is Caperucita Roja And this is the remedy 4

21 The most effective (profitable) computer virus is the human fear. 5

22 It s not funny! The bad guys really exist, but it is not sensible that we stop working just out of fear. Security measures have to leave us a chance to get our data over the Internet. 6

23 Scientific data is very valuable. It must not be damaged. But in most cases it is not very secret. We need effective means to transfer big amounts of scientific data over The Net without compromizing our computers. 7

24 The situation A scientific team gains on a daily basis 250 GB of observation data. In order to be processed and stored the data has to be transmitted over a long distance. Sometimes the destination (HSM) file system runs out of space 8

25 The situation (2) No network is totally reliable! The transmitted data can be changed in the network (very unlikely but not impossible) More catastrophes New day new data 9

26 Keep it small if you are poor! BIG solutions (probably) exist. Big solutions require big ressources (manpower,hardware, $). I m going to propose a small solution (although there is no fame in it). 10

27 Steps Separate the (session and task) control flow from the data flow Secure the control flow Optimize the data flow Keep it simple 11

28 dmscp(2) control and data flow This was done in dmscp(1) Client Access by a (dynamic) passphrase Server dmscp2 Data transfer (send or receive) (unprivileged) data port dmscp2 ssh Session control Access passphrase Negotiating transfer parameters Success check sshd port sshd 12

29 The Trick When using networks with high bandwidth (good) and high latency (poor) or very busy networks splitting the network transfer into parallel streams gets you a remarkable speedup. This is done by other solutions too. (I m not the inventor of the method but it is very good!) 13

30 Optimizing the data flow and data objects Client network streams via 1 or more subnets network buffer Server Switch buffer Switch buffer File Pipe to (from) process HSM disk buffer HSM File File fragments Pipe to (from) process 14

31 Tuning parameters Number of network streams (-streams s) Number of network connections (-srvaddrs <addr1> <addr2>..) Sizeof network buffer (-maxbuf m) The total buffer size is 2*m*s TCP-Windowsize (-wsz w) 15

32 HSM related features of dmscp2 (user exits = perl/sh/ scripts ) Called in case of the following events: disk full error Open fragment (premigration of next fragment) Close file/fragment (trigger migration) Successfull completion (postprocessing on the server) Examples (in perl) come with the source 16

33 More goodies Optional reading/writing from/to pipes at client and server instead of files, Recursive Copy of a file tree (Metadata via the secure line) Network tuning option (copy from core to core) 17

34 Success check Amount of data on server/client is compared Optional checksumming 18

35 Performance Comparing dmscp2 with dmscp(1)/ftp (1.9 MB/s) and scp (0.6 MB/s) Distance 200 Miles Latency 5.7 ms The line contains 100 Mbit components. 3.5 x faster than dmscp(1)/ftp 11.3 x faster than scp streams MB/sec

36 Performance (HPC involved) Local network (1 Gbit/s): Latency < 1ms hpscp 31 MByte/sec (hpscp is a tuned scp *) streams MByte/sec in 2 subnets 232 *) 20

37 Performance (HPC involved) - 2 ~340 km (200 Miles) Bandwidth 1 Gbit/s Latency 3 ms hpscp: 28 MByte/sec streams MByte/sec (101.1 **) **) core to core 21

38 Performance very long distance Comparing Transfer between Berlin and San Diego (ZIB and SDSC) 2 IBM Regattas Distance 4200 Miles Latency 177ms scp: 0.2 MByte/sec streams MByte/sec 1 (like ftp/dmscp1) speedup=40 dmscp2 ist 60 times faster than scp 22

39 dmscp2 needs No root permission for the installation Install it simply in your ~/bin directory! No daemon under root (except sshd) At least one unprivileged IP port on the server that is accessible to the client By default a free port is searched on the server in the range The user can define a specific port. 23

40 Problem Sometimes a network I/O process on the server exits (before or during the data transmission I guess before) This causes (like it should) a panic of dmscp2 In this case the dmscp2 run has to be repeated Searching the reason in the shared memory management 24

41 Work in progress The latest version of dmscp2 has an interface to a GUI When using the xmode parameter dmscp2 creates 2 named pipes One to read commands from the GUI One to write the receipts to the GUI dmscp2 calls xdmscp or the binary defined byte xdmscp parameter. 25

42 The GUI protocol (examples) Task Command Receipt Set local directory LCHDIR <path> LWDIR <current localdirectory> List current remote directory (in ls l format) Delete remote file LISTRDIR RERASE <filename> RDIRLIST <entry> RDIRLIST <entry> END_OF_LIST ERASE_DONE Upload file PUT <filename> ACCEPTING_NEW_COMMAND Download directory GETDIR <dirname> dmscp2: ESTIMATED_SIZE <size> xdmscp: CANCEL DOIT dmscp2:accepting_new_command 26

43 My GUI looks like this called by: dmscp2 -xmode -xdmscp -s dslogin.sdsc.edu -u bzbstoll -streams 25 27

44 Total security. 28

45 comes with a price 29

46 dmscp2 is free and easy to install! contains This presentation a dmscp2 users guide The sources Client and server are the same code (~ 8,100 lines C code) gunzip,tar,configure, make, make install that sall Usually it takes not much more than a minute. A quickinstall script is provided 30

47 dmscp2 is meant to bea back end to solve routine big data transfer problems, build user front ends for typical data transfer tasks. It is not really easy to use (too many features), an all purpose user front end for anybody (scp/sftp has its advantages!) 31

48 Examples (HPC involved!) reading a file from the server bemstoll@berni1-en0 (~/dmscp ):./dmscp2 -r -s bdata.hlrn.de\ -f /fastfs/work/bemstoll/f10g -l /fastfs/work/bemstoll/destfile \ -rdmscp dmscp /dmscp2 Using command: ssh -p 22 bdata.hlrn.de 'true;dmscp /dmscp2 - server' > /tmp/dmscp.pipe bemstoll@bdata.hlrn.de's password: Using the port 1031 Number of network data streams=4 Size of buffers : filebuffer= netbuffer= buffer total= tcp window size is Elapsed time secs --> KB/sec MB/sec State: Success bemstoll@berni1-en0 (~/dmscp ): 32

49 writing 1 stream - local network bemstoll@berni1-en0 (~/dmscp ):./dmscp2 -w -s bdata.hlrn.de\ -f /fastfs/work/bemstoll/f10g \ -l /fastfs/work/bemstoll/destfile -streams 1 Using command: ssh -p 22 bdata.hlrn.de 'true;dmscp2 -server' > /tmp/dmscp.pipe bemstoll@bdata.hlrn.de's password: Using the port 1031 Number of network data streams=1 Size of buffers : filebuffer= netbuffer= buffer total= tcp window size is Elapsed time secs --> KB/sec MB/sec State: Success 33

50 writing 6 streams - local network bemstoll@berni1-en0 (~/dmscp ):./dmscp2 w s bdata.hlrn.de\ -f /fastfs/work/bemstoll/f10g -l /fastfs/work/bemstoll/destfile -streams 4 Using command: ssh -p 22 bdata.hlrn.de 'true;dmscp2 -server' > /tmp/dmscp.pipe bemstoll@bdata.hlrn.de's password: Using the port 1031 Number of network data streams=4 Size of buffers : filebuffer= netbuffer= buffer total= tcp window size is Elapsed time 92.2 secs --> KB/sec MB/sec State: Success 34

51 writing 40 streams distance 300 km (HLRN link) 1 Gbit/sec - latency 3 ms bemstoll@berni1-en0 (~/dmscp ):./dmscp2 -w -s hdata.hlrn.de\ -f /fastfs/work/bemstoll/f10g -l /fastfs/work/bemstoll/destfile\ -rdmscp dmscp /dmscp2 -streams 16 Using command: ssh -p 22 hdata.hlrn.de 'true;dmscp /dmscp2 - server' > /tmp/dmscp.pipe bemstoll@hdata.hlrn.de's password: Using the port 1030 Number of network data streams=6 Size of buffers : filebuffer= netbuffer= buffer total= tcp window size is Elapsed time secs --> KB/sec MB/sec State: Success 35

52 writing 1 stream (ftp performance) distance 300 km latency 3 ms bemstoll@berni1-en0 (~/dmscp ):./dmscp2 -w -s hdata.hlrn.de\ -f /fastfs/work/bemstoll/f10g -l /fastfs/work/bemstoll/destfile\ -rdmscp dmscp /dmscp2 -streams 1 Using command: ssh -p 22 hdata.hlrn.de 'true;dmscp /dmscp2 - server' > /tmp/dmscp.pipe bemstoll@hdata.hlrn.de's password: Using the port 1030 Number of network data streams=1 Size of buffers : filebuffer= netbuffer= buffer total= tcp window size is Elapsed time secs --> KB/sec MB/sec State: Success 36

53 Recursive copy 4 streams - local bemstoll@berni1-en0 (~): dmscp2 -w -s bdata.hlrn.de \ -recursive -src. -dest /fastfs/work/bemstoll/destdir Using command: ssh -p 22 bdata.hlrn.de 'true;dmscp2 -server' > /tmp/dmscp.pipe bemstoll@bdata.hlrn.de's password: Using the port 1031 Number of network data streams=4 Size of buffers : filebuffer= netbuffer= buffer total= tcp window size is Elapsed time 12.5 secs --> KB/sec MB/sec State: Success 37

54 Don t you mess with my data! 38