High-Performance TCP Tips and Tricks

Transcription

1 High-Performance TCP Tips and Tricks TNC 16 June 12, 2015 Brian Tierney, ESnet

2 A small amount of packet loss makes a huge difference in TCP performance Local (LAN) Metro Area With loss, high performance beyond metro distances is essentially impossible Regional ConGnental InternaGonal Measured (TCP Reno) Measured (HTCP) Theoretical (TCP Reno) Measured (no loss) 2

3 Time to Copy 1 Terabyte 10 Mbps network : 300 hrs (12.5 days) 100 Mbps network : 30 hrs 1 Gbps network : 3 hrs (are your disks fast enough?) 10 Gbps network : 20 minutes (need really fast disks and filesystem) These figures assume some headroom lew for other users Compare these speeds to: USB 2.0 portable disk 60 MB/sec (480 Mbps) peak 5-15 MB/sec reported typical performance hours to load 1 Terabyte 3

4 Say NO to SCP Using the right data transfer tool is very important Sample Results: Berkeley, CA to Argonne, IL (near Chicago ) RTT = 53 ms, network capacity = 10Gbps. Tool scp wget, GridFTP, FDT, 1 stream Throughput 330 Mbps 6 Gbps GridFTP and FDT, 4 streams 8 Gbps (disk limited) Notes scp is 24x slower than GridFTP on this path!! to get more than 1 Gbps (125 MB/s) disk to disk requires RAID array. Assume host TCP buffers are set correctly for the RTT 4

5 Parallel Streams Help With TCP Conges;on Control Recovery Time 5

6 Hard vs. SoA Failures Hard failures are the kind of problems every organizagon understands Fiber cut Power failure takes down routers Hardware ceases to funcgon Classic monitoring systems are good at alergng hard failures i.e., NOC sees something turn red on their screen Engineers paged by monitoring systems SoW failures are different and owen go undetected Basic connecgvity (ping, traceroute, web pages, ) works Performance is just poor How much should we care about sow failures? 6

7 Causes of Packet Loss Network CongesGon Easy to confirm via SNMP, easy to fix with $$ This is not a sow failure, but just a network capacity issue OWen people assume congesgon is the issue when it fact it is not. Under-buffered switch dropping packets Hard to confirm Under-powered firewall dropping packets Hard to confirm Dirty fibers or connectors, failing opgcs/light levels SomeGmes easy to confirm by looking at error counters in the routers Overloaded or slow receive host dropping packets Easy to confirm by looking at CPU load on the host 7 6/2/15

8 Sample SoA Failure: failing op;cs normal performance Gb/s degrading performance repair one month 8

9 Sample SoA Failure: Under-powered Firewall Inside the firewall One direcgon severely impacted by firewall Not useful for science data Outside the firewall Good performance in both direcgons 9

10 Sample SoA Failure: Host Tuning 10

11 Tunable Buffers with a Brocade MLXe 1 Buffers per 10G egress port, 2x parallel TCP streams, 50ms simulated RTT, 2Gbps UDP background traffic 11 [1] NI-MLX-10Gx8-M Linecard

12 TCP Tuning hnps://fasterdata.es.net/host-tuning/linux/ Add to /etc/sysctl.conf net.core.rmem_max = net.core.wmem_max = net.ipv4.tcp_rmem = net.ipv4.tcp_wmem = net.core.netdev_max_backlog = # set default to CC alg to htcp net.ipv4.tcp_congestion_control=htcp Add to /etc/rc.local # increase txqueuelen /sbin/ifconfig eth2 txqueuelen And use Jumbo Frames whenever possible! 12 6/12/16

13 CUBIC vs HTCP

14 Useful Network Test Tools tools from perfsonar project bwctl owamp Standard Unix tools ping, traceroute, tracepath Standard Unix add-on tools iperf, iperf3, nuncp To install all of these tools at one Gme yum install perfsonar-tools apt-get install perfsonar-tools InstallaGon instrucgons at hnps://fasterdata.es.net/performance-tesgng/network-troubleshoogng-tools/ 14

15 Default perfsonar Throughput Tool: iperf3 iperf3 (hnp://sowware.es.net/iperf/) is a new implementagon of iperf from scratch, with the goal of a smaller, simpler code base Some key features in iperf3 include: reports the number of TCP packets that were retransmined and CWND reports the average CPU uglizagon of the client and server (-V flag) support for zero copy TCP (-Z flag) JSON output format (-J flag) omit flag: ignore the first N seconds in the results On RHEL-based hosts, just type yum install iperf3 More at: hnp://fasterdata.es.net/performance-tesgng/network-troubleshoogng-tools/iperf-andiperf3/ 15

16 Sample iperf3 output on lossy network Performance is < 1Mbps due to heavy packet loss >iperf3 c hostname [ ID] Interval Transfer Bandwidth Retr Cwnd [ 15] sec 139 MBytes 1.16 Gbits/sec KBytes [ 15] sec 106 MBytes 891 Mbits/sec KBytes [ 15] sec 105 MBytes 881 Mbits/sec KBytes [ 15] sec 71.2 MBytes 598 Mbits/sec KBytes [ 15] sec 110 MBytes 923 Mbits/sec KBytes [ 15] sec 136 MBytes 1.14 Gbits/sec KBytes [ 15] sec 88.8 MBytes 744 Mbits/sec KBytes [ 15] sec 112 MBytes 944 Mbits/sec KBytes [ 15] sec 119 MBytes 996 Mbits/sec KBytes [ 15] sec 110 MBytes 923 Mbits/sec KBytes 16

17 BWCTL BWCTL is the wrapper around all the perfsonar tools Policy specificagon can do things like prevent tests to subnets, or allow longer tests to others. See the man pages for more details Some general notes: Use -c to specify a catcher (receiver) Use -s to specify a sender Will default to IPv6 if available (use -4 to force IPv4 as needed, or specify things in terms of an address if your host names are dual homed) 17

18 bwctl features BWCTL lets you run any of the following between any 2 perfsonar nodes: iperf3, nuncp, ping, owping, traceroute, and tracepath Sample Commands: bwctl -c psmsu02.aglt2.org -s elpa-pt1.es.net -T iperf3 bwping -s atla-pt1.es.net -c ga-pt1.es.net bwping -E -c bwtraceroute -T tracepath -c lbl-pt1.es.net -l s atlapt1.es.net bwping -T owamp -s atla-pt1.es.net -c ga-pt1.es.net -N i.01 18

19 Throughput Expecta;ons Q: What throughput should you expect to see on a uncongested 10Gbps network? A: Gbps, depending on RTT TCP tuning CPU core speed, and rago of sender speed to receiver speed 19

20 BWCTL Example (iperf3) $ bwctl -T iperf3 -t 10 -i 2 -c sunn-pt1.es.net Connecting to host , port 5001 [ 17] local port connected to port 5001 [ ID] Interval Transfer Bandwidth Retransmits [ 17] sec 430 MBytes 1.80 Gbits/sec 2 [ 17] sec 680 MBytes 2.85 Gbits/sec 0 [ 17] sec 669 MBytes 2.80 Gbits/sec 0 [ 17] sec 670 MBytes 2.81 Gbits/sec 0 [ 17] sec 680 MBytes 2.85 Gbits/sec 0 [ ID] Interval Transfer Bandwidth Retransmits Sent [ 17] sec 3.06 GBytes 2.62 Gbits/sec 2 Received [ 17] sec 3.06 GBytes 2.63 Gbits/sec iperf Done. bwctl: stop_tool: /2/15

21 OWAMP OWAMP = One Way AcGve Measurement Protocol E.g. one way ping Some differences from tradigonal ping: Measure each direcgon independently (recall that we owen see things like congesgon occur in one direcgon and not the other) Uses small evenly spaced groupings of UDP (not ICMP) packets Ability to ramp up the interval of the stream, size of the packets, number of packets OWAMP is most useful for detecgng packet train abnormaliges on an end to end basis Loss DuplicaGon Out of order packets Latency on the forward vs. reverse path Number of Layer 3 hops Requires accurate Gme synchronizagon via NTP 21

22 OWAMP (ini;al) > owping sunn-owamp.es.net Approximately 12.6 seconds until results available --- owping statistics from [wash-owamp.es.net]:8885 to [sunn-owamp.es.net]: SID: first: c681fe4ed67f1b3e5faeb249f078ec8a T18:11: last: T18:11: sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 31/31.1/31.7 ms, (err= ms) one-way jitter = 0 ms (P95-P50) Hops = 7 (consistently) no reordering --- owping statistics from [sunn-owamp.es.net]:9027 to [wash-owamp.es.net]: SID: c67cfc7ed67f1b3eaab69b94f393bc46 first: T18:11: last: T18:11: sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 31.4/31.5/32.6 ms, (err= ms) one-way jitter = 0 ms (P95-P50) Hops = 7 (consistently) no reordering 22

23 OWAMP (w/ loss) > owping sunn-owamp.es.net Approximately 12.6 seconds until results available --- owping statistics from [wash-owamp.es.net]:8852 to [sunn-owamp.es.net]: SID: c681fe4ed67f1f c341a2b83f3 first: T18:27: last: T18:27: sent, 12 lost (12.000%), 0 duplicates one-way delay min/median/max = 31.1/31.1/31.3 ms, (err= ms) one-way jitter = nan ms (P95-P50) Hops = 7 (consistently) no reordering --- owping statistics from [sunn-owamp.es.net]:9182 to [wash-owamp.es.net]: SID: c67cfc7ed67f1f09531c87cf38381bb6 first: T18:27: last: T18:27: sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 31.4/31.5/31.5 ms, (err= ms) one-way jitter = 0 ms (P95-P50) Hops = 7 (consistently) no reordering 23

24 OWAMP (w/ re-ordering) Ø owping sunn-owamp.es.net --- owping statistics from [wash-owamp.es.net]:8814 to [sunn-owamp.es.net]: SID: c681fe4ed67f21d94991ea335b7a1830 first: T18:39: last: T18:39: sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 31.1/106/106 ms, (err= ms) one-way jitter = 0.1 ms (P95-P50) Hops = 7 (consistently) 1-reordering = % 2-reordering = % no 3-reordering --- owping statistics from [sunn-owamp.es.net]:8770 to [wash-owamp.es.net]: SID: c67cfc7ed67f21d994c1302dff first: T18:39: last: T18:39: sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 31.4/31.5/32 ms, (err= ms) one-way jitter = 0 ms (P95-P50) Hops = 7 (consistently) no reordering 24

25 BWCTL (owamp) > bwping -T owamp -4 -s sacr-pt1.es.net c wash-pt1.es.net bwping: Using tool: owamp bwping: 42 seconds until test results available --- owping statistics from [ ]:5283 to [ ]: SID: c67cee22d85fc3b2bbe23f83da5947b2 first: last: T08:17: T08:18: sent, 0 lost (0.000%), 0 duplicates one-way delay min/median/max = 29.9/29.9/29.9 ms, (err=0.191 ms) one-way jitter = 0.1 ms (P95-P50) Hops = 5 (consistently) no reordering 25

26 bwctl + tracepath Using bwctl run tracepath very useful to see MTU and asymmetric rougng issues 26 June 12, 2016

27 BWCTL (Tracepath with MTU mismatch) Ø bwtraceroute -T tracepath -c nenest.lbl.gov -s anl-pt1.es.net 1?: [LOCALHOST] pmtu : anlmr2-anlpt1.es.net ( ) 0.249ms asymm 2 1: anlmr2-anlpt1.es.net ( ) 0.197ms asymm 2 2: no reply 3: kanscr5-ip-a-chiccr5.es.net ( ) ms 4: denvcr5-ip-a-kanscr5.es.net ( ) ms 5: sacrcr5-ip-a-denvcr5.es.net ( ) ms 6: sunncr5-ip-a-sacrcr5.es.net ( ) ms 7: et er1-n1.lbl.gov ( ) ms 8: t5-4.ir1-n1.lbl.gov ( ) ms 9: t5-4.ir1-n1.lbl.gov ( ) ms pmtu : nenest.lbl.gov ( ) ms reached Resume: pmtu 1500 hops 9 back 56 27

28 Extra fun for those with a Mac Install Homebrew ruby -e "$(curl -fssl Homebrew/install/master/install)" brew install iperf iperf3 nuttcp bwctl owamp Try some of the bwctl commands in this talk from your laptop. 28

29 Trouble ShooGng Strategy hnp://fasterdata.es.net/performance-tesgng/troubleshoogng/networktroubleshoogng-quick-reference-guide/ 29 June 12, , hnp://

30 1) Look for obvious Packet Loss Problems mtr hostname ping -c i.2 hostname owping hostname owping -c i.01 hostname bwping -T owamp -s send_host -c receive_host -N i. 01 For more informagon: hnps:// , hnp://

31 2) Look for path and MTU problems traceroute hostname tracepath hostname ping -s M do -c 4 hostname bwtraceroute -c recv_host -s send_host bwtraceroute -T tracepath -c recv_host -s send_host More informagon: hnp://fasterdata.es.net/network-tuning/mtu-issues/ , hnp://

32 3) Look for host problems bwctl -c host1 -s host2 bwctl -c host1 -s host2 -w 64M mpstat P ALL 1 More InformaGon: hnp://fasterdata.es.net/host-tuning/ , hnp://

33 4) Look for network buffer problems nuttcp -u -Ri300m/100 -i 1 -T5 -w1m hostname nuttcp -u -Ri300m/300 -i 1 -T5 -w1m hostname More informagon hnp://fasterdata.es.net/network-tuning/router-switch-buffer-sizeissues/switch-buffer-tesgng/ , hnp://

34 5) Look for subtle packet loss problems UDP finds problems that TCP does not bwctl -T iperf3 -u -b500m -c hostname nuttcp -l8972 -T30 -u -w4m -R3G -i1 hostname , hnp://

35 For More Informa;on hnp://fasterdata.es.net /12/16

36 Extra Slides 36 June 12, , hnp://

37 A Fix For scp/sap PSC has a patch set that fixes problems with SSH hnp:// projects/hpn-ssh/ Significant performance increase Advantage this helps rsync too 37

38 SoW Failure: Under-buffered Switches Average TCP results, various switches Buffers per 10G egress port, 2x parallel TCP streams, 50ms simulated RTT, 2Gbps UDP background traffic 1MB Brocade MLXe 1 9MB Arista MB Cisco MB Brocade MLXe 1 90MB Cisco MB Cisco VOQ Arista

39 Buffer Experiment Testbed 10G 10G 10G 10G 10G Add latency on hosts 1 and 2: tc qdisc add dev EthN root netem delay 25ms 39 Host Tuning following: