Fermilab WAN Performance Analysis Methodology. Wenji Wu, Phil DeMar, Matt Crawford ESCC/Internet2 Joint Techs July 23, 2008

Transcription

1 Fermilab WAN Performance Analysis Methodology Wenji Wu, Phil DeMar, Matt Crawford ESCC/Internet2 Joint Techs July 23,

2 The Wizard s Gap 10 years and counting The Wizard Gap (Mathis 1999) is still an issue Users often don t know about: Common OS tuning issues for WAN data movement Wide-area network path, its characteristics, available tools It s still an end-to-end problem, and wizards are still in short supply. Our structured analysis methodology seeks to put some of the wizardry into structured process. 2

3 Factors Affecting Network Performance On the host system: Application software Operating system Hardware In the network path: Local network infrastructures WAN domains (potentially many) It s an end-to-end issue 3

4 Find the performance problem area(s) 4

5 Performance Analysis Methodology Structured approach to performance analysis Model the process like medical diagnosis: Even for wizards, ad hoc analysis seems to be the norm. Structured analysis is a prelude to automation! Collect the physical characteristics. Run diagnostic tests. Record everything; develop a history of the case. Strategic approach: Sub-divide problem space Application-related problems Host diagnosis and tuning Network path analysis then divide and conquer. 5

6 Network Performance Analysis Architecture 6

7 Tools Host diagnosis A web-downloadable script pulls system configuration information Network Diagnostic Tool (NDT) Network path diagnosis OWAMP to collect and diagnose one-way network path statistics. Faulty network connections, malfunctioning NICs, duplex mismatch Packet loss, latency, jitter, path changes Other tools such as ping, traceroute, as needed Packet trace diagnosis Port mirror on border router(s) Tcpdump to collect packet traces Tcptrace to analyze packet traces Xplot for visual examination. 7

8 Host information collected 8

9 Network path characteristics collected Round-trip time Sequence of routers along the paths One-way delay, delay variance One-way packet drop rate Packet reordering 9

10 Network Performance Analysis Methodology Step 1: Definition of the problem space Step 2: Collect host information & network path characteristics Step 3: Host tuning & diagnosis Step 4: Network path performance analysis Route changes frequently? Network congestion: delay variance large? Infrastructure failures: examine the counters one by one Packet reordering: load balancing? Parallel processing? Step 5: Evaluate packet trace pattern 10

11 Tier2/Tier3 Sites worked with UERJ (Brazil) IHEP (China) RAL (UK) UFL (Florida) IFCA (Spain) TTU (Texas) CIEMAT (Spain) UCL (Belgium) ÖAW (Austria) CSCS (Switzerland) 11

12 12

13 CASE STUDY 1: FNAL -> UERJ A walk-thru of the methodology to diagnose the network performance between FNAL and UERJ: Step 1: UERJ contacts US-CMS Tier-1 about poor performance Step 2: UERJ host configurations collected with script Step 3: Host configuration settings analyzed (see following pages ) 13

14 Host Tuning & Diagnosis (step 3): UERJ TCP send buffer & receive buffers too small. The maximum send/receive buffer size was bytes RTT is 250ms Sequence number Receive window Acks Maximum throughput per stream that could be achieved would be 4.2 Mbps (131071*8/0.25)! Segment sent Time sequence diagram with small buffer size (cubic) Time 14

15 Host Tuning (step 3) A second problem TCP congestion control algorithm is Reno/NewReno Ineffective for RTT of 250ms Re-test with more aggressive congestion control algorithm(s) Iperf througput per stream with various congestion control algorithms 15

16 Time sequence diagram (packet trace analysis) Sequence number Receive window FNAL -> UERJ Acks Segment sent Time 16 Time sequence diagram with large buffer size (cubic)

17 Network path performance analysis (step 4): Discovered route flapping in FNAL to UERJ direction Disclosed path between FNAL & UERJ not symmetric Large discrepancy in one-way delay in opposite directions Path problem resolution: Might cause transient loss of connectivity and packet drops Route-flapping problem identified & corrected Path asymmetry remains a product of routing policies Bottom line improvement: ~50Mb/s sustained throughput initially ~500Mb/s when completed 17

18 Path diagnosis FNAL -> UERJ OWAMP indicates route flapping OWAMP indicates markedly differing latencies 18

19 Evaluation of packet trace patterns (step 5): Sequence number Receive window If host tuning & path analysis aren t enough, detailed packet trace is attempted Segment sent Time Sack TCPdump & Xplot graphs Acks Sack Wizards are still useful Retransmit 19 Acks

20 CASE STUDY 2: FNAL -> UFL UFL contacts FNAL WAN about poor performance (step 1) The minimum bandwidth between FNAL and UFL is 10Gbps; however, the overall achievable throughput is less than 1Gbps (between two powerful hosts residing in FNAL and UFL respectively, both with 10G NICs). We follow the methodology to diagnose the network performance between FNAL and UFL Step 2: UFL host configurations collected with script Step 3: Host configuration settings analyzed Step 4: Network path performance analysis 20

21 Surprisingly, We did not find any apparent problems!!! No problem with Host configuration settings The host is the latest powerful hardware in the market The host system parameters have been correctly configured for bulk data transmission in the WAN context. No problem with Network path RTT: 40ms No apparent packet drops detected No apparent packet reordering detected No apparent path changes detected Low link utilization, no apparent traffic congestion detected. 21

22 Step 5: Evaluation of packet trace patterns Experiment system features Run bulk data transmission from FNAL to UFL with IPERF, In the sender, iperf is run as iperf c hostname t 100 w 5M In the receiver, iperf is run as iperf s w 5M We vary the number of TCP streams, the overall throughput is always less than 1Gbps Since RTT is 40ms, theoretically the throughput is 10Mbyte/40ms= 2Gbps/stream if the throughput is not limited by the network 22

23 Step 5: Evaluation of packet trace patterns What is wrong?!!! Packet trace analysis shows: Sequence number 1. No packet drops 2. No packet reordering 3. The performance is limited by the receiver!!! Receive window Limiting receiver window size!!! But TCP/IP parameters have been optimized for WAN!!! Segment sent Maximum receive window size could go as high as 10MBytes Time Acks 23

24 Step 5: Evaluation of packet trace patterns Packet trace analysis does not lie! Host has been correctly configured! Doubled checked! What is wrong? Called network engineer in UFL, and knew their systems are doing TCP-offloading! TCP-offloading is performed in the NIC. However, due to economic issues, NIC does not have as much memory as host. For high BDP context, TCP needs lots of memory for good performance. So TCP-offloading might not be suitable for WAN. 24

25 Solution We disabled the TCP-offloading for Chelsio NIC, recompiled the driver, and re-run the experiments The throughput per TCP stream is around 2Gbps, close to the theoretical analysis. We increased the number of streams to 10 and 20, the overall throughput reach around 4Gbps. A Cisco router in Chicago did unintended rate liming (not discussed here). When the router was updated, the overall throughput reached 6-7 Gbps. 25

26 Status & Summary Currently performing analysis for CMS Tier-2 sites A work-in-progress at this point Focus is on process as well as results Willing to work with others in this area Future areas of effort: Incorporate into work flow & content management system Make use of perfsonar monitoring infrastructure How to get hold of us: Send to WAN@FNAL.GOV Wide Area Work Group video-conf meetings every other Friday 26