Explaining BGP Slow Table Transfers

Transcription

1 1 / 14 Explaining BGP Slow Table Transfers - Implementing a TCP delay analyzer Pei-chun Cheng, Jong Han Park, Keyur Patel +, Shane Amante, Lixia Zhang UCLA, + Cisco, Level3

2 2 / 14 BGP table transfer Massive routing updates... Affect a large portion of a router s BGP table Session resets or major route changes

3 2 / 14 BGP table transfer Massive routing updates... Affect a large portion of a router s BGP table Session resets or major route changes X X

4 2 / 14 BGP table transfer Massive routing updates... Affect a large portion of a router s BGP table Session resets or major route changes... is slow!? anecdotal common knowledge up to minutes reason remains unknown X X

5 2 / 14 BGP table transfer Massive routing updates... Affect a large portion of a router s BGP table Session resets or major route changes... is slow!? anecdotal common knowledge up to minutes reason remains unknown X X Goal: explain the transfer time... effectively!

6 2 / 14 BGP table transfer Massive routing updates... Affect a large portion of a router s BGP table Session resets or major route changes... is slow!? anecdotal common knowledge up to minutes reason remains unknown X X Goal: explain the transfer time... effectively! Idea: investigate the transport (TCP) behavior

7 3 / 14 Why TCP trace? A sample BGP table transfer (low tput: 76 kbps) 5.3 MB 560 sec

8 3 / 14 Why TCP trace? A sample BGP table transfer (low tput: 76 kbps) 5.3 MB 560 sec Delays due to recurring retransmissions

9 3 / 14 Why TCP trace? A sample BGP table transfer (low tput: 76 kbps) 5.3 MB Delays due to receiver awnd 560 sec Delays due to recurring retransmissions

10 3 / 14 Why TCP trace? A sample BGP table transfer (low tput: 76 kbps) 5.3 MB Due to sender cwnd Delays due to receiver awnd 560 sec Delays due to recurring retransmissions

11 4 / 14 TCP induced delay Message delay seen at the BGP level Timestamp Delay Prefix Path / / / / / / /

12 4 / 14 TCP induced delay Message delay seen at the BGP level Timestamp Delay Prefix Path / / / / / / / Sequence (/1000) retransmission time (transport-induced delay) 1500 packet Retransmission ack.seq Time (sec)

13 4 / 14 TCP induced delay Message delay seen at the BGP level Timestamp Delay Prefix Path / / False attribution?... Overlook transport / issues / / / / Sequence (/1000) retransmission time (transport-induced delay) 1500 packet Retransmission ack.seq Time (sec)

14 4 / 14 TCP induced delay Message delay seen at the BGP level Timestamp Delay Prefix Path / / False attribution?... Overlook transport / issues / / / / How to detect? Visual inspection Hard to quantify Call for a delay analysis tool!! Sequence (/1000) retransmission time (transport-induced delay) 1500 packet Retransmission ack.seq Time (sec)

15 T-DAT: TCP Delay Analysis Tool Automatically identify delay contributors Which Application (BGP), TCP, Network Where Sender, Receiver, Network path Series-based approach Convert TCP trace to various event series Represent TCP transmission, retransmission, ACK, etc 2-tuple: [start time, end time], event data Infer delay contributors from the series Series: Advertised window event duration event data [t1, t3] 3 [t3, t5] 4 [t5, -] 2 Series: Outstanding packets event duration event data [t2, t3] 3 [t4, t5] / 14

16 T-DAT: TCP Delay Analysis Tool Automatically identify delay contributors Which Application (BGP), TCP, Network Where Sender, Receiver, Network path Series-based approach Convert TCP trace to various event series Represent TCP transmission, retransmission, ACK, etc 2-tuple: [start time, end time], event data Infer delay contributors from the series Series: Advertised window event duration event data [t1, t3] 3 [t3, t5] 4 [t5, -] 2 Series: Outstanding packets event duration event data [t2, t3] 3 [t4, t5] / 14

17 T-DAT: TCP Delay Analysis Tool Automatically identify delay contributors Which Application (BGP), TCP, Network Where Sender, Receiver, Network path Series-based approach Convert TCP trace to various event series Represent TCP transmission, retransmission, ACK, etc 2-tuple: [start time, end time], event data Infer delay contributors from the series Series: Advertised window event duration event data [t1, t3] 3 [t3, t5] 4 [t5, -] 2 Series: Outstanding packets event duration event data [t2, t3] 3 [t4, t5] / 14

18 6 / 14 T-DAT operations in brief Operation TCP trace Extraction Interpretation adv.seq ack.seq Packet Retransmission Transmission time Upstream loss Series Downstream loss Send BGP limited Send TCP CWD bounded Recv BGP limited Recv TCP ADV bounded Time (sec)

19 6 / 14 T-DAT operations in brief Operation TCP trace Series Transmission time Upstream loss Downstream loss Send BGP limited Send TCP CWD bounded Recv BGP limited Extraction adv.seq ack.seq Interpretation Packet (1) Convert trace to various delay seriesretransmission Represent TCP behaviors Recv TCP ADV bounded Time (sec)

20 6 / 14 T-DAT operations in brief Operation TCP trace Series Transmission time Upstream loss Downstream loss Send BGP limited Send TCP CWD bounded Recv BGP limited Extraction adv.seq ack.seq Interpretation Packet (1) Convert trace to various delay seriesretransmission Represent TCP behaviors Recv TCP ADV bounded (2) Calculate a delay contribution of each series: (series size / analysis period) Ex: 0.3 (30% of time spent in retransmission) Time (sec)

21 7 / 14 Output delay contributors 8 delay contributors Delay vector : V = (r 1, r 2,..., r 8 ), r i = size(series i ) AnalysisPeriod, i = Ex: ( 0.27, 0.43, 0.11, 0.01, 0.00, 0.01, 0.01, 0 ) BGP(2) TCP(2) Network(4) 3 contributing groups Group vector : G = (R s, R r, R n), R g = size( Series i ), g {s, r, n} AnalysisPeriod Ex: ( 0.38, 0.43, 0.01 ) Send Recv Net

22 7 / 14 Output delay contributors 8 delay contributors Delay vector : V = (r 1, r 2,..., r 8 ), r i = size(series i ) AnalysisPeriod, i = Ex: ( 0.27, 0.43, 0.11, 0.01, 0.00, 0.01, 0.01, 0 ) BGP(2) TCP(2) Network(4) 3 contributing groups Group vector : G = (R s, R r, R n), R g = size( Series i ), g {s, r, n} AnalysisPeriod Ex: ( 0.38, 0.43, 0.01 ) Send Recv Net

23 7 / 14 Output delay contributors 8 delay contributors Delay vector : V = (r 1, r 2,..., r 8 ), r i = size(series i ) AnalysisPeriod, i = Ex: ( 0.27, 0.43, 0.11, 0.01, 0.00, 0.01, 0.01, 0 ) BGP(2) TCP(2) Network(4) 3 contributing groups Group vector : G = (R s, R r, R n), R g = size( Series i ), g {s, r, n} AnalysisPeriod Ex: ( 0.38, 0.43, 0.01 ) Send Recv Net

24 7 / 14 Output delay contributors 8 delay contributors Delay vector : V = (r 1, r 2,..., r 8 ), r i = size(series i ) AnalysisPeriod, i = Ex: ( 0.27, 0.43, 0.11, 0.01, 0.00, 0.01, 0.01, 0 ) BGP(2) TCP(2) Network(4) 3 contributing groups Group vector : G = (R s, R r, R n), R g = size( Series i ), g {s, r, n} AnalysisPeriod Ex: ( 0.38, 0.43, 0.01 ) Send Recv Net Apply to BGP table transfers

25 8 / 14 Apply to BGP Data (ISP A and RouteViews) Trace Duration Collector # Pkts/Bytes # Rtrs TCP BGP # BGP Tables Name (M/GB) Transfers ISP A Vendor 1023 / Yes ISP A Quagga 909 / Yes Yes / / RV Vendor 176 / 47 59* Yes - 94 DISCLAIMER: Monitoring sessions!! NOT actual operational BGP sessions

26 8 / 14 Apply to BGP Data (ISP A and RouteViews) Trace Duration Collector # Pkts/Bytes # Rtrs TCP BGP # BGP Tables Name (M/GB) Transfers ISP A Vendor 1023 / Yes ISP A Quagga 909 / Yes Yes / / RV Vendor 176 / 47 59* Yes - 94 DISCLAIMER: Monitoring sessions!! NOT actual operational BGP sessions

27 8 / 14 Apply to BGP Data (ISP A and RouteViews) Trace Duration Collector # Pkts/Bytes # Rtrs TCP BGP # BGP Tables Name (M/GB) Transfers ISP A Vendor 1023 / Yes ISP A Quagga 909 / Yes Yes / / RV Vendor 176 / 47 59* Yes - 94 MCT DISCLAIMER: Monitoring sessions!! NOT actual operational BGP sessions

28 8 / 14 Apply to BGP Data (ISP A and RouteViews) Trace Duration Collector # Pkts/Bytes # Rtrs TCP BGP # BGP Tables Name (M/GB) Transfers ISP A Vendor 1023 / Yes ISP A Quagga 909 / Yes Yes / / RV Vendor 176 / 47 59* Yes - 94 MCT MCT pcap2bgp DISCLAIMER: Monitoring sessions!! NOT actual operational BGP sessions

29 8 / 14 Apply to BGP Data (ISP A and RouteViews) Trace Duration Collector # Pkts/Bytes # Rtrs TCP BGP # BGP Tables Name (M/GB) Transfers ISP A Vendor 1023 / Yes ISP A Quagga 909 / Yes Yes / / RV Vendor 176 / 47 59* Yes - 94 MCT MCT pcap2bgp DISCLAIMER: Monitoring sessions!! NOT actual operational BGP sessions

30 9 / 14 Find delay contributors - ISP A ISP A table transfers Plot the group delay ratio, Ex: (0.7, 0.3, 0.0) Network delay is usually low The failing side of a session tends to have more impact Receiver-side Delay Ratio Vendor collector 1 All Slower receiver Slower sender Sender-side Delay Ratio (only March May 2009) Receiver-side Delay Ratio Quagga collector Network. bounded Sender-side Delay Ratio All

31 9 / 14 Find delay contributors - ISP A ISP A table transfers Plot the group delay ratio, Ex: (0.7, 0.3, 0.0) Network delay is usually low The failing side of a session tends to have more impact Receiver-side Delay Ratio Vendor collector Slower receiver All Sender triggered Slower sender Sender-side Delay Ratio (only March May 2009) Receiver-side Delay Ratio Quagga collector Network. bounded All Sender triggered Sender-side Delay Ratio

32 10 / 14 Find delay contributors - RouteViews RouteViews table transfers From heterogeneous routers Across Internet Receiver-side Delay Ratio All Sender triggered Sender-side Delay Ratio

33 10 / 14 Find delay contributors - RouteViews RouteViews table transfers From heterogeneous routers Across Internet Receiver-side Delay Ratio 1 Slower receiver All Sender triggered TCP trace Slower receiver (30 seconds) 16KB adv. window for 86% of time tput adv. window / RTT (0.04, 0, 0.02, 86.6, 0, 0, 0.05, 0) adv ack Sender-side Delay Ratio POI series Retransmission Recv.BGP bounded Recv.ADV bounded Send.BGP bounded Send.CWD bounded Time (sec)

34 10 / 14 Receiver-side Delay Ratio Find delay contributors - RouteViews RouteViews table transfers From heterogeneous routers Across Internet All Sender triggered Slower sender Sender-side Delay Ratio TCP trace POI series Lazy sender (160 seconds) The sender often pauses transmission Remains idle for 92% of time (0.92, 0, 0.02, 0, 0, 0, 0, 0) Retransmission Recv.BGP bounded Recv.ADV bounded Send.BGP bounded Send.CWD bounded Recovery adv ack Time (sec)

35 10 / 14 Find delay contributors - RouteViews RouteViews table transfers From heterogeneous routers Across Internet Receiver-side Delay Ratio Retransmission All Sender triggered TCP trace Congestion limited (70 seconds) 67% time in recovering the packet loss (0.08, 0.15, 0.15, 0.50, 0, 0, 0.67, 0) Recovery adv ack Sender-side Delay Ratio POI series Retransmission Recv.BGP bounded Recv.ADV bounded Send.BGP bounded Send.CWD bounded Time (sec)

36 11 / 14 Example Screenshot $ tcptrace -G tcp.pcap > tcp tsg.xpl $ t-dat tcp tsg.xpl > tcp ser.xpl $ BGPlot tcp tsg.xpl tcp ser.xpl

37 12 / 14 Summary A TCP delay analysis tool Require TCP trace (tcpdump) Automatically suggest the reasons of transfer delay Demonstrate the usage using BGP monitoring data One ISP and RouteViews Can apply to operational sessions The tool and detail report will be available at (

38 13 / 14 Potential Usage Augment the existing BGP monitoring settings Enable TCP investigation Reduce transport-induced artifact Allow potential in-band monitoring

39 13 / 14 Potential Usage Augment the existing BGP monitoring settings Enable TCP investigation Reduce transport-induced artifact Allow potential in-band monitoring (1) TCP processor (2) Visualization tool (3) BGP extractor (4) T-DAT

40 13 / 14 Potential Usage Augment the existing BGP monitoring settings Enable TCP investigation Reduce transport-induced artifact Allow potential in-band monitoring (1) TCP processor (2) Visualization tool (3) BGP extractor (4) T-DAT

41 14 / 14 On-going Work Improve the tool Parameter / threshold settings Support other TCP variations? Collect BGP/TCP traces ISPA RouteViews UCLA Explore the tool usage for other TCP applications

42 Thanks! 14 / 14