Making TCP more Robust against Packet Reordering Alexander Zimmermann <alexander.zimmermann@netapp.com> Lennart Schulte <lennart.schulte@aalto.fi> TCPM, IETF-9, Honolulu, HI, USA November, 24
TCPM s feedback at IETF 89 & 9 to the reordering problem in general Strong consensus that we should make TCP more robust against reordering Question: what kind of packet reordering do we want to make TCP robust against? to reordering data & TCP-aNCR measurements Measuring reordering as a fraction of time Considering loss mixed with reordering TCP-aNCR performance vs. Linux Kernel 3.x 2
Reordering in Mobile Networks an Update Netradar.org measurement platform Provides information about the quality of mobile Internet connections and mobile devices App available for many mobile device OSes Measurement setup Measurement server @ Aalto University, Finland Performs sec bulk TCP transfer PCAP files are analyzed for reordering <https://github.com/lennart-schulte/tcpdump-analyzer> Implements draft-zimmermann-tcpm-reordering-detection 3
Reordering Events per Connection (3G & 4G) Observations 3G: w/ Operator 6% of connections with reordering 4G: even more connections with reordering Reordering is operator and network depended.8.8.6.4.6.4.2 Oper (2888) Oper 2 (28) Oper 3 (384)..2 Oper (8277) Oper 2 (8487) Oper 3 (2788). Number of reorderings per connection (#) Number of reorderings per connection (#) 3G Connections 4G Connections 4
Absolute Reordering Extent (3G) Observations Absolute reordering extent can be huge Absolute reordering extent is operator depended Operator : ± 78% of reordering events < 3 segments Operator 2: ± 7% of reordering events > 3 segments.9.8.8.7.6.5.4.3.6.4.2. 2 3 4 5 Reordering extent (Byte) Oper (27786) Oper 2 (5) Oper 3 (32782).2 Oper (27786) Oper 2 (5) Oper 3 (32782) 2 3 4 5 6 7 8 9 Reordering extent (Byte) 5
Reordering Delay (3G & 4G) Observations Two different kinds of packet reordering can be seen Operator : higher, fine grained reordering Operator 2: lower, coarse grained reordering Illustrate: absolute reordering extent is a func. of CWND..9.9.8.8.7.6.5.4.3.7.6.5.4.3.2. Oper (27776) Oper 2 (48) Oper 3 (32784) 2 3 4 5 6 7.2. Oper (77233) Oper 2 (4534) Oper 3 (54536) 2 3 4 5 6 7 Reordering delay (ms) Reordering delay (ms) 3G Connections 4G Connections 6
Relative Reordering Extent (3G & 4G) Observations Relative reordering extent less in > 9% of all cases No strong need to delay a Fast Retransmit more than ± CWND ( ± RTT)...9.9.8.8.7.6.5.4.7.6.5.4.3.3.2. Oper (27789) Oper 2 (47) Oper 3 (32784).5.5 2 2.5 3 3.5 4.2. Oper (77243) Oper 2 (45648) Oper 3 (54536).5.5 2 2.5 3 3.5 4 Relative reordering extent Relative reordering extent 3G Connections 4G Connections 7
Bursts and Reordering mixed w/ Losses (3G) Observations Reordering events can be bursty, or just a lot in one recovery / disorder phase Reordering events can be co-located with losses à no spurious recovery All potential recoveries (3G).95 8 Loss Reorder Mixed.9 % of total 6 4.85.8 Oper (24646) Oper 2 (838) Oper 3 (247) 2 Oper Oper 2 Oper 3 Burst size 8
Number of Avoidable Recoveries (3G) Putting all together Operator : in 4% of all connections one or more recovery phases could have been avoided For other operators: fewer connections were affected (% &.2%).98.96.94.92.9 2 4 6 8 2 4 Reorder phases (#) Oper (2888) Oper 2 (28) Oper 3 (384) 9
Summary & Next Steps We should tackle reordering in all dimension à besides prevention strategies, we need a state recovery strategy too detected all kinds of reordering, despite of any reaction à draft-zimmermann-tcpm-reordering-detection start to work a reordering detection to gain more inside à adapt draft-zimmermann-tcpm-reordering-detection We may make TCP not robust against any kind of reordering, i.e. reordering > ± RTT revive draft-blanton-tcp-reordering as overview draft?
Document Status draft-zimmermann-tcpm-reordering-detection-2 Extended algorithm for calculation of reordering extents greater than one RTT (steps C.2, S.5 and D.3) Moved reasoning for relative reordering extent to discussion draft-zimmermann-tcpm-reordering-reaction-2 Specify interaction between TCP-aNCR and PRR Fix typo in DupThresh calculation (steps I.5 and E.9)
Appendix 2
Additional 4G measurement results
Absolute Reordering Extent (4G) Observations Operator : ± 98% of reordering events < 3 segments Operator 2: % of reordering events > 3 segments! Same two different kinds of packet reordering as in 3G networks.9.8.8.7.6.5.4.3.6.4.2. 2 3 4 5 Reordering extent (Byte) Oper (7745) Oper 2 (435) Oper 3 (5444).2 Oper (7745) Oper 2 (435) Oper 3 (5444) 2 3 4 5 6 7 8 9 Reordering extent (Byte) 4
Bursts and Reordering mixed w/ Losses (4G) Observations Operator : reordering events are more bursty than in the 3G networks Reordering events are as much co-located with losses as in the 3G networks All potential recoveries (4G) Loss Reorder Mixed.9 8.8.7 % of total 6 4.6.5 Oper (74386) Oper 2 (4374) Oper 3 (24749) 2 Oper Oper 2 Oper 3 Burst size 5
Number of Avoidable Recoveries (4G) Observations Worst is Operator : in 2% of all connections one or more recovery phases could have.98 been avoided Reflects same behavior as in 3G networks.96.94.92.9.88.86 2 3 4 5 6 7 8 9 Reorder phases (#) Oper (8277) Oper 2 (8487) Oper 3 (2788) 6
TCP-aNCR Measurement Results draft-zimmermann-tcpm-reordering-detection-2 draft-zimmermann-tcpm-reordering-reaction-2
Evaluation Methodology Traffic model One unidirectional long-lived bulk TCP NewReno flow No competing flows; no cross traffic, no short-lived flows Traffic generation: flowgrind <http://www.flowgrind.net> Network emulation: netem Flows under test Linux 3.6 (w/(o) Timestamps); TCP-(a)NCR (CF & AG) Duration 2s; Repetition: x Bottleneck Bandwidth 2 Mbps, RTT 4ms Reordering Rate 2%, Reordering Delay 2ms 8
Bandwidth Variation No Reordering 2 Native Linux DS Native Linux TS TCP-NCR CF TCP-NCR AG TCP-aNCR CF TCP-aNCR AG 8 Throughput [ Mb /s] 6 4 2 2 3 4 5 6 7 8 9 Bottleneck Bandwidth [ Mb /s] 9
Bandwidth Variation Reordering 2 Native Linux DS Native Linux TS TCP-NCR CF TCP-NCR AG TCP-aNCR CF TCP-aNCR AG 8 Throughput [ Mb /s] 6 4 2 2 3 4 5 6 7 8 9 Bottleneck Bandwidth [ Mb /s] 2
Round-Trip Time Variation 25 Native Linux DS Native Linux TS TCP-NCR CF TCP-NCR AG TCP-aNCR CF TCP-aNCR AG 2 Throughput [ Mb /s] 5 5 2 4 6 8 2 4 6 Round-Trip Time [ms] 2
Reordering Rate Variation 25 Native Linux DS Native Linux TS TCP-NCR CF TCP-NCR AG TCP-aNCR CF TCP-aNCR AG 2 Throughput [ Mb /s] 5 5 5 5 2 25 3 35 4 Reordering Rate [%] 22
Reordering Delay Variation 25 Native Linux DS Native Linux TS TCP-NCR CF TCP-NCR AG TCP-aNCR CF TCP-aNCR AG 2 Throughput [ Mb /s] 5 5 2 3 4 5 6 7 8 Reordering Delay [ms] 23
ACK Loss Variation 25 2 Native Linux DS Native Linux TS TCP-NCR CF TCP-NCR AG TCP-aNCR CF TCP-aNCR AG Throughput [ Mb /s] 5 5 5 5 2 25 3 35 4 ACK Loss Rate [%] 24