Assessing the Nature of Internet traffic: Methods and Pitfalls Wolfgang John Chalmers University of Technology, Sweden together with Min Zhang Beijing Jiaotong University, China Maurizio Dusi Università degli Studi di Brescia, Italy kc claffy, Nevil Brownlee CAIDA, SDSC, UCSD, USA Traffic classification (TC) Introduction Bittorrent HTTP SMTP
Introduction (cont.) Why traffic classification Network design and provisioning QoS assignment and traffic shaping Accounting Security monitoring: IDS/IPS Network Forensics Trends and changes in network applications Introduction (cont.) Today s Internet evolving in scope and complexity applications adapt rapidly to detection attempts emerging obfuscation techniques Many classification approaches in literature using whatever traffic samples available no systematic integration of results
Outline Classification Methods Research review and taxonomy Survey analysis: P2P Pitfalls Systematic shortcomings Re-validate assumptions UDP rising Routing (a)symmetry on backbone links Research Review and Taxonomy Research review create a structured taxonomy of traffic classification papers and their datasets help to answer popular questions reveal open issues and challenges http://www.caida.org/research/traffic-analysis/classification-overview
Research review and taxonomy: Overview 64 papers published between 1994 and 2008 Definition: traffic classification Methods to classify traffic data sets based on features passively observed in the traffic, according to specific classification goals. Research review and taxonomy: Datasets and Goals Data sets: >80 data sets used for 64 papers! Time of collection, link type, capture environments, geographic location, (payload, anonymization), etc. Classification goals: Coarse or fine-grained classification Applications or protocols
Research review and taxonomy: Features Features Reacting on application development Research review and taxonomy: Methods Methods exact matching port number, payload, etc heuristic methods e.g. on connection patterns machine learning methods supervised and unsupervised
Survey analysis: P2P How much P2P 1.3% to 93% across the 18 (out of 64) papers Survey analysis: P2P (contd.) So how much of modern Internet traffic is P2P "there is a wide range of P2P traffic on Internet links; see your specific link of interest and classification technique you trust for more details."
Survey analysis: P2P (contd.) SUNET: April till Nov. 2006 Outline Methods Research review and taxonomy Survey analysis: P2P Pitfalls Systematic shortcomings Re-validate assumtions UDP rising Routing (a)symmetry on backbone links
Systematic Shortcomings Poor comparability of results!!! 80 data sets by 64 papers lack of shared, modern data sets as reference data no clear definitions (P2P or file-sharing ) lack of standardized measures lack of defined classification goals Assumption: TCP dominates traffic Current TC approaches consider mainly TCP Assumptions TCP is dominating traffic Bulk (data) transfer is done via TCP Advantage TCP has a clear notion of sessions
Assumption: TCP dominates traffic (cont.) There might be a shift (soon): IPTV applications PPLive, PPStream: switched to UDP in Oct. 2008 VA (Video Accelerator): UDP for data transfer P2P applications utp: Micro Transport protocol, based on UDP Part of utorrent 1.9 beta, expected during 2010 All on high, random ports (of course ) Assumption: TCP dominates traffic (cont.)
Assumption: TCP dominates traffic (cont.) CDF of UDP flows per Port number Indeed, high ephemeral ports are common today! Assumption: TCP dominates traffic (cont.) Avg. Packets/Flow for top 10 UDP ports No substantial data portions carried (on these links - yet)
Assumption: TCP dominates traffic (cont.) Current situation (on the links measured) TCP dominating pkts (bytes), UDP dominating flows UDP for P2P overlay signaling This might change soon: UDP based IPTV already common in China, utp UDP for bulk and streaming data transfer TC methods can no longer ignore UDP Assumption: routing symmetry Current approaches consider bidirectional traffic Assumption Traffic is routed symmetrically Same path for forward and backward direction Advantage Bi-directional information offers more features for classification For TCP, bi-directional information allows easier inference of sessions (connections)
Assumption: routing symmetry (cont.) Degree of symmetry 4 link locations (Sweden and USA) 2 samples each Assumption: routing symmetry (cont.) Beyond Intranets and access links (edge networks), there is little symmetry Degree of symmetry decreases with level of coreness of the link TC methods for backbone links need to master unidirectional data flows
Summary Research review structured taxonomy of traffic classification papers Current systematic shortcomings lack of shared, modern data sets as reference data lack of standardized measures lack of defined classification goals Upcoming technical challenges TC methods can no longer ignore UDP TC methods should handle unidirectional flows Traffic classification overview: http://www.caida.org/research/traffic-analysis/classification-overview/ Observations on UDP traffic on Internet backbone links: soon to be published on www.caida.org ( News section) Estimation of routing asymmetry on Internet links: http://www.caida.org/research/traffic-analysis/asymmetry/ or Email: johnwolf@chalmers.se