Measuring, modeling and troubleshooting Quality of Experience at Internet access:

Transcription

1 Measuring, modeling and troubleshooting Quality of Experience at Internet access: Inferring traffic differentiation with ChkDiff Riccardo Ravaioli, I3S/CNRS/UNS Guillaume Urvoy-Keller, I3S/CNRS/UNS, INRIA

2 Generalities What is Quality of Experience (QoE)? - A subjective measure of human experience Good, Medium, Poor for an audio conversation 0, 1, 2, for a video streaming - Measured over a panel of testers (e.g., lab, crowdsourcing) How can QoE be estimated/modeled? - By linking it to measurable metrics (Quality of Service or QoS) - Application level, network level, device level, etc Diverse factors impacting QoE. We focus on two main ones: - (Global) Network performances issues ACQUA: Application for predicting Quality of User experience at Internet Access - Application-Specific Network performance issues (middleboxes, firewals, shapers) ChkDiff: Checking Traffic Differentiation at Internet Access - 2

3 Controlled Experimentation Crowdsourcing ACQUA in a nutshell For more details: Appli 1 QoE Model 1 Internet Expected Quality 1 Appli 2 QoE Model 2 Expected Quality 2 (Neutral) Network and device level measurements Appli N QoE Model N Measurements re-utilization Expected Quality N Classification vs. Network Measurements Measurements at end-user Troubleshooting by analyzing Landmarks Learning/Calibration Vary network performance - 3

4 But quality is not only result of global performance issues Evidences that ISPs degrade the performance of selected traffic - [2007] Comcast interrupts P2P upload transfers - [2011, 2012] Free throttles bandwidth allocated to YouTube during evening hours - [2014] Comcast slows down traffic on OpenVPN default port - [2014] Verizon deteriorates connections to Netflix These actions are never advertised ChkDiff: Is my network access neutral? Which applications are impacted by traffic differentiation? - 4

5 Existing work Application flow (Skype, BitTorrent) user Control flow (randomized payload, different port) server Compare - Max throughput - Delay and loss distribution at original and high rate - Received rate to infer token bucket parameters Does not scale to all user applications (one run per application) Does not preserve IP addresses - 5

6 Key Ideas in ChkDiff Goal: be agnostic to - Targeted applications - Differentiation mechanisms at layer 3 (IP) How? - Consider real user traffic, not synthetic traces - Replay outgoing traffic to routers at hops close to the user - Replay incoming traffic from a measurement server - Analyse per-flow delay distribution and losses - Compare flows to each other (ground truth = other flows) Any shaping at layers 3 and 7 will result in higher delays and losses Check for differentiation on the exact set of applications run by the user - 6

7 ChkDiff: outline user hop h server 1. Dump user traffic 2. Arrange into 5-tuple flows 3. Set same size to all packets - Same transmission time when replayed 4. Shuffle packets according to flow sizes - PASTA property holds - Each flow sees the same network conditions 5. Replay upstream traffic 6. Replay downstream traffic 7. Run per-flow statistical analysis on delay distribution and losses - 7

8 Example: ChkDiff upstream user hop h server for each hop h {1,2 k} do for each run r {1,2,3} do replay trace with TTL:=h at a constant rate > original rate collect ICMP time-exceeded packets detect shaped flows at hop h Compare delay distribution (Kolmogorov-Smirnov test) of each flow to that of all other trace delays Check if losses of each flow are consistent with overall losses of the trace (binomial approximation) Flow is differentiated if rejected in all 3 runs - 8

9 Responsiveness to TTL-limited probes At any given probing rate, routers are - Unresponsive - Rate-limited - Fully responsive ICMP rate-limitation appears as: Characterizing ICMP Rate Limitation on Routers, ICC 15 Large-scale campaign to characterize routers - Probed 850 routers at hops 1-5 from PlanetLab nodes - 9

10 ICMP rate limitation: characterization TTL-fingerprinting on routers to get vendor name - Cisco (59%), Juniper (30.5%), others (10.5%) Analysis of rate-limitation parameters - Burst size, inter-burst time, answering rate 20, 100 and 500 pps for most cisco routers 50 pps for 75% of Juniper - 10

11 ICMP rate limitation: delays Are RTTs (Round-Trip Times) biased by our probing rate? No correlation between probing rate and resulting RTT Not hitting any capacity limits of routers - 11

12 ChkDiff: validation in neutral set-up user shaper router Why packets with the same size? - For routers at hops 1,2 the delay variability is comparable to the variability of transmission delays - Flows with large packets would appear as shaped Why 3 runs? - False positives disappear with 2 runs - Safer margin of error - 12

13 ChkDiff: validation in non-neutral set-up Throttled bandwidth - applied to selected flows (fraction f r of the trace) - bw as a function of the sending rate of flows to differentiate (bw = k bw * rate) Throttled bandwidth + ICMP rate limitation - router responds at 20 pps max - user replays at 30, 50, 80, 100 pps When up to 60% of traffic is shaped, no flow is incorrectly flagged 80% is too much for baseline to work Similar results as in the first case ICMP rate limitation affects losses, delays are not altered - 13

14 ChkDiff: validation in non-neutral set-up Losses Losses + ICMP rate limitation - uniform drops at rate lr shaped on each flow to differentiate - uniform drops at rate lr all on the whole trace - router responds at 20 pps max - user replays at 30, 50, 80, 100 pps Replaying up to 5x the ICMP rate limitation Results are not significantly altered - 14

15 ChkDiff: download it! Code available on: Runs on Linux, requires tcpdump & tcpreplay Sample output: - 15

16 Thank you - 16

17 ACQUA in a nutshell Model Calibration Phase Controlled experimentation in the lab Application e.g. Skype Vary artificially network performance Ask end users for their feedback Write down QoE Write down QoE and network conditions Model for QoE (Decision Tree, SVM, BayesianNet) Crowdsourcing QoE Estimation/Prediction Phase Measure network performance Model for QoE Expected QoE - 17

18 Network measurements in ACQUA Path-level measurements - Bandwidth, delay and loss, upload and download, Device-level measurements - Signal strength, type of connection, traffic in/out, Measurements inside and from the device Measurements to Landmarks - Measurement servers - Expected QoE per landmark - Statistics of QoE over landmarks - Troubleshooting by landmark elimination Internet - Dozen of landmarks for a good span of QoE - 18

19 Model calibration by controlled experimentation Varying network conditions in ACQUA Space of experimentation can be huge - One dimension per performance metric - Complexity power of the number of metrics A two-layer approach for space sampling - Fourier Amplitude Sensitivity Analysis (FAST) method for a fair coverage of the space and for sample suggestion - Active learning for sample acceptance/rejection (Vowpal Wabbit implementation) Only accepted samples transform into scenarios to experiment with - 19

20 Experimenting with Skype Five measurable path metrics - Bandwidth and loss rate, both upload and download - Round-trip delay QoE = Skype quality meter DummyNet - Four levels - Good, Medium, Poor, No Call Controlled experimental setup - DummyNet at access point - Both ways - Local Skype traffic - Around 600 experiments A local skype call One network configuration (5 values) Controlled experimentation One experiment QoE of Skype (Good, Medium, Poor, No Call) - 20

21 Modeling Skype Quality of Experience A variety of machine learning techniques - Decision Tree, Naïve Bayesian, Lazy learner, Support Vector Machine, etc. Focus on Decision Trees for their readability Performance metrics: Precision and Recall per Quality class - 21

22 Precision Precision Skype QoE prediction accuracy Recall Recall - Best performance for the Medium class, no particular technique outperforming - Almost 70% prediction accuracy (or recall) on average - 22

23 Skype tree sample - 23

24 12kbs a critical rate ARQ/FEC Skype Quality Rules Rule = set of branches from root to leaf 20 rules (after pruning) - Rule 1: Download Bandwidth > 1078, Download Delay <= 94 class Excellent [84.1%] - Rule 2: Upd Bandwidth > 1903, Dwn Bandwidth > 1078 class Excellent [70.7%] - Rule 3: Dwn Bandwidth <= 1078, Dwn Delay <= 665, Upd Loss > 0, Upd Loss <= 2, Dwn Loss > 0, Dwn Loss <= 2 class Excellent [66.2%] - Rule 4: Dwn Bandwidth <= 12 class No Call [90.6%] - Rule 5: Upd Bandwidth <= 14, Upd Loss <= 27 class No Call [75.7%] - Rule 6: Upd Delay <= 506, Upd Loss > 27, Upd Loss <= 46, Dwn Loss > 45 class No Call [61.2%] - Default class: Good Skype can easily deal with one-way losses if bandwidth is available one-way delay up tp 400ms - 24

25 Frequency of quality results - 25

26 Sensitivity analysis (offered by FAST) Participation of each metric to the overall variability of the quality 30% 25% 20% 15% 10% 5% 27% 24% 17% 14% 10% 8% 0% Upd Loss Dwn Loss Upd Bandwidth Dwn Bandwidth Upd Delay Dwn Delay - 26

27 Sampling the space of parameters Fair coverage of the six-dimensional space - With random selection, the probability to pick a corner is as low as 10-6! FAST: Fourier Amplitude Sensitivity Analysis - Virtual time - Each parameter is a sinusoid of virtual time, with different frequency FAST provides sensitivity analysis for free Energy of a parameter = Energy of the corresponding frequency in the output spectrum + its replicas 538 experiments with repetitions - 27

28 Decision Tree Building Chosen for its efficiency, readability and ease of implementation C4.5 algorithm: - Numerical attributes and binary tree - Top down tree building - Start with attributes providing the maximum information gain (best compression of the tree if attribute removed) - Pruning: remove low frequency leafs - Cross validation under-going (less accuracy announced by still acceptable) - 28

29 PlanetLab experiments Dummynet is finally not reality PlanetLab nodes - Real paths different than emulated ones - Metrics unknowns, to be measured PlanetLab-driven path conditions - Tunneling via PlanetLab instead of emulation - Running measurement tools - Almost same accuracy as in the lab - 29

30 The ACQUA application Client in C#, Server in Java - Clients proactive in performing measurements - Servers reactive Measurements - Trains of UDP packets, periodically - From Client to Server***s*** Metrics per path (client <-> server) - Upload Delay, Download Delay, Round Trip Delay - Upload Available Bandwidth, Download Available Bandwidth - Upload Packet Loss Rate, Download Packet Loss Rate An XML file per application modelling its Expected Quality Lines evolving with time and showing - Mean/Variance Measurements over paths - Mean/Variance Expected Quality of Experience - 30

31 ACQUA GUI Augmented with two application profiles: Skype and Video-Like - 31

32 ACQUA GUI Augmented with two application profiles: Skype and Video-Like - 32

33 Concluding remarks A new framework for QoE estimation/prediction starting from network-level measurements Methodology to be applied to other applications as well - Meters might not be present - Reuse existing models for QoE First calibration of models in the lab, then crowd sourcing for refinement Measurements themselves pose lot of problems: - Choice of measurement servers - Choice of metrics - Consideration of new technologies for content delivery (CDN, Cloud, etc) - Overhead of measurements - Collaboration of users and network - 33

34 Network Neutrality A network is neutral if it treats all traffic flows equally No discrimination on - header information (IP src, IP dest, port) - payload (reveals application) Differentiation techniques - Blocking, deprioritizing, packet dropping, modification of TCP advertised window size, application-level mechanisms - 34