From network-level measurements to expected Quality of Experience. the Skype use case

Transcription

1 From network-level measurements to expected Quality of Experience the Skype use case 2015 IEEE 2015 International IEEE International Workshop Workshop on Measurements on Measurements & Networking & Networking (M&N) (M&N)

2 INTRODUCTION Quality of Experience of Internet Services - 2

3 INTRODUCTION Quality of Experience of Internet Services A measure of user performance based on both objective and subjective psychological measures of using an ICT service or product. [ETSI TR V1.0.1] - 2

4 INTRODUCTION Quality of Experience - 3

5 INTRODUCTION Quality of Experience vs. Network Performance - 3

6 EXAMPLE - 4

7 EXAMPLE hello? icons by: Javier Sánchez - javyliu from the Noun Project - 4

8 EXAMPLE are you there?! icons by: Javier Sánchez - javyliu from the Noun Project - 4

9 EXAMPLE why your connection is always so bad!? icons by: Javier Sánchez - javyliu from the Noun Project - 4

10 EXAMPLE NO! It s not mine! icons by: Javier Sánchez - javyliu from the Noun Project - 4

11 GOAL Predict applications QoE without running them! - 5

12 POSSIBILITIES - study network dimensioning - understanding of application needs - give user the ability to track its connectivity - 6

13 STATE OF THE ART Several contributions in the QoE domain - 7

14 STATE OF THE ART mostly - limited scope / usage - target single applications - having application data - passive measurements - 8

15 STATE OF THE ART nobody targets prediction of QoE - 9

16 STATE OF THE ART nobody aims to relate QoS of connections to the QoE of multiple services - 10

17 OUR PROJECT ACQUA

18 OUR PROJECT ACQUA Monitoring platform for tracking QoE of Internet Services - 11

19 OUR PROJECT starts with active measurements for detecting network conditions at the edges of the Internet Landmark Server Client - 12

20 OUR PROJECT starts with active measurements for detecting network conditions at the edges of the Internet Landmark Server Client - 12

21 OUR PROJECT predicts multiple applications QoE with only one test test result Client - 12

22 OUR PROJECT predicts multiple applications QoE with only one test Youtube Model test result Spotify Model Facebook Model Client - 12

23 OUR PROJECT predicts multiple applications QoE with only one test test result Youtube Model Spotify Model Facebook Model Client - 12

24 CHALLENGES - find the right metrics for prediction - define a robust methodology to collect data and create models (has to be applied on several services) - deploy the framework - 13

25 METHODOLOGY Data Analytics - 14

26 METHODOLOGY - create and analyze a data set - see how to collect such data in real world scenarios - compare performances of different classifiers derived from such data set - 15

27 THIS CONTRIBUTION focus on audio calls - 16

28 WHY SKYPE? Skype gives us a feedback about the QoE during each call - 17

29 WHY SKYPE? Skype gives us a feedback about the QoE during each call - 17

30 WHY SKYPE? Skype gives us a feedback about the QoE during each call Basically we will reverse engineer Skype QoE model - 17

31 DATA SET Measures obtained in a controlled environment collecting Skype s application feedback given - latency - throughput - packet loss Image from: Marta Carbone, Luigi Rizzo, An emulation tool for PlanetLab, Computer Communications, Elsevier, Oct.2011, doi: /j.comcom

32 DATA SET QoE label Measures obtained in a controlled environment collecting Skype s application feedback given - latency - throughput - packet loss Image from: Marta Carbone, Luigi Rizzo, An emulation tool for PlanetLab, Computer Communications, Elsevier, Oct.2011, doi: /j.comcom

33 DATA SET QoE label Measures obtained in a controlled environment collecting Skype s application feedback given - latency - throughput - packet loss Static Conditions Image from: Marta Carbone, Luigi Rizzo, An emulation tool for PlanetLab, Computer Communications, Elsevier, Oct.2011, doi: /j.comcom

34 DATA SET Composed by 5 input metrics + 1 output QoE label - Round Trip Time (RTT) - passing throughput - packet loss rate - 19

35 DATA SET Composed by 5 input metrics + 1 output QoE label - Round Trip Time (RTT) - passing throughput - packet loss rate } upload and download - 19

36 DATA SET Composed by 5 input metrics + 1 output QoE label - Round Trip Time (RTT) - passing throughput - packet loss rate } upload and download - QoE label {NoCall, Poor, Medium, Good} - 19

37 DATA SET FAST sampling (Fourier Amplitude Sensitivity Testing) to cover the configuration space uniformly - 20

38 DATA SET (training set) 538 entries (training) - 21

39 DATA SET (training set) 538 entries (training) entries (validation) - 21

40 MODEL GENERATION Data-Driven Approach - compare different ML techniques using the Weka toolkit - check classifier - performances using independent test set - stability using Cross Validation - 22

41 HOW A CLASSIFIERS WORKS? ML algorithm - 23

42 HOW A CLASSIFIERS WORKS? Step 1: calibration Training Data ML algorithm Classifier - 23

43 HOW A CLASSIFIERS WORKS? Step 2: validation Test Data Test Results??? fail ok ok Classifier - 24

44 HOW A CLASSIFIERS WORKS? Step 3: deploy Input Output Classifier - 25

45 CONSIDERED TECHNIQUES a classifier for each different family - Decision Trees (C4.5) and Rule Inference (JRip, FURIA) - Lazy Learners (knn) - Probability based (Naive Bayes, Neural Networks) - Support Vector Machines (SVM) - Random Forests - Meta Techniques (Boosting, Bagging) - 26

46 PERFORMANCE METRICS How to consider the effectiveness of a classifier? - 27

47 PERFORMANCE METRICS How to consider the effectiveness of a classifier? ACCURACY = #Correctly Classified Instances #Instances - 27

48 PERFORMANCE METRICS - 28

49 PERFORMANCE METRICS Classifying all the instances as medium - 28

50 PERFORMANCE METRICS Classifying all the instances as medium over 60% accuracy - 28

51 PERFORMANCE METRICS Classifying all the instances as medium over 60% accuracy Not enough to have a global overview - 28

52 PERFORMANCE METRICS How to consider the effectiveness of a classifier? - Accuracy - 29

53 PERFORMANCE METRICS How to consider the effectiveness of a classifier? - Accuracy - 29

54 PERFORMANCE METRICS How to consider the effectiveness of a classifier? - Accuracy - Precision / Recall for each class - 29

55 PERFORMANCE METRICS RECALL (completeness) True Positives (TP) Instances of D C - 30

56 PERFORMANCE METRICS PRECISION (quality) True Positives (TP) Number of Instances classified as C - 31

57 Recall INTERPRETATION C 1 A dot represents a classifier Precision - 32

58 Recall INTERPRETATION C 1 All instances of C correctly classified A dot represents a classifier with low errors Precision - 32

59 Recall INTERPRETATION No instances of C correctly classified C 2 Precision - 32

60 Recall INTERPRETATION overestimation of C C 3 most instances of C correctly classified with high error Precision - 32

61 Recall INTERPRETATION Few instances of C recognized correctly but without errors C 4 Precision - 32

62 RESULTS - Cross Validation NoCall Poor Medium Good - 33

63 Recall RESULTS - Cross Validation Precision - 33

64 RESULTS - Cross Validation - 33

65 RESULTS - Validation - 34

66 RESULTS - Validation excellent results - 34

67 RESULTS - Validation lower performances - 34

68 CAN WE GET MORE? - 35

69 CAN WE GET MORE? Unbalanced training set could affect classification - 35

70 CAN WE GET MORE? Unbalanced training set inflate data to reduce bias could affect classification - 35

71 CAN WE GET MORE? Unbalanced training set could affect classification inflate data to reduce bias shrink data to reduce bias - 35

72 CAN WE GET MORE? Unbalanced training set could affect classification inflate data to reduce bias shrink data to reduce bias WORSE RESULTS - 35

73 CAN WE GET MORE? Data Set could be too small - 36

74 CAN WE GET MORE? Data Set could be inflate data too small - 36

75 CAN WE GET MORE? Data Set could be inflate data too small WORSE RESULTS - 36

76 FINAL ACCURACY about 70% of accuracy in general - 37

77 CLUSTERED RESULTS no outstanding performances of a single classifier - 38

78 LIMITED ERROR SPREADING 93% of instances classified correctly or in the adjacent classes (C4.5 classifier) - 39

79 ANALYSIS 12kbps minimum bandwidth to start a call - 40

80 ANALYSIS up to 800ms of RTT do not affect quality - 41

81 FUTURE WORKS Enrich our data set with crowd based feedbacks (Android application under development) Currently applying an extended methodology to the video streaming domain - 42

82 FUTURE WORKS Deploy ACQUA framework - 43

83 THANK YOU Title From network-level measurements to expected Quality of Experience: the Skype use case Authors Thierry Spetebroot (INRIA Sophia Antipolis, France) Salim Afra (University of Calgary, Canada) Nicolás Aguilera (NIC Chile Research Labs, Chile) Damien Saucez (INRIA Sophia Antipolis, France) Chadi Barakat (INRIA Sophia Antipolis, France)

84 BACKUP SLIDES - 84

85 DATA SET QoE label Measures obtained in a controlled environment collecting Skype s application feedback given - latency - bandwidth - packet loss Image from: Marta Carbone, Luigi Rizzo, An emulation tool for PlanetLab, Computer Communications, Elsevier, Oct.2011, doi: /j.comcom

86 DATA SET Composed by 6 input metrics + 1 output QoE label - latency - bandwidth - packet loss rate } upload and download - QoE label {NoCall, Poor, Medium, Good} - 86

87 MEASUREMENT FEASIBILITY In real world is difficult to measure - One Way Delays clock synchronization is required - Link Capacity due to link level losses - 87

88 DATA SET PREPARATION Adaptation of - One Way Delays Round Trip Time (RTT) merging both delays - Link Capacity Passing Throughput convolution of bandwidth with packet loss rate - 88

89 FINAL DATA SET Composed by 5 input metrics + 1 output QoE label - Round Trip Time (RTT) - passing throughput - packet loss rate - QoE label - 89

90 TRAINING SET - 90

91 TEST SET 100 random instances - 91

92 RESULTS - Cross Validation Recall 60% Poor results in 3 cases... NoCall Poor Good - 92

93 RESULTS - Cross Validation Only cluster with excellent results - 93

94 WHY? It could be due to the Data Set preparation process... do we have information loss? - 94

95 INFORMATION LOSS - 95

96 INFORMATION LOSS Same results... data set transformations don t affect the results - 96

97 WHY? It could be a Class Imbalance Problem Where we have better results we have considerably more data - 97

98 CLASS IMBALANCE PROBLEM Unbalanced training set could affect classification inflate data to reduce bias - replication - interpolation - 98

99 CLASS IMBALANCE PROBLEM Worse results... see all algorithms - 99

100 CLASS IMBALANCE PROBLEM Worse results... back - 100

101 CONCLUSIONS Limited Error Spreading 93% of instances classified correctly or in the adjacent class (C4.5 classifier) - 101

102 X DRAFTS - 102