Performance Characterization of a Commercial Video Streaming Service

Transcription

1

2 Performance Characterization of a Commercial Video Streaming Service Mojgan Ghasemi, Princeton University P. Kanuparthy, 1 A. Mansy, 1 T. Benson, 2 J. Rexford 3 1 Yahoo, 2 Duke University, 3 Princeton University 1

3 First study to measure both sides Video makes up 70% of the tra c! 2

4 Yahoo s Video Streaming System

5 Yahoo s Video Streaming System Client receives the manifest manifest Player Network CDN 3

6 Yahoo s Video Streaming System HTTP requests for chunks share a TCP connection Each chunk is 6 seconds HTTP requests Player Network CDN 3

7 Yahoo s Video Streaming System CDN servers use Apache Tra cserver(ats),lrupolicy Chunk Player Network CDN Backend 3

8 Yahoo s Video Streaming System Chunks pass client s download and rendering stack Chunk Player Network CDN Backend 3

9 Our Dataset: Yahoo Videos

10 Yahoo Videos 4

11 Our Dataset VoD Dataset: Over 18 days, Sept CDN servers across the US 65 million VoD sessions, 523m chunks Users: Non-mobile users, no proxy Predominantly in North America (over 93%) Video Streams: Popularity: 66% of requests for 10% of titles Duration: most videos less than 100 sec 5

12 Our Goal Identify performance problems that impact video Player Network CDN 6

13 Our Goal Identify performance problems that impact video Player Network CDN 6

14 Our Goal Identify performance problems that impact video Player Network CDN A content provider (e.g., Yahoo) controls both sides 6

15 Our Approach: e2e Per-chunk Measurement End-to-end Instrumenting both sides (player, CDN servers) Per-chunk Unit of decision making (e.g., bitrate, cache hit/miss) Sub-chunk is too expensive TCP statistics Sampled from CDN host s kernel Operational at scale 7

16 Our Approach: e2e Per-chunk Measurement Player OS CDN Backend WAN 8

17 Our Approach: e2e Per-chunk Measurement Player OS CDN Backend WAN 8

18 Our Approach: e2e Per-chunk Measurement Player OS WAN CDN Backend HTTP Get 8

19 Our Approach: e2e Per-chunk Measurement Player OS WAN CDN Backend HTTP Get Cache miss D FB D CDN + D BE D DS 8

20 Our Approach: e2e Per-chunk Measurement Player OS WAN CDN Backend HTTP Get Cache miss D FB D CDN + D BE D DS D LB 8

21 Studying QoE Factors Individually Factors: Video startup time Rebu ering rate Video quality (bitrate, framerate) We look at individual metrics, because: Type of content Length of video 9

22 Outline Introduction Measurement Dataset Server-side Problems Network Performance Problems Client s Performance Problems Take-aways and Conclusions 10

23 Server-side Performance Problems

24 Monitoring CDN Performance Direct measurement Player Session ID Chunk ID Startup time Re-buffering Video quality CDN Session ID Chunk ID Server latency (D CDN ) Backend latency (D BE ) Cache hit/miss 11

25 Monitoring CDN Performance Direct measurement Player Session ID Chunk ID Startup time Re-buffering Video quality CDN Session ID Chunk ID Server latency (D CDN ) Backend latency (D BE ) Cache hit/miss 11

26 Impact of CDN on Startup Time Only possible via data from both ends Startup time vs. server latency in first chunk 12

27 1. Cache Misses request memory disk backend Cache misses increase server latency 40X median, 10X average Server latency can be worse than network Caused by cache misses (40% miss rate) 13

28 2. Persistent Problems in Unpopular Videos Cache misses are persistent: Average: 2% After one miss: 60% Unpopular titles have significantly higher cache misses 14

29 Network Performance Problems

30 Network Measurement CDN s host kernel Polling tcp_info Every 500ms per-chunk tcp_info maintained by OS: weighted average of RTT (SRTT) congestion window packet retransmissions Challenges: Smoothed average of RTT: SRTT Infrequent network snapshots Packet traces cannot be collected 15

31 1. Network Latency Problems Persistent high latency: /24 IP prefixes, recurring in 90 th percentile 25% of prefixes are located in the US, with the majority close to CDN nodes High latency variation: Enterprise networks have higher latency variation 16

32 2. Earlier Packet Losses Cause More Rebu ering Packet loss is more common in the first chunk (4.5X) Packet loss in the first chunk causes more rebu ering 17

33 3. Throughput is a Bigger Problem than Latency perf score = chunk duration D FB + D LB D FB : measure of latency, D LB : measure of throughput Player OS WAN CDN Backend HTTP Get Cache miss D FB D CDN + D BE D DS D LB 18

34 3. Throughput is a Bigger Problem than Latency perf score = chunk duration D FB + D LB D FB : measure of latency, D LB : measure of throughput perf score > 1:Morethan1secofvideodeliveredpersec perf score < 1:Lessthan1secofvideopersec 18

35 3. Throughput is a Bigger Problem than Latency perf score = chunk duration D FB + D LB D FB : measure of latency, D LB : measure of throughput perf score > 1:Morethan1secofvideodeliveredpersec perf score < 1:Lessthan1secofvideopersec D LB has a major contribution (orders of magnitude) 18

36 Client s Download Stack Performance Problems

37 Download Stack Latency Network NIC Download Stack OS Browser Player Cannot observe download stack latency (D DS )directly Detecting outliers D FBi >µ DFB +2 D FB 19

38 Download Stack Latency Network NIC Download Stack OS Browser Player Cannot observe download stack latency (D DS )directly Detecting outliers D FBi >µ DFB +2 D FB TP insti >µ TPinst +2 TP inst 19

39 Download Stack Latency Network NIC Download Stack OS Browser Player Cannot observe download stack latency (D DS )directly Detecting outliers D FBi >µ DFB +2 D FB TP insti >µ TPinst +2 TP inst Similar network and server performance 19

40 LDtency (ms) Download Stack Latency: Case Study srtt server first-byte delay Chunk ID 100 connection TP download TP 7P (0bps) Chunk ID 21 20

41 Client s Download Stack Problems Transient: Outlier: 1.7M chunks (0.32%) First chunks have higher D DS Persistent: In most cases, D DS is higher than network and server latency 21

42 Client s Rendering Stack Performance Problems

43 Rendering Stack Player Demux (audio/video) Rendering Stack (CPU or GPU) Decode Render Screen If CPU is busy, rendering quality drops (high frame drops) If video tab is not visible, browser drops frames Per-chunk data: vis (is player visible?), dropped frames Per-session data: OS, browser 22

44 1. Good Rendering Requires 1.5 sec sec Download Rate De-multiplexing, decoding, and rendering takes time. % Dropped FrDmes average median DownloDd rdte of Fhunk, sec sec 5 23

45 2. Higher Bitrates Show Better Rendering Paradox: Higher bitrates put more load on the CPU Showed better rendering framerate Higher bitrates are often requested in connections: Lower RTT variation Lower retransmission rate 24

46 3. Unpopular Browsers Have Worse Rendering Chunks with good performance (rate > 1.5 sec sec ) Player is visible (i.e., vis = true) 25

47 3. Unpopular Browsers Have Worse Rendering Chunks with good performance (rate > 1.5 sec sec ) Player is visible (i.e., vis = true) % Fhunks in SOaWIorP % drossed IraPes 3erFenWage ChroPe IE FireIox Edge 2Wher 6aIari ChroPe FireIox 2Wher Windows 0aF 25

48 Take-aways

49 Take-aways: CDN Problem Cache miss impact Cache miss persistence Take-away Cache-eviction policy Pre-fetch subsequent chunks 26

50 Take-aways: Network Problem Nearby clients with high latency Prefixes with persistent high latency or variation Throughput the major bottleneck Take-away Avoid over provisioning servers for nearby clients Adjust ABR algorithm accordingly (more conservative bitrate, increase buffer size) Good news for ISPs (e.g., establish more peering points) 27

51 Take-aways: Client Problem Download stack latency Rendering is resourceheavy Take-away Can cause over-shooting or under-shooting by ABR, incorporate server-side TCP metrics Use 1.5!"# video arrival!"# rate as a rule-of-thumb 28

52 Conclusion Instrumenting both sides Uncover range of problems for the first time Per-chunk and per-session data Uncover persistent vs. transient problems Our findings have been used to enhance performance in Yahoo 29

53 Thank You!

54 Network Problems Impact QoE Data from both sides show the impact Startup time vs. SRTT of first chunk Network latency significantly impacts video startup time 30