It s Not the Cost, It s the Quality! Ion Stoica Conviva Networks and UC Berkeley 1
A Brief History! Fall, 2006: Started Conviva with Hui Zhang (CMU)! Initial goal: use p2p technologies to reduce distribution costs and improve the scale! Slowly, realized our customers (content premium producers & aggregators) value more quality than cost! Today: maximize distribution quality, distribution management, and provide real-time analytics 2
Where is the Data Coming From?! Content Providers and Aggregators! CDNs 3
Trends 4
Trends: CDN Pricing! CDN pricing has decreased x1.5-2 every year over the last 5 year cents/gb 45 40 35 30 25 20 15 10 5 0 2006 2007 2008 2009 2010 5
Trends: Streaming Rate for Premium Content! Average streaming rate has increased 20-40% every year Kbps 1600 1400 1200 1000 800 600 400 200 0 2006 2007 2008 2009 2010 6
Trends: Per-hour Streaming Cost! Per-hour streaming cost has decreased 15-35% every year cents/hour 10 9 8 7 6 5 4 3 2 1 0 2006 2007 2008 2009 2010 7
HTTP Chunking! Trend accelerated by switching from proprietary streaming technologies (e.g., Adobe s FMS) to HTTP Chunking: Move Networks (2005) Apple (2008) Microsoft (2008/2009) Adobe (2010, 2 nd half) 8
How Does HTTP Chunking Work? origin CDN1 CDN2 http cache ISP A ISP B Viewer Viewer Viewer 9
HTTP Chunking Advantages! Chunks: immutable, relative large objects (hundreds of KB) Great for caching! Leverage existing HTTP infrastructure CDNs ISP deployed caches Enterprise http proxies! Low cost and high scale 10
What Does this Mean?! Ad supported premium content CPM (cost per thousand of ad impressions) for premium content has reached: $20-$40 One ad covers one hour of streaming!! Paid content $0.99 episode, distribution cost < 3%! Subscription based premium content Distribution, usually a few percents of total cost It costs $1.6 per month to stream content to an user watching 2 hours per day! Production & rights costs dominate 11
Quality Matters 12
Quality Matters! Better quality Increase viewing time more ad opportunities Increase retention rate Protect brand! Quality Join time Buffering ratio Rendering quality Streaming rate 13
Analysis! Load: Four channels of a premier video-on demand (VoD) content producer Four days Number of sessions (views): 1,176,049 A large live event: ~250,000 concurrent viewers! Metrics Content length distribution Viewer Hour Loss (VHL): number of viewer hours lost due to quality issues
VoD Object Length Distribution 300 250 Short clips: [2min, 3min] Medium clips: [9min, 11min] Full Episodes: [42min, 45min] Number of Objects 200 150 100 50 0 10 40 70 100 130 160 190 220 250 280 310 340 370 400 430 460 490 520 550 580 610 640 680 750 880 1240 1330 2540 2570 2600 2640 4040 5410 5550 Object Length 15
Quality Metrics! Buffering Quality (BQ): PlayingTime/(PlayingTime + BufferingTime) Rendering Quality (RQ): RenderingRate/EncoderRate Good session BQ > 95% RQ > 60%
Analysis Underestimates Quality Impact! For most analysis use BQ only RQ only a small part of quality issues due to low bit rate (500-700Kbps)! Ignore connection failures
Short Clip (2-3min) Analysis 2 1.8 Average Playing Time (Minutes) 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 0 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 18
Short Clip (2-3min) Analysis 300000 250000 Number of Sessions 200000 150000 100000 50000 0 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 19
Viewer Hour Gain! D : Average duration of sessions with high quality (0.98 <= quality < 1)! D q : Average duration of sessions with quality = q! N q : Number of sessions with quality = q! Viewer hour gain for sessions with quality q! Total viewer hour gain N q x (D D q ) q N q x (D D q Viewer hour loss for 1-2 minute ) clips: 20
Medium Clip (9-11min) Analysis 8 7 Average Playing Time (Minutes) 6 5 4 3 2 1 0 Viewer hour loss for 9-11min clips: 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 21
Full Episodes (42-45min) Analysis 25 Average Playing Time (Minutes) 20 15 10 5 0 Viewer hour loss for episodes: 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Viewer hour loss for Quality all content: 22
Large Scale Live Event 35 Average Playing Time (Minutes) 30 25 20 15 10 5 0 Viewer hour loss: 98<=Q<=100 95<=Q<98 90<=Q<95 75<=Q<90 50<=Q<75 0<=Q<50 Quality 23
Large Scale Live Event: Engagement Funnel 1 0.9 0.8 Fraction of Sesions 0.7 0.6 0.5 0.4 0.3 0.2 1 0.99 0.8624 0.7364 0.6602 0.5509 0.4357 0.3639 0<=Q<=0.5 0.5<=Q<=0.75 0.75<=Q<=0.9 0.9 <= Q <= 0.95 0.95<=Q<=0.98 0.98<=Q<=1.0 Quality = 1.0 Continuing 0.1 0 Half people leave due to quality issues All 0-0 0-1 1-3 3-5 5-10 10-20 20-30 Session Duration (Mins) 24
Another Case Study: Live Event 20000 15000 Peak Concurrent Views Quality Engagement Total views 151,980 25 minutes Failed views 13,815 (9%) 0 minutes 10000 5000 0 7:00 PM 7:15 PM 7:30 PM 7:45 PM 8:00 PM 8:15 PM 8:30 PM 8:45 PM 9:00 PM 9:15 PM 9:30 PM 9:45 PM 10:00 PM 10:15 PM 10:30 PM 10:45 PM 11:00 PM 11:15 PM 11:30 PM 11:45 PM 12:00 AM 12:15 AM 12:30 AM 12:45 AM 1:00 AM Total sessions 151,980 Unique viewers 73,942 Sessions per viewer 1.9 Total viewer hours 58,436 Quality impacted views 21,584 (14%) 16 minutes Good views 116,581 (77%) 27 minutes Unique viewers 75,328 48 minutes Failed viewers 1,386 (2%) 0 minutes Quality impacted viewers 14,309 (19%) 30 minutes Good viewers 59,633 (79%) 51 minutes Total viewer hours Lost viewer hours 58,436 hours 5,134 hours (9%) Viewer with poor quality watch 41% less minutes! 25
Does High Bit Rate Video Help?! Comparing Engagement of low and high bitrates Viewers watch longer on average on 1500Kbps minutes 35 30 25 20 15 10 5 0 Average Play Time (minutes) 10-15% increase 500 kbps 1500 kbps
Summary! Quality impact: BQ can impact viewer engagement by up to 40% Higher bit-rates can increase viewer engagement by up to 15%! Engagement loss due to quality issues: between 4 and 30% Even a 4% improvement, may offset distribution costs Ignore other quality issues, like connectivity and media failures 27
Root Cause Analysis 28
Viewers vs. Buffering Quality 250000 200000 150000 100000 50000 0 Light Period 100.00% 90.00% 80.00% 70.00% 60.00% Heavy Period 50.00% 40.00% 30.00% 20.00% 10.00% 0.00% 0:00:00 0:15:00 0:30:00 0:45:00 1:00:00 1:15:00 1:30:00 1:45:00 2:00:00 2:15:00 2:30:00 2:45:00 3:00:00 3:15:00 3:30:00 3:45:00 4:00:00 4:15:00 4:30:00 4:45:00 5:00:00 What causes quality issues? Viewers Average Buffering Quality % 5:15:00 5:30:00 5:45:00 6:00:00 6:15:00 29
Root Cause Analysis Viewer Last Mile + ISP Peering ISP CDN! Root cause a quality issue to: Viewer machine (CPU) Last mile + ISP (Autonomous System Number) CDN! Note: Cannot differentiate between edge and core ISPs Use only passive measurements, no IP traceroute
Metrics and Definitions! Quality metrics Buffering quality (BQ) playing time/(playing time + buffering time) Rendering quality (RQ) rendering frame rate/encoded frame rate! Session classification: Good: (BQ >= 95%) AND (RQ >= 60%) Low BQ: (BQ < 95%) Low RQ: (BQ >= 95%) AND (RQ < 60%)
Methodology: Root Causing Viewer Machine! CPU likely to be the issue when: Rendering quality low Buffering quality high! Conclude CPU is the issue when session s RQ < 60% BQ > 95%
Quality Issues: Light Period Network/CDN issues 4% (CPU issue) 21.5% 74% Good Low BQ Low RQ
Quality Issues: High Period Network/CDN issues 6.2% (CPU issue) 31.5% 62% Good Low BQ Low RQ
Explaining Buffering Issues! Assume buffering quality issues are either due to: CDN, or ISP! Recall: a session has buffering quality issues if BQ < 95%
Methodology: Root Causing CDN (1/2)! Viewers connected to same ASN but using two CDNs! Intuition: if quality experienced by CDN 1 viewers is significantly lower than of CDN 2 viewers for same ASN, CDN 1 has quality issues Last Mile + ISP Peering ISP Peering ISP CDN 1 CDN 2
Methodology: Root Causing CDN (2/2)! Select all ASNs who have more than 50 sessions for each CDN If difference between quality of viewers in CDN1 and CDN2 for same ASN is > 10% Lower quality CDN is root cause at current time
Methodology: Root Causing ASN/ISP! Two CDNs: Conclude ASN A has quality issues if ASN A s viewers connected to either CDN1 or CDN2 experience bad quality Average quality of viewers connected to other ASNs higher! One CDN ASN A s viewers connected to CDN have much lower quality than the average quality of viewers connected to CDN
Buffering Quality: Light Period 5% 16% 33% 46% Unqualified CDN ISP Uknown
Buffering Quality: Heavy Period 15.4% 8.2% 39.7% 36.7% Unqualified CDN ISP Uknown 40
Some Findings! Most of ASNs who had quality issues were enterprise ASNs Expected given that the large scale event was during the workday One ASN had 44% buffering quality!! No CDN was uniformly bad (see next)
CDN Comparison! Quantify quality difference between CDNs! Methodology: 1. Select all ASNs which have more than 50 sessions on both CDNs 2. Compute average quality for CDN1 and CDN2 viewers per ASN 3. Order ASNs by difference in quality between CDN1 and CDN2
Internet delivery is more variable than realized! Content Delivery Networks all have problems sometime! Even in the same viewer session the best quality changed many times during the event CDN 1 was best CDNs were even CDN 2 was best
Summary 24-38% of Total Sessions have Quality Issues Quality Issues Classification CDN (7-12% of total sessions) End-Host CPU (4-6% of total sessions) ISP (2-3% of total sessions) Unqualified (9-11% of total sessions) Unknown (1-4% of total sessions) Solution Resource switching Bit-rate switching Localize traffic, bit-rate switching Mitigated by above N/A
Conclusions! At least for premium content Reducing cost is important, but improving quality is even more important! P2P can play an important role Localize traffic Highly robust to source failures! Great opportunity Adobe has announced full p2p support for Flash Player 10.1 No need for client download! 45