Improving Video Quality by Information Sharing!! Ion Stoica, CTO! (also at UC Berkeley and Databricks)!!

Size: px

Start display at page:

Download "Improving Video Quality by Information Sharing!! Ion Stoica, CTO! (also at UC Berkeley and Databricks)!!"

Robert Fields
6 years ago
Views:

1 Improving Video Quality by Information Sharing!! Ion Stoica, CTO! (also at UC Berkeley and Databricks)!!

2 Across many sites We ve seen 3 patterns While monitoring more than 4 Billion streams per month

3 1 Viewers watch more when video is not interrupted 28 UGC TV Channel Sport Aggregator % more % more % more % more * June 2012 * June 2012 Buffering Impacted Views (BIV): > 2% buffering or > 5s continuous buffering non-biv BIV

4 2 Viewers watch more when video is higher definition TV Channel Pre. Channel TV Channel News 89 46% more % more % more % more * July 2012 * June 2012 Live >1 Mbps <1 Mbps

5 3 Viewers leave and do not return to sites when video fails to start Viewers who experience a single video start-up failure return 54% less

Every step of the delivery chain presents unique

and high bit-rate Encoders CDNs ISPs Devices Inaccurate

resolution to bit-rates Wide variation in CDN

traffic and time of day Great diversity in access link

) Heterogeneity of screen sizes and rendering

6 Every step of the delivery chain presents unique challenges to delivering video with low interruptions and high bit-rate Encoders CDNs ISPs Devices Inaccurate choice of bit-rates & poor matching of screen sizes & resolution to bit-rates Wide variation in CDN performance across various ISPs with peak load, type of traffic and time of day Great diversity in access link speeds (fiber, cable, dsl, 3G/Wi-fi, corporate networks ) Heterogeneity of screen sizes and rendering performance across PCs, mobile hand-sets, tablets, game consoles, set top boxes, connected TVs

from each CDN DMA: Designated Market Area Percentage of DMA--hour partitions a CDN has lowest

7 CDNs Vary in Performance over Geographies and Time Metric: buffering ratio One month aggregated data-set (2011) Multiple Flash (RTMP) customers Three major CDNs 31,744 DMA--hour with > 100 streams from each CDN DMA: Designated Market Area Percentage of DMA--hour partitions a CDN has lowest buffering ratio CDN 1 There is no single best CDN across geographies, network, and time 50% CDN 2 25% 25% CDN 3

8 Washington, DC viewer experience differed greatly Comcast viewers got the best streams from CDN 1 51% of the time and only 9% from CDN 2 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Washington DC (Hagerstown): -CXA-ALL Washington DC (Hagerstown): VZGNI-TRANSIT (19262) Verizon users got the best streams from CDN 1 only 17% of the time and 77% from CDN 2 There is no single best CDN in the same geographic region or over time

9 CDN Streaming Failures Are Common Events " % of stream failures: % of streams with video start failures " Three months dataset (May-July, 2011) for a premium customer using Flash CDN (relative) performance varies greatly over time

10 Performance Changes Minute-by-minute September 24 th, 2013 (1 minute data- points over 24 hours) CDN1 CDN2 Buffering Ra5o Video Start Failures Average Bitrate " For the same ISP, CDN performance varies minute by minute over the course of a day Different quality metrics no always correlated

11 Conviva Approach to Optimize Viewer Experience Viewer Centric Position " True end-point sensor to see what the viewer sees and inform the source in real time

Real-time Measurement from Every Viewer Player States Joining Playing Buffering Playing Events time Network/ stream connection established

12 Real-time Measurement from Every Viewer Player States Joining Playing Buffering Playing Events time Network/ stream connection established JoinTime (JT) Video buffer filled up BufferingRa2o(BR) RateOfBuffering(RB) Video buffer empty Buffer replenished sufficiently Stopped/ Exit User action

source in real time " Adjustments to streams every second to account for local environment and

13 Conviva Approach to Optimize Viewer Experience Viewer Centric Position Continuous Optimization Global View and Policy Control " True end-point sensor to see what the viewer sees and inform the source in real time " Adjustments to streams every second to account for local environment and Internet variables " Dynamic policy control based on real-time patterns across viewers, affiliates, and networks

14 IP Video Streaming Architecture Business Policy Input Conviva Pla.orm Real- 5me data ingest and processing Real- 5me metric computa5on Real- 5me global op5miza5on Big- data analysis infrastructure Real- 5me Feedback/ac5on Real- 5me Heartbeats Pulse: Real- 5me Visibility for publishers Conviva Team PMC: Real- 5me Visibility for CDNs Offline Analysis and Recommenda5ons CDN Team Content Publisher Team CDN 1 HDS/HLS Fragments Mezzanine Crea5on / Transcoding CMS / Origin Storage CDNs: Origin storage HDS fragmenta5on HLS fragmenta5on SmoothStreaming CDN 2 HLS Fragments

15 IP Video Streaming Architecture Business Policy Input Conviva Pla.orm Real- 5me data ingest and processing Real- 5me metric computa5on Real- 5me global op5miza5on Big- data analysis infrastructure Real- 5me Feedback/ac5on Real- 5me Heartbeats Pulse: Real- 5me Visibility for publishers Conviva Team PMC: Real- 5me Visibility for CDNs Offline Analysis and Recommenda5ons CDN Team Content Publisher Team CDN 1 HDS/HLS Fragments Mezzanine Crea5on / Transcoding CMS / Origin Storage CDNs: Origin storage HDS fragmenta5on HLS fragmenta5on SmoothStreaming CDN 2 HLS Fragments

16 Conviva Platform Con5nuous real- 5me measurements from every client Real- 5me Global Data Aggrega5on and Correla5on (Streaming Map- reduce) Real- 5me Alerts Real- 5me and historical Insights Historical Data Aggrega5on and Analysis (Hadoop+Hive+Spark) Global Inference, Decision & Policy Engine Real- 5me global op5miza5ons Localize issues by region, network, CDN, and 5me Inference & Predic5on Engine Decision Engine DMA Akamai Level3 DMA DMA Limelight Time of day CDNs Bit Rates Op5mize viewer performance by selec5ng the best op5on within the set of bit rates and CDNs

17 Conviva Platform Con5nuous real- 5me measurements from every client Real- 5me Global Data Aggrega5on and Correla5on (Streaming Map- reduce) Real- 5me Alerts Real- 5me and historical Insights Historical Data Aggrega5on and Analysis (Hadoop+Hive+Spark) Global Inference, Decision & Policy Engine Real- 5me global op5miza5ons Localize issues by region, network, CDN, and 5me Inference & Predic5on Engine Decision Engine DMA Akamai Level3 DMA DMA Limelight Time of day CDNs Bit Rates Op5mize viewer performance by selec5ng the best op5on within the set of bit rates and CDNs

18 Key Idea behind Inference & Prediction Engine Share quality information across views Use quality information from existing views to predict performance of new viewers at join time performance of existing views if we were to switch bitrate or CDN Create a model consisting of a set of decision tables Reactively adapting after failure too late!

19 Inference and Decision Engines Update decision tables every 1min (5-8 sec processing) Make decisions in constant time (<1ms) Inference & Prediction Engine Decision Tables Spark Cluster Decision Engine

20 Spark In-memory computation engine APIs in Scala, Python, Java Unifies batch, streaming, interactive computations Powerful machine learning (MLlib) and soon graph (GraphX) libraries Used by tens of companies including Yahoo!, Intel, Streaming Interactive Sophisticated algos. Iterative Spark Streaming Batch, Interactive Shark SQL Spark MLlib GraphX

(3G, 4G, wifi) Geo (DMA) Protocol Player version Etc. Quality (e.g.

21 Use Case 1: Best Starting Bit Rate for Single CDN Pick the best starting bit rate based on decision tables Device Connection type (3G, 4G, wifi) Geo (DMA) Protocol Player version Etc. Quality (e.g., buf ra5o) of desktop users in [1] x rate[1] Desktop ipad iphone rate rate rate

22 Use Case 1: Best Starting Bit Rate for Single CDN Pick the best starting bit rate based on decision tables Device Connection type (3G, 4G, wifi) Geo (DMA) Protocol Player version Etc. For an ipad in [1] select highest bit rate providing good quality (i.e., rate[3]) Desktop ipad iphone rate rate rate

23 Use Case 2: Best Starting CDN for Multi-CDNs Pick the best CDN based on decision tables Device Connection type (3G, 4G, wifi) Geo (DMA) Protocol Player version Etc. Desktop ipad iphone DMA DMA DMA CDN1 Desktop ipad iphone DMA DMA DMA CDN2

24 Use Case 2: Best Starting CDN for Multi-CDN Pick the best CDN based on decision tables Device Connection type (3G, 4G, wifi) Geo (DMA) Protocol Player version Etc. Desktop ipad Assign viewers in [2]xDMA[2] iphone to CDN1 DMA DMA DMA CDN1 Desktop ipad CDN1 >> CDN2 for [2]xDMA[2] iphone DMA DMA DMA CDN2

Use Case 3: CDN Switch on Quality Degradation CDN1 CDN2 Desktop

Buffer Length Trend 30 20 10 0 Detect and track buffer drop and

centric predic=ve algorithms track recent trends in available

quality problems Global Inference & Predic=on algorithms track

25 Use Case 3: CDN Switch on Quality Degradation CDN1 CDN2 Desktop DMA Desktop DMA 3000 Available Bandwidth Trend Buffer Length Trend Detect and track buffer drop and bandwidth trends and take ac5on to prevent buffering Client centric predic=ve algorithms track recent trends in available bandwidth, buffer length, rendering performance and predict quality problems Global Inference & Predic=on algorithms track global trends by geography, network (), CDN iden5fy anomalies and predict quality of views

26 Use Case 4: Asset Publishing and Caching Issues CDN1 DMA % Video Start Failures Asset CDN2 DMA Asset Asset Global inference algorithms track individual asset failures by device geography, network (), CDN and iden5fy any regional anomalies Asset CDN3 DMA

27 Use Case 5: ISP Saturation Bandwidth Fluctua5on CDN1 CDN2 DMA DMA Peak Concurrent Viewers CDN3 DMA Network Satura5on Point This /DMA is saturated on all three CDNs è Don t switch CDN. Reduce bit rates and maintain

28 Use Case 5: ISP Saturation Bandwidth Fluctua5on CDN1 CDN2 DMA DMA Peak Concurrent Viewers CDN3 DMA Network Satura5on Point This /DMA is saturated on all three CDNs è Don t switch CDN. Reduce bit rates and maintain

29 Use Case 5: ISP Saturation Bandwidth Fluctua5on Bandwidth Fluctua5on Peak Concurrent Viewers Network Satura5on Point Unsaturated Network Peak Concurrent Viewers CDN1 CDN2 CDN3 DMA DMA DMA This /DMA is saturated on all three CDNs è Don t switch CDN. Reduce bit rates and maintain

30 Challenges " What happens if a partition doesn t has enough data? 1) Spatial aggregation Which partition use for prediction? DMA? CDN1 (buffering rabo)

31 Challenges " What happens if a partition doesn t has enough data? 1) Spatial aggregation Which partition use for prediction? 2) Temporal aggregation (increase window) What window size? DMA? CDN1 (buffering rabo) DMA CDN1 (buffering rabo)

32 Example 1: CDN and Bit Rate Selection

33 Example 1: Impact on CDN Usage " 7922 (Comcast) for all of Aug 19 th Metric Buffering Ra=o Avg. Bitrate Video failures CDN % 2270 K 9.3% CDN % 2668 K 8.12% CDN % % Precision picks CDN2 64% of the 5me " (AT&T Wireless) for all of Aug 19th Metric Buffering Ra=o Avg. Bitrate Video failures CDN % 1832 K 10.7% CDN % 1874 K 11.0% CDN % 1830 K 9.6% Precision picks CDN1 & CDN3 74% of the 5me

34 Example 1: Aggregate over 5 Days Metric Non- Precision Precision Precision Improvement Buffering Ra5o 2.3 % 1.5 % 33% Average Bitrate 1692 K 2287 K 35% Failures and Exits 11.5% 10.6 % 8% Buffering Impacted Views 13.4% 9.4 % 30%

35 Example 2: Preserving Quality in Presence of Failures! " Precision ensures that audience quality is NOT impacted by CDN failures " Content brands and audience are protected " Content owners can be more aggressive in using capacity from CDN vendors With Precision Video, CDN problem has no effect on viewers Without Precision Video, CDN problem has big effect on viewers

36 Summary " Key transition of main-stream video to the Internet " Video quality presents opportunity and challenge " Premium video on big screens à zero tolerance for poor quality " Ability to infer and predict viewer quality key to maximize quality perceived by users " Reacting after the fact too late!

38 Conviva Precision Protects Brands, Audience & CDN! " Precision ensures that audience quality is NOT impacted by CDN failures " Content brands and audience are protected " Content owners can be more aggressive in using capacity from CDN vendors With Precision Video, CDN problem has no effect on viewers Without Precision Video, CDN problem has big effect on viewers

39 Conviva Precision: Simultaneously Increasing Resolution & Reducing Buffering 100 Highest Quality/Most Engaged Audience Uninterrupted Views (%) (before Precision) (aker Precision) Other VoD Entertainment Competitors Average Bitrate (kbps)

40 With Conviva Precision, Viewers Watch More, Come Back More Often

41 Lift with Precision 25.00% 20.00% 15.00% 10.00% 5.00% 0.00% Views Impacted by Buffering Reduced views! impacted by buffering! from 16.13% to 5.56% Average Bit Rate Increased average bitrate! from 1.7 mbps to 2.1 mbps First Full Month Results Views Uniques Audience Viewed Minutes Minutes per View Minutes per Unique 19% 15% 36% 14% 18% Raised engagement by 36%

42 The Truth " Video delivery over the internet is hard CDN variability makes it nearly impossible to deliver high quality all the time with just one CDN SIGCOMM 12

43 The Idea " Where there is heterogeneity, there is room for optimization " For each viewer we want to decide what CDN to stream from " But it s difficult to model the internet, and things can rapidly change over time " So we will make this decision based on the real-time data that we collect

45 The Truth " Video delivery over the internet is hard CDN variability makes it nearly impossible to deliver high quality everywhere with just one CDN SIGCOMM 12

46 The Truth " Video delivery over the internet is hard CDN variability makes it nearly impossible to deliver high quality all the time with just one CDN SIGCOMM 12

47 The Idea " Where there is heterogeneity, there is room for optimization " For each viewer we want to decide what CDN to stream from " But it s difficult to model the internet, and things can rapidly change over time " So we will make this decision based on the real-time data that we collect

48 The Idea " For each CDN, partition clients by City " For each partition compute Buffering Ratio Seall e San Francisco New York Londo n Hong Kong Avg. buff ra5o of users in CDN1 (buffering rabo) Region[1] streaming from CDN1 CDN2 Avg. (buffering buff ra5o rabo) of users in Region[1] streaming from CDN2

49 The Idea " For each partition select best CDN and send clients to this CDN Seall e San Francisco New York Londo n Hong Kong CDN1 (buffering rabo) CDN2 (buffering rabo)

50 The Idea " For each partition select best CDN and send clients to this CDN Seall e San Francisco New York Londo n Hong Kong CDN2 >> CDN1 CDN1 (buffering rabo) CDN2 (buffering rabo)

51 The Idea " For each partition select best CDN and send clients to this CDN Seall e San Francisco New York Londo n Hong Kong CDN1 (buffering rabo) CDN2 (buffering rabo)

52 The Idea " For each partition select best CDN and send clients to this CDN Seall e San Francisco New York Londo n Hong Kong CDN1 (buffering rabo) Best CDN (buffering rabo) CDN2 (buffering rabo)

53 The Idea " What if there are changes in performance? Seall e San Francisco New York Londo n Hong Kong CDN1 (buffering rabo) Best CDN (buffering rabo) CDN2 (buffering rabo)

54 The Idea " Use online algorithm respond to changes in the network. Seall e San Francisco New York Londo n Hong Kong CDN1 (buffering rabo) Best CDN (buffering rabo) CDN2 (buffering rabo)

How? " Coordinator implementing an optimization

client based on Individual client Aggregate

realtime data Business Policies Content owners (CMS &

55 How? " Coordinator implementing an optimization algorithm that dynamically selects a CDN for each client based on Individual client Aggregate statistics Content owner policies " All based on realtime data Business Policies Content owners (CMS & Origin) Coordinator CDN 1 CDN 2 CDN 3 control Clients Con5nuous measurements

56 What processing framework do we use? " Twitter Storm Fault tolerance model affects data accuracy Non-deterministic streaming model " Roll our own Too complex No need to reinvent the wheel " Spark Easily integrates with existing Hadoop architecture Flexible, simple data model Writing map() is generally easier than writing update()

57 Spark Usage for Production Spark Cluster Spark Node Spark Node Spark Node Spark Node Storage Layer Spark Node Decision Maker Decision Maker Decision Maker " Compute performance metrics in spark cluster " Relay performance information to decision makers Clients

58 Results Non- buffering views 100% 95% 90% 85% 80% 75% Average Bitrate (Kbps)

59 Spark s Role " Spark development was incredibly rapid, aided both by its excellent programming interface and highly active community " Expressive: Develop complex on-line ML decision based algorithm in ~1000 lines of code Easy to prototype various algorithms " It has made scalability a far more manageable problem " After initial teething problems, we have been running Spark in a production environment reliably for several months.

60 Problems we faced " Silent crashes " Often difficult to debug, requiring some tribal knowledge " Difficult configuration parameters, with sometimes inexplicable results " Fundamental understanding of underlying data model was essential to writing effective, stable spark programs

61 Enforcing constraints on optimization " Imagine swapping clients until an optimal solution is reached CDN1 (buffering rabo) 50% Constrained Best CDN (buffering rabo) CDN2 (buffering rabo) 50%

62 Enforcing constraints on top of optimization " Solution is found after clients have already joined. " Therefore we need to parameterize solution to clients already seen for online use. " Need to compute an LP on real time data " Spark Supported it 20 LPs Each with 4000 decisions variables and 350 constraints 5 seconds.

63 Tuning " Can t select a CDN based solely on one metric. Select utility functions that best predict engagement " Confidence in a decision, or evaluation will depend on how much data we have collected Need to tune time window Select different attributes for separation

64 Tuning " Need to validate algorithm changes quickly " Simulation of algorithm offline, is essential

Spark Usage for Simulation HDFS with Produc5on traces Spark Node Spark Node Spark Node Spark Node Spark Node " Load production traces with randomized initial

65 Spark Usage for Simulation HDFS with Produc5on traces Spark Node Spark Node Spark Node Spark Node Spark Node " Load production traces with randomized initial decisions " Generate decision table (with artificial delay) " Produce simulated decision set " Evaluate decisions against actual traces to estimate expected quality improvement

66 Future of Spark and Conviva " Leverage spark streaming " Unify live and historical processing " Develop platform to build various processing apps (e.g. Anomaly Detection, Customer Tailored Reporting) Can share the same data API Will all have consistent input

67 In Summary " Spark was able to support our initial requirement of fast fault tolerant performance computation for an on-line decision maker " New complexities like LP calculation just worked in the existing architecture " Spark has become an essential tool in our software stack

68 Real-time Measurement from Every Viewer Player States Joining Playing Buffering Playing Events time Network/ stream connection established JoinTime (JT) Video buffer filled up BufferingRa2o(BR) RateOfBuffering(RB) Video buffer empty Buffer replenished sufficiently Stopped/ Exit User action

69 An UK ISP: Top Metros by Metric over time Greater London Op5mizing for Buffering Ra5o Slough Birmingham Greater London Op5mizing for Video Start Failures Slough Birmingham Greater London Op5mizing for Average Bitrate Slough Birmingham Time (1 min data- points over 24 hours)

70 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% Measurements of 3 Leading CDN Show Significant Variation Over Geo and ISP Washington DC (Hagerstown):CMCS(33657) Milwaukee:ROADRUNNER- CENTRAL(20231) Green Bay - Appleton:SCRR- 7015(7017) Denver:- QWEST(209) Charlole:SCRR (11426) Washington DC (Hagerstown):- CXA- ALL- Philadelphia:VZGNI- TRANSIT(19262) San Diego:SBIS- AS(7132) Las Vegas:- CXA- LV CBS(13432) Madison:CHARTER- NET- HKY- NC(20115) Indianapolis:SBIS- AS(7132) Providence - New Bedford:- CXA- ALL- CCI Washington DC (Hagerstown):VZGNI- TRANSIT(19262) Harrord & New Haven:SBIS- AS(7132) Houston:SBIS- AS(7132) Grand Rapids - Kalamazoo - Balle Creek:SBIS- Atlanta:BELLSOUTH- NET- BLK(6389) Honolulu:HAWAIIAN- TELCOM(36149) Atlanta:COMCAST- 7725(7725) Washington DC (Hagerstown):SPCS(10507) Orlando - Daytona Beach - Melbourne:EMBARQ- Denver:CMCS(33652) Norfolk - Portsmouth - Newport News:- CXA- ALL- Saint Louis:CHARTER- NET- HKY- NC(20115) Cincinna5:FUSE- NET(6181) Phoenix:- QWEST(209) Dallas - Fort Worth:VZGNI- TRANSIT(19262) Pilsburgh:CCCH- 3(7016) Saint Louis:SBIS- AS(7132) San Francisco - Oakland - San Jose:SBIS- AS(7132) San Diego:ROADRUNNER- WEST(20001) Greenville - Spartansburg - Asheville - Kansas City:SBIS- AS(7132) Columbus - OH:SCRR (10796) Louisville:INSIGHT- COMMUNICATIONS- CORP- Chicago:ATT- INTERNET3(6478) Kansas City:- CXA- ALL- CCI RDC(22773) Miami - Fort Lauderdale:BELLSOUTH- NET- BLK(6389) Cleveland:NEO- RR- COM(11060) Chicago:VZGNI- TRANSIT(19262) Columbia - SC:SCRR (11426) Dallas - Fort Worth:SBIS- AS(7132) Detroit:SBIS- AS(7132) Kansas City:SCRR (11955) Los Angeles:CHARTER- NET- HKY- NC(20115) Miami - Fort Lauderdale:COMCAST (20214) Cleveland:SCRR (10796) West Palm Beach - Fort San Diego:- CXA- ALL- CCI RDC(22773) Sealle - Tacoma:- QWEST(209) New York:CABLE- NET- 1(6128) Portland - OR:- QWEST(209) Oklahoma City:SBIS- AS(7132) Phoenix:- CXA- ALL- CCI RDC(22773) Chicago:SBIS- AS(7132) Greensboro - High Point - Winston- Albany - Schenectady - Troy:RR- NYSREGION- Tyler - Longview (Lutin & Grand Rapids - Kalamazoo - Balle Creek:CMCS(33668) Houston:CMCS(33662) Harrord & New Haven:COMCAST- 7015(7015) Eugene:- QWEST(209) San Francisco - Oakland - San Jose:CMCS(33651)

11/16/11. Bringing Internet Video to Prime Time. Background : Internet Video Going Prime Time. 2005: Beginning of the Internet Video Era

11/16/11. Bringing Internet Video to Prime Time. Background : Internet Video Going Prime Time. 2005: Beginning of the Internet Video Era 11/16/11 Background Aditya Ganjam Vice President of Engineering at Conviva Conviva Startup founded by Hui Zhang (Prof. at CMU) and Ion Stoica in 2006 Conviva optimizes video quality for premium content