On the Efficacy of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol Extended Version

Transcription

1 On the Efficacy of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol Exted Version James Martin, Yunhui Fu, Gongbing Hong School of Computing Clemson University Clemson, USA Abstract Ten years ago the term video streaming implied UDP transport. Now, video streaming typically refers to adaptive HTTP-based adaptive streaming (HAS). Various approaches for HAS have evolved from companies such as Netflix, Microsoft, Apple, and Google. It was observed that the different approaches have more similarities than differences. This motivated the development of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol. In the research reported in this paper, we evaluate the efficacy of DASH. Ongoing studies are quick to point out the issues of current implementations but fail to fully address the real issue: do the adaptations make any difference? The contributions of this research includes the following: first, we propose metrics that help quantify the effectiveness of an adaptation algorithm; second we provide evidence that in certain situations DASH appears to improve the user experience; third we propose an adaptation algorithm that demonstrates the tradeoffs involved when changing the algorithm s constraints. Collectively our results provide evidence of the efficacy of DASH. Keywords Internet Video Streaming; HTTP-based Adaptive Streaming; Simulation Modeling; Video Performance Assessment I. INTRODUCTION Sandvine s recent Internet usage report estimates that 5% of downstream traffic during peak usage times for fixed access networks is real-time entertainment [SAN]. This traffic category represents streamed content which primarily consists of Netflix and YouTube traffic. Ten years ago the term video streaming implied UDP transport. Now, video streaming typically refers to adaptive HTTP-based adaptive streaming (HAS). Various approaches for HAS have evolved from companies such as Netflix, Microsoft, Apple, and Google. It was observed that the different approaches have more similarities than differences. This motivated the development of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol. DASH provides a standard method for containing and distributing video content over the Internet [3GPP, ISO11, SOD11]. While it is not clear when or if the current set of HAS applications will converge towards a single standard, it is clear that HAS applications will be the dominant consumer of bandwidth in broadband access networks in the foreseeable future. For the remainder of this paper we use the term DASH instead of HAS. Ongoing academic research is providing foundations for understanding how DASH applications behave and how they might be improved. Various issues and problems have been identified. The fairness and stability of how DASH streams compete with each other and with other TCP applications is under study. The interaction between TCP (and in particular the TCP congestion control variants) and application control is under investigation. Some of the early results are contradictory. The work in [HHH] provides measurement evidence suggesting that the dual control algorithm in effect by a session s TCP and by the client adaptation are in conflict and contribute to pathologically wrong bitrate selection by DASH when competing with greedy TCP flows. The suggested solution is to use large playback buffers and ensure large client request sizes. However, VoD usage studies confirm that users are not static and instead will frequently move the session in time or move to a different choice of content [YLQ9, YZZ]. To ensure efficient use of network bandwidth, it is necessary to minimize the playback buffer. Taking a step back, the idea behind DASH is that matching the video content bitrate to the available path bandwidth leads to a better user experience and to reduced bandwidth consumption compared to if the video was streamed at a fixed bitrate. This implies that the application voluntarily gives up available TCP bandwidth with the assumption that this improves the user experience. The literature provides insight rationalizing this behavior: the work in [DSA11] suggests that buffer stalls have the biggest impact on user engagement; the work in [CPM] suggests that frequent adaptations is distracting; the work in [MCL11] suggests that sudden changes in video quality triggers poor subjective scores. There are several recent performance studies of DASH (e.g., [JSZ, MLC, HHH]) that do consider Quality of Experience (QoE). However, in our opinion, the efficacy of DASH (i.e, if video bitrate adaptation improves the user experience) has not been robustly established. Determining the perceived QoE of a video streaming session is very complex as the assessment deps on many factors including the person, the video encoding details, and the content. The first step towards establishing the efficacy of DASH requires an understanding of the objectives. A well designed DASH solution must meet the following objectives: 1. Maximize the video playback rate while minimizing the amount of wasted bandwidth; 2. Minimize playback buffer underruns; This research has been sponsored in part by CableLabs.

2 3. Minimize the frequency of bitrate adaptations;. Minimize the average playback buffer size; 5. Maintain fairness between flows; In prior work we have characterized the bandwidth consumption of a widely deployed DASH application (i.e., Netflix) [MFW13]. This work provides insight in how different implementations and different access networks can impact bandwidth consumption. We found that the basic behaviors of Netflix across devices and access networks were similar. In general, Netflix sessions are quick to respond to congestion and once congestion subsides the session cautiously consumes bandwidth as it becomes available. In the research reported in this paper, we evaluate the efficacy of DASH through a simulation analysis that involves assessments that quantify how well each objective is achieved. The results reported in this paper focus on the first four objectives listed above and leaves the issue of fairness to future work. The contributions of the research include the following: We propose metrics that help quantify the effectiveness of an adaptation algorithm. We provide evidence that in certain situations DASH appears to improve the user experience. We propose an adaptation algorithm that demonstrates the tradeoffs involved when changing the algorithm s constraints This paper is organized as follows. Section II provides an overview of DASH. Section III describes the simulation model we have developed. Section IV introduces our experimental methodology. Section V presents the results. Section VI highlights the related work. The final section provides conclusions and identifies our next steps. A. Overview II. DASH Figure 1 illustrates the main components of a DASH-based video distribution system. Multimedia content is encoded to one or more representations (different bit rates), allowing the client (i.e., the media player) to request any part of a representation in units of variable size blocks of data. The data content is made available to client devices by standard HTTP requests. The DASH server-side can be as simple as a standard HTTP web server. The available representations are maintained in a summary file referred to as the Media Presentation Description (MPD). MPD describes media types, resolutions, a manifest of the alternate stream and its URLs, minimum and maximum bandwidths, and required digital rights management (DRM) information. As illustrated in Figure 1a, the DASH client contains three main functional areas: the HTTP access, the media engine that decodes and rers the video stream, and the control engine. The latter component monitors the arriving video stream and determines when a lower or higher quality stream should be requested. The client maintains a playback buffer that serves to smooth variable content arrival rates to support the video playback. The client requests a new segment (tagged with the desired bitrate) once the playback buffer drops below a certain threshold. When the playback buffer is almost empty, the client is likely to go into a buffering state where it requests segments at a high rate. The session therefore operates in one of at least two states: buffering or steady state. While in a buffering state, the client requests data at a rate up to the available bandwidth over the path between the server and the client. If conditions permit, the session will attempt to find a steady state where it requests segments at a rate necessary to playback the content at a given encoded bitrate. We briefly diverge to clarify terminology. The available bandwidth (availbw) represents the available TCP bandwidth over a path. The bandwidth consumed by the session is referred to as the client throughput. We consider a DASH session to be in steady state if it converges to a stable client throughput for at least 5 seconds (i.e., the observed mean over at least one 5 second interval is within one standard deviation of the mean). We assume that content is always requested and stored by the player in units of segments. A segment represents a consecutive block of encoded video data and is identified by several attributes including 1)location of the video relative to the start of the video; 2)duration of the video contained in the segment; 3)the bitrate at which the video is encoded. We refer to the latter as the segment bitrate. DASH assumes that a segment is self-contained meaning that it can be rered indepent of prior or future segments. We assume that a player can attempt to rer a partial segment. In this case, the rering will likely suffer from artifacts. When the player decides to switch to a lower quality encoding of the content, it simply specifies the desired bitrate in the request. When the first client request is sent with a bitrate that is greater than what was in the previous request, this is a switchup event. If the client requests a segment with a lower bitrate, this is a switchdown event. The rate at which data is dequeued from the playback buffer is the videoplaybackrate. Fig. 1. DASH System B. Observed Behaviors Figure 2 illustrates the observed behavior of a Netflix client operating on a Windows Desktop PC located at

3 Bandwidth Consumption (Mbps) Bandwidth Consumption (Mbps) Clemson University. The figure visualizes the results from two sessions that were run separately. Each session views the same content and runs for 1 seconds. The client is located on a private network and uses a Linux router to access Clemson s network (and the Internet). Packets associated with the sessions were captured using tcpdump at the Linux router. We used the Linux netem capability to emulate wide area network conditions. We configured netem to limit the inbound and outbound channel capacity to 1 Mbps. For session 1 (illustrated in Figure 2a) we configured netem to maintain a Bernoulli random loss process (3% loss rate) during the time to seconds. Figure 2b illustrates the impact on the Netflix session when a competing TCP flow becomes active from time seconds until seconds. We ran iperf on a pair of Linux machines, one on the Clemson network and one on the private network. The default Linux Cubic congestion control algorithm with SACK is used on all Linux machines. The Linux machine on the Clemson network had netem configured to add an artificial latency to outbound packets such that the RTT experienced by iperf was similar to the uncongested RTT in effect during the Netflix session. Using Ping, we observed the RTT between the Window Netflix client under observation and the Netflix servers was 1.5 milliseconds. In both experiments, we ensure that the client device was the only device generating or consuming traffic over the link. We confirmed that the network outside of the private testbed (i.e., the Clemson network and the path to the Netflix servers) was not congested by issuing ping before and after each session Time (seconds) second samples 5 second samples a. Measurement Result: 3% loss rate - seconds 2 Mean: bps Std: 92 bps iperf: Mean: 75 bps Std: 3315 bps 2. second samples 5 second samples Netflix: Mean: 219 bps Std: 3151 bps Time (seconds) b. Measurement Result: Competing iperf flow - seconds Fig. 2. Netflix sessions over Netem 1 Mbps emulated Link Figure 2a plots the bandwidth consumed by the first session for the 1 seconds. As reported in [MFW13], Netflix might involve multiple connections. Our analysis tools aggregate all data sent downstream by the Netflix servers. The first 1 seconds reflects the client in the buffering state. The Windows client uses a large playback buffer (on the order of 2 seconds). Once the playback buffer is filled, the client goes into steady state. We observe the average bandwidth consumed during time 2- seconds to be.2 Mbps. The average throughput while packet loss was active (i.e., during the time - seconds) was 3. Mbps. At time seconds, we see the throughput resume to the original steady state. Figure 2b plots the results observed in the second scenario. The behavior during the first seconds is similar to that observed in Figure 2a. Once the iperf flow starts it consumes over % of the available bandwidth at the bottleneck link (the emulated channel maintained by netem). The Netflix session drops to a steady state throughput of 2.5 Mbps. This is similar to the unfairness described in [HHH]. III. SIMULATION MODEL In order to evaluate the efficacy of DASH adaptation, we use simulation. We have developed a simulation model of DASH using the ns2 simulation tool. The model captures the key application elements allowing us to generate realistic DASH session traffic in simulated network scenarios. Our current DASH model assumes that once a session starts, it is active until a desired termination time. The model does not account for user interactions such as pausing, rewind or fast forward, or changing channels (i.e, switching content). The model assumes one TCP connection per DASH session. The ns2 TCP/Full/SACK transport protocol is used. It is reasonable to assume that the video encoder that is likely to be used in a DASH system is highly optimized such that the encoded stream conforms to the target bitrate with a controlled level of variability. In our simplified video player model, encoded segments arrive and are inserted into the playback buffer. Since a segment can be large (specified by a configuration parameter SegmentSize, we use.5 seconds by default), we define a configuration parameter (PlayerInterval) that specifies how frequently the player consumes data from the playback buffer. The default setting is ¼ of the segment size. This allows the player to operate on chunks of video data smaller than a segment. If the playback buffer does not have the entire amount of data requested by the player, this is an underrun. In most cases, an underrun occurs when the playback buffer is empty. The player will go into buffering mode if the playback buffer level drops to less than a configured threshold (specified by a configuration parameter lowbufferthreshold which is set to twice the segment size by default). We monitor the frequency and duration of rebuffering operations. A client request specifies the desired bitrate encoding, the range of data in the stream, and the amount of data to be sent.

4 Client request message conform to the DASH (and HTTP 1.1) specifications. In our model, the data sent by the server has no meaning to the client. When the server receives a client request, it transmits the requested number of bytes (based on the segment s bitrate and duration) at the rate allowed by TCP. The client requests data as the video player consumes data from the playback buffer. The adaptation algorithm (described in the next subsection) runs prior to when the client issues a new request. Parameter Table 1. Model Terminology Definition Client throughput sample at time interval i (units in bps). Available bandwidth estimate at time interval i (bps). Discrete bitrate encoder options in bps (1 through N). The bitrate of the segment that is being rered by the player at time interval i. The amount of time that the adaptation logic must wait before it performs the next adaptation (seconds). This variable is updated by the adaptation algorithm. Defines the short time scale that sets the throughput sample intervals ( set to 2 seconds by default). The time when the last bitrate change occurred (seconds). purposely set T up to a value that is below B i. Alternatively, T threshold. If the AvailBW is checked if it exceeds the i down either of these conditions are true, the logic will generate either a switchup or a switchdown event. The second step, described in Figure, involves an adaptation controller that operates at longer timescales. The controller is in one of two states: either it is looking to adapt the bitrate to a higher quality level (CautiousUp state) or it is looking to adapt the bitrate to a lower quality level (CautiousDown state). The controller maintains the time before the next adaptation can occur (i.e. dynamicada ptationtime i ). The controller is frozen (i.e., it will not allow adaptation) if the following condition holds: currenttim e lastbitratechangetime dynamictime i SwitchEvent The minimum time that is required after switching between the long time scale controller states (in either direction). The adaptation decision result: (SwitchUp, SwitchDown, or NoChange) When considering changing to a lower segment bitrate, the current bandwidth estimate must be greater than this threshold (bps). When considering changing to a higher segment bitrate, the current bandwidth estimate must be less than this threshold (bps). ( A. Algorithm Table 1 defines the terminology that we use to describe the algorithm. We assume the algorithm operates in discrete time intervals based on a statically configured time scale I st. The client samples throughput ( A i ) are based on byte counts of data that arrive in time interval i. Throughput samples based on timescales of seconds can exhibit widely varying samples due to the on/off nature of DASH. Therefore, we apply a low pass filter to smooth the samples: i> i= The δ parameter is set to. by default. The variable AvailBW provides a smoothed estimate of the available TCP i bandwidth. The adaptation algorithm consists of two sequential steps that are executed with each new client request (but not more frequently than I st ). The first step, described in Figure 3, compares the availbw to the Tup and Tdown thresholds to determine if the bitrate could change. The B[ j] array holds the range of available bitrate encodings (based on our prior measurement study, we assume the following set of bitrates: Kbps, Kbps, 5Kbps, 1 Mbps, 1.5 Mbps, 2. Mbps, 3.5 Mbps,.2 Mbps). The algorithm first checks if the session can switch up to the next higher bitrate, then if it can switch down. As described in [HHH], it can be difficult to reliably assess the true available path bandwidth due to the bursty nature of DASH. Therefore, we assume that the AvailBW i underestimates the actual bandwidth. Consequently, we (1) Fig. 3. Switch Event Decision The principles implemented by the controller are as follows: SwitchUp or switchdown events that occur while the controller is frozen are ignored. While in CautiousUp state, consecutive bitrate adaptations to the next higher level cause the dynamictime to decrease linearly. The rational is that at this point there is an indication that bandwidth is available so it is safe to be aggressive. While in CautiousDown state, consecutive bitrate adaptations to the next lower level causing the dynamictime to increase exponentially. The rational is that at this point there is evidence that congestion has occurred and that the client should not give up bandwidth unnecessarily. Therefore the next bitrate reduction is delayed in hopes that network congestion subsides. When we change states, we reset the dynamictime back to the configured value (minstabilitytime). When transitioning from CautiousUp to CautiousDown, this has the effect of increasing the

5 Bandwidth Consumption (Mbps) Bandwidth Consumption (Mbps) dynamictime. When transitioning from CautiousDown to CautiousUp, this has the effect of decreasing the dynamictime. minstabilitytime parameter. For the simulations illustrated in Figure, the parameter was set to seconds. 2 if frozen TRUE; else frozen FALSE; switch (ControllerState) case(cautiousup) if (frozen==true) && (switchevent==switchup)) SwitchEvent NoChange if (frozen==false) && (switchevent==switchup)) dynamictime dynamicadaptiontime.25 * dynamicadaptiontime if (switchevent==switchdown) dynamictime minstabilitytime C CautiousDown case(cautiousdown) if (frozen==true) && (switchevent==switchdown)) SwitchEvent NoChange If (frozen==false) && (switchevent==switchdown)) dynamictime 2 * dynamicadaptiontime if (switchevent==switchup) dynamictime minstabilitytime C CautiousUp second samples 5 second samples Time (seconds) Mean: 17 bps Std: 159 bps a. Simulation Result: 3% loss rate - seconds 2. second samples 5 second samples Fig.. Controller Node 1 1 Iperf(equivalent): Mean: 572 bps Std: 32 bps Netflix: Mean: 997 bps Std: bps Netflix Servers 1 Gbps or 1Mbps, Mbps,.5ms prop delay.5ms prop delay Router 1 Router 2 Router 3 2 Competing Traffic Generators/Sinks Data rates: 1 Mbps (US and DS) Node n Time (seconds) b. Simulation Result: Competing Iperf flow - seconds Fig. 5. DASH System Fig.. Simulation Calibration Results over emulated 1 Mbps link Figure 5 illustrates the simulated network that is used in our study. The model loosely represents the system under observation during the measurement analysis. Each node configured to sink either a DASH stream, a TCP/FTP stream, or a UDP-based traffic stream. Figure visualizes results from simulations that are meant to duplicate (as close as possible) the measurement results illustrated in Figure 2. For the results illustrated in Figure a, the link between the Router 2 and Router 3 was set with a link speed of 1 Mbps. Router 2 s output interface was configured with a queue capacity of packets. The link was configured with a random Bernoulli loss process that was applied to all packets traveling in the downstream direction during time through seconds. Comparing Figure a with Figure 2a, we observe similar behaviors. The steady state bandwidth consumed during time - seconds is lower compared to the measurement result. We attribute this to differences in how we implemented the adaptation algorithm. Similarly, Figure b shows similarities and differences to Figure 2b. Our objective is not to duplicate the observed Netflix algorithm. Instead, we used it as a guide to help us develop our own adaptation algorithm. One particularly important behavior to point out in Figure is the time it takes for the DASH flow to consume available bandwidth as it becomes available (i.e., after time seconds). This time is determined in large part by the IV. METHODOLOGY The system under study is quite complex. Due to space constraints, we only show a portion of our analysis (an exted version of this paper is available [MFH13B]). For the results presented in the next section, all simulations involve three flows: the DASH flow, a competing TCP flow, and a competing UDP that generates on/off traffic. All three flows s application data in the downstream direction (i.e., from servers to the node s connected to Router 3). The competing TCP flow starts at time and stops at time. The competing UDP flow starts at time seconds and the DASH flow starts at time.7 seconds. Table 2 summarizes the three scenarios. Scenario 1 uses the ns2 exponential traffic generator to s data at an average rate of 5 Mbps with an average off time of 1. second. Aggregate traffic causes the link between Router 2 and Router 3 to be heavily congested. We conduct a total of 3 simulations, varying the minstabilitytime and the playbackbuffercapacity. Scenario 2 is similar, however the SegmentSize and the playbackbuffercapacity are varied. Scenario 3 removes the randomness from the UDP on/off traffic generator such that the traffic generator ss at a constant rate for a fixed amount of time and then goes idle for a fixed off time (and repeats the cycle forever). For Scenario 3, we set the fixed rate to 7 Mbps. The on time is set to 5 seconds and the off time is varied such that the average sing

6 rate of the UDP flow ranges from 57 Kbps to 3.2 Mbps. The link between Router 2 and Router 3 is the bottleneck, however it is not as congested as in the previous scenarios. The playbackbuffercapacity is set to seconds and the minstabilitytime is varied as in the previous scenario. In our analysis, we describe a relatively simple extension to the adaptation algorithm and illustrate the results using Scenario 3. Scenario ID Description Table 2. Experiment Scenarios 1 UDP mean 5 Mbps, idle time 1. second. Vary two params: Vodapp_player_min_stabilize_time: 25,5, 75,, 5,15 Playback Buffer Capacity: 5, 9, 13,17,21, 25 2 UDP mean 5 Mbps, idle time 1. second. Vary two params: Segment size: 3:,,9,,15,1 Playback Buffer Capacity: 5, 9, 13,17,21, 25 3 UDP variable mean, VBR_BURST_RATE: 7Mbps, VBRONTIME: 5 seconds, VBRIDLETIME: 55, 5,35,25,15,5 Vodapp_player_min_stabilize_time: 25,5, 75,, 5,15 A. Performance Metric Definitions UDP-based video streaming typically involves objective metrics that are either data oriented (like PSNR), vision-based metrics (like Structural Similarity), or packet-stream oriented metrics that do not require video decoding [WM]. These metrics attempt to quantify how the data arrives and possibly a description of the artifacts produced. Some are based on a reference video while others do not require a reference. Objective metrics cannot provide an accurate quality of experience measure as perceived by an user over a range of content and conditions. Subjective video quality assessment, which requires human involvement, are the only reliable method to assess the video quality perceived by users. The challenge is finding an objective metric that accurately estimates the perceived quality observed by the average User. This latter form of assessment is inherently subjective and therefore viewed as highly complex [DLD, NEE11B, XST9]. Further complexity arises with DASH because TCP and the adaptive application control algorithms are in effect. As pointed out in [DAJ11, BSA, OS], a QoE assessment that is appropriate for DASH must include components such as average bitrate, the rate of playback buffer underruns, the rate of bitrate adaptations, and the time it takes to fill the buffer before rering can begin (or restart). The preliminary work in this area suggests that: 1)The QoE must be a function of the metrics that map to a utility value; 2)the utility function differs significantly for live streams and for VoD streams; 3)The utility function differs with the perspective (i.e., user or a network operator). For example, a user s QoE will be formulated differently from a network operator s utility function. Our model abstracts the impacts of impairment to the stream using the following set of metrics. Video quality: We use mean squared error (MSE) to quantify how the received video stream compares to the original source stream. We assess the video session by breaking the signal into fixed intervals, such as the DASH segment size parameter. We define the source signal, X i, to be the highest possible video bitrate for segment i (equation 2). The video stream is received and decoded by the player. The physical representation of this signal, Y, is depicted by equation 3. The MSE of the received signal is defined in equation. We define a peak signal-to-noise ratio (PSNR) as given in equation 5. In the literature, L represents the dynamic range of the signal. When applied to DASH, the range of the signal deps on the allowed bitrates. Therefore, we define L to be the maximum allowed bitrate. This provides us a convenient objective metric that does not require a reference but leverages the fact that DASH is generally encoded to meet a target bandwidth consumption. In our results, we find that the value ranges from about to about 115. For each interval obtained when an underrun occurs, the sampled signal has a value of. This causes the DPSNR to be biased on the low side as the underrun rate grows. This makes intuitive sense however further study is required to better calibrate the metric. Y MSE(x,y) Underrun and Rebuffering rate: The video player reads data from the playback buffer at fixed intervals although the amount of data that is transferred from the playback buffer deps on the bitrate of the segment. If the video player is able to read the entire PlayerInterval amount of data from the playback buffer, the quality assessment is captured by the DPSNR (i.e., the distance from the highest quality reference). If the playback buffer does not contain a PlayerInterval amount of data, an artifact such as a stutter occurs. The model assumes the player can potentially rer a partial segment (i.e., if not all of the segment data arrives in time). Setting the player interval to ¼ of the segment size potentially provides more information about the impairment. Each time the player does not receive the amount of data requested in a read operation on the playback buffer, we increment an underrun count. Similarly, each time the player successfully completes a rebuffering operation, a rebuffering count is incremented. At the of the simulation, the underrun and rebuffering rates computed (normalized to an hourly basis). Also at the of the simulation we calculate the average duration of all rebuffering events experienced by a session. Frequency of adaptation: This captures the frequency of bit rate changes (in units of bitrate switches per hour). Playback buffer resources consumed: The video player manages a playback buffer. An attribute of an adaptation algorithm is the average size of the buffer utilized by the session. V. RESULTS AND ANALYSIS A. Calibration Results The first two data rows shown in Table 3 document the metric results for the calibration simulations visualized in Figure. The next pair of rows in the table shows the results when we run the same calibration simulations but without the DASH adaptation (the client always requests data with the (2) (3) () (5)

7 Average Playback Buffer Size DASH PSNR Average s / Hour highest bitrate of.2 Mbps). The single flow simulation demonstrates the efficacy of DASH. Without adaptation, the average videoplayerrate is higher than when adaptation is enabled (3.53 Mbps versus 3.9 Mbps respectively). However since there are more underrun events the DPSNR is low in both cases. The playbackbuffercapacity was set to seconds. The average buffer size is similar for both cases (about seconds). The rebuffering event rate is slightly higher with no adaptation but more significantly, the average duration of a rebuffering event is larger (7.93 seconds versus.57 seconds). The final column statistic indicates that with no adaptation the session is in buffering mode 1.3% of the time versus.2% of the time when adaptation is enabled. The two flow results in the table illustrate a case where DASH adaptation might actually decrease the perceived quality (the DPSNR is 22 when adaptation is disabled while 19 when adaptation is enabled). The last two rows in the table are with a modified adaptation algorithm we describe later in the paper. Description Single Flow, (Figure 7a) Two Flows, (Figure 7b) Single Flow, No Two Flows, No Single Flow, Enhanced Two Flows, Enhanced Video Playback Rate (Mbps) Table 3. Calibration Metric Results DPSNR Average Playback Buffer Size (seconds) Underrun Rate (per hour) Rebuffering Event Rate (per hour) Average Rebuffering Event Time (seconds) % % % % % % Percent Session Time in Buffering Mode B. Scenario 1 and 2 Results Figure 7 illustrates the results of Scenario 1. Figure 7a shows that the DPSNR drops slightly as the playbackbuffercapacity gets smaller. The figure also suggests that the DPSNR drops slightly as the minstabilitytime gets smaller. DPSNR is more sensitive to the playbackbuffercapacity than to the minstabilitytime. Figure 7b shows that the minstabilitytime parameter has the desired dampening effect on the frequency of adaptations. Figure 7c shows that the average playback buffer size is sensitive to the minstabilitytime only at the higher settings of the playbackbuffercapacity Figure 7d shows that rebuffering events begin to occur once the playbackbuffercapacity is 15 seconds (or lower). The rebuffering rate increases as the minstabilitytime increases. This is because the adaptation reacts more slowly as the minstabilitytime increases Fig. 7. Scenario 1 Simulation Results a. DPSNR b. Rate c. Average Playback Buffer As observed in Scenario 1, Figure a (Scenario 2) illustrates that the rebuffering rate increases as the playbackbuffercapacity is set to 15 seconds or less. The Scenario results also suggest that rebuffering occurs more frequently as the segmentsize gets smaller, especially for small playbackbuffercapacity settings. We ran Scenario 2 with DASH adaptations disabled. Comparing this result (Figure b) with the previous (Figure a) suggests that DASH does improve the QoE. The other metrics (not shown) are supportive of this conclusion d. Rebuffering Event Rate 15

8 DASH PSNR DASH PSNR DASH PSNR playback buffer size. The algorithm monitors the size of the playback buffer (referred to a curbuffersize). We define the state of the playback buffer, bufferstate, to be either GREEN, YELLOW, or RED. A bufferstate of GREEN indicates the curbuffersize is at least 2/3 the dynamicplaybackbuffer- Capacity. A bufferstate of YELLOW indicates the curbuffersize is between 1/3 and 2/3 the capacity. Finally, a bufferstate of RED implies the curbuffersize is less than 1/3 the capacity Segment Size (seconds) 2 a. Scenario 2 with UDP CBR Off Time (seconds) UDP CBR Off Time (seconds) a. Scenario 3 DPSNR b. Scenario 3 Rebuffering Rate b. Scenario 2 without Segment Size (seconds) UDP CBR Off Time (seconds) UDP CBR Off Time (seconds) d. Scenario 3 No Rebuffering Rate c. Scenario 3 No DPSNR 2 15 Fig.. Scenario 2 Simulation Results C. Scenario 3 Results Figure 9 illustrates the results associated with Scenario 3. The mean sing rate of the exponential UDP traffic generator is varied by adjusting the traffic generator s off time. The idea is to produce controlled bursts of congestion. We ran the Scenario three times. First, using the adaptation algorithm described in Figures 3 and. Second, with the adaptation algorithm disabled. Third, with an enhanced adaptation algorithm (described below). Comparing Figures 9a and 9b suggests that adaptation can decrease the frequency of rebuffering events. Additionally, the DPSNR result (not shown) suggests that adaptation increases the video quality slightly. Prior work suggests that rebuffering events (i.e., stalls) have a greater impact on perceived quality than reduced video quality. Scenarios 1 and 2 show that increasing the playbackbuffercapacity can reduce the rate of rebuffering. However, there are reasons (identified earlier in the paper) to minimize the average buffer size. We introduce the following incremental modifications to the adaptation algorithm. We assume the configured playbackbuffercapacity now represents the maximum allowed capacity and is set by default to a large value such as 5 seconds. We introduce a variable referred to as the dynamicplaybackbuffercapacity to represent the best UDP CBR Off Time (seconds) 5 UDP CBR Off Time (seconds) e. Scenario 3 Enhanced Algorithm DPSNR f. Scenario 3 Enhanced Algorithm Rebuffering Rate Fig. 9. Scenario 3 Simulation Results When the adaptation controller generates a SwitchDown event and if the bufferstate is RED, the bitrate reduction is set to three bitrate levels below the current (or it is set to the lowest quality setting). If the bufferstate is YELLOW, the bitrate reduction is set to two levels below the current. Finally, if the bufferstate is GREEN, a single bitrate level reduction is performed. When a buffer underrun occurs and the controller is in RED state, the dynamicplaybackbuffercapacity is doubled When a buffer underrun occurs and the controller is in the YELLOW state, the buffer capacity is increased by 5%. If the playback buffer is in the GREEN state for 5 consecutive buffer times, the dynamiplaybackbuffercapacity is increased by 25%. The updates are not allowed to set the dynamicplaybackbuffer- Capacity to be less than minplaybackbuffercapacity or greater than playbackbuffercapacity. By default, we set the minplayerbuffercapacity to a value of 2*segmentSize. Figure 9c illustrates Scenario 3 with the modified adaptation algorithm. The results show that the enhanced algorithm is able to reduce the rate of rebuffering events. The average playback buffer size is kept to roughly 75 seconds

9 compared to an average size of seconds in the base algorithm. The improvements, however, come at a cost of decreased video quality. The DPSNR drops by more than 5%. with the enhanced algorithm. research. The challenge however is finding the best weights to assign to the various decision components. Our next step is to conduct subjective user tests to establish a basis for formulating the optimization. RELATED WORK Early research in DASH involved empirical studies that characterized the traffic associated with HAS applications such as YouTube and Netflix [AJZ1, AJZ11, AN11, CKRAM7, CDL7, FMM11, GALM7, GALM]. Quite a bit of recent work has focused on protocol performance evaluations [ABD11, ACH, CMP11, IKG11, JSZ, KKH1, KKH11, LBG11, LMT, RLB11, SSH11]. Perhaps of more relevance is the work that specifically focuses on the adaptation algorithm [AAD, CMP11, HHH, LBG11, MLC, JSZ, TL]. The work in [HHH] is particularly helpful in understanding the complex interactions between the adaptation algorithm and TCP. We however do not agree with the authors recommations to use large playback buffers. The work in [JSZ] and [MLC] are similar to our approach as they incorporate QoE assessment in the analysis methodology. [JSZ] introduces an instability metric that quantifies the level and relative weight of bitrate switches. This exts our simpler frequency of bitrate adaptations metric by looking at the evolution of bitrate adaptations. Our rebuffering artifact assessments, in particular the average artifact duration, addresses the temporal effects of artifacts. The work in [MLC] performed subjective tests to determine which bitrate adaptation behaviors lead to the highest overall perceived quality. We note that none of these recent studies that address QoE directly addresses the core question of if bitrate adaptations can improve the perceived quality. CONCLUSION In this paper we address the question if the DASH adaptation can (and under what circumstances) lead to beneficial results. Prior research that has developed methods for assessing the perceived quality of video does not directly apply to DASH. We have introduced a set of metrics that provide assessments of the required components: video quality (average videoplaybackrate, DPSNR), artifact assessment (playback buffer underrun rate, rebuffering rate, average rebuffering event duration), and playback buffer usage (average playback buffer size). We have studied (through simulation) an adaptation algorithm that exposes several key parameters for experimentation. Our results suggest that the playbackbuffercapacity and the minstabilitytime parameters have a significant impact on observed performance. Our analysis included comparing results both when DASH adaptation is enabled and then disabled. The results suggest that in some situations DASH can increase video quality and reduce the rate of observed artifacts. Although in other situations, the rate of observed artifacts decreases but at a cost of reduced video quality (i.e., DPSNR). In summary, we provide evidence that supports the efficacy of DASH, however finding an algorithm that ensures the right decision is always made is quite challenging. In ongoing work, we treat the adaptation algorithm as a multi-attribute decision making problem. This is a well-established branch of operations REFERENCES [1] [3GPP] 3GPP TS 2.27 version 1.1. Release 1: Transparent End-to- Packet Switched Streaming Service (PSS); Progressive Download and Dynamic Adaptive Service over HTTP, 3GPP, January 2, available online at /ts_27vp.pdf [2] [AAD] S. Akhshabi, L. Anantakrishnan, C. Dovrolis, A. Begen, What Happens when HTTP Adaptive Streaming Players Compete for Bandwidth, Proceedings of ACM NOSSDAV, June 2. [3] [ABD11] S. Akhshabi, A. Begen, C. Dovrolis, An Experimental Evaluation of Rate- Algorithms in Adaptive Streaming over HTTP, Proceedings of ACM MMSys, February 211. [] [ACH] T. Andelin, V. Chetty, D. Harbaugh, S. Warnick, D. Zappala, Quality Selection for Dynamic Adaptive Streaming over HTTP with Scalable Video Coding, Proceedings of ACM MMSys, February 2. [5] [AJZ1] V. K. Adhikari, S. Jain, and Z.-L. Zhang, YouTube Traffic Dynamics and Its Interplay with a Tier-1 ISP: An ISP Perspective, in IMC, 21. [] [AJZ11] V. K. Adhikari, S. Jain, and Z. L. Zhang, Where Do You Tube? Uncovering YouTube Server Selection Strategy, in IEEE ICCCN, 211. [7] [AN11] S. Alcock, R. Nelson, Application Flow Control in YouTube Video Streams, ACM SIGCOMM Computer Communication Review, April 211. [] [BSA] A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, H. Zhang, A Quest for an Internet Video Quality-of-Experience Metric, Proceedings of the ACM HotNets, October 2. [9] [CRC] M. Cha, P. Rogriguez, J. Crowcraft, S. Moon, X. Amatriain, Watching Television Over an IP Networks, Proceedings of the IMC, October 2. [1] [CDL7] X. Cheng, C. Dale, and J. Liu, Understanding the Characteristics of Internet Short Video Sharing: YouTube as a Case Study, in IMC, 27. [11] [CKRAM7] M. Cha, H. Kwak, P. Rodriguez, Y.-Y. Ahn, and S. Moon, I Tube, You Tube, Everybody Tubes: Analyzing the Worlds Largest User Generate, Content Video System, in IMC, 27. [] [CPM] N. Cranley, P. Perry, L. Murphy, User Perception of Adapting Video Quality, Internation Journal of Human-Computer Studies, 2. [13] [CMP11] L. Cicco, S. Mascolo, V. Palmisano, Feedback Control for Adaptive Live Video Streaming, Proceedings of MMSys 11, February 211. [1] [DAJ11] F. Dobrian, A. Awan, D. Joseph, A. Ganjamm J. Zhan, V. Sekar, I. Stoica, H. Zhang, Understanding the Impact of Video Quality on User Engagement, Proceedings of SIGCOMM 11, August 211. [15] [DLD] N. Degrande, K. Laevens, D. De Vleeschauwer, and R. Sharpe, Increasing the user perceived quality for iptv services, Communications Magazine, IEEE, vol., no. 2, pp. 9, 2. [1] [EKG11] K. Evensen, D. Kaspar, C. Griwodz, P. Halvorsen, A. Hansen, P. Engelstad, Improving the Performance of Quality-Adaptive Video Streaming over Multiple Heterogeneous Access Networks, Proceedings of MMSys 11, February 211. [17] [FMM11] A. Finamore, M. Mellia, M. Munafo, R. Torres, S. Rao, YouTube Everywhere: Impact of Device and Infrastructure Synergies on User Experience, Proceedings of IMC 11, November 211. [1] [GALM7] P. Gill, M. F. Arlitt, Z. Li, and A. Mahanti, YouTube Traffic Characterization: A View From the Edge, in IMC, 27.

10 [19] [GALM] P. Gill, M. Arlitt, Z. Li, and A. Mahanti, Characterizing User Sessions on YouTube, in ACM/SPIE MMCN, 2. [2] [GPAC] Open source multimedia framework, available at [21] [HDD] R. Huysegems, B. De Vleeschauwer, K. De Schepper, C. Hawinkel, T. Wu, K. Laevens, and W. Van Leekwijck, Session Reconstruction for HTTP Adaptive Streaming: Laying the Foundation for Network-based QoE Monitoring, in Proceedings of IWQoS, June, 2. [22] [HG] R. Houdaille and S. Gouache, Shaping HTTP Adaptive Streams for a Better User Experience, in Proceedings of the MMSys, February, 2 [23] [HHH] T. Huang, N. Handigol, B. Heller, N. McKeown, R. Johari, Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard, Proceedings of the IMC, November, 2. [2] [ISO11] ISO/IEC, Information technology MPEG systems technologies Part : Dynamic adaptive streaming over HTTP (DASH), Jan 211, available online at [25] [JSZ] J. Jiang, V. Sekar, and H. Zhang, Improving Fairness, Efficiency, and Stability in HTTP-based Adaptive Video Streaming with Festive, Proceedings of CoNEXT, December, 2. [2] [KKH1] R. Kuschnig, I. Kofler, H. Hellwagner, Improving Internet Video Streaming Performance by Parallel TCP-based Request-Response Streams, Proceedings of IEEE CCNC 21, January 21. [27] [KKH11] R. Kuschnig, I. Kofler, H. Hellwagner, Evaluation of HTTPbased Request-Response Streams for Internet Video Streaming, Proceedings of MMSys 11, February 211. [2] [LBG11] C. Liu, I. Bouazizi, M. Gabbouj, Rate for Adaptive HTTP Streaming, Proceedings of ACM MMSys, February 211. [29] [LMT] S. Lederer, C. Muller, C. Timmerer, Dynamic Adaptive Streaming over HTTP Dataset, Proceedings of ACM MMSys, February 2. [3] [MFW13] J. Martin, Y. Fu, N. Wourms, T. Shaw, Characterizing Netflix Bandwidth Consumption, Proceedings of the IEEE CCNC, January 213. [31] [MF13] J. Martin, Y. Fu, G. Hong, On the Efficacy of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol Exted Version, TechnicalReport,Availableonline at: ~jmarty/papers/efficacydashexted.pdf, March 213. [32] MLC R. Mok, X. Luo, E. Chan, R. Chang, QDASH: A QoE-aware DASH System, Proceedings of the the ACM MMSys, December, 2. [33] [MT11A] C. Muller, C. Timmerer, A Testbed for the Dynamic Adaptive Streaming over Http Featuring Session Mobility, Proceedings of ACM MMSys, February 211. [3] [MT11B] C. Muller, C. Timmerer A VLC Media Player Plugin Enabling Dynamic Adaptive Streaming over HTTP, Proceedings of ACM MM 11, December 211. [35] [OS] O. Oyman, S. Singh, Quality of Experience for HTTP Adaptive Streaming Services, IEEE Communications Magazine, April 2. [3] [RLB11] A. Roa, Y. Lim, C. Barakat, A. Legout, D. Towsley, W. Dabbous, Network Characteristics of Video Streaming Traffic, Proceedings of ACM CoNEXT Conference, December 211. [37] [SAN] Global Internet Phenomena Report, Sandvine Corporation, 2H 2, available online: Sandvine_Global_Internet_Phenomena_Report_2H_2.pdf. [3] [SAR] K. D. Singh, Y. H. Aoul, G. Rubino, Quality of Experience Estimation for Adaptive HTTP/TCP Video Streaming Using H.2/AVC, Proceedings of CCNC, January, 2. [39] [SOD11] I. Sodagar, The MPEG-DASH Standard for Multimedia Streaming Over the Internet, IEEE Multimedia magazine, October/December 211. [] [SSH11] Y. Sanchez, T. Schierl, C. Hellge, T. Wiegand, D. Vleeshauwer, W. Keekwijck, Y. Louedec, idash: Improved Dynamic Adaptive Streaming over HTTP using Scalable Video Coding, Proceedings of ACM MMSys, February 211. [1] [STO11] T. Stockhammer, Dynamic Adaptive Streaming over HTTP Standards and Design Principles, Proceedings of the MMSys 11, February 211. [2] [WBS] Z. Wang, A. Bovik, H. Sheikh, et. al., Image Quality Assessment: From Error Visibility to Structure Similarity, IEEE Transactions on Image Processing, vol 1 number, 25, pp [3] [XSTH9] J. Xia, Y Shi, K. Teunissen, I. Heynderickx, Perceivable Artifacts in Compressed Video and Their Relation to Video Quality, Signal Processing : Image Communications vol. 2, 29, pp [] [YZZ] H. Yu, D. Zheng, B. Zhao, W.Zheng, Understanding User Behavior in Large-Scale Video-on-Demand Systems, Proceedings of Eurosys, 2. [5] [YLQ9] H. Yin, X. Liu, F. Qiu, N. Xia, C. Lin, H. Zhang, V. Sekar, G. Min, Inside the Bird s Nest: Measurements of Large-Scale Live VoD from the 2 Olympics, Proceedings of the ACM IMC, 211. [] [YZZ] H. Yu, D. Zheng, B. Zhao, W. Zheng, Understanding User Behavior in Large-Scale Video-on-Demand Systems, Proceedings of EuroSys, April, 2.