On the Efficacy of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol

Transcription

1 On the Efficacy of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol James Martin, Yunhui Fu, Gongbing Hong School of Computing Clemson University Clemson, USA Abstract Ten years ago the term video streaming implied UDP transport. Now, video streaming typically refers to adaptive HTTP-based adaptive streaming (HAS). Various approaches for HAS have evolved from companies such as Netflix, Microsoft, Apple, and Google. It was observed that the different approaches have more similarities than differences. This motivated the development of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol. In the research reported in this paper, we evaluate the efficacy of DASH. Ongoing studies are quick to point out the issues of current implementations but fail to fully address the real issue: do the adaptations make any difference? The contributions of this research includes the following: first, we propose metrics that help quantify the effectiveness of an adaptation algorithm; second we provide evidence that in certain situations DASH appears to improve the user experience; third we propose an adaptation algorithm that demonstrates the tradeoffs involved when changing the algorithm s constraints. Collectively our results provide evidence of the efficacy of DASH however we also find that finding an algorithm that always makes the right decision is extremely difficult. Keywords Internet Video Streaming; HTTP-based Adaptive Streaming; Simulation Modeling; Video Performance Assessment I. INTRODUCTION Sandvine s recent Internet usage report estimates that 5% of downstream traffic during peak usage times for fixed access networks is real-time entertainment [1]. This traffic category represents streamed content which primarily consists of Netflix and YouTube traffic. Ten years ago the term video streaming implied UDP transport. Now, video streaming typically refers to adaptive HTTP-based adaptive streaming (HAS). Various approaches for HAS have evolved from companies such as Netflix, Microsoft, Apple, and Google. It was observed that the different approaches have more similarities than differences. This motivated the development of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol. DASH provides a standard method for containing and distributing video content over the Internet [, 3]. While it is not clear when or if the current set of HAS applications will converge towards a single standard, it is clear that HAS applications will be the dominant consumer of bandwidth in broadband access networks in the foreseeable future. For the remainder of this paper we use the term DASH instead of HAS. Ongoing academic research is providing foundations for understanding how DASH applications behave and how they This research has been sponsored in part by CableLabs. might be improved. Various issues and problems have been identified. The fairness and stability of how DASH streams compete with each other and with other TCP applications is under study. The interaction between TCP (and in particular the TCP congestion control variants) and application control is under investigation. Taking a step back, the idea behind DASH is that matching the video content bitrate to the available path bandwidth leads to a better user experience and to reduced bandwidth consumption compared to if the video was streamed at a fixed bitrate. This implies that the application voluntarily gives up available TCP bandwidth with the assumption that this improves the user experience. The literature provides insight rationalizing this behavior: the work in [] suggests that buffer stalls have the biggest impact on user engagement; the work in [5] suggests that frequent adaptations is distracting; the work in [] suggests that sudden changes in video quality triggers poor subjective scores. There are several recent performance studies of DASH (e.g., [7-9]) that do consider Quality of Experience (QoE). However, in our opinion, the efficacy of DASH (i.e, if video bitrate adaptation improves the user experience) has not been robustly established. Determining the perceived QoE of a video streaming session is very complex as the assessment deps on many factors including the person, the video encoding details, and the content. The first step towards establishing the efficacy of DASH requires an understanding of the objectives. A well designed DASH solution must meet the following objectives: 1. Maximize the video playback rate while minimizing the amount of wasted bandwidth;. Minimize playback buffer underruns; 3. Minimize the frequency of bitrate adaptations;. Minimize the average playback buffer size; 5. Maintain fairness between flows; In prior work we have characterized the bandwidth consumption of a widely deployed DASH application (i.e., Netflix) [1]. This work provides insight in how different implementations and different access networks can impact bandwidth consumption. We found that the basic behaviors of Netflix across devices and access networks were similar. In general, Netflix sessions are quick to respond to congestion and once congestion subsides the session cautiously consumes

2 bandwidth as it becomes available. In the research reported in this paper, we evaluate the efficacy of DASH through a simulation analysis that involves assessments that quantify how well each objective is achieved. The results reported in this paper focus on the first four objectives listed above and leaves the issue of fairness to future work. The contributions of the research include the following: We propose metrics that help quantify the effectiveness of an adaptation algorithm. We provide evidence that in certain situations DASH appears to improve the user experience. We propose an adaptation algorithm that demonstrates the tradeoffs involved when changing the algorithm s constraints. This paper is organized as follows. Section II provides an overview of DASH. Section III describes the simulation model we have developed. Section IV introduces our experimental methodology. Section V presents the results. Section VI highlights the related work. The final section provides conclusions and identifies our next steps. A. Overview II. DASH Figure 1 illustrates the main components of a DASH-based video distribution system. Multimedia content is encoded to one or more representations (different bit rates), allowing the client (i.e., the media player) to request any part of a representation in units of variable size blocks of data. The data content is made available to client devices by standard HTTP requests. The DASH server-side can be as simple as a standard HTTP web server. The available representations are maintained in a summary file referred to as the Media Presentation Description (MPD). MPD describes media types, resolutions, a manifest of the alternate stream and its URLs, minimum and maximum bandwidths, and required digital rights management (DRM) information. As illustrated in Figure 1, the DASH client contains three main functional areas: the HTTP access, the media engine that decodes and rers the video stream, and the control engine. The latter component monitors the arriving video stream and determines when a lower or higher quality stream should be requested. The client maintains a playback buffer that serves to smooth variable content arrival rates to support the video playback. The client requests a new segment (tagged with the desired bitrate) once the playback buffer drops below a certain threshold. When the playback buffer is almost empty, the client is likely to go into a buffering state where it requests segments at a high rate. The session therefore operates in one of at least two states: buffering or steady state. While in a buffering state, the client requests data at a rate up to the available bandwidth over the path between the server and the client. If conditions permit, the session will attempt to find a steady state where it requests segments at a rate necessary to playback the content at a given encoded bitrate. We briefly diverge to clarify terminology. The available bandwidth (availbw) represents the available TCP bandwidth over a path. The bandwidth consumed by the session is referred to as the client throughput. We assume that content is always requested and stored by the player in units of segments. A segment represents a consecutive block of encoded video data and is identified by several attributes including 1)location of the video relative to the start of the video; )duration of the video contained in the segment; 3)the bitrate at which the video is encoded. We refer to the latter as the segment bitrate. DASH assumes that a segment is selfcontained meaning that it can be rered indepent of prior or future segments. We assume that a player can attempt to rer a partial segment. In this case, the rering will likely suffer from artifacts. When the player decides to switch to a lower quality encoding of the content, it simply specifies the desired bitrate in the request. When the first client request is sent with a bitrate that is greater than what was in the previous request, this is a switchup event. If the client requests a segment with a lower bitrate, this is a switchdown event. The rate at which data is dequeued from the playback buffer is the videoplaybackrate. Fig. 1. DASH System B. Observed Behaviors Figure illustrates the observed behavior of a Netflix client operating on a Windows Desktop PC located at Clemson University. The figure visualizes the results from two sessions that were run separately. Each session views the same content and runs for seconds. The client is located on a private network and uses a Linux router to access Clemson s network (and the Internet). Packets associated with the sessions were captured using tcpdump at the Linux router. We used the Linux netem capability to emulate wide area network conditions. We configured netem to limit the inbound and outbound channel capacity to 1 Mbps. For session 1 (illustrated in Figure a) we configured netem to maintain a Bernoulli random loss process (3% loss rate) during the time to 1 seconds. Figure b illustrates the impact on the Netflix session when a competing TCP flow becomes active from time seconds until 1 seconds. We ran iperf on a pair of Linux machines, one on the Clemson network and one on the private network. To better match our TCP simulation

3 Bandwidth Consumption (Mbps) Bandwidth Consumption (Mbps) model, we configured Linux to use the Reno/SACK control algorithm. The Linux machine on the Clemson network had netem configured to add an artificial latency to outbound packets such that the RTT experienced by iperf was similar to the uncongested RTT in effect during the Netflix session. Using Ping, we observed the RTT between the Window Netflix client under observation and the Netflix servers was 1.5 milliseconds. In both experiments, we ensure that the client device was the only device generating or consuming traffic over the link second samples 5 second samples a. Measurement Result: 3% loss rate -1 seconds Mean: 3399 bps Std: 9 bps iperf: Mean: 75 bps Std: 3315 bps. second samples 5 second samples Netflix: Mean: 9 bps Std: 351 bps b. Measurement Result: Competing iperf flow -1 seconds Fig.. Netflix sessions over Netem 1 Mbps emulated Link Figure a plots the bandwidth consumed by the first session for the seconds. As reported in [1], Netflix might involve multiple connections. Our analysis tools aggregate all data sent downstream by the Netflix servers. The first seconds reflects the client in the buffering state. The Windows client uses a large playback buffer (on the order of seconds). Once the playback buffer is filled, the client goes into steady state. We observe the average bandwidth consumed during time - seconds to be. Mbps. The average throughput while packet loss was active (i.e., during the time -1 seconds) was 3. Mbps. At time 1 seconds, we see the throughput resume to the original steady state. Figure b plots the results observed in the second scenario. The behavior during the first seconds is similar to that observed in Figure a. Once the iperf flow starts it consumes over 7% of the available bandwidth at the bottleneck link (the emulated channel maintained by netem). The Netflix session drops to a steady state throughput of.19 Mbps. This is similar to the unfairness described in [9]. III. SIMULATION MODEL In order to evaluate the efficacy of DASH adaptation, we use simulation. We have developed a simulation model of DASH using the ns simulation tool. The model captures the key application elements allowing us to generate realistic DASH session traffic in simulated network scenarios. Our current DASH model assumes that once a session starts, it is active until a desired termination time. The model does not account for user interactions such as pausing or rewinding. The model assumes one TCP connection per DASH session. In our simplified video player model, encoded segments arrive and are inserted into the playback buffer. Since a segment can be large (specified by a configuration parameter SegmentSize, we use 1.5 seconds by default), we define a configuration parameter (PlayerInterval) that specifies how frequently the player consumes data from the playback buffer. The default setting is ¼ of the segment size. This allows the player to operate on chunks of video data smaller than a segment. If the playback buffer does not have the entire amount of data requested by the player, this is an underrun. In most cases, an underrun occurs when the playback buffer is empty. The player will go into buffering mode if the playback buffer level drops to less than a configured threshold (specified by a configuration parameter lowbufferthreshold which is set to twice the segment size by default). We monitor the frequency and duration of rebuffering operations. A client request specifies the desired bitrate encoding, the range of data in the stream, and the amount of data to be sent. Client request message conform to the DASH (and HTTP 1.1) specifications. In our model, the data sent by the server has no meaning to the client. When the server receives a client request, it transmits the requested number of bytes (based on the segment s bitrate and duration) at the rate allowed by TCP. The client requests data as the video player consumes data from the playback buffer. The adaptation algorithm (described in the next subsection) runs prior to when the client issues a new request. Parameter SwitchEvent Table 1. Model Terminology Definition Client throughput sample at time interval i (units in bps). Available bandwidth estimate at time interval i (bps). Discrete bitrate encoder options in bps (1 through N). The bitrate of the segment that is being rered by the player at time interval i. The amount of time that the adaptation logic must wait before it performs the next adaptation (seconds). This variable is updated by the adaptation algorithm. Defines the short time scale that sets the throughput sample intervals ( set to seconds by default). The time when the last bitrate change occurred (seconds). The minimum time that is required after switching between the long time scale controller states (in either direction). The adaptation decision result: (SwitchUp, SwitchDown, or NoChange) When considering changing to a lower segment bitrate, the current bandwidth estimate must be greater than this threshold (bps). When considering changing to a higher segment bitrate, the current bandwidth estimate must be less than this threshold (bps).

4 Bandwidth Consumption (Mbps) Bandwidth Consumption (Mbps) A. Algorithm Table 1 defines the terminology that we use to describe the algorithm. We assume the algorithm operates in discrete time intervals based on a statically configured time scale I st. The client samples throughput ( A i ) are based on byte counts of data that arrive in time interval i. Throughput samples based on timescales of seconds can exhibit widely varying samples due to the on/off nature of DASH. Therefore, we apply a low pass filter to smooth the samples: The δ parameter is set to. by default. The variable AvailBW provides a smoothed estimate of the available TCP i bandwidth. The adaptation algorithm consists of two sequential steps that are executed with each new client request (but not more frequently than I st ). The first step compares the AvailBW to the Tup and Tdown thresholds to determine if the i bitrate could change. The B[ j] array holds the range of available bitrate encodings (based on our prior measurement study, we assume the following set of bitrates: Kbps, Kbps, 5Kbps, 1 Mbps, 1.5 Mbps,. Mbps, 3.5 Mbps,. Mbps). As described in [9], it can be difficult to reliably assess the true available path bandwidth due to the bursty nature of DASH. Therefore, we assume that the AvailBW i underestimates the actual bandwidth. Consequently, we purposely set Tup to a value that is below B i. Alternatively, the AvailBW i is checked if it exceeds the Tdown threshold. If either of these conditions are true, the logic will generate either a switchup or a switchdown event. if frozen TRUE; else frozen FALSE; switch (ControllerState) case(cautiousup) if (frozen==true) && (switchevent==switchup)) SwitchEvent NoChange if (frozen==false) && (switchevent==switchup)) dynamictime dynamicadaptiontime.5 * dynamicadaptiontime if (switchevent==switchdown) dynamictime minstabilitytime C CautiousDown case(cautiousdown) if (frozen==true) && (switchevent==switchdown)) SwitchEvent NoChange If (frozen==false) && (switchevent==switchdown)) dynamictime * dynamicadaptiontime if (switchevent==switchup) dynamictime minstabilitytime C CautiousUp Fig. 3. Controller i> i= The second step, described in Figure 3, involves an adaptation controller that operates at longer timescales. The controller is in one of two states: either it is looking to adapt the bitrate to a higher quality level (CautiousUp state) or it is looking to adapt the bitrate to a lower quality level (CautiousDown state). The controller maintains the time (1) before the next adaptation can occur (i.e. dynamicada ptationtim ). e i Figure illustrates the simulated network that is used in our study. The model loosely represents the system under observation during the measurement analysis. Each node configured to sink either a DASH stream, a TCP/FTP stream, or a UDP-based traffic stream. For all TCP connections, the ns TCP/Full/Sack model was used. Figure 5 visualizes results from simulations that are meant to duplicate (as close as possible) the measurement results illustrated in Figure. For the results illustrated in Figure 5a, the link between the Router and Router 3 was set with a link speed of 1 Mbps. Router s output interface was configured with a queue capacity of 1 packets. The link was configured with a random Bernoulli loss process that was applied to all packets traveling in the downstream direction during time through 1 seconds. Comparing Figure 5a with Figure a, we observe similarities and differences. The steady state bandwidth consumed during time -1 seconds is about 5% lower compared to the measurement result. Figure 5b does not illustrate the same level of unfairness as exhibited in Figure b. We could modify our implementation so that the simulation results are much closer to the measurement results. However we found those settings did not work well in all scenarios that we examined. Our objective was simply to use the measurement results as a guide. Netflix Servers Competing Traffic Generators/Sinks Fig.. DASH System Router 1 Mean: 17 bps Std: 159 bps. second samples 5 second samples a. Simulation Result: 3% loss rate -1 seconds Iperf(equivalent): Mean: 57 bps Std: 3 bps Mbps,.5ms prop delay 1Mbps,.5ms prop delay Router Router 3. second samples 5 second samples Netflix: Mean: 997 bps Std: bps b. Simulation Result: Competing Iperf flow -1 seconds Data rates: 1 Mbps (US and DS) Fig. 5. Simulation Calibration Results over emulated 1 Mbps link Node 1 Node n

5 IV. METHODOLOGY The system under study is quite complex. Due to space constraints, we only show a portion of our analysis (an exted version of this paper is available [11]). For the results presented in the next section, all simulations involve three flows: the DASH flow, a competing TCP flow, and a competing UDP that generates on/off traffic. All three flows s application data in the downstream direction (i.e., from servers to the node s connected to Router 3). The competing UDP flow starts at time seconds and the DASH flow starts at time.7 seconds. Scenario 1 uses the ns exponential traffic generator to s data at an average rate of 5 Mbps with an average off time of 1. second. The competing TCP flow starts at time and stops at time 1. The simulation time is seconds. Aggregate traffic causes the link between Router and Router 3 to be heavily congested. We conduct a total of 3 simulations for Scenario 1, varying the minstabilitytime and the playbackbuffercapacity. Scenario removes the randomness from the UDP on/off traffic generator such that the traffic generator ss at a constant rate for a fixed amount of time and then goes idle for a fixed off time (and repeats the cycle forever). The UDP fixed rate to 7 Mbps. The on time is set to 5 seconds and the off time is varied such that the average sing rate of the UDP flow ranges from 57 Kbps to 3. Mbps. The competing TCP flow starts at time and runs until the of the simulation (3 seconds). The playbackbuffercapacity is set to 1 seconds and the minstabilitytime is varied as in the previous scenario. A total of 3 simulations are performed. A. Performance Metric Definitions As pointed out in [, 1, 13], a QoE assessment that is appropriate for DASH must include components such as average bitrate, the rate of playback buffer underruns, the rate of bitrate adaptations, and the time it takes to fill the buffer before rering can begin (or restart). Our simulation model maintains counts of bitrate adaptions, playback buffer underruns, and rebuffer events. For the results presented in this paper, we show only the rebuffering rate (normalized to events per hour). In addition we assess video quality using a straightforward application of PSNR as described below. We use mean squared error (MSE) to quantify how the received video stream compares to the original source stream. We assess the video session by breaking the signal into fixed intervals. We define the source signal, X i, to be the highest possible video bitrate for segment i (equation ). The video stream is received and decoded by the player. The physical representation of this signal, Y, is () depicted by Y (3) equation 3. The MSE of the MSE(x,y) () received signal is defined in equation. We define a peak signal-to-noise ratio (PSNR) as given in equation 5. In the literature, L represents the dynamic range of the (5) signal. When applied to DASH, the range of the signal deps on the allowed bitrates. Therefore, we define L to be the maximum allowed bitrate. This provides us a convenient objective metric that does not require a reference but leverages the fact that DASH is generally encoded to meet a target bandwidth consumption. In our results, we find that the value ranges from about to about 115. For each interval obtained when an underrun occurs, we assign a value of. This causes the DPSNR to be sensitive to possible artifact situations. This makes intuitive sense however further study is required to better calibrate the metric. V. RESULTS AND ANALYSIS A. Calibration Results The first two data rows shown in Table document the metric results for the calibration simulations visualized in Figure 5. The next pair of rows in the table shows the results when we run the same calibration simulations but without the DASH adaptation (the client always requests data with the highest bitrate of. Mbps). The single flow simulation demonstrates the efficacy of DASH. Without adaptation, the average videoplayerrate is higher than when adaptation is enabled (3.53 Mbps versus 3.9 Mbps respectively). However since there are more underrun events the DPSNR is low in both cases. The playbackbuffercapacity was set to 1 seconds. The average buffer size is similar for both cases (about seconds). The rebuffering event rate is slightly higher with no adaptation but more significantly, the average duration of a rebuffering event is larger (7.93 seconds versus 1.57 seconds). The final column statistic indicates that with no adaptation the session is in buffering mode 1.3% of the time versus.% of the time when adaptation is enabled. The two flow results in the table illustrate a case where DASH adaptation might actually decrease the perceived quality (the DPSNR is when adaptation is disabled while 19 when adaptation is enabled). The last two rows in the table are with a modified adaptation algorithm we describe later in the paper. Description Single Flow, (Figure 7a) Two Flows, (Figure 7b) Single Flow, No Two Flows, No Single Flow, Enhanced Two Flows, Enhanced Video Playback Rate (Mbps) DPSNR Table. Calibration Metric Results Average Playback Buffer Size (seconds) Underrun Rate (per hour) Rebuffering Event Rate (per hour) Average Rebuffering Event Time (seconds) % % % % % % Percent Session Time in Buffering Mode B. Scenario 1 and Results Figure illustrates the results of Scenario 1. Figure a shows that the DPSNR drops slightly as the playbackbuffercapacity gets smaller. The figure also suggests that the DPSNR drops slightly as the minstabilitytime gets

6 Average Rebuffering Rate (per hour) Average Rebuffering Rate (per hour) Average Playback Buffer Size Average Rebuffering Rate (per hour) Average Rebuffering Rate (per hour) DASH PSNR Average s / Hour smaller. DPSNR is more sensitive to the playbackbuffercapacity than to the minstabilitytime. Figure b shows that the minstabilitytime parameter has the desired dampening effect on the frequency of adaptations. Figure c shows that the average playback buffer size is sensitive to the minstabilitytime only at the higher settings of the playbackbuffercapacity Figure d shows that rebuffering events begin to occur once the playbackbuffercapacity is 15 seconds (or lower). The rebuffering rate increases as the minstabilitytime increases. This is because the adaptation reacts more slowly as the minstabilitytime increases Playback Buffer Capacity (seconds) Playback Buffer Capacity (seconds) a. DPSNR b. Rate YELLOW, or RED. A bufferstate of GREEN indicates the curbuffersize is at least /3 the dynamicplaybackbuffer- Capacity. A bufferstate of YELLOW indicates the curbuffersize is between 1/3 and /3 the capacity. Finally, a bufferstate of RED implies the curbuffersize is less than 1/3 the capacity. When the adaptation controller generates a SwitchDown event and if the bufferstate is RED, the bitrate reduction is set to three bitrate levels below the current (or it is set to the lowest quality setting). If the bufferstate is YELLOW, the bitrate reduction is set to two levels below the current. Finally, if the bufferstate is GREEN, a single bitrate level reduction is performed. When a buffer underrun occurs and the controller is in RED state, the dynamicplaybackbuffercapacity is doubled When a buffer underrun occurs and the controller is in the YELLOW state, the buffer capacity is increased by 5%. If the playback buffer is in the GREEN state for 5 consecutive buffer times, the dynamiplaybackbuffercapacity is increased by 5%. The updates are not allowed to set the dynamicplaybackbuffer- Capacity to be less than minplaybackbuffercapacity or greater than playbackbuffercapacity. By default, we set the minplayerbuffercapacity to a value of *segmentsize Playback Buffer Capacity (seconds) 5 Playback Buffer Capacity (seconds) 5 d. Rebuffering Event Rate c. Average Playback Buffer Fig.. Scenario 1 Simulation Results UDP CBR Off a. Scenario Figure 7 illustrates the results associated with Scenario. The mean sing rate of the exponential UDP traffic generator is varied by adjusting the traffic generator s off time. The idea is to produce controlled bursts of congestion. We ran the Scenario three times. First, using the adaptation algorithm described in section III. Second, with the adaptation algorithm disabled. Third, with an enhanced adaptation algorithm (described below). Comparing Figures 7a and 7b suggests that adaptation can decrease the frequency of rebuffering events. Additionally, the DPSNR result (not shown) suggests that adaptation increases the video quality slightly. Prior work suggests that rebuffering events have a greater impact on perceived quality than reduced video quality. Scenarios 1 and show that increasing the playbackbuffer- Capacity can reduce the rate of rebuffering. However, there are reasons (identified earlier in the paper) to minimize the average buffer size. We introduce the following incremental modifications to the adaptation algorithm. We assume the configured playbackbuffercapacity now represents the maximum allowed capacity and is set by default to a large value such as 51 seconds. We introduce a variable referred to as the dynamicplaybackbuffercapacity to represent the best playback buffer size. The algorithm monitors the size of the playback buffer (referred to a curbuffersize). We define the state of the playback buffer, bufferstate, to be either GREEN, 5 3 Fig. 7. Scenario Simulation Results 1 UDP CBR Off UDP CBR Off b. Scenario 3 No c. Scenario 3 Enhanced Figure 7c illustrates Scenario with the modified adaptation algorithm. The results show that the enhanced algorithm is able to reduce the rate of rebuffering events. The average playback buffer size is kept to roughly 75 seconds compared to an average size of seconds in the base

7 algorithm. The improvements, however, come at a cost of decreased video quality. The DPSNR drops by more than 5%. with the enhanced algorithm. RELATED WORK Quite a bit of recent work has focused on protocol performance evaluations [7-9, 1-17]. The work in [9] is helpful in understanding the complex interactions between the adaptation algorithm and TCP. We however do not agree with the authors recommations to use large playback buffers. The work in [7] and [] is similar in spirit with our work as they incorporate QoE assessment in the analysis methodology. [7] introduces an instability metric that quantifies the level and relative weight of bitrate switches. This exts our simpler frequency of bitrate adaptations metric by looking at the evolution of bitrate adaptations. The average rebuffering event duration that we compute addresses the temporal effects of artifacts. The work in [] performed subjective tests to determine which bitrate adaptation behaviors lead to the highest overall perceived quality. We note that none of these recent studies directly address the core issue of if bitrate adaptations actually improve the perceived quality. CONCLUSION In this paper we address the question if the DASH adaptation can (and under what circumstances) lead to beneficial results. Prior research that has developed methods for assessing the perceived quality of video does not directly apply to DASH. We have introduced a set of metrics that provide assessments of the required components: video quality (average videoplaybackrate, DPSNR), artifact assessment (playback buffer underrun rate, rebuffering rate, average rebuffering event duration), and playback buffer usage (average playback buffer size). We have studied (through simulation) an adaptation algorithm that exposes several key parameters for experimentation. Our results suggest that the playbackbuffercapacity and the minstabilitytime parameters have a significant impact on observed performance. Our analysis included comparing results both when DASH adaptation is enabled and then disabled. The results suggest that in some situations DASH can increase video quality and reduce the rate of observed artifacts. Although in other situations, the rate of observed artifacts decreases but at a cost of reduced video quality (i.e., DPSNR). In summary, we provide evidence that supports the efficacy of DASH, however finding an algorithm that ensures the right decision is always made is quite challenging. In ongoing work, we treat the adaptation algorithm as a multi-attribute decision making problem. This is a well-established branch of operations research. The challenge however is finding the best weights to assign to the various decision components. Our next step is to conduct subjective user tests to establish a basis for formulating the optimization. REFERENCES [1] SAN1] Global Internet Phenomena Report, Sandvine Corporation, H 1, available online: documents/phenomena_h_1/sandvine_global_internet_phenomen a_report_h_1.pdf. [] [3GPP1] 3GPP TS.7 version 1.1. Release 1: Transparent End-to- Packet Switched Streaming Service (PSS); Progressive Download and Dynamic Adaptive Service over HTTP, 3GPP, January 1, available online at ts/1_199/17/1.1._/ts_17vp.pdf [3] [ISO11] ISO/IEC, Information technology MPEG systems technologies Part : Dynamic adaptive streaming over HTTP (DASH), Jan 11, available online at chiariglione.org/working_documents/mpeg-dash/dash-dis.zip [] [DAJ11] F. Dobrian, A. Awan, D. Joseph, A. Ganjamm J. Zhan, V. Sekar, I. Stoica, H. Zhang, Understanding the Impact of Video Quality on User Engagement, Proceedings of SIGCOMM 11, August 11. [5] [CPM] N. Cranley, P. Perry, L. Murphy, User Perception of Adapting Video Quality, Internation Journal of Human-Computer Studies,. [] [MT11A] C. Muller, C. Timmerer, A Testbed for the Dynamic Adaptive Streaming over Http Featuring Session Mobility, Proceedings of ACM MMSys, February 11. [7] JSZ1] J. Jiang, V. Sekar, and H. Zhang, Improving Fairness, Efficiency, and Stability in HTTP-based Adaptive Video Streaming with Festive, Proceedings of CoNEXT 1, December, 1. [] MLC1 R. Mok, X. Luo, E. Chan, R. Chang, QDASH: A QoE-aware DASH System, Proceedings of the the ACM MMSys, December, 1. [9] [HHH1] T. Huang, N. Handigol, B. Heller, N. McKeown, R. Johari, Confused, Timid, and Unstable: Picking a Video Streaming Rate is Hard, Proceedings of the IMC 1, November, 1. [1] [MFW13] J. Martin, Y. Fu, N. Wourms, T. Shaw, Characterizing Netflix Bandwidth Consumption, Proceedings of the IEEE CCNC, January 13. [11] [MF13] J. Martin, Y. Fu, G. Hong, On the Efficacy of the Dynamic Adaptive Streaming Over HTTP (DASH) Protocol Exted Version, TechnicalReport,Availableonline at: ~jmarty/papers/efficacydashexted.pdf, 13. [1] [BSA1] A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica, H. Zhang, A Quest for an Internet Video Quality-of-Experience Metric, Proceedings of the ACM HotNets 1, October 1. [13] [OS1] O. Oyman, S. Singh, Quality of Experience for HTTP Adaptive Streaming Services, IEEE Communications Magazine, April 1. [1] ABD11] S. Akhshabi, A. Begen, C. Dovrolis, An Experimental Evaluation of Rate- Algorithms in Adaptive Streaming over HTTP, Proceedings of ACM MMSys, February 11. [15] [LBG11] C. Liu, I. Bouazizi, M. Gabbouj, Rate for Adaptive HTTP Streaming, Proceedings of ACM MMSys, February 11. [1] [LMT1] S. Lederer, C. Muller, C. Timmerer, Dynamic Adaptive Streaming over HTTP Dataset, Proceedings of ACM MMSys, February 1. [17] [AAD1] S. Akhshabi, L. Anantakrishnan, C. Dovrolis, A. Begen, What Happens when HTTP Adaptive Streaming Players Compete for Bandwidth, Proceedings of ACM NOSSDAV 1, June 1.