Contents Executive Summary... 1 Overview... 2 Progressive Video Primer: The Layers... 2 Adaptive Video Primer: The Layers... 3 Measuring the Stall: A TCP Primer... 4 Conclusion... 5 Questions to Ask of a Video Quality Reporting Product... 6 Executive Summary TCP is a fine protocol that has been with us for many years. In its earliest incarnation, it just went as fast as it could, all the time, and if packets got dropped they were retransmitted. In the mid 1980s, the Internet began to collapse under the load of congestion. Then an American computer scientist named Van Jacobson invented a method by which TCP will, on packet drop, slow down so it stops dropping. Now, when your 1 Mbps home connection tries to pull content from a server capable of 1 Gbps, you don t experience 99.9% packet drops. Nevertheless, even people in the industry continue to believe that when demand exceeds capacity, the entire difference is simply dropped. The assumption is that network throughput capability and reliable quality measurements can be derived by examining TCP packet loss, but this is not the case. In fact, TCP rarely drops any packets, and the drops that do occur are not correlated at all to a subscriber s perceived quality of experience for what is currently the most prevalent and bandwidth-intensive form of real-time Internet entertainment over-the-top video. Still, there are those who promote TCP Packet Loss (typically presented as a percentage of dropped packets) as the ultimate measure of network quality. Such a claim is either deliberately misleading or the result of genuine ignorance. Regardless of the actual cause, like neon colors and big hair, it belongs in the 80s. In this paper, Sandvine thoroughly explores the issues surrounding the confusion about TCP protocol behavior and presents the actual requirements for the accurate measurement of OTT video quality.
Overview Third-party Video applications that run over the top of a communication service provider s transport layer, such as Netflix and Youtube, are the dominant drivers of bandwidth on today s consumer Internet, as shown repeatedly by Sandvine s semi-annual Global Internet Phenomena Reports. This type of video is responsible for two fundamental shifts in consumer behaviour: higher peak bandwidth levels (video is on demand), and higher quality demand (you notice when video is lower quality than expected). As a consequence, it is driving a lower efficiency in the network, with wasted off-peak capacity and the need for oversupply at peak times to avoid congestion effects. There are two primary means of streaming video on the Internet today - adaptive and progressive. With the adaptive video method, the display quality of the video is modulated to the bandwidth available in a piece-wise fashion. In the progressive method, a single large file is burst-paced into the network, with no consideration for available bandwidth. There are other video methods available (e.g., RTP multicast over UDP), but they are not in common use today on the over-the-top (OTT) Internet. HTTP over TCP is the dominant transport mechanism. As a consequence of video being delivered by HTTP over TCP, there are two primary aspects to video quality: 1. Display quality (fidelity) is the image quality sufficient for the device s screen size? 2. Transport quality (stalling) how long does the video take to start, and does it play smoothly? Jitter and packet-loss are not directly relevant in a TCP-delivered video world since TCP itself will retransmit and buffer. As a consequence, traditional test and measurement methods are not effective for determining OTT video quality. Progressive Video Primer: The Layers With progressive video, the media asset is a single file stored on a server, and HTTP byte-ranges are used for seeking through the video. For situations where videos are presented with different resolution quality choices (bitrate), the video client must manually select a different file on the server. For example, with Youtube a subscriber can select the fidelity when they do this, a separate file is loaded from the server with a different quality and bitrate. Figure 1: Progressive video stream anatomy Sandvine obtains the following info from each layer: IP: Subscriber, CDN, BGP AS Path Subscriber: physical location on network, service plan, device type TCP: <nothing> HTTP: asset (used to link multiple chunks together), duration, stall information (transport quality Container: codec, resolution, bitrate (display quality), CDN Elementary stream: bytes transferred Page 2
The video client can seek in the file by selecting alternative byte-ranges, either because the subscriber moves a control (such as a slider) or to re-buffer on a stall. Video duration can only be determined by keeping track of this information dynamically for all HTTP transactions and not just the initial GET on an HTTP flow. In addition, since HTTP typically allows pipelining and persistent requests, to accurately capture video Figure 2: HTTP Progressive video 'flows' duration the Network Policy Control solution must take multiple measurements throughout the life of a TCP flow. It is important to be able to correlate the same asset across multiple GET transactions issued into the same, as well as different, TCP connections as a matter of accurate state tracking (associating videos with the subscribers who are accessing them). Since video is typically long in duration, interim measurements are required. In practice, ~15s - ~30s interim reporting provides a good contour of delivered progressive video. Adaptive Video Primer: The Layers Adaptive video is similar to progressive. The key difference lies in an additional protocol layer. Adaptive OTT video also uses several layers to get its data to the client. In addition to the traditional IP & TCP layers, HTTP is used to find the media asset. A protocol layer is tacked Figure 3: Adaptive video stream anatomy on top of HTTP, giving parameters for seeking, obtaining meta-data, etc. After this, a container is sent which contains one or more elementary streams (e.g., video, audio, sub-title, etc). Sandvine obtains the following info from each layer: IP: Subscriber, CDN, BGP AS path (same as progressive) Subscriber: physical location on network, service plan, device type (same as progressive) TCP: (same as progressive) HTTP: asset (used to link multiple chunks together), protocol, CDN Protocol: duration, stall information (transport quality) Container: codec, resolution, bitrate (display quality) Elementary stream: bytes transferred (same as progressive) In the same fashion as progressive video, each chunk may either come in the same TCP connection or a new connection. Byte ranges are replaced by chunk names encoded in the protocol. The most common adaptive video methods are Microsoft Silverlight (used by Netflix), Apple HTTP Live Streaming (HLS), and Adobe Real-time Media Protocol (RTMP). They each operate in a similar fashion: the video asset is stored on disk in multiple target bitrate/resolutions. Page 3
The video is then split up into chunks (approximately 10-100MB in size, depending on bitrate). Figure 4: Adaptive video and the network The video client requests the first chunk, and starts playing it. If a chunk takes too long to deliver, the client will request a lower-quality chunk in the next time interval. To properly measure the delivered quality of adaptive video (i.e., display quality and transport quality), a network device must be protocol-aware (e.g., HLS), container-aware (to see the lowered resolution/bitrate/codec), and also measure stalling in the backchannel from the client. In addition, to measure the duration of the video, the size of the transfer is irrelevant (since it changes rates dynamically). Only the container information is valid. Since video is typically long in duration, interim measurements are required. To avoid a Nyquist effect, whereby insufficient resolution produces only a crude approximation of the true video quality, there must be two or more measurements per chunk of video. In practice, reporting on ~15s-30s intervals is sufficient. Measuring the Stall: A TCP Primer In the earliest days of the Internet, TCP had no congestion management: it would simply go as fast as it could with no throttle. In 1984, RFC 896 predicted the need for a congestion control algorithm, and this was implemented in 1987 by American computer scientist Van Jacobson. The Van Jacobson algorithm used packet loss to signal that a link was operating at maximum capacity and cause the transmitter to slow. In general, much confusion still exists regarding the significance of packet loss. Many find it odd that in both normal and congested situations packet loss occurs on the network. How can this be? The answer lies in the behavior of TCP. Consider Figure 5, where a client application has a 100Mbps connection to its first hop, and a server has a 1 Gbps connection to its first hop what does TCP do when we fetch a file? Is 900 Mbps dropped, or 990 Mbps? In fact the quantity of dropped packets is negligible. The server starts accelerating, going faster and faster until a packet is dropped, and then it slows down slightly. In this diagram, the achieved rate is 10Mbps because the end-to-end connection adjusts to the slowest Figure 5: Typical single Internet path Page 4
link in the chain. Thus we cannot use TCP packet loss as a means of finding the narrowest path - it will stop dropping once it determines the rate of the narrowest link. If we assume the server shown in Figure 5 has a video stream which must be delivered @ 12Mbps to avoid stalling, there will still be no TCP packet loss to indicate the stalls. The only indication will come from the higher layer signaling in the client. If a quality measurement solution cannot detect that signaling, then it has no ability to measure this aspect of video quality. In fact, TCP packet loss only exists to any amount when there are drops which are uncorrelated to bitrate (e.g., when you are using your home WiFi and someone generates interference by turning on their microwave or cordless phone). Conclusion There are those who promote TCP Packet Loss as the ultimate measure of network quality. Such a claim is either deliberately misleading or the result of genuine ignorance. Regardless of the actual cause, like neon colors and big hair, it belongs in the 80s. TCP packet loss is not an accurate indicator of network throughput capability or OTT video quality. To measure video transport quality, one must have a view of where the video decoder is situated in its clock cycle (which picture is being decoded), and the buffer s progression. Figure 6: Video progress Page 5
Questions to Ask of a Video Quality Reporting Product 1. Can it inspect multiple HTTP GET transactions in the same TCP connection? 2. Can it inspect and correlate multiple HTTP GET transactions across multiple TCP connections? a. On a single device? b. On multiple devices in an asymmetric network (since each TCP connection will have a different path in general)? c. When video header information is split across multiple TCP connections? 3. Does it handle adaptive video streaming such as: a. Apple HTTP Live Streaming (HLS)? b. Microsoft Silverlight (smooth streaming)? c. Adobe RTMP? 4. 5. 6. 7. 8. For adaptive protocols, does it keep track of the display quality per chunk? Does it report on an interim basis or just on fixed time grid / end of video? Does it record true duration, or merely interpolate from bytes and average bit-rate? Does it measure transport stalls based on the protocol, or merely use TCP packet loss? Can it segment measurements simultaneously? a. Per device? b. Per CDN? c. Per provider Page 6
Headquarters Sandvine Incorporated ULC Waterloo, Ontario Canada Phone: +1 519 880 2600 Email: sales@sandvine.com European Offices Sandvine Limited Basingstoke, UK Phone: +44 0 1256 698021 Email: sales@sandvine.co.uk Copyright 2013 Sandvine Incorporated ULC. Sandvine and the Sandvine logo are registered trademarks of Sandvine Incorporated ULC. All rights reserved. 2013-08-13