Adaptive Forward Error Correction for Real-time Internet Video Mathias Johanson Alkit Communications Sallarängsbacken 2 S-431 37 Mölndal, Sweden mathias@alkit.se Abstract Real-time video communication over packet networks is attracting increasingly more attention for applications such as distributed collaborative work, teleteaching and telemedicine. Due to its sensitivity to delay, real-time communication is usually based on unreliable transport protocols, like UDP, since retransmission of lost packets is considered to take too long time. Instead of retransmissions forward error correction schemes have been proposed to make the applications more resilient to packet loss. Since the packet loss rate on the Internet typically fluctuates depending on the load of the network, the amount of redundancy applied by the error correction coding should ideally be adaptive to the amount of loss experienced on an end-to-end connection. In this paper an adaptive forward error correction scheme, based on Reed-Solomon erasure coding and receiver feedback, is presented. Moreover, an optimal packetization strategy for error protected video communication is suggested and an efficient algorithm implementing this packetization scheme is deduced. The performance gain of the scheme, in terms of increased network bandwidth utilization, is analyzed and found to be as high as around 10% for some relevant usage situations. The combination of adaptive forward error correction schemes with congestion control algorithms is also discussed in the paper. 1. Introduction Traditional computer communication applications, like file transfers or email, typically require reliable transmission of data. In packet-switched networks this is achieved through transport protocols implementing some sort of automatic repeat request (ARQ) technique. In the Internet the transmission control protocol (TCP) provides this functionality transparently to applications. However, for delay sensitive applications like real-time audio and video tools, ARQ schemes are not suitable since the delay imposed by retransmissions cannot be tolerated. Moreover, a limited amount of packet loss is generally not catastrophic for these types of applications, but rather leads to a graceful degradation of performance. In the case of persistent loss, however, audio and video quality will be continually low. In response, open loop error control schemes based on forward error correction (FEC) have been proposed for both audio and video applications [1][2][3][4]. FEC techniques rely on the transmission of redundant information, usually in the form of error correcting codes, from which lost data packets can be recovered. In addition to reducing the time needed to recover lost data packets compared to ARQ schemes, FEC-based mechanisms also have better scalability properties for multicast applications, since no feedback is necessary [5]. Transmission errors in computer networks can be divided into two classes: Random bit-errors due to noisy communication channels and packet drops due to congested links. For packet networks, like the Internet, biterrors will be detected by cyclic redundancy checksums (CRCs) and the corrupted packets will be discarded. Thus, both random bit-errors and packet drops due to congestion will show up as packet loss to the applications. Consequently, Internet applications are only required to deal with packet loss, not with bit-errors. In wired networks (in particular optical networks) bit-errors are extremely rare, making congestion the overwhelmingly dominant cause for packet loss. This means that care must be taken when designing an adaptive FEC-scheme, so that the applications not simply increase the amount of redundancy transmitted when the packet loss rate increases, since this will effectively exacerbate the congestion. Instead, the data transmission rate must be lowered in favour of more redundant information, and the total transmission rate must be controlled by the transport protocol s congestion control algorithm. This suggests that a unified approach to congestion control and error control is needed for real-time Internet applications, much like the combination of the congestion avoidance algorithm with the retransmission scheme in the TCP protocol [6]. For wireless networks, however, bit errors can be relatively frequent. Thus, a lost packet cannot automatically be assumed to be the result of congestion, indicating that congestion control algorithms need more input than packet loss events to be efficient in wireless networks.
2. An Adaptive FEC Model for Internet Video Although audio and video applications are certainly more tolerant to packet loss compared to data transmission applications that require reliable transport, even moderate loss rates can have a significant impact on the perceived audio and video quality if the loss is persistent. This is most pronounced for video, since video codecs are often more sensitive to loss. Moreover, since a video frame typically spans a number of packets, losing only one of these packets will often result in the whole frame being discarded. To make things even worse, interframe compression schemes, wherein a video frame is temporally predicted from a past frame, are often used for Internet video applications, causing the loss of one packet to potentially result in several frames being dropped. Using a video coding with only intra-coded frames makes the video more robust to packet loss, but at the expense of requiring significantly higher bandwidth. Even for intra-coded video persistent packet loss will be a problem, indicating than an FEC-scheme is needed. FEC schemes can be used both to correct bit-errors and to reconstruct lost source data symbols (known as erasures). As previously mentioned, we only need to consider erasures for packet data networks, not bit-errors. Since IP packets usually are very long (typically several hundred bytes) compared to the symbol length used for the FEC coding, a lost packet corresponds to a very long burst of erasures. Although robust FEC-schemes have been developed that can handle relatively long bursts, applying the FEC coding directly to the IP packets would be highly wasteful, since we know that whenever one symbol in a packet is lost all other symbols in that packet will also be lost. The solution is to buffer a number of packets before transmission and to apply the erasure codes orthogonally across the packets, as illustrated in Figure 1. For real-time, interactive applications like teleconferencing, sender side buffering is generally problematic since it introduces delay. However, for video applications, a video frame typically needs to be fragmented into a number of packets, because its size is generally larger than the network s maximum transmission unit (MTU). The higher the resolution of the video, the more packets per frame will be needed. In a typical application a video image will roughly require from around five to around thirty packets, depending on the resolution of the video and the frame coding mode (intra- or inter-coding) and the efficiency of the compression scheme. Thus, without requiring any additional buffering, we can apply the FEC coding across the packets corresponding to one frame of the video. For real-time audio applications, this method will generally not work very well, since the size of the encoded audio data sampled over the packetization interval (i.e. the time between two consecutive audio packets) is much smaller and cannot efficiently be fragmented into as many packets as would be needed. symbol length video data codeword redundancy k data packets n-k redundant packets Figure 1 Organization of packets for FEC coding 2.1. Adaptive Reed-Solomon Coding An error correction scheme where k source data symbols are encoded at the sender to produce a codeword of n symbols of encoded data, such that any subset of k encoded symbols suffices to reconstruct the source data, is called an (n, k) code (cf. Figure 1). There are a number of different ways to construct the error correcting codes. One of the most popular schemes is the Reed-Solomon (RS) code, which is very well suited for error protection against packet loss, since it is a maximum distance separable code, which means that no other coding scheme can recover lost source data symbols from fewer received code symbols [7]. For packet video the symbol length used when calculating the RS codes is usually eight bits. Generally an encoded video frame doesn t fit perfectly into an integral number of MTU-sized packets. When calculating the FEC packets, the last packet of the frame is simply padded to the same size as the other packets. (The padded bytes are only needed when computing the FEC codewords and are thus not transmitted.) If a verbatim copy of the source data is contained in the encoded data the code is called a systematic code. This is a very useful property, since it makes the decoding process extremely simple in case none of the k data packets are lost. The redundant packets are then just discarded. This also means that receivers of the video stream not capable of FEC decoding can still decode the video in the absence of packet loss. The adaptive FEC scheme proposed in this paper is based on systematic RS(n, k) codes, where n and k are reassigned for each
frame to be transmitted based on the optimal number of packets to fragment the frame into and the current loss rate of the communication channel. In order for a receiver to be able to reconstruct the original packets from the received packets, the positions in the packet stream of the lost packets must be known. For applications using the Real-time Transport Protocol (RTP), this information is easily obtained by monitoring the sequence number field of the RTP header. In the absence of mechanisms for guaranteed Quality-of-Service (QoS) in the Internet, the packet loss rate as experienced by applications fluctuates with the load on the network. In order for the video transmitter to apply an adequate amount of redundancy, the receiver monitors the loss rate and continually reports this back to the sender. For multicast applications, feedback from a set of heterogeneous receivers must be harmonised by the sender. To this end, different strategies can be adopted, such as adapting the redundancy to meet the loss rate of the receiver experiencing the highest loss rate, or adapting the redundancy to the mean loss rate calculated over all receiver reports. Also, as for all feedback mechanisms in multicast communication, the feedback interval must be scaled to the size of the session to avoid feedback implosion. 2.2. Error correction and congestion control As previously mentioned it is important that the application keeps a unified strategy for congestion control and error correction when responding to packet loss rate reports from the receiver(s). Since the cause of the packet loss is likely to be congestion, a naïve implementation responding to increased packet loss solely by making the FEC code stronger will in effect worsen the congestion. This will cause the application to increase redundancy even more and this loop will continue ad nauseam, until all available bandwidth is consumed. The simplest remedy for this problem is to keep the output rate of the sender constant, reducing the video codec s rate when increasing the FEC redundancy. Alternatively, a rate based congestion control algorithm could be used to control the total output rate. For instance, the TCP-friendly rate control algorithm (TFRC) [8] proposed by Floyd et al. could be used in circumstances where fairness to TCP flows is desirable. TFRC tries to model TCP s long term performance while allowing the application to change its sending rate more smoothly, compared to TCP s halving of the throughput in response to a loss event. It is important to note that while both the adaptive FEC scheme presented here and TFRC react on receiver reports of packet loss, the FEC scheme should be based on the long term, persistent loss rate, whereas congestion control algorithms like TFRC are based on the loss event rate. An interesting opportunity when integrating adaptive error correction and congestion control algorithms is the possibility of increasing the robustness of the FEC coding proactively in situations when the congestion control algorithm is probing for bandwidth, e.g. the slowstart phase of the TFRC algorithm or when conducting join experiments in receiver-driven layered multicast [9]. Since all loss-based congestion control algorithms in effect provoke packet loss in order to detect congestion, the negative effects of this can be reduced by applying a suitable amount of error correction redundancy when packet loss can be foreseen by the congestion control algorithm. 3. Optimal packetization for error protected video In the adaptive FEC scheme described above the packet size was assumed to be equal to the MTU size of the network, i.e. around 1500 bytes for most practical situations. Generally, when packetizing video data the packet size should be kept as large possible to keep the overhead imposed by IP/UDP/RTP headers at a minimum. However, since the FEC coding is performed at packet level the granularity of the error protection is determined by the packet size and the size of the coded video frame. As an example, if a video frame to be transmitted is 10 kbytes in size a packet payload size of one kilobyte will result in ten frames. This makes the precision at which the error protection can be applied be in units of 10% (i.e. one packet of redundancy will make the transmission robust against packet loss rates up to 10%, two packets of redundancy can withstand up to 20% losses, and so on). Thus, if we measure a loss rate of for instance 5% we will have to use an overly strong FEC code, wasting bandwidth unnecessarily. If we instead use a packet size of 500 bytes we will get a protection granularity of 5% matching the measured loss rate perfectly, but causing higher IP/UDP/RTP header overhead. The optimal packetization strategy, from a bandwidth utilization standpoint, is thus a trade-off between error protection granularity and header overhead. To find this optimal packetization strategy, let I be the size of the video frame in bytes, let s be the packet size in bytes, let h be the header size in bytes and let p be the current packet loss rate. The utilization, u, i.e. the proportion of the bandwidth consumed by the application that is used for actual video data, will then be given by I u(i,s, h,p) =. (1) I p I + s s h s h
To achieve an optimal packetization strategy we should select s so that u is maximized for the given values of I, h and p. In Figure 2, u is plotted against s for I=10000 and h=40 at four different loss rates (p=0, p=0.01, p=0.05 and p=0.1 respectively). 1.0 utilization 0.8 0% 1% 5% 10% 0.5 0 500 1000 1500 packet size (bytes) Figure 2 Bandwidth utilization as a function of the packet size for error protected transmission of a 10 kbytes video frame at loss rates of 0%, 1%, 5% and 10% As can be expected, the utilization increases monotonically with the packet size when there is no loss. For the more interesting cases where there is packet loss, the utilization can be seen to be given by piecewise continuous functions generated from the functions obtained by setting the ceilinged expression in (1) to 0, 1, 2, etc. Although there is no closed form expression for the maximum of the utilization function, we can derive a simple algorithm to calculate it. To do this we introduce the helper functions I u h (s) =, where a is a nonnegative integer. (2) I + a s s h Plotting u h (s) for I=10000, h=40 and a=0, 1, 2, 3 gives the functions shown in Figure 3. 1.0 u h 0.8 0.5 0 500 1000 1500 s a=0 a=1 a=2 a=3 Figure 3 Helper functions It is easy to see that the utilization functions at different packet loss rates (depicted in Figure 2) can be generated by a certain combination of the helper functions (depicted in Figure 3). The endpoints of the intervals where the utilization function coincides with the same helper function can be found by solving for s in p I p I p I a =. This gives + h < s + h. (3) s h a + 1 a Now, it is readily realized that the maximum of the utilization function is achieved either at one of the transition points between the helper functions, defined by (3), or at the local maximum of one of the helper functions. The local optima of the helper functions can be easily found using standard analytical methods to be given by I h s = h +. (4) a We can now find the maximum of the utilization function by iteratively examining the candidate optima given by the interval endpoints of (3) and the local optima given by (4). In Figure 4 a pseudocode segment is listed that finds the optimal packet size given the current loss rate, p, the current video frame size, I, the header size, h, and the maximum packet size, S. The result will be in the variable s. The algorithm works iteratively by testing candidate maximum values, starting with points located on the helper function given by setting a=1 in (2)
and then increasing a by one for each iteration. The loop terminates when the best candidate maximum value tested is found to be greater than the maximum of the helper function given by the previous value of a. Since the helper functions never intersect, this condition guarantees that the maximum value has been found and the algorithm is done. k := ceil(p*i) if (k=0) then s := S else s := h+1 a := 1 prev_sa := s while (a<=k and u(i, h, p, prev_sa) >= u(i, h, p, s)) do Sa := ceil(h + sqrt(i*h/a)); if (p*i/(a+1)+h < Sa and (Sa <= p*i/a+h or a=1) and u(i, h, p, Sa) > u(i, h, p, s) and Sa <= S) then s := Sa Se := ceil(p*i/a+h) if ( u(i, h, p, Se) > u(i, h, p, s) and Se <= S) then s := Se prev_sa := Sa a := a+1 done Figure 4 Pseudocode implementation of an algorithm that finds the optimal packet size for FEC-coded video. The function ceil( ) implements the ceiling operation and the function u( ) is the function given in (1). The algorithm described above is executed by the sender for each video frame to be transmitted. For nonpathological choices of the variables I, p, h and S, the algorithm finds the optimal packet size within only a few iterations of the loop. Compared to the complexity of the video compression and the FEC encoding process the added complexity of the packetization algorithm is negligible. The only requirement on the receivers is that they must be able to handle video packets of variable size. The optimal packet size, as calculated by the algorithm above, is plotted in Figure 5 for I=10000, h=40 at different loss rates. It is clear from Figure 5 that the optimal packet size varies between about 750 bytes and the maximum packet size (1500 bytes). 2000 packet size 1500 1000 500 0 0.0 0.5 1.0 loss rate Figure 5 Optimal packet size at different loss rates 3.1. Performance gain The performance gain in terms of efficient utilization of network bandwidth when using an optimal packet size is plotted in Figure 6 for I=10000, h=40, S=1500 at different loss rates. The performance gain is calculated as the difference between the bandwidth utilization when using the maximum packet size and the utilization when using the optimal packet size. As can be seen in Figure 6, the performance gain is between 0 and 7.5% depending on the loss rate. The cases where the performance gain is zero correspond to the situations where the optimal packet size is equal to the maximum packet size.
0.08 utilization gain 0.06 0.04 0.02 0 0.0 0.5 1.0 loss rate Figure 6 Performance gain of optimal packetization, when I=10000, h=40 and S=1500 To see how the performance gain of optimal packetization depends on the size of the encoded video frame, the average performance gain at different loss rates is calculated over a range of video frame sizes. In Figure 7 the average performance gain is plotted for video frame sizes ranging from 2000 bytes to 30000 bytes. 0.12 average performance gain 0.10 0.08 0.06 0.04 0.02 0.00 0 10000 20000 30000 video frame size Figure 7 Average performance gain for different video frame sizes From looking at Figure 7 it is evident that the average performance gain decreases with the video frame size. With a video frame size of only 2 kbytes, the average performance gain is as high as 12%, while a 30 kbyte frame size gives an average performance gain of about 1%. 4. Discussion The adaptive FEC framework proposed in this paper is primarily targeted at real-time video applications, like videoconferencing. For applications with more relaxed real-time requirements, like streaming of stored video material, substantially larger sender-side buffers can be used. This means that the FEC coding can be applied over a range of video frames buffered before transmission. When the FEC coding is applied to larger data units, the performance gain of the optimal packetization scheme decreases (cf. Figure 7). In other words, when the data unit over which the FEC coding is applied becomes large enough, the optimal packet size will be equal to the MTU-size. Consequently, it is for delay-sensitive video applications that the optimal packetization scheme will be most useful. Rhee and Joshi propose an FEC-based loss recovery scheme for interactive video applications wherein the interval over which the FEC packets are computed is longer than a single frame interval [10]. This can be done without introducing delay by not requiring the receiver to be able to recover a lost packet in time for the playout deadline of the frame the packet belongs to. Thus, the FEC-coding will not prevent loss of quality in the reconstructed video, but will limit error propagation due to inter-frame dependencies. This scheme could be combined with the optimal packetization strategy presented in this paper, to shorten the interval over which to compute the FEC codes, improving overall performance. When it comes to error protection of other real-time media than video, audio is the first that leaps to mind. Many different approaches for robust audio transport have been suggested [1][11][12]. The packetization scheme presented here for video could certainly be applied to audio too. Since the target bandwidth of a typical audio codec is much lower than that of video codecs the packet size imposed by the packetization interval is usually smaller than the optimal packet size as calculated by the algorithm presented here. However, audio applications with less stringent delay-bounds, e.g. audio streaming applications, might benefit from choosing an audio packet payload size based on the algorithm in question.
5. Conclusions In this paper an adaptive error control scheme for packet video based on Reed-Solomon erasure coding and receiver reports has been proposed. An optimal packetization scheme was introduced, that improves the utilization of the network by adapting the video packet size depending on the loss rate as measured by the receiver. The FEC scheme improves the robustness of the video communication without introducing any additional sender-side buffering. This is important for real-time applications, like videoconferencing, that can tolerate only minimal buffering delays. An algorithm was presented that provides an efficient implementation of the optimal packetization scheme. The performance gain of the scheme was found to be biggest when the encoded video frame size is reasonably small, with an average utilization improvement of as much as 12% for a 2 kbyte frame size. For larger frame sizes, the improvement in utilization is still substantial (e.g. about 3% for a 10 kbyte frame size). The importance of keeping a unified strategy for error correction and congestion control must be emphasized. Since one of the main reasons for packet loss in the Internet is congestion, simply increasing the amount of redundancy in the FEC-coding will in effect exacerbate the congestion. Instead the total bandwidth allocated by an application for both video (or other data) and error correction redundancy should be under the control of the congestion control algorithm. 6. References [1] J.-C. Bolot and A. Vega-Garcia, "The case for FEC-based error control for packet audio in the Internet," ACM Multimedia Systems, 1997. [2] U. Horn, K. Stuhlmüller, M. Link, and B. Girod, "Robust Internet video transmission based on scalable coding and unequal error protection," Image Com., vol. 15, no. 1-2, pp. 77-94, September 1999. [3] Y. Wang and Q. F. Zhu, "Error control and concealment for video communication: A review," Proceedings of the IEEE, vol. 86, no. 5, pp. 974-997, May 1998. [4] J-C. Bolot and T. Turletti, "Adaptive error control for packet video in the internet," Proceedings of International Conference on Internet Protocols, Lausanne, September 1996. [5] G. Carle and E. Biersack, "Survey of error recovery techniques for IP-based audio-visual multicast applications," IEEE Network, December 1997. [6] V. Jacobson, "Congestion Avoidance and Control," ACM Computer Communications Review, vol. 18, no. 4, pp. 316-329, 1988. [7] R. E. Blahut, "Theory and practice of error-control codes," Addison-Wesley, Reading, MA, 1983. [8] S. Floyd, M. Handley, J. Padhye, and J. Widmer, "Equation-based congestion control for unicast applications," ACM SIGCOMM 2000, Stockholm, August 2000. [9] V. Jacobson S. McCanne and M. Vetterli, "Receiver-driven layered multicast", Proc. of ACM SIGCOMM'96, pp. 117-130, Stanford, CA, August 1996. [10] I. Rhee and S. R. Joshi, "FEC-based loss recovery for interactive video transmission experimental study," ICMCS, vol. 1, pp. 250-256, 1999. [11] K. Synnes, P. Parnes, D. Schefström, "Robust audio transport using maudio", Research Report 1999:04, Luleå University of Technology, April 1999. [12] C. Perkins, O. Hodson, and V. Hardman, "A survey of packet-loss recovery techniques for streaming audio," IEEE Network Magazine, September 1998. [13] L. Rizzo, "Effective Erasure Codes for Reliable Computer Communication Protocols," Computer Communication Review, 27(2):24--36, April 1997. [14] I. Rhee, "Error control techniques for interactive low-bit rate video transmission over the Internet," Proceedings of the ACM SIGCOMM '98 conference on Applications, technologies, architectures, and protocols for computer communication, Vancouver, Canada, 1998. [15] G. Ramamurthy and D. Raychaudhuri, "Performance of packet video with combined error recovery and concealment," Proceedings of IEEE INFOCOM, pp. 753 761, April 1995.