An Unequal Packet Loss Protection Scheme for H.4/AVC Video Transmission Xingjun Zhang, Xiaohong Peng School of Engineering and Applied Science, Aston University Aston Triangle, Birmingham, B4 7ET, UK {x.zhang, x-h.peng}@aston.ac.uk Abstract In this work, we present an unequal packet loss protection scheme for robust H.4/AVC video transmission over lossy networks. This scheme combines erasure coding, H.4/AVC error resilience techniques and importance measures in video coding. The unequal importance of the video packets is identified in the group of pictures (GOP) and the H.4/AVC data partitioning levels. Using a fixed amount of redundancy, more important packets of a video stream are protected with a more powerful erasure code than the less important packets. The effectiveness of this approach is demonstrated by system implementation and performance evaluation. We show that the received video quality, as measured by PSNR, is significantly improved with the packet loss rate increasing when the unequal loss protection () scheme is used. More importantly, we have observed that can achieve higher PSNR values and better user perceived quality, even though using less redundancy than the equal loss protection () scheme. I. INTRODUCTION Multimedia applications become popular in most networks. The majority of networks such as IP (Internet protocol) networks operate in a lossy environment without qualityof-service (QoS) guarantees. Thus, how to ensure robust transmission of compressed video has been a big technical challenge. When data is transmitted over lossy networks, packets may be lost due to errors as well as congestion. This problem can become very serious and even catastrophic in the transmission of compressed bitstreams with strong spatio-temporal dependency, where errors will propagate and substantially deteriorate the received video quality [1]. Error control techniques, such as forward error correction (FEC), retransmission, error resilience, and error concealment, can be used to enhance video performances in error prone environments. Conventional retransmission based schemes such as automatic repeat request (ARQ) are not viable options for conversational and streaming services with constraints on realtime delay and jitter [2]. Error resilience and error concealment is used from the video compression coding perspective. Error resilient schemes limit the scope of damages caused by transmission error by encoding each frame based on the semantic of the compression layer elaborately. Error concealment is a post-processing technique used by the decoder. Combining FEC algorithms with appropriate error resilient tools are often shown to be advantageous for transmission of H.4/AVC coded streams, while maintaining the computational cost at reasonable levels [3], [4], [5]. Several protection schemes for H.4/AVC video transmission [5], [6], [7] were proposed based on Flexible Macroblock Ordering (FMO), one of the error-resilient features of the H.4/AVC codec. The data partitioning of H.4/AVC and rate compatible punctured convolutional codes (RCPC) were proposed for video transmission over wireless channels in [8]. The RCPC codes were applied at the network adaptation layer (NAL) and data partitions were unequally protected according to their importance. By analyzing the existing unequal importance settings in H.4/AVC coding schemes, and exploiting the H.4/AVC data partition resilience tool, we present an unequal loss protection () scheme that is implemented over an emulation testbed. The scheme can differentiate the importance of the stream packets according to the derivations of the packets. Firstly, the steam packets are classified by the GOP sequence number of the, which indicates the picture group the packets come from. The packets to be transmitted earlier in the sequence are more important. The packets are then classified again by the H.4/AVC data partition type. Reed-Solomon (RS) codes with different recoverability are used in our erasure coding approach. As a dropped packet can be regarded as an erasure, we call the FEC code or FEC coding used in the video transmission system the erasure code or erasure coding. The performance in terms of PSNR (peak signal-to-noise ratio) is analyzed in the process of system implementation. The paper is organized as follows. Section 2 identifies the unequal importance in the different levels of H.4/AVC video encoding schemes. Section 3 presents a novel scheme and system implementation. In Section 4, we provide experimental results and performance analysis. Section 5 concludes the paper. II. UNEQUAL IMPORTANCE IDENTIFICATION Firstly, different importance of video exists in the GOP level. In compressed video, different frame types (I, P and B) are assigned different levels of importance. The I- frame is most important, and the P-frame is more important than the B-frame. In addition, the in a GOP have a descending order of importance from the beginning to the ending. A GOP sequence starts with an I-frame, and all the other depend on it. The first P-frame is predicted using the I-frame. Subsequent P- use the previous P- frame as their reference until the next GOP starts. B-
are predicted from the preceding and following I-frame or P- frame. Due to this temporal dependency, the decoding of the current frame strongly depends on its preceding in a GOP. The earlier a frame is lost in a GOP, the more that will be corrupted afterwards [2]. Secondly, the unequal importance of video packets can be found by using H.4 data partitioning error resilience. Normally, each coded slice is encapsulated into one NALU (Network Abstraction Layer Unit). In the case of data partitioning, each coded slice is split into three partitions, which are each encapsulated in a NALU of their own. The H.4/AVC specification defines three data partitions (A, B & C): partition A (PA) contains the slice header, macroblock types, quantization parameters, prediction modes, and motion vectors; partition B (PB) contains residual information of intracoded macroblocks; and partition C (PC) contains residual information of inter-coded macroblocks. In decoding, PA is independent of PB and PC, but not the inverse. Data partitioning with constrained intra prediction option makes decoding PB independent of the corresponding PC. However, no option exists to make the decoding of PC also independent of PB [9]. The purpose of data partitioning is to divide the coded data into several data streams with different importance. A network that can provide different transmission or protection priorities to the packets with corresponding importance is able to protect the important data in a more efficient way. III. UNEQUAL LOSS PROTECTION SCHEME A. The Architecture Figure 1 shows the proposed transmission system framework. The whole system is composed of three parts: the server, the packet loss controller and the client. The server includes the NALU classification component, RS encoder, embedded H.4/AVC encoder and the RTP server. The raw footage is encoded based on the partitioned data. There are in total twelve different partitions. Except for PA, PB and PC, we call all others key partitions, including SPS (Sequence Parameter Set), PPS (Picture Parameter Set), IDR (Instantaneous Decoder Refresh) data slices and setup parameters. The system assumes that these key partitions are transmitted using a security channel. All PA, PB and PC partitions are classified according to the criteria analyzed in section 2. One of the motivations for this architecture is to keep the method consistent with the H.4/AVC design idea, which specifies the video coding layer and the network abstraction layer separately. The system intends to enable robust video transmission without the details of video source coding. According to the importance of the classification, the RS encoder generates parity packets by choosing suitable RS codec parameters. NALU and parity packets are packed into the RTP format and transmitted through the packet loss controller that is created by encapsulating the Linux networking traffic controller (TC) and used to alter network traffic in terms of bandwidth, delay, jitter, packet loss rates, and packet loss patterns. The client is equipped with the RS decoder, the embedded H.4/AVC decoder and the RTP client. The lost packets are recovered H.4/AVC Encoder Server H.4/AVC Decoder Client NALU Classification RS Decoder Fig. 1. RS Encoder Transport (RTP) System Architecture Transport (RTP) Packet Loss Controller by utilizing an appropriate RS code according to the NALU header information. Finally, H.4/AVC coding layer takes over to complete the remaining decoding tasks. Setting up proper classification ranks for the GOP level at the server is important for implementing the unequal protection scheme. In general, the more classified levels that are used, the more delicate unequal protection that will be achieved. The system assembles the classified packets into different blocks of packets (BOP) in order to implement the RS code across packets. More classification stages will result in smaller BOP sizes, which directly affect the capability of the RS code in coping with burst packet losses. For example, suppose that there are 15 packets belonging to one RS (15, 11) BOP transmitted over a lossy network, and this code can recover up to 4 burst packet losses. But if those 15 packets are used to form two RS (7, 5) BOPs (with one packet unused), they can only recover up to 2 burst packet losses per BOP. Thus, BOPs with large sizes are more robust than the small ones. However, a larger BOP would result in a longer delay. In practice, the total number of video packets in a BOP should be determined by considering the tradeoff between delay and robustness [11]. For Internet applications, the ideal number of video packets in one BOP can be determined according to the end-to-end system delay constraint [12]. I Frame Security Channel FFH FH FSH GOP P Frames SFH SH SSH RS(15,13) B Frames IDR Partition PA PB/PC PA PB/PC PA PB/PC PA PB/PC PA/PC Fig. 2. RS(15,9) B. System Implementation RS(15,11) classified partitions Vs. RS codes We implemented the system based on a modified version of the test model coder JM12.4 which is provided by the Joint Video Team (JVT). The system parameters are fixed for validating the scheme. For simplicity and clearness, we
will calculate the PSNR values only using the I-frame and P- of a GOP for the purpose of performance comparison. This is because the loss of packets in B- does not affect any other. We configure the system in four stages for GOP-level classifications. Firstly, in the GOP are divided into the First Half (FH) and the Second Half (SH) ; and then the FH are further divided into the First Half FH (FFH) and the Second Half FH (FSH) ; and the SH are further divided into the First Half SH (SFH) and the Second Half SH (SSH). Data partitioning is carried out according to the H.4/AVC standard. In the system implementation, the GOP-level classification is performed after data partitioning. The outputs of VCL are NALUs. Each NALU takes one data partition. The system is aware of the GOP position and the source from which a NALU is created. When decoding the received data using erasure codes, the receiver is assumed to know the exact location of the lost packets. This information is not required in a general FEC scheme. Erasure codes are typically used for sending packets through the Internet since the receiver can detect the location of the lost packets by notifying their sequence numbers. In a typical erasure encoder, parity packets are generated and then sent out with the original packets to the receiver. The receiver can reconstruct the original packets upon receiving a fraction of the total packets. For RS (N, K) erasure codes, the encoder produces (N K) parity packets using the K original packets, resulting in a total of N packets. If K or more packets are received, all the original packets can be completely reconstructed. Hence, a larger N/K ratio leads to a higher level of protection for data [10]. We implement the systematic RS erasure codes the mapping of the classified partitions onto the RS code chosen is shown in Figure 2. IV. EXPERIMENTAL RESULTS In the experiments, the first 100 of a standard video sequence in the 4:2:0 YUV CIF format has been encoded at per seconds (fps) in an IPBPBPB... structure. The encoder generated 4017 partitions in total, including 11 key parameter partitions (such as SPS, PPS), 2178 PA and 18 PB/PC. The packet loss controller generates random packet losses and the packet loss rate is tuned from 1% to %. For system validation, an equal loss protection () scheme is also implemented by fixing all the K values of the RS (N, K) codes used in the to 11 except for the one used for B- frame protection. Figure 3 shows the statistical features of packet losses and recovery in the experiments. For both and schemes, the bar denotes the number of packets lost; the dark portion of each bar denotes the number of packets unrecovered among those lost. The numbers of lost packets shown in the and experiments are not exactly the same for a particular PLR, that is because the total numbers of packets transmitted for the two schemes are not identical, but the same loss rate can still be obtained for both cases. Figure 3 also shows the percentage of unrecoverable rate. We can see that overall the Lost Packets vs. Unrecoverable Packets Unrecoverable Rate 800 700 600 500 400 0 0 100 01 0.8 0.6 0.4 0.2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Fig. 3. Packet Loss Rate (%) Unrecoverable Packets Statistics unrecoverable rate of is relatively lower than that of. This is because used slightly more redundancy than, i.e., with the redundancy rates.667% for and.631% for, respectively. Average PSNR (db) 1 5 Fig. 4. 10 Packet Loss Rate (%) 15 Average PSNR Values Figure 4 plots the measured average PSNR values, which clearly shows that the scheme has much better PSNR values as the packet loss rate increases. We can see that when the packet loss rate is low, with more than % redundancy, both and can recover almost all the losses; the frame PSNR values of are dramatically higher than the method when the packet loss rate is high; With the packet loss rate increasing and the number of errors aggregating, the PSNR values of and begin to converge, while s PSNR is still marginally higher than that of others schemes. As an example, Figures 5 and 6 show the PSNR values with respect to the frame sequence number for three different packet loss rates, respectively. The results are consistent with the average PSNR values shown in Figure 4. Some parts of, the curves in Figures 5 and 6 are prone to jitter. This is because although these protection schemes are compared against the same packet loss rate, the locations of the lost packets may differ between schemes, resulting in different numbers of aggregated errors. In addition, the frame structure (IPBPBPB... ) chosen in the H.4 encoding scheme is quite sensitive to the packet loss and therefore any displayed error
will be propagated throughout the entire video sequence. Thus, we used the motion vector copy concealment algorithm to deal with the losses that and fail to recover. This algorithm belongs to frame level concealment and is not an elaborate concealment algorithm. Therefore, the accuracy of the comparisons in PSNR may be affected slightly due to these three reasons. PSNR (db) PSNR (db) 38 Without Protection 18 0 10 40 50 60 70 80 90 100 Frame Number 38 Fig. 5. PSNR comparison at 11% PLR Without Protection 18 0 10 40 50 60 70 80 90 100 Frame Number Fig. 6. PSNR comparison at % PLR of the are all improved in both and against the scheme without any protection; and that the frame quality for is much better than that of, implying that user perceived quality is significantly improved when using the scheme. The decoding complexity and added redundancy are the main costs in the system. Rizzo had showed the feasibility of FEC coding in software at high speeds [13]. There are two kinds of redundancy in the system: parity packet redundancy and padding redundancy due to the difference in packet size of the video footage used. The padding redundancy does not need to be transmitted by using the virtual padding process [11]. The system uses no complex interleaving or FEC assignment algorithms, allowing for fast processing to meet the requirements on transmission throughput and latency. V. CONCLUSIONS An unequal packet loss protection scheme for robust H.4/AVC video transmission over lossy networks is presented. This novel approach efficiently utilizes erasure coding, H.4/AVC data partitioning and unequal importance identification techniques in video coding schemes. The objective and subjective video quality are assessed for the system implemented. The experimental results demonstrate that the received video quality, as measured by PSNR, can be significantly improved with the scheme, especially for the environment with high packet loss rates. In addition, we show that can even achieve higher PSNR values and better user perceived quality with less amount of redundancy than that required in the scheme. VI. ACKNOWLEDGMENTS This work was supported by the grants from the Engineering and Physical Sciences Research Council of the United Kingdom and Xyratex Ltd. The authors wish to thank Richard Haywood, Tim Porter, Dave Milward and Tim Courtney for useful discussions. We found that s PSNR is higher than that of when the packet loss rate is lower than 8.3% in Figure 4. This is because is better in coping with the burst losses in low PLR. In the experiments, there are 1193 out of 4006 packets being protected by the RS (15, 13) code in the scheme. In transmitting these packets, if there are more than two packets lost consecutively, those losses will not be recovered. uses the RS (15, 9) code, which can recover up to 6 consecutive losses and hence more powerful than the RS (15, 13) code used by, to protect the more important 539 out of 4006 packets. However, this advantage of the code cannot be exerted in the low packet loss rate region as it is very rare to have around consecutive 6 out 15 packets lost when the lost rate is low. We have also investigated the user perceived video quality. Figures 7 show the snapshots of the 100th for the proposed, and uncoded transmission schemes at the packet loss rate of %. We can see that the visual qualities REFERENCES [1] X. Yang, C. Zhu, Z. Li, X. Lin, and N. Ling, An unequal packet loss resilience scheme for video over the internet, IEEE Trans. Multimedia, vol. 7, no. 4, pp. 753 765, August 05. [2] Y. Xu and Y. Zhou, H.4 video communication based refined error concealment schemes, IEEE Trans. on Consumer Electronics, vol. 50, no. 4, pp. 11 1141, November 04. [3] I. E. G. Richardson, H.4 and MPEG-4 Video Compression. John Wiley & Sons Ltd, 03. [4] S.Wenger, H.4/avc over ip, IEEE Transaction on Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 645 656, 03. [5] N. Thomos, S. Argyropoulos, N. V. Boulgouris, and M. G. Strintzis, Robust transmission of h.4 streams using adaptive group slicing and unequal error protection, EURASIP Journal on Applied Signal Processing, vol. 06, no. 1, pp. 1 1, 06. [6] H. K. Arachchi, W. Fernando, S. Panchadcharam, and W. Weerakkody, Unequal error protection technique for roi based h.4 video coding, in Proceedings of IEEE Canadian Conference on Electrical and Computer Engineering, Ottawa, Canada., May 06, pp. 33. [7] Y. Dhondt, P. Lambert, and R. Van de Walle, A flexible macroblock scheme for unequal error protection, in Proceedings of the 06 IEEE International Conference on Image Processing, October 06, pp. 829 8.
(a) (b) (c) Fig. 7. Frame 100 at % PLR using: (a) without protection; (b) ; (c) [8] T. Stockhammer and M. Bystrom, H.4/avc data partitioning for mobile video communication, in Proceedings of the International Conference on Image Processing, Singapore, 04, pp. 545 548. [9] M. Stefaan, D. Yves, V. D. W. Dieter, D. S. Davy, and V. D. W. Rik, A performance evaluation of the data partitioning tool in h.4/avc, in Proceedings of the SPIE/Optics East conference, Boston, 06. [10] T. Nguyen and A. Zakhor, Distributed video streaming with forward error correction, in 12th International Packet Video Workshop, 02. [11] X. Yang, C. Zhu, Z. Li, X. Lin, G. Feng, S. Wu, and N. Ling, Unequal loss protection for robust transmission of motion compensated video over the internet, Signal Processing: Image Communication, vol. 18, no. 3, pp. 157 167, March 03. [12] S.-W. Yuk, M.-G. Kang, B.-C. Shin, and D.-H. Cho, An adaptive redundancy control method for erasure-code-based real-time data transmission over the internet, IEEE Transactions on Multimedia, vol. 3, no. 3, pp. 6 374, September 01. [13] L. Rizzo, On the feasibility of software fec, University of Pisa, Tech. Rep., 1997.