Robust Video Transmission over Lossy Network by Exploiting H.264/AVC Data Partitioning

Transcription

1 Robust Video Transmission over Lossy Network by Exploiting H.4/AVC Data Partitioning Xing-Jun Zhang, Xiao-Hong Peng, Richard Haywood and Tim Porter Electronic Engineering, School of Engineering and Applied Science, Aston University, UK x.zhang, Department of Computer Science, Xi an Jiaotong University, China Abstract In this paper, a robust H.4/AVC video transmission system over lossy packet network is investigated and implemented. The H.4/AVC standard has defined a new data partition scheme which can be used to perform unequal loss protection () in video transmission systems. However, few actual implementation and experiments which integrate data partition and loss protection algorithms have been reported. Based on the H.4/AVC data partitioning mechanism, we implement three loss protection schemes using Reed-Solomon and XOR parity codes, namely equal loss protection (), and partition A protection only ().The performance of each scheme is evaluated under different network environments. Experimental results and performance analysis show that has higher Peak Signal-to-Noise Ratio (PSNR) values than and, but requires the least redundancy and offers the best tradeoffs between PSNR values and redundancy among all the three schemes. I. INTRODUCTION Multimedia traffic is increasing rapidly in nearly every type of network. The majority of networks such as IP (Internet protocol) networks are in a lossy environment, and packet losses or erasures can significantly affect the performance of realtime applications. In video transmission, for example, a single loss could corrupt both current and subsequent frames, due to error propagation and the motion prediction and compensation techniques used in the video compression coding. Error control techniques, which consist of forward error correction (FEC), retransmission, error resilience, and error concealment, can be used to enhance the video performance in error prone environments. Conventional retransmission based schemes such as automatic repeat request (ARQ) are not viable options for conversational and streaming services with constraints on real-time delay and jitter [1]. Error resilience and error concealment is used from the video compression coding perspective. Error resilient schemes limit the damage scope caused by transmission error by encoding each frame based on considering the semantic of the compression layer elaborately. Error concealment is a post-processing technique used by the decoder. The latest standard H.4/AVC [2] is likely to become the most promising video compression standard for a large range of video applications. The standard supports valuable error-resilient tools to cope with erased packets, such as flexible macroblock ordering, data partitioning, etc., while it outperforms previous coding standards (H.3, MPEG-4). Unfortunately, these tools increase the computational complexity, which is undesirable for real-time video applications, and have a negative impact on compression efficiency. Therefore, schemes combining FEC algorithms with appropriate selection of error-resilient tools are often shown to be advantageous for transmission of H.4/AVC-coded streams, while maintaining the computational cost at reasonable level [3, 4, 5]. A dropped packet can be regarded as an erasure, so we call the FEC code used in the video transmission system as an erasure code and FEC coding as erasure coding correspondingly. In recent research [5], the Flexible Macroblock Ordering (FMO), one of the error-resilient features of H.4/AVC codec, was used to do. Some related works were presented in [6] [7], which also used the FMO mechanism. The data partitioning of H.4/AVC and high-memory rate compatible punctured convolutional codes (RCPC) were proposed for video transmission over wireless channels in [8]. RCPC codes were applied to the network adaptation layer (NAL). Data partitions were unequally protected according to their significance. The most related work is based on simulation. In this paper we exploit the H.4/AVC data partition resilience tool and implement three different protection schemes using Reed-Solomon and XOR (exclusive-or) parity codes for video transmission. The performance in terms of PSNR (peak signal-to-noise ratio) for the three schemes: (equal loss protection), (unequal loss protection) and (partition A protection only) are analysed under different network environments. The paper is structured as follows. Section II gives a brief introduction to the H.4 data partition resilience techniques and the erasure coding background. Section III describes the whole system implemented. In Section IV, we provide the experimental results and performance analysis. Finally, a conclusion is given in Section V. II. DATA PARTITIONING AND ERASURE CODING A. H.4/AVC Data Partitioning H.4/AVC makes a distinction between a Video Coding Layer (VCL) and a Network Abstraction Layer (NAL). The output of the encoding process is VCL data which is a sequence of bits representing the coded video data. The VCL data is then mapped to NAL units (NALU) prior to transmission or storage. The purpose of separately specifying

2 the VCL and NAL is to distinguish between coding-specific features (at the VCL) and transport-specific features (at the NAL) [4] [9]. Normally, each coded slice is encapsulated into one NALU. In the case of data partitioning, each coded slice is split into three partitions, which are each encapsulated in a NALU of their own. The H.4/AVC specification defines the three data partitions. Partition A contains the slice header, macroblock types, quantization parameters, prediction modes, and motion vectors. Partition B contains residual information of intra-coded macroblocks and partition C contains residual information of inter-coded macroblocks. The purpose of data partitioning is to divide the coded data into several partitions depending on the importance of the data. A network, which can give different priorities to different packets, can then protect the important data in a better way [1]. H.4/AVC supports the partitioning of I-slices into two partitions, P- slices into three partitions, and B-slices into two partitions. The decoder processes the received data partitions as follows. In case of the loss of partition A, regular error concealment is carried out, otherwise the header and motion information is decoded and used in the motion compensation process. If partition C is available the displaced frame difference is reconstructed and added to the motion compensated frame. If partition B is also available the intra macroblocks are added to the reconstructed frame. However, if partition B is lost, temporal error concealment is applied. A more advanced error concealment strategy for partition B losses is currently under investigation [8]. B. Erasure Coding on Video Data Erasure codes are a form of FEC used for communication through a lossy medium. When decoding the encoded data using erasure codes, the receiver is assumed to know the exact location of the lost packets. This information is not needed in a general FEC technique. Erasure codes are typically used for sending packets through the Internet since the receiver can detect the location of the lost packets by noting the missed packet sequence number. In a typical erasure code, the sender encodes redundant packets before sending both the original and redundant packets to the receiver. The receiver can reconstruct the original packets upon receiving a fraction of the total packets. Standard erasure codes such as the RS (N, K) erasure codes; take K original packets and produces (N K) redundant packets, resulting in a total of N packets. If K or more packets are received, then all the original packets can be completely reconstructed. Hence, a larger N/K ratio leads to a higher level of protection for data [11]. In this paper, we use RS codes as a typical example. RS codes as maximum distance separable (MDS) codes have been suggested to be applied to packet loss protection in many papers, such as [12]- [15]. As shown in Figure 1, a video frame is supposed as being transmitted in K packets where K varies with frame type, encoding method, and media content. RS code adds (N K) redundant packets to the K original packets and sends the N packets over the network. Although some packets K K K K Video Coding Layer (Encoder) Data Partition Enabled H.4 Encoder H.4 Decoder Video Coding Layer (Decoder) Decoder SPS PPS parameters & its RTP encapsulation PA NALU & its RTP encapsulation Encoder K+1... N K+1... N Fig. 1. Reed-Solomon Codes RS Encoder XOR Encoder XOR Decoder RS Decoder Network Abstraction Layer (Encoder) Network Abstraction Layer (Decoder) PB NALU & its RTP encapsulation Original Video Frame Packets After Adding Parity Packets (systematic RS coding) After Transmission (some packets lost) After Reconstruction Security Channel Transport (RTP) Transport (RTP) Redundancy & its RTP encapsulation Fig. 2. System Architecture NTuner Security Channel PC NALU & its RTP encapsulation may be lost, e.g. packet 3 in Figure 1, the frame still can be completely reconstructed if any K or more packets are successfully received. A. System Structure III. SYSTEM OVERVIEW The system is presented in detail in Figure 2. The whole system is mainly composed of three parts: the server, NTuner and the client. The server includes the RS encoder and XOR parity encoder embedded H.4/AVC encoder and the RTP server implementation. The packet loss controlled transmission platform, named as NTuner (Network traffic Tuner), which can be used to configure parameters for network bandwidth, delay, jitter, packet loss rate (PLR), and packet loss pattern. The receiver client includes the RS decoder and XOR parity decoder embedded H.4/AVC decoder and the RTP receiver implementation. The raw footage is encoded by exploiting data partition function in the server side. There are twelve types of partition in total. Except PA, PB and PC, we call all the others key partitions which are mainly about SPS (Sequence Parameter Set), PPS (Picture Parameter Set) and setup parameters. The system assumes these key partitions are transmitted using a security channel. The system can be configured to do, and loss protection. The partition A are protected using RS codec. The partition B and C are protected by XOR parity. The NALU and redundancy packets are packed into RTP format then transmitted over the NTuner platform, which

3 Packets In Input De-Multiplexing TCP, UDP FORWARDING Fig. 3. Linux Traffic Control TRAFFIC CONTROL OUTPUT QUEUE Packets out we created by encapsulating the Linux advanced networking traffic controller into the Web-based application. NTuner can alter the network traffic according to the configuration. The receiver recovers the lost partitions by utilizing the proper loss protection codec according to the NALU header information. Then, H.4/AVC coding layer takes over to do the following decoding. B. Traffic and Packet Loss Control We build up the network emulator upon the traffic control component of Fedora 4 that supports a number of advanced networking features [16]. Figure 3 shows how the Linux kernel processes incoming packets, and how it generates packets to be sent to the network. The input de-multiplexer examines the incoming packets and determines if the packets are destined for the local node. If so, they are sent to the higher layer for further processing, otherwise to the forwarding block. The forwarding block, which may have also received locally generated packets from the higher layer, looks up the routing table and determines the next hop for the packets. After this, it queues the packets to be transmitted on the output interface. It is at this point that Linux traffic control comes into play, which can be used to build a complex combination of queuing disciplines, classes and filters, which control the packets that are sent onto the output interface. We encapsulate traffic control into the Web-based application NTuner. C. Erasure Coding Implementation As a typical example, we implement a systematic RS code in our system. RS codes are described in numerous coding theory books and papers [17][18]. Given a data polynomial a(x) of degree k < n, n = 2 m (k is the number of information symbols and n is the code length), in Galois field GF(2 m ) and a code generating polynomial g(x) of degree p, where p n - k and p 1 ( g(x) = x + α i ) k, α(x) = α i x i (1) i= i= with α i successive unity roots in GF(2 m ) and α i elements of the same field, the systematic encoding of α(x) is given by C(x) = α(x)x n k R(x) (2) where R(x) is the remainder of the division α(x)x n k by g(x). Galois field arithmetic is fundamental to RS encoding and decoding. There are several approaches to performing Galois field multiplication in software [18]. In our implementation, we represent the Galois field elements either in the index or polynomial format. In the index format, the number is the power of the Galois field primitive element. This format is convenient for multiplication, which just needs to add the powers modulo 2 m 1. In the polynomial format, the bits represent the coefficients of the polynomial representation of the number. This is the most convenient format for addition. The two formats are swapped via lookup tables. Encoding implementation is based on the use of a Linear Feedback Shift Register (LFSR) that provides a convenient way to perform polynomial division [19]. Decoding utilizes the Berlekamp- Massey algorithm [2]. The RS code offers optimal efficiency such that any available parity element can be substituted for any erased data element in the block. Parity-check information is generated through operations on a Galois field [21]. The computational cost of this process is related to the size of the field, where typical RS code implementations operate in a field of size 2 8, or one with 255 elements. The implementation in our testbed is for a generic RS code; user can configure the parameters n, k and t (t is the erasure correction capability of the code) according to different requirements for video source codes and bit rates. In the following experiments, the Galois field size is 2 3, so n=7, k=5 and t=2. D. Loss Protection Schemes The system supports three types of loss protection schemes:, and. The objective video quality assessment metric PSNR is used to evaluate the video quality. Figure 4 shows the error correction encoding blocks in different schemes. In the scheme, RS codec is used to protect all PA, PB and PC NALU packets equally. This strong protection means the same scale of redundancy being used for protecting each partition. More redundancy results in higher bit rates, and consequentially leads to higher packet loss rates. Because PA is more important than PB and PC, PA packets are protected with a more powerful erasure code than those for PB and PC packets in the scheme. In this way, the overall redundancy is reduced as a result. In the scheme, only the most important partitions (i.e., PA) are protected and the redundancy is reduced dramatically. The system can also work without involving any of the protection schemes when the loss protection encoders are bypassed on the server side. The protected NALU packets are encapsulated into RTP packets according to RFC 3984 RTP Payload Format for H.4 Video []. IV. EXPERIMENTAL RESULTS A. Experimental Environment The system is built up based on a modified version of the test model coder JM12.4 which is provided by the Joint Video Team (JVT). JM12.4 is chosen because it supports data partitioning for the purposes of error resilience. As an example, the first 1 frames of a standard video sequence, in the 4:2: YUV CIF format has been encoded at frames per seconds (fps) applying an IPBPBPBstructure, which means the error caused by packet loss in any frame will be propagated throughout the whole video sequence. The encoder generated 419 partitions in total, which include 13 key parameter

4 Block Block Block TABLE I REDUNDANCY IN SYSTEM Parity Redundancy (Bytes) Alignment Redundancy (Bytes) Redundancy Percentage (Bytes) % 59.21% Partition A Partiton B Partition C RS Parity Redundancy XOR Parity Redundancy Alignment Redundancy % Fig. 4. Loss Protection Schemes partitions (such as SPS, PPS), 2178 PA and 18 PB/PC. A RS (7, 5) code, capable of recovering 2 lost packets per 7 transmitted packets, is implemented and applied in the, and schemes. Since each frame has been divided into eleven slices and each slice is divided into three partitions according to the H.4/AVC encoding mode, using RS (7, 5) code, the system can control the buffering delay within one slice of processing time. The system needs only to buffer five packets to employ the RS encoding and decoding at the server and client side respectively. In the scheme, every five PB/PC packets are used to produce one XOR parity packet which provides transmission protection. On the NTuner platform, the random packet loss pattern is adopted and the packet loss rate is varied between 1% to 2% B. Experimental Results In Figure 5 the PSNR value with respect to frame number, using a selection of packet loss rates, are shown. We found: (1) the PSNR is improved for all protection schemes. (2) Figure 5a and 5b show that when the packet loss rate is low, the and can recover the majority of all lost packets. (3) Normally, the yields a higher PSNR than the other schemes, especially in initial frames, which shows that the error aggregation caused by packet loss is not too high. Figure 5d, 5e, 5f, 5g and 5i shows that the frame PSNR values of is dramatically higher than other protection methods when the frame number less than 4. (4) The PSNR curves of the and are close in most cases. This effect can be seen in Figures 5d to 5j. (5) With the packet loss rate increasing and the error aggregating, the PSNR values of the, and begin to converge, but the PSNR is still marginally higher than the others schemes. This phenomenon can be seen in Figures 5h, 5i and 5j. In a few cases, such as some parts in the Figure 5f and 5h, the curves are prone to jitter. This phenomenon is because: (1) Despite each protection scheme using the same packet loss rate, the location of the lost packets may differ between schemes, resulting in the aggregated error being very different (2) The frame structure chosen in the H.4 encoding is quite sensitive to the packet loss and therefore any displayed error will be propagated throughout the entire video sequence. These two reasons may reduce the accuracy of the PSNR comparison. The user perceived quality was investigated using a range of packet loss rates. It was found that the frame quality significantly improved when using the scheme. This can be seen in Figures 6a and 6b, which show 1 frames with and without protection. We also found and can improve the video quality to the same level as if the packet loss rates were 5% and 1%. Figure 6c, 6d and 6e show the effects of 1 frames with 1% packet loss. Similar results can be found when packet loss rate is 5%. If the error caused by packet loss is not extensively aggregated (i.e. the amount of error early in the sequence is limited), the and schemes can exert their loss recovery capability with excellent results. Figures 6f, 6g and 6h shows this phenomenon in the th frame with 15% packet loss rate. Normally, gets better playing effects than others. The perceived results when using, and are not tolerable if the packet loss rate is greater than 2%. The decoding complexity and added redundancy are the main cost in the system. Rizzo had showed the feasibility of FEC coding in software at high speeds [23]. With the development of computer hardware, decoding complexity is rapidly becoming less of an issue in many circumstances due to the availability of inexpensive high speed CPUs. In this paper we focus mainly on the trade-off between the amount of added redundancy and the video quality improvement. There are two kinds of redundancy in the system. The first one is the parity packet redundancy. Two parity packets are added for every five payload packets when using a (7,5) RS code and one parity packet is added for every five payload packets when using XOR code. H.4/AVC employs variable motion block sizes, therefore the length of the video packets are very different which lead to much more FEC alignment redundancy than conventional video standards. The system uses no complex interleaving or FEC alignment algorithms, the redundancy is added sequentially. Table 1 shows the redundancy and its percentage proportion in the whole sequence. For strong erasure codes, such as RS (7, 5) codes, we found the level of redundancy excessive for real world applications - its theoretical redundancy overheads occupy.57% of the transmitted data. We calculated the PSNR differences between each protection scheme and the transmission without any protection under different packet loss rate. We found the PSNR values can be improved by 5.99, 7.63 and 1.52 (db) respectively at most in

5 , and respectively. If we average the increased values from packet loss rates 1% to 2% for each scheme, the values are 4.13, 5.5 and 6.47 (db) for, and respectively. If bandwidth limitations are not an issue, and the application is real-time related, and can be used with excellent results. If the reliability, real-time feature and bandwidth factors are all to be considered, then the scheme is a practical approach. V. CONCLUSIONS In this paper, we built up a robust streams transmission system by exploiting H.4/AVC data partition and error resilience mechanisms. Erasure coding and H.4/AVC coding with data partitioning are jointly investigated, resulting in three different protection schemes (, and ) for video transmission. The PSNR and user perceived video quality are used to assess the performance of the system. We have compared the performance between the different protection schemes and found that has better PSNR than and but requires much higher redundancy than the others. This could hamper the realisation of the scheme in a practical system, especially a real-time system. Considering both performance in terms of PSNR and redundancy requirement, is shown to be the best choice, compared to and, for this application. This work can also be extended to testing other FEC schemes in combination with H.4/AVC partition and error resilience methods for video transmission. ACKNOWLEDGMENTS This work was supported by the grants from the Engineering and Physical Sciences Research Council of the United Kingdom and Xyratex Ltd. The authors wish to thank Saty Mukherjee and Gu-Chun Zhang for useful discussions. [9] ThomasWiegand, Gary J. Sullivan, Gisle Bjontegaard, and Ajay Luthra, Overview of the H.4/AVC Video Coding Standard, IEEE Transactions on Circuits and System for Video Technology, VOL. 13, NO. 7, JULY 23. [1] Mys, S., Dhondt, Y., Van de Walle, D., De Schrijver, D., and Van de Walle, R, A performance evaluation of the data partitioning tool in H.4/AVC, in Proceedings of the SPIE/Optics East conference, Boston (). [11] T. Nguyen and A. Zakhor, Distributed Video Streaming with Forward Error Correction, Packet Video Workshop, April. [12] T. Zhang and Y. Xu, Unequal packet loss protection for layered video transmission, IEEE Trans. Broadcast., June [13] XU Y, ZHANG T. Variable shortened-and-punctured Reed-Solomon codes for packet loss protection, IEEE Transactions on Broadcasting,, 48(3): [14] Y. Xu and T. Zhang, An adaptive redundancy technique for wireless indoor multicasting, in Proc. Fifth IEEE Symp. Computers and Commun., Antibes-Juan les Pins, France, July 3-6, 2. [15] Huahui Wu, Mark Claypool, Robert E. Kinicki, Adjusting forward error correction with quality scaling for streaming MPEG, NOSSDAV 25: [16] Saravanan Radhakrishnan, Linux - Advanced Networking Overview Version 1, August, [17] S. Lin and D. J. Costello, Error Control Coding: Fundamentals and Applications, Englewood Cliffs, New. Jersey: Prentice Hall, [18] S. Mamidi, M. Schulte, D. Iancu, A. Iancu, and J. Glossner, Instruction Set Extensions for Reed-Solomon Encoding and Decoding, IEEE 16th International Conference on Application-specific Systems, Architectures and Processors, pp. 4-9, July 25. [19] Y. Katayama and S. Morioka, One-Shot Reed-Solomon Decoder, in Annual Conference on Information Science and Systems, vol. 33, pp. 7-75, [2] Richard E. Blahut, Theory and Practice of Error Control Codes, Addison-Wesley Publishing Company, [21] J. A. Cooley, J. L. Mineweaser, L. D. Servi, and E. T. Tsung, Softwarebased Erasure Codes for Scalable Distributed Storage, in 2th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, San Diego, CA, 23. [] S. Wenger, M.M. Hannuksela, T. Stockhammar, M. Westerlund, and D. Singer, RTP payload format for H.4 video, IETF, Request for Comments: 3984, 25. [23] L. Rizzo. On the Feasibility of Software FEC, Technical Report DEIT Technical Report LR-9713, University of Pisa, Available at luigi/softfec.ps. REFERENCES [1] Yanling Xu and Yuanhua Zhou, H.4 Video Communication Based Refined Error Concealment Schemes, IEEE Trans. on Consumer Electronics, vol.5, no.4, Nov., pp [2] Wiegand Thomas, Draft ITU-T Recommendation and Final Draft International Standard of Joint Video Specification (ITU-T Rec. H.4; ISO/IEC; AVC), Document: JVT-G5; 7th Meeting, Pattaya, Thailand, Mar. 7-14, 23. [3] S.Wenger, H.4/AVC over IP., IEEE Transaction on Circuits and Systems for Video Technology, vol. 13, no. 7, pp , 23. [4] Iain E G Richardson, H.4 and MPEG-4 Video Compression, John Wiley & Sons, Sept. 23. [5] N. Thomos, S. Argyropoulos, N. V. Boulgouris, and M. G. Strintzis, Robust transmission of H.4 streams using adaptive group slicing and unequal error protection, EURASIP Journal on Applied Signal Processing,, 1 (Jan. ), [6] H. Kodikara Arachchi, W.A.C. Fernando, and S. Panchadcharam, Unequal Error Protection Technique for ROI Based H.4 Video Coding, in Proceedings of IEEE Canadian Conference on Electrical and Computer Engineering, Ottawa, Canada, May. [7] Dhondt, Y., Lambert, P., and Van de Walle, R., A Flexible Macroblock Scheme for Unequal Error Protection, in Proceedings of the IEEE International Conference on Image Processing,. pp [8] T. Stockhammer andm. Bystrom, H.4/AVC data partitioning for mobile video communication, in Proceedings of the International Conference on Image Processing (ICIP 4), pp , Singapore, October.

6 Fig. 5a. PSNR with 1% PLR Fig. 5e. PSNR with 9% PLR Fig. 5b. PSNR with 3% PLR Fig. 5f. PSNR with 11% PLR Fig. 5c. PSNR with 5% PLR Fig. 5g. PSNR with 13% PLR Fig. 5d. PSNR with 7% PLR Fig. 5h. PSNR with 15% PLR 9 1

7 Fig. 5i. PSNR with 17% PLR Fig. 6c. Frame 1, 1% PLR, Fig. 5j. PSNR with 19% PLR Fig. 6d. Frame 1, 1% PLR, Fig. 6a. Frame 1, 1% PLR, Fig. 6e. Frame 1, 1% PLR, Fig. 6b. Frame 1, 1% PLR, Fig. 6f. Frame, 15% PLR,

8 Fig. 6g. Frame, 15% PLR, Fig. 6h. Frame, 15% PLR,