Performance comparison of multiplexing techniques for MPEG-4 objectbased

Performance comparison of multiplexing techniques for MPEG-4 objectbased content Seán Murphy, Stefan Goor, Liam Murphy Department of Computer Science, University College Dublin, Dublin 4, Ireland. Email: sean.murphy@iname.com; stefangoor@gmail.com; liam.murphy@ucd.ie Abstract A study of the performance of a number of different multiplexing schemes was conducted in the context of streaming of MPEG-4 object-based content, with particular emphasis on streaming content to mobile devices. The comparison involved six different schemes. The schemes differed in terms of how they packed the data for each arbitrarily shaped video object into packets. An experimental testbed was constructed to compare the performance of the different schemes. The experiments showed that the scheme in which no effort is made to maximise the amount of data in the packet performs worse than the others in terms of the amount of overhead it generates and the video quality obtained when it is used to stream content. The other schemes did exhibit small differences in terms of the amount of overhead generated and video quality obtained, but the differences were not sufficiently large to be able to identify any as being clearly better than the others. 1 Introduction There has been much interest in streaming of multimedia content to mobile devices quite recently. However, not all issues relating to streaming such content have been solved. One important issue with streaming of multimedia content to mobile devices is determining how to multiplex the content appropriately. MPEG-4 has been selected by the 3GPP standardisation group as the format of choice for distributing content to mobile devices. While MPEG-4 has the capability to work with a number of multiplexing schemes, there is no recommended approach to multiplexing MPEG-4 content to mobile devices. In this paper, a number of different approaches to multiplexing MPEG-4 arbitrarily shaped object based content for streamed transmission are compared. As such object support is one of the major new functionalities provided by MPEG-4, it is interesting to determine how it interacts with multiplexing and transmission schemes. In particular, it is interesting to determine how a number of different MPEG-4 multiplexing schemes operate in the presence of object-based content. That is the focus of this work. The paper is laid out as follows. Related work is discussed in the next section. This is followed by a discussion of approaches that have been proposed for multiplexing of object-based content in section 3. In section 4, the experimental testbed that was used to compare the different schemes is presented. Section 5 describes some aspects of the experimental approach including the video content that was used. The results that were obtained from the experiments are then discussed in section 6. Finally, the document is concluded in section 7. 2 Related work A number of contributions have been made to different fora in which approaches to packetisation of MPEG-4 content for transmission on IP networks are proposed [1][2][3][4][5][6]. Most of these contributions simply describe approaches to multiplexing the content; no analysis of the performance of the approach is given. In [6] and [3], however, there is a more detailed consideration of the performance of the multiplexing scheme. In [6], the emphasis is on the use of diffserv for transporting MPEG-4 content which has differing requirements for different streams. As QoS support is not assumed in this work, some aspects of the multiplexing scheme chosen therein are not relevant here. In [3], Nafaa et al assess the performance of their RTP4Mux scheme. Their work differs from that described here in that they put more emphasis on the audio content and they also do not consider the resulting quality of the received video content. Finally, we have presented some earlier work in [10] which studies the impact of these different multiplexing schemes in a non-arbitrarily shaped-video context. 3 Approaches to multiplexing MPEG-4 content Four approaches to multiplexing MPEG-4 content were compared in this work: an RFC 3016 compliant scheme, an approach based on the Reduced SL Header of RFC 3640 [5], an approach based on the FlexMux proposal in [1] and an approach based on the so-called RTP4Mux proposal in [6]. Interleaved variants of the latter two schemes were also compared. Each of these approaches is described here, with an emphasis on the difference between them and some comments on the difference between these approaches and that proposed in [2]. The RFC 3016 [4] scheme can then be described as a relatively simple scheme in which a single MPEG-4

Access Unit (AU) is inserted into an individual RTP packet in most cases. An important deficiency of RFC 3016 is that it does not provide support for multiplexing data from different Elementary Streams (ESs) in a single RTP stream; the RFC 3016 encoding does not include enough information to enable a receiver to determine which ES a particular packet is associated with. Typically, then, each ES uses a separate RTP stream when RFC 3016 is employed. A more sophisticated approach is defined in RFC 3640 [5]. It provides better support for multiple AUs to be carried in a single packet. Extra header information for one or more AUs is included at the start of each RTP packet. This is then followed by the AU data for the AUs that are present in the packet. The RFC 3640 approach does not provide support for multiple ESs to be carried in a single RTP stream, although it does provide for AUs with different timestamps to be carried in a single packet. The approach proposed in [1] provides for more information in the RTP packet such that the receiver is able to differentiate between packets from different ESs which are transmitted in the same RTP packet. MPEG-4 FlexMux PDUs are generated which include a field which can be used as an ES identifier. Thus, it is possible to determine which ES a FlexMux PDU is associated with. In [1], the authors advocate an approach which differs slightly from that of RFC 3640 in that the header information for each RTP packet is inserted just before each FlexMux PDU, rather than aggregating the header information for all AUs at the start of the RTP packet. The RTP4Mux approach differs a little from the FlexMux approach described above in that, in this case, all AUs associated with a particular ES are placed one after the other in each RTP packet. This approach results in a slight decrease in the amount of header information required, since less information is required to identify the ES associated with the packet. Finally, a variant of both the FlexMux and RTP4Mux approaches was used in which the data was inserted into the RTP packets using interleaving. An interleaving stride of 2 was used in these experiments. In the following, these different approaches are termed as follows: Simple refers to the RFC3016 approach, Multi- SL refers to the approach based on RFC 3640, FlexMux refers to the approach described in [1] and RTP4Mux refers to the approach described in [6]. The interleaved variants are referred to as FlexMux-I and RTP4Mux-I. 4 Experimental testbed In order to compare the different approaches to multiplexing MPEG-4 content, an experimental testbed was constructed. This testbed comprised of a client, a video server and a network emulator as shown in Figure 1. Data was streamed from server to client over RTP. Figure 1: The experimental testbed used in this work A Java Media Framework (JMF) based client was developed that could request content from the server and decode the received content. The client was based on Microsoft s MPEG-4 reference codec. The client contained functionality to process the received RTP packets and demultiplex them according the multiplexing scheme that was being used. The demultiplexed data was then passed to written to a disk for off-line decoding, as the available decoders which had arbitrary shaped object support were neither robust nor fast enough for real-time decoding. The Microsoft reference codec was used to decode the data into YUV format. A modified version of the Apple Darwin Streaming Server (DSS) was used in these experiments. The standard distribution of the server is RFC 3016 compatible; the server that was used for these experiments was modified to support the multiplexing schemes described in section 3 above. The more sophisticated multiplexing schemes mentioned above were configured such that an RTP packet was sent when the addition of one more AU resulted in the packet size exceeding the permitted MTU of 1500 bytes. The NistNet network emulator [8] was used in the testbed to mimic the effects of a network between the client and server. 5 Experimental approach Two particular aspects of the experimental approach warrant some discussion: the selection of the test video sequences and the approach used to determine video quality. These are discussed here. Four video sequences were selected that exhibited contrasting spatial and temporal complexities. The content was also selected to be representative of the content which might be streamed in a mobile environment. These were: a movie trailer, a news broadcast, a cartoon and a weather forecast. Each video was approximately 2 minutes in duration.

All of the videos were encoded using the Mpegable Video SDK [7] with a target bitrate of 65kbps and a frame rate of 15 frames/s. The video was encoded using an IBBPBBPBBPBB frame sequence. A manual approach was used to segment the video into different objects. 3 of the pieces of video content were segmented into 2 video objects and one was segmented into 3 video objects. While MPEG-4 does provide some support for error resilience, no specific error resilience schemes are included in the standard. For this reason, no error resilience mechanisms are included in this work. Determining video quality is a non-trivial problem and the best approach is always to use subjective testing. However, this was not possible within the available resources and hence an objective approach was used. As results have shown that PSNR measurements can be as accurate as more sophisticated approaches [9], a simple PSNR approach was used here. While PSNR is a well known and well understood metric, two specific points relating to our PSNR calculations are worth mentioning here. Firstly, the PSNR measurements were bounded above with a 40dB threshold. This made it easier to perform reasonable comparisons (as PSNR comparisons can return very high values if the sequences are very similar). Secondly, the PSNR tools which are readily available do not operate well in environments in which there is data loss. Specifically, the tools have difficulty if there is a loss of synchronisation between transmitted sequence and the received sequence; such a loss of synchronisation occurs when a frame is dropped from the received sequence. To overcome this problem, when dropped frames occur, the frame preceding the dropped frame is duplicated in the received video sequence resulting in transmitted and received sequences of the same length. All of the results presented in section 6 below were obtained by streaming the content from server to client through the network emulator 5 times and averaging the results. 6 Results and discussion Experiments were done to assess two aspects of the performance of the multiplexing schemes: the overhead they introduce and the resulting video quality under different network conditions. These are discussed below. 6.1 Analysis of overhead The 4 content types were streamed from the server to the client. For each content type, loss rate and multiplexing scheme, the mean overhead statistics were obtained; these results were then averaged over all content types to obain an overall average. Figure 2: Packets generated by each packetisation technique for video D with 3 arbitrarily shaped video objects. The first set of results (see Figure 2) obtained shows the amount of packets that were transmitted using each of the schemes. Clearly, RFC 3016 generates many more packets than do the other multiplexing schemes. All of the other schemes generate similar amounts of packets. Analysis of the packet sizes showed that the RFC 3016 approach resulted in mean packet sizes of just over 280 bytes for the 2 VO case and approximately 185 bytes per packet for the 3 VO case. Also, the packet sizes for the non-rfc 3016 approaches are similar to each other as the server attempts to generate packets which are as close to the MTU as possible. It is worth noting that each RTP packet contained an average of 4-5 AUs in these experiments, but, in some cases, there were as many as 21 AUs in a single RTP packet. Figure 3 shows how the data transmitted from the server to the client is broken down into the different headers and payloads for content comprising of 2 VOs. The RFC 3016 solution results in very significant amounts of UDP/IP and RTP overhead about 8% of the total transmitted data. This arises from the fact that so many packets are transmitted using this multiplexing scheme. The other schemes result in approximately the same amount of Figure 3: Header overheads of the different packetisation approaches for video sequence A with a two arbitrarily shaped VOs. overhead (about 5%). Similar results were obtained for the case with 3 arbitrarily shaped VOs. In that case, however, the amount of overhead generated by the RFC

3016 approach is even greater than before approximately 15% of the total transmitted data while the other schemes generate approximately the same amount of data as in the 2 VO case. This demonstrates the scalability issues that arise with RFC 3016 in this context. information is lost for even a single frame, it can have a very detrimental impact on subsequent frames. An example of this is shown in Figure 6. Figure 4: Fraction of the data transmitted that is used for header information. This scalability issue is demonstrated more clearly in Figure 4 where the fraction of overhead is shown as the numbers of VOs in the content increases. There is a small increase in the amount of overhead generated for non- RFC 3016 schemes which results from the increase in the amount of AUs, and associated headers, per video frame. There is a much more significant increase in the amount of overhead generated by RFC 3016 as the number of VOs scales up; an increase from 13% to 19% can be seen as the number of VOs increases from 2 to 3. The difference between the overhead introduced by all of the non-rfc 3016 schemes is not very significant. The RTP4Mux scheme does appear to exhibit slightly more efficient behaviour, but since the difference is so small, it is probably not sufficiently large to argue that this would be good reason to chose RTP4Mux over the other approaches. 6.2 Received video quality The different multiplexing schemes were compared in terms of the impact they had on received video quality in the presence of varying loss and jitter. A playout buffer at the receiver is assumed: for the purposes of this work, the playout buffer was chosen to be sufficiently large to accommodate 100ms of video content. If a packet was delayed so long such that it when it arrived at the receiver, it had passed its playout time, the packet was discarded. In this way, jitter can have a significant impact on the quality of the video played out at the receiver. Shaped video differs somewhat from rectangular video in the context of packet loss and jitter: two key differences are apparent. Firstly, losses can be localised when shaped video is used: if the loss is local to a specific object, it typically does not impact other objects in the frame. An example of this is shown in Figure 5. Secondly, shaped video can exhibit great sensitivity to loss due to the importance of the shaped video information. If this Figure 5: Localisation of impairments with multiple VOs. the original unimpaired image; the object representing the birds is flawed; the background object remains undamaged. Figure 6: Damaged shape vectors can result in much damage to a VO. the unimpaired VO image; the decoded image from the VO with damage shaped vectors. In Figure 7 -(c), the variation in video quality with frame number for a particular video sequence is shown for the different multiplexing schemes in the presence of loss. There are some similarities between all 3 graphs: all graphs show some variation in the PSNR between 20-40dB with a bias towards 40dB. On closer inspection, it is possible to see some subtler differences between the graphs. RFC 3016 exhibits quite a lot of variation; for example, there is a prolonged period between frames 800 and 1200 during which the received video quality is less than 40dB. A number of factors combine to give this result: since RFC 3016 causes many packets to be generated and the packet loss rate is (more or less) fixed, more packets will be lost. This will mean that more frames will be impacted, albeit only partially, since typically only a single object in a frame is affected. Since there is a dependency between frames, these errors get propagated and, often, another packet loss occurs before these frame errors are resolved. The Multi-SL exhibits longer periods of better quality. There is more variation in the interleaved variant, which arises from the fact that the loss of a single RTP packet containing multiple AUs can impact a number of consecutive frames. This, in turn, can impact further frames due to the differential coding schemes used for video.

appear to perform better. As against this, the two RTP4Mux variants exhibit slightly poorer performance in the presence of substantial amounts of jitter. In an attempt to further compare the performance of the systems in terms of received video quality, the amount of Figure 8: Percentage frames damaged for each of the multiplexing schemes different loss conditions; different jitter conditions. (c) Figure 7: Video quality against frame number for content with 3 VOs for the different multiplexing schemes RFC 3016; Multi-SL; (c) RTP4Mux-Interleaved. Approximately 5% packet loss between server and client. Some more analysis of the received video was performed to determine how many frames suffered some damage through the transmission process for each of the different schemes. Two sets of experiments were performed; one in which the loss rate was varied and one in which the jitter experienced by the packets was varied. Figure 8 shows the results obtained for these two sets of experiments. In both cases, it is quite clear that RFC 3016 exhibits much poorer performance than the other approaches, which is consistent with what was observed previously. The interleaved FlexMux and Multi-SL cases also exhibit more sensitivity to packet loss while the two RTP4Mux variants and the non-interleaved FlexMux Figure 9: Number of occurrences damaged for each of the packetisation techniques under lossy conditions for videos with multiple VOs. sequences of damaged frames of length 1 or more was determined for each multiplexing scheme. This, coupled with the fraction of damaged frames can be used to better understand how the different multiplexing schemes result in different patterns of damaged frames. Unsurprisingly, the RFC 3016 scheme results in large number of damage sequences. The Multi-SL approach results in more damage sequences than the other schemes for smaller

packet loss rates. There is not a very significant difference between the other multiplexing schemes. Figure 10: Average PSNR of damaged frames for the various approaches in lossy environments for multiple VOs. Figure 11: Average PSNR of damaged frames for the various approaches subjected to jitter for multiple VOs A further analysis of the data was performed so as to determine the quality of the damaged frames received for each of the multiplexing schemes. This was done for both the packet loss experiments and the jitter experiments as shown in Figure 10 and Figure 11. There is quite some variation in the results obtained for each of the multiplexing schemes and some of them do not exhibit very clear trends which makes it difficult to make very strong inferences. In Figure 10, it can be seen that the frames damaged when transmitted using RFC 3016 are damaged least. This can probably be attributed to the fact that using RFC 3016 transmission, only data from a single object is lost whenever an RTP packet is lost. Thus, not even a whole frame is adversely affected. In contrast, for the other schemes, if an RTP packet is lost multiple AUs can be lost which can often result in data for an entire frame being lost or data for multiple objects in consecutive frames to be lost. In either case, since the amount of data lost is greater, it is not surprising that it has a more significant impact on those frames that are damaged. It is also notable that the Multi-SL approach results in damaged frames which have a higher PSNR than the other more sophisticated schemes for variations in both loss and jitter. 7 Conclusion A study of the performance of a number of different multiplexing schemes was conducted in the context of streaming of MPEG-4 object-based content, with particular emphasis on streaming content to mobile devices. Of the schemes that were compared, it was very clear that the performance of the RFC 3016-based scheme was significantly worse than that of the others in terms of the amount of overhead generated and impact of loss and jitter on video streamed using this scheme. Since the other schemes were very similar, it is not surprising that the performance of them was quite similar. They all generated very similar amounts of overhead and had quite a similar impact on the received video quality. There were some differences relating to the distribution of frames lost and how much each damaged frame was impacted, but these differences did not appear to be sufficiently large to enable any of the schemes to be identified as being better than the others in terms of the performance criteria used above. Acknowledgement The financial support of Irish Research Council for Science, Engineering and Technology and Enterprise Ireland are gratefully acknowledged. 8 References [1] C. Guillemot, P. Christ, S. Wesner and A. Klemets, RTP payload format for MPEG-4 with flexible error resiliency, IETF Internet-Draft, March 2000, expired September 2000. [2] D. Curet, E. Gouleau, S. Relier, C. Roux, P. Clement and G. Cherry, RTP payload format for MPEG-4 flexmultiplexed streams, IETF Internet Draft, expired May 8th, 2001. [3] A. Nafaa, T. Ahmed, Y. Hadjadj-Aoul and A. Mehaoua, RTP4mux: a novel MPEG-4 RTP payload for multicast video communications over wireless IP, in Proc. IEEE Packet Video(PV 03), April 2003. [4] IETF RFC 3016, RTP payload format for MPEG-4 audio/visual streams, November 2000. [5] IETF RFC 3640, RTP payload format for the transport of MPEG-4 elementary streams, November 2003. [6] T. Ahmed, G. Buridant and A. Mehaoua, Encapsulation and marking of MPEG-4 video over IP differentiated services, IEEE ISCC 01, Tunisia, pp. 346-352, July 2001. [7] Dicas Digital Image Coding GmbH, Mpegable MPEG-4 video SDK version 1.46, Holsteinsche Staße 39, D-12161, Berlin, Germany, March 2004. [8] National Institute of Standards and Technology (NIST), NISTNet: network emulation software, NIST, 100 Bureau Drive, Stop 3460, Gaithersburg, MD, June 2002. [9] VQEG, Final report from the video quality experts group on the validation of objective models of video quality assessment, http://www.vqeg.org, March 2000. [10] S. Goor, S. Murphy and L. Murphy, "Experimental Performance Analysis of RTP-Based Transmission Techniques for MPEG-4", 14th International Packet Video Workshop (PV2004), Irvine, California, December 13-14, 2004.