FINAL PROJECT. x x x x x x x x x x x. TITLE : Application of SP and SI frames in wireless multimedia communication

Transcription

1 x x x x x x x x x x x FINAL PROJECT TITLE : Application of SP and SI frames in wireless multimedia communication TITULATION: Enginyeria Tècnica de Telecomunicació, especialitat Sistemes de Telecomunicació AUTHOR: Maria Salvat Perarnau DIRECTOR: Markus Rupp SUPERVISOR: Luca Superiori DATE: July 25, 2007

2 Títol : Application of SP and SI frames in wireless multimedia communication Autor: Maria Salvat Perarnau Director: Markus Rupp Supervisor: Luca Superiori Data: 25 de juliol de 2007 Resum L objectiu d aquest treball és el de realitzar un estudi sobre les imatges SP i SI introduïdes pel codec H.264/AVC. Estudiarem les seves característiques, el seu comportament, així com les diferències amb les seqüències de vídeo constituïdes únicament per imatges I i P. Aquest dos tipus d imatges pretenen introduir una millora en aspectes del video streaming com poden ser random acces, el switching entre diferents bitrates,... En el primer capítol farem un repàs dels conceptes bàsics i els aspectes més importants de l estandard H.264/AVC. Comentarem les aplicacions del codec, els profiles que el defineixen i els tipus d imatges que introdueix. En els tres següents capítols estudiarem el comportament d una seqüència de vídeo, Foreman, introduint només imatges I i P en primer lloc. En segon lloc afegint-hi les imatges SP, i, finalment, una seqüència amb imatges I, P i SI. Tot seguit, pasarem al capítol 5 on buscarem les característiques més òptimes per aconseguir els amples de banda utilitzats per la tecnologia UMTS. En el capítol 6 ens centrarem en l estudi del switching entre una seqüència de alta qualitat i una de baixa. Després, analitzarem els canvis necessaris en el codi de l encoder per tal de poder realitzar el switching en uns punts definits. Finalment, trobarem les conclusions.

3 Title : Application of SP and SI frames in wireless multimedia communication Author: Maria Salvat Perarnau Director: Markus Rupp Supervisor: Luca Superiori Date: July 25, 2007 Overview The objective of this thesis is to make a study about the SP and SI pictures introduced by the codec H.264/AVC. We are going to study their characteristics, their behavior, and also, the differences between the video streaming sequences only formed by I and P frames. These two types of frames, SP and SI, introduce an improvement in some applications of the video streaming as random access and switching between different bitrates. In the first chapter, there is a resume of the basic concepts and the most important aspects of the standard H.264/AVC. We are going to talk about the codec applications, its profiles and the images types. In the next three chapters there is the study of a video sequence behavior, Foreman, introducing, at first, I and P frames only. Then, we are going to add SP frames, and finally, a sequence with I, P and SI frames. In the chapter 5 we are going to find the better characteristics to achieve the bandwidth used by the UMTS technology. In the chapter 6, there is the study of the switching between a high and a low quality sequences. Next, we are going to analyze the necessary changes in the encoder code in order to let the switch happen in defined points. Finally, we will discuss the conclusions.

4

5 To my family and Jordi.

6

7 CONTENTS INTRODUCTION CHAPTER 1. H.264/AVC Overview Applications H.264/AVC in wireless environments Transport in Wireless systems Profiles Baseline Main Extended Frames types and format Frame types definition CHAPTER 2. I and P frames Encoding works in baseline profile I frames P frames Standard sequence description Simulation and results of I and P sequences Bitstream switching in baseline profile CHAPTER 3. SP frames Advantages and disadvantages of using SP frames Primary and secondary SP frames Encoding and decoding SP frames Encoding and decoding process of primary SP frames Encoding and decoding process of secondary SP frames How does an SP frame work? Simulations and results of bitstream switching

8 3.6. Comparison of I switching and SP switching CHAPTER 4. SI frames Advantages and disadvantages on using SI frames Encoding and decoding process of SI frames How does an SI frame work? Simulation and results CHAPTER 5. Simulation Scenario Kbps Kbps Kbps Comparison of results CHAPTER 6. Switching simulations High and Low Quality Simulation Graphics Switching simulation CHAPTER 7. Code improvements Original code Modified code CHAPTER 8. Conclusions BIBLIOGRAPHY APPENDIX A A.1. Abbrebiations A.2. Table of results for the simulation scenario

9 APPENDIX B B.1. Configuration file: encoder.cfg B.2. Configuration file: decoder.cfg

10

11 LIST OF FIGURES 1.1 Standardization Scheme H.264/AVC standard in transport environment H.264/AVC Profiles Subdivision of a frame into slices Spatial Prediction Full macroblock prediction Multi-frame motion compensation Standard Sequence Relation between I & P frames and the QP On the right, the I frame size vs the GOP size. On the left, the P frame size Switching by means of I frames, first step Switching by means of I frames, second step An SP frame in a stream Secondary SP frames, SPABn Block diagram of the encoding process of primary SP frames Block diagram of the decoding process of primary SP frames Resumed block diagram of the encoding process of secondary SP frames Block diagram of the decoding process of secondary SP frames Temporal sequence of switching on the right. The two bitrates on the left Bitstream switching process, steps 1 and Bitstream switching process, steps 3 and The size of I frames versus SP rate, for two different QP, 23 and The quality of I frames versus SP rate, for two different QP, 23 and The size of P frames versus SP rate, for two different QP, 23 and The quality of P frames versus SP rate, for two different QP, 23 and The size of SP frames versus SP rate, for two different QP, 23 and Graphic with the size of the three frame types Comparison of the qualities between I and SP switching Comparison of the bitrates between I and SP switching Resumed block diagram of the encoding process of SI frames Block diagram of the decoding process of secondary SP frames Sending the video stream What happens without SI frames when an error appears SI frames: the client finds the error and sends a warning SI frames: an SI frame is generated SI frames: the error is corrected I frame size versus SI rate for two different QP, 23 and I frame quality versus SI rate for two different QP, 23 and P frame size versus SI rate for two different QP, 23 and Comparison graphic of the size of the three frame types Videostream visualization of the pair selected, QP 37 and GOP

12 5.2 Videostream visualization of the pair selected, SP rate of 15 and QP Videostream visualization of the pair selected, SIrate 45 and QP Videostream visualization of the pair selected, QP of 30 and a GOP Videostream visualization of the pair selected,sp rate 8 and QP Videostream visualization of the pair selected, QP 38 and SI rate Videostream visualization of the pair selected QP of 23 and a GOP value of Videostream visualization of the pair selected, SP rate 45 and QP Videostream visualization of the pair selected, QP 23 and SI rate Comparison graphic of P and SP frames Visualization on High and Low quality Quality comparison between sequences with and without SP frames, for high and low quality Visualization of the frames, 1st frame of the streaming Visualization of the frames, 8th frame of the streaming Text visualization of the sequence Quality of switching simulation Comparison of the Qualities Frames of the switching video streaming encoder.cfg: SP rate variable encoder.cfg: change QP variable Encoder code: SetImgType function Text file of a switching simulation Example of a file with switching point Text file of the switching simulation with the modified code Function SetImgType modified A.1 Results for I & P sequences A.2 Results for I & P & SP sequences A.3 Results for I & P & SI sequences

13 LIST OF TABLES 5.1 Results for 44 Kbps by I&P sequence Results for 44 Kbps by I&P&SP sequence Results for 44 Kbps by I&P&SI sequence Results for 105 Kbps by I&P sequence Results for 105 Kbps by I&P&SP sequence Results for 105 Kbps by I&P&SI sequence Results for 360 Kbps by I&P sequence Results for 360 Kbps by I&P&SP sequence Results for 360 Kbps by I&P&SI sequence Frame sizes for a high quality simulation Frame sizes for a low quality simulation Frame sizes for a low quality simulation Frame sizes for a low quality simulation Pair number definition Frame sizes for a switching simulation

14

15 1 INTRODUCTION H.264/AVC is the newest coding standard based on hybrid block video compression. It is the result of the standardization effort of the ISO Moving Pictures Group (MPEG) and ITU-T Video Coding Experts Group (VCEG). The main purpose of this standard is the attempt to improve the compression efficiency of the video streaming. Because of this characteristic, H.264/AVC is the best standard to use in wireless systems. Slow variance due to distance, shadowing, handover, etc. transform the wireless channel in a slowly varying variable-bit-rate channel. As a consequence of these variations of the bitrate, the study of the new frames introduced by the standard, SP and SI frames, is a fact to take into account to provide a better switching between the bitrates. It is also important to introduce other applications such random access, error recovery, etc. We start this work with the study of a standard sequence formed by I and P frames only. I frame only depends on itself, but P frames depend on the previous frames encoded. The size of the sequence, as in any other sequence type, depends on the Quantization Parameter (QP) and the GOP (Group Of Pictures) value. The GOP value is the space (in terms of pictures) between two I frames. So, if there is an error during the transmission of the stream we have to wait until the next I frame, which can be several seconds later, to continue sending the video without any error. Not to reduce the effects of the error transmission but to make error propagation smaller, the standard introduces the two new images types. In the standard H.264, the bitstream switching can be produced by meanings of I frames or SP frames. SP frames are smaller than I frames for same quality. They provide better bit charge because we can introduce them more often without increasing the bitstream size as the way it is increased by introducing the I frames.there are two types of SP frames, primary and secondary. To each primary SP frame, corresponds a secondary. Secondary SP frames are only introduced when a switching is produced. SI frames share the instant refresh properties of I frames but are only sent after a frame is lost and in a random access. The main advantage of the SI frames is that they do not need any reference frame to be encoded, because they use intra prediction. One of the most important problems introduced by SI frames is that they are bigger even than I frames, so that makes its use more restricted. In this thesis we have also analyzed simulation scenario used by UMTS. The reason is that it is supposed to introduce these frame types in this technology environment. We also studied bitstream switching because is an important application of SP frames. Finally, we present our code improvements in order to let the switch happen in defined points, rather than each switching period as it was previously implemented. After this work, we can conclude that SP frames are better than I frames in a switching scenario. The reason is that the SP switching simulation gives a more constant level of

16 2 Application of SP and SI frames in wireless multimedia communication the bitrate values than I switching, which introduces very high peaks. Also, the SP frames introduce less bits than SI frames. For the SI frames we can conclude that their use is more restricted because SI frames introduce more bits than any other frame, they are even bigger than I frames.

17 H.264/AVC Overview 3 CHAPTER 1. H.264/AVC OVERVIEW H.264/AVC is the newest coding standard based on hybrid block video compression. It is the result of the standardization effort of the ISO Moving Pictures Group (MPEG) and ITU-T Video Coding Experts Group (VCEG). The official title is Advanced Video Coding for the MPEG4 and H.264 for the ITU-T, but it is called and known as standard H.264/AVC. The purposes of the standardization effort are to improve the compression behavior, to develop a unique and a simple video coding design and to provide a network-friendly video representation which addresses conversational (telephony) and non-conversational (storage, broadcasting or streaming) applications [1]. Together with this purposes, the demanded services and the popularity of high definition TV have produced the need of a higher coding efficiency, high quality and high bitrate. For the earlier standards, transmission media such as xdsl, UMTS or Cable Modem offer much smaller data rates than broadcast channels. Even for DVB-T, there is insufficient spectrum available. In the past years, the two groups had developed their own standard. MPEG had developed focusing its achievements in the video storage, while the main target of the VCEG was the video streaming. Below, there is a graphic with the developed standards of each organization. Figure 1.1: Standardization Scheme. In 1991, the ISO Moving Pictures Group introduced the standard MPEG-1. The MPEG-1 is the initial standard of video and audio compression, which is used by Video CD (VCD) and includes the popular format of audio compression MP3. The quality obtained by the VCD is similar to a domestic VHS. As an extension of this first standard, in 1994 appeared the MPEG-2. MPEG-2 is focused on the generic codification of moving pictures and audio information. It is generally used for the video and audio compression, which includes: terrestrial TV (DVB-T), satellite TV (DVB-S), cable TV (DVB-C), High Definition TV (HDTV) and, it is also used by SVCD (Super VCD) and DVD (Digital Versatile Disc). After this second standard, ISO MPEG designed the standard MPEG-3. MPEG-3 was designed to

18 4 Application of SP and SI frames in wireless multimedia communication be a video compression standard for the High Definition TV (HDTV), but the advantages on the use of MPEG-2 demonstrated that it was possible to achieve similar results with this earlier standard; so, they did not continue enhancing MPEG-3. While MPEG was developing its own standards, the ITU-T Video Experts Group developed its own too. They started in 1990 with the developing of H.261. The H.261 is a standard originally designed for transmission over ISDN lines. H.261 supports two image resolutions, QCIF (Quarter Common Interchange Format) which is 144x176 pixels and CIF (Common Interchange Format) which is 288x352 pixels. After this one, appeared the H.262 standard, which is identical to the one developed for the MPEG for the HDTV and DVD, MPEG-2. The two contents of these standards are exactly the same because it was developed between the ITU-T and the ISO organizations, as has happened with the H.264/AVC standard. In 1996, after this common standardization, the ITU-T designs the new standard H.263. H.263 is low-bitrate compressed format standard for videoconferencing. Originally, it was designed as an enhanced standard based on H.261, the previous ITU-T standard for video compression, MPEG-1 and MPEG-2. Once these two organization developed their standards, they agreed to design a new video compression standard as the product of a collective partnership effort known as the Joint Video Team (JVT). The H.264 name follows the ITU-T naming convention, while the MPEG-4 AVC name relates to the naming convention in ISO/IEC MPEG. The first complete version of the H.264/AVC was presented in May The intention of the organization was to develop a new standard without increasing the complexity of design, only enhancing some parts to make it more efficient, and to make it more compatible with much more applications than the previous standards. The basic functional elements are little difference from the previous standards, the important changes in H.264/AVC occur in the details of each functional element [2] Applications The purpose of H.264/AVC is to make it compatible with most of the existing applications, many of them possible with the previous standards. Next, there is a list with the most important applications defined in Draft ITU-T Recommendations [3]: 1. Cable TV on optical networks (CATV). 2. Direct broadcast satellite video services (DBS). 3. Digital subscriber line video services (DSL). 4. Digital terrestrial television broadcasting (DTTB). 5. Interactive Storage Media (ISM): optical disks, etc. 6. Multimedia Mailing (MMM).

19 H.264/AVC Overview 5 7. Multimedia services over packet networks(mspn). 8. Real-time conversational services (RTC): videoconferences, videophone, etc. 9. Remote video surveillance (RVS). 10. Serial Storage Media (SSM). The development of H.264/AVC has opened the doors of new markets and industrial opportunities. Nowadays, many enterprises and companies have introduced the using of the standard in their own developing [4]. As an illustration of this development, consider the case of mobile TV for the reception of audio-visual content on cell phones or portable devices, presently on the verge of commercial deployment. Several such systems for mobile broadcasting are currently under consideration: Digital Multimedia Broadcasting (DMB) in South Korea. Digital Video Broadcasting - Handheld (DVB-H) in Europe and United States of America. Multimedia Broadcasting/Multimedia Service (MBMS)as specified in Release 6 of 3GPP. For these three mobile TV services, the use of H.264/AVC is focused to obtain a better video compression. If they can achieve this better video compression, then,it is possible to achieve better error robustness and better quality on the transmission systems. Another field of application is for satellite TV services. Important enterprises and companies of satellite TV distributors have announced deployments of H.264/AVC H.264/AVC in wireless environments Because of the enhanced video compression efficiency and error resilience features, H.264/AVC is the best standard to use in wireless systems. The troubles caused by the limited bandwidth over the radio-link require a better video compression which is the main requirement for a video coding standard to be successful in a mobile environment. The video compression performance of the H.264/AVC video coding layer typically provides a significant improvement. The network-friendly design goal of H.264/AVC is addressed via the network abstraction layer that has been developed to transport the coded video data over any existing and future networks including wireless systems. This two layers are described in section Transport in wireless systems. Slow variance due to distance, shadowing, handover, etc. transform the wireless channel in a slowly varying variable-bit-rate channel. With an appropriate setting of the initial delay and receiver buffer a certain quality of service is guaranteed [5].

20 6 Application of SP and SI frames in wireless multimedia communication The latter techniques such as switching predictive to achieve a channel adaptive streaming are supported by H.264/AVC. As the streaming server is in general aware of the current channel bitrate, the transmitter can decide to send one of several pre-encoded versions of the same content taking into account the expected channel behavior Transport in Wireless systems This new standard, as in the earlier ones, does not explicitly define a CODEC ( COder DECoder). What is defined is the syntax for encoding a bitstream and the method of decoding it. H.264/AVC is designed in two layers. The first one is the Video Coding Layer (VCL) which represents efficiently the coded video contents. The second one is the Network Abstraction Layer (NAL) which is designed to adapt the format of the video details to the transmission support. It also provides adapted header informations for different transport layers or storage media [6]. The figure 1.2 depict the standard in transport environment: Figure 1.2: H.264/AVC standard in transport environment Profiles Profiles and levels specify conformance points that provide interoperability between encoder and decoder implementations within applications of the standard and between various applications that have similar functional requirements. Currently, in the standard there are defined six profiles, but the most important are: baseline (BP), main (MP) and extended (EP). The other three are: high, high 10 and high 4:2:2

21 H.264/AVC Overview 7 profile. Each one of them contemplates its own characteristics and every one has its own applications. Next, there is a figure with the contents of each one (figure 1.3). All profiles define a set of coding tools or algorithms that can be used to generate a compliant bitstream. All decoders complying a specific profile have to support all features in that profile. Encoders are not required to make use of any particular set of features supported in a profile but have to provide conforming bitstreams. The High profile is used for an 8x8 spatial prediction and transform, the monochrome format and scaling matrices. The High 10 is used in the case the bit depth is up to 10b. Finally, the High 4:2:2 is applied for 4:2:2 chroma format. Next there is an explanation of the three foremost profiles. Figure 1.3: H.264/AVC Profiles Baseline Baseline profile is, typically considered, the simplest profile and includes the basic features of the H.264/AVC standard: Only I and P slice types may be present. Flexible Macroblock Ordering (FMO). Arbitrary Slice Ordering (ASO). Redundant Pictures. Motion-compensated prediction. In-loop deblocking. Intra-prediction. Context Adaptative Variable Length Coding (CAVLC). This profile emphasizes coding efficiencies and robustness with low computational complexity. The features not supported by the baseline profile are:

22 8 Application of SP and SI frames in wireless multimedia communication B slices. Weighted prediction. Picture or macroblock adaptive switching between frame and field coding. SP and SI slices Main The Main profile emphasizes primarily coding efficiency alone. It typically allows the best quality at the cost of higher complexity (essentially due to the B-slices and CABAC) and delay. This second profile contains all the features of the baseline with the exception of: FMO, ASO and Redundant Slices. It also includes: B slices. Field coding. Weighted prediction. Macroblock adaptive frame-field (MBAFF). Coding Adaptative Binary Arithmetic Coding (CABAC). Only a subset of the coded video sequences that are decodable by a Baseline profile decoder can be decoded by a Main profile decoder Extended The Extended emphasizes robustness and flexibility with high coding efficiency. This profile is a superset of the Baseline and main profiles supporting all tools in the specification with the exception of CABAC. The SP/SI slices and slice data partitioning tools are included only in this profile Frames types and format In the existing video coding standards, such as MPEG-2, H.263 and MPEG-4, three main types of frames are defined: I, P and B slices. H.264/AVC supports these three types and adds two types more which are new: SP and SI slices.

23 H.264/AVC Overview 9 A picture of a video sequence, a frame, is divided into macroblocks. Each macroblock has a fixed size that cover a rectangular picture area of 16x16 samples of the luma component (brightness) and 8x8 samples of each of the two chroma components. The macroblocks are organized in slices, which represent regions of a given picture that can be decoded independently of each other. Here there is an example: Figure 1.4: Subdivision of a frame into slices. To know more about what is the luma and chroma component see [7] Frame types definition I or Intra slices are the simplest ones. They are coded using Intra prediction. They are not referred to any previous slice of the video sequence, they only contain reference from themselves. The first frame of a sequence have to be Intra coded. All profiles support this type. P or Predicted slices are coded using Inter prediction. Inter prediction creates a prediction model from one or more previously encoded video frames. The model is formed by shifting samples in the reference frame(s) (motion compensated prediction). The AVC CODEC uses block-based motion compensation, the same principle adopted by every major coding standard since H.261. with at least one motion compensated prediction signal per prediction block [8]. All profiles support this type. B or Bi-predicted slices are coded using Inter prediction with two motion-compensated prediction signals per prediction block that are combined using a weighted average. All profiles except the Baseline profile supports this type. SP or Switching P slices permit an efficient switching between two different bitstreams coded at different bitrates, without the large numbers of bits required by I slices. They are only supported by the Extended profile. SI or Switching I slices are encoded only using Intra prediction, allow exact match with SP slices for random access or error recovery. They are only supported by the Extended profile.

24 10 Application of SP and SI frames in wireless multimedia communication

25 I and P frames 11 CHAPTER 2. I AND P FRAMES 2.1. Encoding works in baseline profile Nowadays, UMTS allows only the baseline profile reference TS 260. So, at first, we are going to start the study with baseline profile but not because we are going to use it. Baseline profile only accepts I and P frames, so SP and SI frames are not allowed, and we are going to start with and study of the basic frames I frames In contrast with the previous standards, in H.264/AVC intra prediction is always conducted in the spatial domain, while, previously, was in transform domain (frequency domain coefficients). The idea of spatial prediction is based on the observation that adjacent macroblocks tend to have similar textures. It is possible to predict the current macroblock to be encoded by the surrounding macroblocks. Figure 2.1: Spatial Prediction. One of the advantages of using spatial prediction is the improvement of the predicted signal quality and permits to take as reference areas that are not temporally predicted. There are two basic definitions for Intra prediction: 1. Full macroblock intra prediction (16x16 luma prediction) 2. 4x4 luma prediction Intra 16x16 luma prediction defines only one spatial prediction scheme for the whole macroblock. Pixels may be filled from surrounding macroblocks at the left and the upper edge using one of four possible prediction modes. Intra prediction is also performed for the chroma planes using the same range of prediction modes. However, different modes may be selected for luma and chroma [9]. There are four types for full macroblock prediction:

26 12 Application of SP and SI frames in wireless multimedia communication horizontal, vertical, DC and plane. The best way to understand the four ways is viewing an example: Figure 2.2: Full macroblock prediction. The vertical mode (0) predict the current macroblock to be encoded by the vertical adjacent macroblocks. The horizontal mode (1) is the same than mode (0) but using the horizontal adjacent macroblocks. The DC mode (2) makes the prediction averaging the values of the neighbor macroblocks. Finally, the plane mode (3) is defined by a three-parameter curve-fitting equation, having a brightness, slope in the horizontal direction, and slope in the vertical direction that approximately matches the neighboring pixels P frames P frames are encoded by using Inter prediction. Inter prediction is based on a creation of a prediction model from one previously picture. In the previous standards, such MPEG-2 and its predecessors was predicted only by using one previously picture to predict the values of the incoming pictures. The standard H.264/AVC gives the freedom of choosing the reference frame among various pictures. The new design is developed to enable efficient coding by allowing an encoder to select among a larger number of pictures as reference (motion compensation model). These pictures have been decoded and stored at the decoder. The motion compensation model is formed by shifting samples in the reference frame. The new features of motion compensation introduced by H.264/AVC are: Multiple reference picture motion compensation: in this new standard, inter prediction is based on prediction by using a larger number of pictures (figure 2.3). Quarter-sample-accuracy motion compensation: motion compensation describes a picture by the origin section of that picture in a previous sample. The frames are partitioned in blocks of pixels. The offset accuracy in the previous standards is about a half of a pixel, but with H.264/AVC is about a quarter of a pixel. The offset between the two areas has 1 4-pixel resolution.

27 I and P frames 13 Figure 2.3: Multi-frame motion compensation Standard sequence description A standard sequence of the H.264/AVC is defined in baseline profile by I and P slices. If we want to introduce the others types of frames we have to change the profile, but we are going to see it in the next chapters. The size of the sequence depends on the Quantization Parameter (QP) and the GOP (Group of Pictures) number. The QP is a parameter used for determining the quantization of transform coefficients in H.264/AVC. This parameter can take 52 values and allows the changing of the coding sequence quality. The bigger is the QP, the lower is the quality of the video streaming. The GOP size is the number of frames between two consecutive I frames. It determines the error resilience: if we have a bigger GOP it is able to compress better, but the error can propagate longer. The error is propagated until the next I frame, which starts a new GOP. It can be defined using two variables: N+1, which is the number of pictures inside of the GOP, including the I frame. N, which is the spacing between I frame. Here, there is an example: Figure 2.4: Standard Sequence.

28 14 Application of SP and SI frames in wireless multimedia communication Simulation and results of I and P sequences In this section, there is a study of I and P frames behavior. We want to analyze the dependency of these frames on the Quantization Parameter and the GOP size. All the simulations are made with the entire Foreman sequence (400 frames). As a first analysis, we will discuss the relation between I and P frames size and the QP. In this case, the video sequence was simulated with a GOP value of 20. Figure 2.5: Relation between I & P frames and the QP. As it was expected, the size of the frames depends on the QP of the encoded streaming. The reason is just a fact of the quality of the resulting frame.the bigger is the QP value, the lower is the quality of the sequence, and vice versa. Another thing that is very important to remark is, as is obvious in the graphic, I frames are much bigger than P frames. It has an easy explanation because the frame size follows the spatial and temporal prediction efficiency. More complex frames require more bits for their description, I frames prediction, while others are described by fewer bits, P frames. Now, let s see the results of I and P frames depending on the GOP size. For these simulations the QP is fixed in 36. For the GOP size, if we take a look to our results (figure 2.6), we can notice that I frames do not depend on the GOP size, they only depend on the characteristics of the I frames selected to be encoded. For the P frames, we realize that there is some kind of dependency with the GOP size, the bigger is the GOP size, the bigger is the size of the frames. Prediction is less efficient.

29 I and P frames 15 Figure 2.6: On the right, the I frame size vs the GOP size. On the left, the P frame size Bitstream switching in baseline profile In a wireless video streaming system, may be difficult to achieve a guaranteed end-to-end quality of service over the entire streaming period. Datarate in wireless multimedia communication channels changes very often, so, to get a better quality of the service for the user, a change between the transmitted video datarates is needed. Such switching of the channel characteristics optimizes the use of the radio resources and facilitates providing the rewired quality of service. In the previous standards and in baseline profile, perfect bitstream switching is only possible at I frames, mis-match free switchings. Now, let s see an example. There are two encoded bitstreams and we want to change from the one encoded with high quality to the low. In the case the switching is necessary, we have to wait until the next I frame because P frames are predicted by temporal prediction, taking as reference the previous processed frames. Figure 2.7: Switching by means of I frames, first step.

30 16 Application of SP and SI frames in wireless multimedia communication We start transmitting the highest bitrate sequence until we arrive to the I frame. Once we have arrived to the I frame, we change from the first sequence to the other sending the I frame of the lowest bitrate bitstream. Then we continue transmitting the P frames of the second bitstream. Figure 2.8: Switching by means of I frames, second step. The drawback of using I frames in these applications is that they require much larger number of bits than P frames at the same quality. Therefore I switching gives an unacceptable bitrate. That is why H.264/AVC has introduced the SP and SI frames, in order to improve the quality using less resources.

31 SP frames 17 CHAPTER 3. SP FRAMES SP frames are specially-coded frames which enable efficient switching between video streams and efficient random access for video decoders. They are only permitted in the extended profile. The difference between SP frames and P frames is that SP frames allow identical frames to be constructed even when they are predicted using different reference frames. SP frames are smaller than I frames and they are designed to support switching between similar coded sequences without the increased bitrate, penalty of I slices. They are classified as secondary SP frames and primary SP frames. For each primary SP frame, a corresponding secondary SP frame is generated, which has the same identical reconstructed values as the primary, but they are only set during bitstream switching Advantages and disadvantages of using SP frames The using of SP frames gives some advantages which make the SP frames more efficient and comfortable than other frames for working : They require fewer bits than I frames to achieve the same quality. For example, to achieve a quality of 36 db, an SP frame employs 4 8 Kbits, while the I frame employs 24 Kbits. It is possible to reconstruct a picture with different reference frames. They can be used instead of I frames in switching, fast forward, fast backward, random access and error resilience and recovery. In the other hand, they have some disadvantages: They are not allowed in baseline profile. They are less efficient than normal P pictures: the overall coding efficiency is degraded if many switching points are assigned Primary and secondary SP frames SP frames are designed to support bitstream switching. Besides, as it s shown in the figure 3.1, they can be placed in a single bitstream even when there is no foreseen bitstream switching.

32 18 Application of SP and SI frames in wireless multimedia communication Figure 3.1: An SP frame in a stream. There are two types: primary and secondary SP frames. Now, assume that there are two bitstream of the same video sequence but encoded with different encoding parameters, and we want to switch from the bitstream A to the bitstream B. At the switching point there are three SP frames (figure3.2). Figure 3.2: Secondary SP frames, SPABn. The first one is SP An which is generate in bitstream A, and its reference is the previous encoded frame of its own bitstream, A n-1. SP An is going to be the reference of A n+1. The second SP frame is the frame named as SP Bn. It is encoded in the second bitstream. It is referred to the previous frame B n-1 and is going to be the reference of B n+1. This two SP frames are primary SP frames. Then, there is the third SP frame, SP ABn, which is a secondary SP frame. This frame is only generated when the switching is proceed. As a reference to encode it, the server takes a previous frame of the origin bitstream, A n-1. But, in this case, SP ABnis going to be the reference for the next frame which belongs to bitstream B, B n-1. The secondary SP frames are frames only used when switching from one bitstream to another. If there is no bitstream switching they are not applied. It can be resumed as when a switching is needed, from the primary SP frame of the first bitstream, a secondary SP frame is generated to change the bitrate. In the next sections, we are going to know exactly how the switching is performed.

33 SP frames Encoding and decoding SP frames In this section there is a description of the encoding and decoding process for primary and secondary SP frames. In both case we assume that we are encoding and decoding nonintra blocks in SP. For intra blocks the process is identical as the I frames Encoding and decoding process of primary SP frames In figure 3.3 there is the block diagram of the encoder for primary SP frames[10]. Figure 3.3: Block diagram of the encoding process of primary SP frames. The encoding process starts subtracting a motion-compensated version of the last frame reconstructed. Then, with the original image, the block P is predicted by motion-compensation. After predicting P, a forward transform is applied and the transform coefficients are quantized and dequantized with SPQP as the quantization parameter. The results obtained after the quantization are marked in the figure as dpred. After processing the predicted block P, the encoder substracts the results dpred from the transform coefficients of the original image, cerr. Then, they are quantized using PQP as quantization parameter and sent to the multiplexer with the motion information (motion vector). Once, we have seen the encoding process of the primary SP frames, let s see the decoding. The decoding process follows the same steps as the encoder but in another order (figure

34 20 Application of SP and SI frames in wireless multimedia communication 3.4)[10]. Figure 3.4: Block diagram of the decoding process of primary SP frames. By motion-compensation, a predicted block P is obtained and transformed, cpred. The cpred results are added to the received inverse quantized error coefficients, crec. The result of the addition is quantized and dequantized with the SPQP as the quantization parameter. The quantization parameter of the inverse quantization, PQP, of the error coefficients is not necessarily be the same as the quantization parameter used to quantize the addition result, SPQP Encoding and decoding process of secondary SP frames The secondary SP frames follow the same scheme than primary SP frames in the encoding process, but there are two details that make it different. The first one is that secondary SP frames predict the block P using as the previously reconstructed frame the frame of the origin bitstream (bitstream A in the previous section of the chapter). And the second one is that the transform coefficients of the predicted block P are subtracted from the original image of the destination bitstream, bitstream B. Here, there is a simple diagram of the encoding process [11]:

35 SP frames 21 Figure 3.5: Resumed block diagram of the encoding process of secondary SP frames. The decoding process is quite different from the decoding process of the primary SP frames. Let s see the scheme[10]: Figure 3.6: Block diagram of the decoding process of secondary SP frames. If we compare this diagram with the block scheme of the primary SP frames (figure 3.4), we can appreciate two important differences. With the primary SP frames the predicted block P is transformed and added to the inverse quantized error coefficients. Then, the result of the addition, cpred, is quantized. For the secondary SP frames, the predicted block is transformed and quantized before the addition, lpred. Then, this result is added to the predicted error coefficients without being inverse quantized. The result of the addition is inverse quantized and inverse transformed How does an SP frame work? The best way to understand how an SP frame work is by an example. First of all, in figure 3.7 is drawn the temporal visualization of the bitstream switching that we want to do.

36 22 Application of SP and SI frames in wireless multimedia communication Figure 3.7: Temporal sequence of switching on the right. The two bitrates on the left. In the streaming server,there are two encoded sequences, one of high bitrate, 128 Kbps, and another of low bitrate, 64 Kbps, both of them with a SP rate of 7. If we want to switch without SP frames, we have to wait until the next I frame, which can be several seconds later. But if we make it with SP frames we only have to wait for an SP frame, which probably is going to appear and be encoded before an I frame. Due to the channel conditions we cannot continue using the bitrate of 128 Kbps, so at time t1we have to switch to the lowest bitrate. We send the coded sequence from the high bitrate stream until a primary SP frame (number one in figure 3.8). This primary SP frame, SP An, is referred to the previous P frame encoded. SP An is going be the reference for the next frame. In the case there is no switching, SP An is going to be the reference of the next P frame of th 128 Kbps stream. But, when the switching is proceed, from 128 Kbps to 64 Kbps, a secondary SP frame is generated, SP ABn (number two in figure 3.8). SP ABnis referred to the primary SP frame, SP An. Figure 3.8: Bitstream switching process, steps 1 and 2. The secondary SP frame is going to be the reference for the next P frame in the 64 Kbps stream (number three in figure 3.9). If we want to switch again, from 64 Kbps to 128 Kbps, we have to do the same way back (number four in figure 3.9). We have to wait until the next SP frame, SP Bn, which is going to be the reference of the secondary frame generated to switch, SP BAn. In the decoding process, when we want to decode a P frame that follows a switching,

37 SP frames 23 Figure 3.9: Bitstream switching process, steps 3 and 4. it does not matter which type of SP frame is chosen. We can use both, primary and secondary SP frames for reconstruct the picture. We need to know which one is used only to know if there have been a switching or not Simulations and results of bitstream switching In this section there is an analysis of the impact of SP frames in the size and the quality of the frames in the video sequences. Exactly, we are going to examine the dimension of the frames relying on SP rate and QP. For these simulations, the GOP size is fixed in 50 and we have used the complete Foreman sequence (400 frames). Let s see what happens with the I frames in figure The graphics represent the size of the I frames for two fixed QP, 23 and 37, versus the SP rate. Figure 3.10: The size of I frames versus SP rate, for two different QP, 23 and 37. As we can observe on the graphics, there is no relation between the SP rate and the

38 24 Application of SP and SI frames in wireless multimedia communication bit number of the I frames, it does not change if we modify the SP rate. For the quality happens exactly the same (figure 3.11). Figure 3.11: The quality of I frames versus SP rate, for two different QP, 23 and 37. For P frames the results are quite different (figure 3.12). Figure 3.12: The size of P frames versus SP rate, for two different QP, 23 and 37. On the contrary, we have noticed that there is a relation between the SP rate and the size of the P frames. The bigger is the SP rate, the lower is the number of bits of the P frames. Therefore, we can conclude that SP frames degrade the quality of bitstreams (figure 3.13).

39 SP frames 25 Figure 3.13: The quality of P frames versus SP rate, for two different QP, 23 and 37. Now, the analysis of SP graphics in figure Figure 3.14: The size of SP frames versus SP rate, for two different QP, 23 and 37. The size of the SP frames does not depend on the SP rate value, they only depend on the frequency of the SP frames to be encoded. Of course, if the size does not depend on the SP rate, with the quality happens also the same. Finally, let s take a look to the graphic comparing the size of the three types of frames with the SP rate. As we know, the size in bits of the SP frames is smaller than the size of I frames, but are bigger than the number of bits of P frames, for a given QP (figure 3.15). So, we can say that if we are talking about sizes with the reference of the QP, the SP frames have to be in the middle, between the I frames and P frames size. But, if the reference is the SP rate the smallest frames are SP frames. For a given QP, with the size of the frames

40 26 Application of SP and SI frames in wireless multimedia communication depending only on the SP rate, the size of the SP frames is the smallest one. Figure 3.15: Graphic with the size of the three frame types Comparison of I switching and SP switching In this part of the chapter we are going to compare the results obtained between a SP switching and an I switching. What we want to do is a switching between high and low quality sequences. The high quality sequence corresponds to a video streaming with a QP of 28, while the low sequence corresponds to 49. As a first analysis, there is the graphic of the quality over time (figure 3.16). On one hand, as we can observe, with an I switching we can reach higher quality than with a SP switching. The two values of qualities, the high and the low, are bigger with an I switching than with SP switching. So, we can conclude that SP switching degrades the quality of the video streaming. In both cases, the difference between the high and the low quality is between 15 and 16 db. On the other hand, as we can observe, with an I switching we can reach highest peaks of bitrate than with a SP switching. But, introducing SP frames the bitrate is less variable, the difference of the peak values is lower. When the switching take place, the bitrate achieved by the I frame is higher than the bitrate achieved by the SP frame, the difference is about 3 25 Kbps (figure 3.17). We only represent the first 100 frames in both graphics because of its periodicity.

41 SP frames 27 Figure 3.16: Comparison of the qualities between I and SP switching. Figure 3.17: Comparison of the bitrates between I and SP switching.

42 28 Application of SP and SI frames in wireless multimedia communication

43 SI frames 29 CHAPTER 4. SI FRAMES SI frames are frames that allow an exact match with an SP slice for random access or error recovery purposes, while using only Intra prediction. SI frames share the instant refresh properties of I frames but are only sent after a frame is lost. SI frames, in some applications, like, video streaming switching, are used in conjunction with SP-frames, but their using gives some advantages in other fields. An SI frame uses intra prediction as an I-frame and still reconstructs identically the frame which follows it, the corresponding P frame, which uses motion-compensated prediction Advantages and disadvantages on using SI frames The main advantage of the SI frames is that they do not need any reference frame to be encoded, because they use intra prediction. This advantage make them very useful in such applications as: Error recovery: if there is some error we can reconstruct the following stream without any reference frame. Random Access: we can restart the playing from any SP frame, which means that an SI frame is going to be generated. Another advantage is that we can use them only when they are needed by a warning message from the client. As disadvantages we can say that their size is much bigger than any other frames, they are even bigger than I frames. This disadvantage makes them less useful than SP frames in a switching scenario because SP frames can achieve the same results with lower number of bits Encoding and decoding process of SI frames In this section there is a description of the encoding and decoding process for SI frames. As it happens with the secondary SP frames, SP frames predict the block P using as the previously reconstructed frame the frame of the origin bitstream (bitstream A), predicted by motion-compensation. But to encode an SI frame, the original image with which the encoder makes the subtraction, frame B, must be intra-predicted[11].

44 30 Application of SP and SI frames in wireless multimedia communication Figure 4.1: Resumed block diagram of the encoding process of SI frames. Now, let s see the scheme of the decoder[10]: Figure 4.2: Block diagram of the decoding process of secondary SP frames. By intra-prediction, a predicted block P is obtained, transformed and quantized, lpred. The lpred results are added to the received error coefficients, lerr. The result of the addition,lrec, is dequantized with the SPQP as the quantization parameter, and finally, inverse transformed How does an SI frame work? As we have done with the SP frames, the best way to understand how an SI frame works is by an example. There is one video stream to be sent. The streaming server send an encoded sequence over an RTP (Real Time Protocol) stream to the client. Now, suppose that an erroneous frame is transmitted. If we were using a standard sequence, we should wait until the next I frame, which could be several seconds later.

45 SI frames 31 Figure 4.3: Sending the video stream. Figure 4.4: What happens without SI frames when an error appears. But, introducing the SI frame, when an error is found, the client sent to the server an RTCP with a warning. Figure 4.5: SI frames: the client finds the error and sends a warning. Once the server receives the RTCP (Real Time Control Protocol), we only have to wait until an SP frame. Then, the server, instead of the SP frame, sends an SI frame synchronized with the SP frame to continue transmitting correct frames for the rest of the sequence.

46 32 Application of SP and SI frames in wireless multimedia communication Figure 4.6: SI frames: an SI frame is generated. Even introducing the SI frames, as we have just seen, the server transmits some error pictures to the client, but, the number of erroneous frames could be much lower than the number if we use the standard sequence because we can introduce SP frames with more periodicity than I frames. So, the final sequence transmitted from the server to the client is the one that follows: Figure 4.7: SI frames: the error is corrected Simulation and results We have simulate a sequence with SI frames. As we have done with SP frames, we are going to analyze the dependency of the size and the quality of the frames with the SI rate. For these simulations, the GOP size is fixed in 50 and we have used the complete Foreman sequence (400 frames). We start with the I frames. Below there are the graphics of the size of the I frames for two fixed QP, 23 and 38, versus the SP rate.

47 SI frames 33 Figure 4.8: I frame size versus SI rate for two different QP, 23 and 38. As it has happened with the SP rate, the I size does not depend on the rate of the SI frames. There is no connexion between them, but as we saw on the I and P sequence file, the size depends on the random selected frame to be I-encoded. If we take a look to the quality we can conclude exactly the same: Figure 4.9: I frame quality versus SI rate for two different QP, 23 and 38. The results for P frames are quite similar with the ones obtained for the SP rate. There is a relation between the SI rate and the size of the P frames, the bigger is the SP rate, the lower is the number of bits of the P frames. The SI frames, as it happens with SP frames and their bitrate, do not depend on the rate which they are introduced in the encoding process.

48 34 Application of SP and SI frames in wireless multimedia communication Figure 4.10: P frame size versus SI rate for two different QP, 23 and 38 Finally, there is a graphic comparing the sizes of this three types of frames: Figure 4.11: Comparison graphic of the size of the three frame types. As we have already said in the section of disadvantages, the sizes of the SI frames are bigger even than the I frames. In this graphic there is a clear representation of the difference between the sizes of the frames.

49 Simulation Scenario 35 CHAPTER 5. SIMULATION SCENARIO Usually, the user profile of UMTS is 64, 128 and 384 Kbps, but the bandwidth for the real multimedia transmission is 44, 105 and 360 Kbps, because of the headers of the packets and the audio. We know that the bitrate is a function of SP and SI rate and QP, so we made several simulations trying to reach these demanded bitrates. In the case of sequence with only I and P frames the bitrate is function of the GOP and QP. The method used for doing this is finding the optimal pair of SP rate/qp. For us, the optimal one, is the one that has the best quality with the bitrate demanded as the threshold. We have done this study to the three types of sequences which are analyzed in this work: I & P frames sequences. I & P & SP frames sequences. I & P & SI frames sequences. For all the simulations, the GOP size is 50, except in I and P frames sequence where the GOP size is the variable. To see all the results of the simulations of this section see Appendix B Kbps Here, in this section we are going to find the best pairs for each type of sequence to achieve the bitrate demanded: 44 Kbps. Let s start with I and P frames sequence. We have defined this bitrate with different pairs of QP-GOP. We have fixed the QP number, and then changing the GOP value until we manage the appropriate bitrates. We have used the complete Foreman sequence (400 frames). The best pair of QP-GOP is the one which has the best quality without surpassing the threshold of 44 Kbps. So, in this case, the optimal result is the one achieved by the pair of QP 37 and GOP 40, because the bitrate does not exceed the value of 44 Kbps. The quality of this pair is db. Here, there is the table with the pairs which achieve, approximately, the bitrate demanded:

50 36 Application of SP and SI frames in wireless multimedia communication Let s see the video stream: QP GOP Bitrate [Kbps] Y-PSNR [db] Size [bits] ,99 29, ,06 29, ,30 29, ,04 29, ,11 29, ,80 28, ,77 28, ,38 28, ,78 28, ,30 27, Table 5.1: Results for 44 Kbps by I&P sequence. Figure 5.1: Videostream visualization of the pair selected, QP 37 and GOP 40. For the I, P and SP frames sequence the method is quite different. We have fixed the GOP size, to 50, and then we changed the QP value, while also changing the SP rate, until we managed the 44 Kbps. We have also used the complete Foreman sequence (400 frames). With the following pairs of SP rate - QP we have achieved the bitrate of 44 Kbps: QP SP rate Bitrate [Kbps] Y-PSNR [db] Size [bits] ,73 29, ,35 29, ,95 29, ,78 29, ,46 29, ,14 29, Table 5.2: Results for 44 Kbps by I&P&SP sequence. Like we have done with the I and P sequence, we are going to choose the best pair for the bitrate. To choose the better we have to take into account that the bitrate is not bigger than 44 Kbps, and the best quality is achieved. Taking a look to the table we can say that the best pair is the one with a SP rate of 15 and QP 37. The quality obtained is db. It is not the highest Bitrate, but the quality is much better. Let s see the visualization in figure 5.2. Finally, let s see the results for the sequence which introduces the SI frames. As we have done with the previous sequence, we have fixed the GOP size, and then changing the QP value until we manage the appropriate bitrates.

51 Simulation Scenario 37 Figure 5.2: Videostream visualization of the pair selected, SP rate of 15 and QP 37. Here there is the table with the QP - SI rate pairs which achieve the bitrate demanded: QP SP rate Bitrate [Kbps] Y-PSNR [db] Size [bits] ,80 29, ,54 29, ,28 29, ,37 29, ,60 29, Table 5.3: Results for 44 Kbps by I&P&SI sequence. Now, that we have got the table with the bitrates, we are going to select the best one with the best quality, without surpassing the limit of 44 Kbps. The pair selected is the one with SI rate 45 and QP 38, its quality is the best one and, even, the result of the bitrate is so close with the reference value. The quality is db. Despite of that the SI frames introduce a huge number of bits in the sequence, they are much bigger than the SP, the quality is quite the same, as we can see on the visualization of the video stream: Figure 5.3: Videostream visualization of the pair selected, SIrate 45 and QP Kbps Like we have done with the previous bitrate, 44 Kbps, we are going to find the best results to achieve the 105 Kbps. First, we have got the sequence formed only by I and P frames. We get these pairs of QP-GOP to achieve the demanded bitrate (Table 5.4). The best pairs are the ones with a QP of 30 and a GOP value of 68 and 69. Now, we must choose the one with less number of bits, so, the selected one is with a GOP of 68. We can see the visualization of the results in figure 5.4. Let s see the results of the sequence formed by I, P and SP frames. Next, there is the table of the pairs obtained with the simulations which we can get the bitrate of 105 Kbps.

52 38 Application of SP and SI frames in wireless multimedia communication QP GOP Bitrate [Kbps] Y-PSNR [db] Size [bits] ,88 34, ,19 34, ,65 34, ,78 34, ,42 34, ,7 34, ,7 33, ,19 32, ,83 31, Table 5.4: Results for 105 Kbps by I&P sequence. Figure 5.4: Videostream visualization of the pair selected, QP of 30 and a GOP 68. As it happens with the other bitrate, the result of the simulation must be less or equal to 105 Kbps, not higher. The pair selected is: SP rate 8 and QP 31. The quality achieved is db. The bitrate obtained is the lowest one, Kbps, but what we want is to get the best PSNR possible. QP SP rate Bitrate [Kbps] Y-PSNR [db] Size [bits] ,20 34, ,28 34, ,03 34, ,07 34, ,04 33, ,86 33, ,62 33, ,47 33, ,61 33, Table 5.5: Results for 105 Kbps by I&P&SP sequence. Below, there is the visualization of the video stream for these values:

53 Simulation Scenario 39 Figure 5.5: Videostream visualization of the pair selected,sp rate 8 and QP 31. Finally, the sequence formed by I, P and SI frames. Here there is the table with the best pairs: QP SP rate Bitrate [Kbps] Y-PSNR [db] Size [bits] ,06 32, ,37 32, ,20 32, ,94 32, ,18 32, Table 5.6: Results for 105 Kbps by I&P&SI sequence. Only taking a look to it, we can already say which is the best pair to achieve the bitrate demanded, 105 Kbps. The pair is: QP 38 and SI rate 24. The quality obtained is db. The bitrate is the lowest compared with the reference but it has the best quality. With this bitrate, the quality, if we compare to the one of the I & P & SP sequence, it is lower. In the sequence with SP frames we can obtain the demanded bitrate in a quality rate between db, so, in some cases there is a difference of 2 db. But the difference it is not so evident if we visualize the video stream: Figure 5.6: Videostream visualization of the pair selected, QP 38 and SI rate Kbps Now, as the last analysis we are going to find the best pairs to get the bitrate of 360 Kbps. The results that we have obtained to achieve the 360 Kbps with the different simulations by the I and P frames sequence are the ones in the table 5.7. To find the optimal result we have to consider that the maximum bitrate we have to achieve is 360Kbps. If there is some bitrate bigger it is not valid. Then, once we have got the correct bitrates we must take a look to the quality. There are only two pairs that they do not exceed the bitrate value.

54 40 Application of SP and SI frames in wireless multimedia communication QP GOP Bitrate [Kbps] Y-PSNR [db] Size [bits] ,88 39, ,68 39, ,55 39, ,93 38, ,66 38, ,13 38, ,00 38, ,72 38, Table 5.7: Results for 360 Kbps by I&P sequence. The one selected is the one with a QP of 23 and a GOP value of 20. Here there is the visualization: Figure 5.7: Videostream visualization of the pair selected QP of 23 and a GOP value of 20. Now, the sequence formed by I, P and SP frames. In the table below we have got the pairs which get this value: QP SP rate Bitrate [Kbps] Y-PSNR [db] Size [bits] ,55 38, ,86 38, ,98 38, ,38 38, ,53 38, ,32 38, ,86 39, Table 5.8: Results for 360 Kbps by I&P&SP sequence. We need a maximum bitrate of 360 Kbps. We are going to choose the pair of SP rate and QP, which achieve this bitrate, with the highest quality. So, the pair selected is: SP rate 45 and QP 23, its quality result is db. With these bitrate we can achieve a very good quality of the video stream:

55 Simulation Scenario 41 Figure 5.8: Videostream visualization of the pair selected, SP rate 45 and QP 23. As the last analysis, the results of I, P and SI frames. With this bitrate we are going to do the same we have done with the others. Let s see the table: QP SP rate Bitrate [Kbps] Y-PSNR [db] Size [bits] ,52 39, ,15 39, ,88 39, ,66 39, ,74 39, ,28 39, ,13 39, Table 5.9: Results for 360 Kbps by I&P&SI sequence. The selected pair the one with QP 23 and SI rate 49. The quality given by this pair is db. This result is the same result of the I, P and SP sequence. So the visualization has the same quality: Figure 5.9: Videostream visualization of the pair selected, QP 23 and SI rate Comparison of results If we compare the results of the three types of sequences, it is easy to see that if we introduce any other frame than the normal ones, I and P frames, to achieve the same bitrate the quality decreases. It is an evident conclusion because if we introduce SP frames, as it is discussed in chapter 3, without SP frames, the sequence achieve better quality. The same happens with the SI frames, as it is discussed in chapter 4.

56 42 Application of SP and SI frames in wireless multimedia communication

57 Switching simulations 43 CHAPTER 6. SWITCHING SIMULATIONS In this chapter we are going to describe the results of three qualities simulation. The first one is a High Quality simulation, then it is a Low Quality simulation, and the third is a switching simulation between high and low quality. The three simulations are performed using the whole Foreman sequence (400 frames), with a GOP of 50 and a SP rate of High and Low Quality Simulation In this section, there are the analysis of high and low quality results. For the High Quality simulation the QP parameter is initialized for I and P frames to 28. The QP parameter of the SP frame is 26. For the Low Quality Simulation the QP parameter is initialized to 51 for I and P frames. The QP parameter is 49. The reason of introducing this huge difference between QPs is just to make more evident the degradation of the video stream with the QP value. As it was expected, the number of bits of SP frames are less than the bits of I frames, but they are bigger than the ones of P frames on the high quality simulation: I size: P size: 4757,3 SP size: 7987,3 Table 6.1: Frame sizes for a high quality simulation. For the Low Quality simulations the results are not exactly the same, the size of the SP frames are lower than any other frame: I size: 1994 P size: 246,7331 SP size: 137,3 Table 6.2: Frame sizes for a low quality simulation. Because of this strange behavior, we have made a simulation with an intermediate QP of 35, and the SPQP is 33: I size: P size: 1657,1 SP size: 2227,9 Table 6.3: Frame sizes for a low quality simulation. As we can see, the results are quite similar to the ones of high quality. The number of bits of SP frames are bigger than the number of bits of P frames but smaller than the ones in I frames.

58 44 Application of SP and SI frames in wireless multimedia communication To make sure our results we have made another simulation with a QP of 50 for I and P frames and the SPQP 48. We have got the same result as the first simulations of low quality, the SP size is smaller than P size: I size: 2253 P size: 303,5322 SP size: 170,2857 Table 6.4: Frame sizes for a low quality simulation. After this results we can deduce that there should be somewhere where the size of P and SP frames are the same. So, we draw the graph with some points to know the crossing one. The first pair is the one corresponding to PQP= 39 and SPQP=37 and the last one PQP=50 and SPQP=48. Figure 6.1: Comparison graphic of P and SP frames. So, the crossing point is the pair number 4 which corresponds to PQP 42 and SPQP 40. Pair number PQP SPQP Table 6.5: Pair number definition.

59 Switching simulations Graphics Now, there is the representation of the first GOP of each simulation, the high and the low, in terms of quality. As we can observe on the graphics below, the quality of the SP frames is smaller than the quality of the other types of frames: Figure 6.2: Visualization on High and Low quality. Now, there is the comparison between a high quality simulation with the same characteristics but without introducing SP frames. Also for the low quality simulations. Figure 6.3: Quality comparison between sequences with and without SP frames, for high and low quality. Now, we are going to review the visualization of these sequences. If we visualize the

60 46 Application of SP and SI frames in wireless multimedia communication first sequence of each simulation, high and low quality, is so easy to know which one corresponds to the high and which one corresponds to the low quality. Figure 6.4: Visualization of the frames, 1st frame of the streaming. When we visualize the 8th frame, where the SP frame appears because of the SP rate, we observe the following: Figure 6.5: Visualization of the frames, 8th frame of the streaming. While we are watching the video stream we can not appreciate when an SP frame appears. It makes no difference in the video stream if we see an SP frame, a P frame or an I frame. The whole video stream is visualized with the same perceptive quality Switching simulation In this part we are going to analyze a simulation done with the switching flag activated. In this simulation the SP rate and the switching rate are synchronized, both of them at 8. It make no sense to simulate without synchronizing these two rates because the switching has to be made at SP frames, and with a GOP of 50 the switching will be produced at a P frame, which will lead to noticeable impairments. First, let s see the mean of number of bits for each frame. We start with the synchronized simulation. I size: P size: 2461,1 SP size: Table 6.6: Frame sizes for a switching simulation. In this case, the size of SP frames is bigger than the size I frames. The reason is that the size of SP frames corresponds to the addition of primary and secondary SP frames. In the text visualization of the sequence is impossible to distinguish between the two types of SP frames:

61 Switching simulations 47 Now, we are going to observe the graphic: Figure 6.6: Text visualization of the sequence. Figure 6.7: Quality of switching simulation. In the synchronized switching, as it was expected, when in the sequences appears the SP frame, the quality of the video stream changes from low quality to high, or in the contrary, from high to low quality. The changes between the qualities are produced at the SP frame because the SP frame is synchronized at the same rate of the switching period, as it was told, both of them each 8th frame. The position of these frames in the graphic it is one position more than in the real sequence because in the simulation the first position of the sequence is number zero, and in the graphic the first is number one.

62 48 Application of SP and SI frames in wireless multimedia communication And now, we are going to compare the switching simulation the results of the high and low simulations without SP frames. Until the first SP frame, the values of the qualities are the same for the low quality and the switching simulation, which starts in low quality. From here the qualities values of the switching simulation is lower than the original ones, the values of the simulations without SP frames. Here is the graphic: Figure 6.8: Comparison of the Qualities. Now, we are going to observe some frames of the video streaming. Figure 6.9: Frames of the switching video streaming. The picture marked as 1st corresponds to the first frame of the sequence, the IDR frame. We cannot see the image clearly because we are in low quality, but if we observe the next picture, the 8th, corresponding to an SP frame, the quality is already high. The last picture, marked as 16th, which corresponds to the 16th frame of the sequence and the 2nd SP, is in low quality.

63 Code improvements 49 CHAPTER 7. CODE IMPROVEMENTS The purpose of modifying the code of the encoder is in order to let the switch happen in defined points, rather than in each switching period. Our intention is to introduce the number of the frames in which we want to make the switching, without a periodicity of the SP frames, with a text file. The text file contains the position of the switching frames Original code In the original code, the SP rate is market by the variable SPPicturePeriodicity. Figure 7.1: encoder.cfg: SP rate variable. And the variable in which we can change from one quality to another is ChangeQPStart. Figure 7.2: encoder.cfg: change QP variable. With this variable we mark the frame in which we want to change the Quantization Parameter. The variables marked as ChangeQPI and ChangeQPP are the variables of the new QP. For example, if the SPPicturePeriodicity variable is 8 means that every 8 frames an SP frame is introduced. And, if the ChangeQPstart is 10 means that every 10 frames, the QP of the frames changes, so, the quality of the sequence changes. Both variables are initialized in encoder.cfg. As we have already seen in the section 6.2 Switching simulation, it is important that the SPPicturePeriodicity and the ChangeQPStart must be synchronized. In the text code, the main function to set the type of the frame is the next one:

64 50 Application of SP and SI frames in wireless multimedia communication Figure 7.3: Encoder code: SetImgType function. Finally, let s see the text file of the simulation: Figure 7.4: Text file of a switching simulation. We can see that every 8 frames an SP frame is introduced and the switching is proceed Modified code As we have already said in the introduction of this chapter, our purpose is to achieve the switching without enabling the SPPicturePeriodicity and the ChangeQPStart.

65 Code improvements 51 We focused our studies on the SetImgType function, introduced on the Original Code section. In order to make everything more easy without touching the code, to introduce the position of the switching points, we use a text file. Figure 7.5: Example of a file with switching point. About the SetImgType function we have changed its definition. In the original code an input parameter is not defined, while in our definition the function is: SetImgType(int aux), where aux is the variable used to read every row of the text file. Every time a switching point takes the same value as a frame position an SP is introduced and the variable aux is incremented in one point (if(img-type== SP SLICE)aux++;). The first part of the function is exactly the same as the original one. Then, we open the text file and read it. While the file is being read, the switching points are saved in the local vector frame[400]. The extension of the vector is 400 because is the maximum number of frames that we can get (Foreman sequence is formed by 400 frames). If the frame number is the same as the value saved at the frame vector we introduce an SP frame and the index of the secondary SP frames is enabled. Then, we update the input-qp2start with the current frame number if the qp2start is 0, so it changes from the first quality to the second one. After this, we disable the secondary SP frames indicator. The third one is only used in case we want to change from the secondary quality to the first one. The code of this function is shown in figure 7.7 at the end of this section.

66 52 Application of SP and SI frames in wireless multimedia communication Now, let s see the text file of the simulation. Figure 7.6: Text file of the switching simulation with the modified code. As it is defined in the text file of the switching points, we change from one quality to the other in the frames: 3, 5, 7, 14 and 25. So, there is no primary SP period. We only introduce SP frames in case we wan to make a switching. If we compare the bits per picture between the frames obtained by the original code and these one they are almost the same.

67 Code improvements 53 Figure 7.7: Function SetImgType modified.

68 54 Application of SP and SI frames in wireless multimedia communication

69 Conclusions 55 CHAPTER 8. CONCLUSIONS In this work we have made a study of the SP and SI frames introduced by the extended profile oh the standard H.264. The purpose was to learn the behavior of these two types of frames and implementing the existing code in order to make it more efficient. As a main conclusion we can say that the size of any frame type depends on the QP. Anyway, the SP frames depend also in its own rate. The bigger is the SP rate, the lower is the number of bits of the SP frames. The same happens with the SI frames and the SI rate. The other two type of frames, the I and P, do not depend on any of these two rates. We can conclude that SP frames are better than I frames in a switching scenario. The reason is that the SP switching simulation gives a more constant level of the bitrate values than I switching, which introduces very high peaks. If we compare the SP and the SI frames,the SP frames are better in switching applications because they are smaller than Si frames, but, in random access and back and fast forward SI frames are better because they are predicted using intra prediction, so, they do not need any reference frame to be predicted. In the case of High and Low quality simulations, without using the switching, we can say that the insertion of SP frames in the sequences reduces the overall quality of the video stream. In the case of the switching simulations, we can say that it is better to simulate the switching sequence coordinating the SP rate and the switching rate in order to prevent strange behavior in the simulation, and, of course it is much more effective to make the switching in an SP frame than in any other frame because we are able to know how it is going to be the behavior of the stream. After discussing all the results, we have modified the code in order to work with the basic streaming sequence, formed by I and P frames, but introducing the SP frames when they are needed. Finally, we have to say that to profit the introduction of this two new types of frames we have to take into account where and when we are going to use them, and what is more important, to introduce a better quality or to decrease the number of bits trying to maintain the same quality.

70 56 Application of SP and SI frames in wireless multimedia communication

71 BIBLIOGRAPHY 57 BIBLIOGRAPHY [1] Klaus von Klitzing. The quantised hall effect. RMP, 58:519, [2] Iain E.G. Rchardson. H.264 / mpeg-4 part 10 white paper - h.264 overview. page 1, [3] Thomas Wiegand, Gary Sulliivan, and Ajay Luthra. Draft itu-t recommendation and final draft international standard of joint video specification (itu-t rec. h.264 iso/iec avc). Joint Video Team, [4] Detlev Marpe, Thomas Wiegand, and Gary Sullivan. The h.264/mpeg4 advanced video coding standard and its applications. IEEE Communications Magazine, page 134, [5] J Ribas-Corbera, P.A. Chou, and S. Regunathan. A generalized hypothetical reference decoder for h.264/avc. IEEE Trans. Circuits Syst. Video Technol., 13:674, [6] Thomas Wiegand, Sullivan Gary, Gisle Bjøntegaard, and Ajay Luthra. Overview of the h.264/avc video coding standard. IEEE Tansactions on Circuits and Systems for Video Technology, 13:2, [7] Gary Sullivan and Thomas Wiegand. Video compression - from concepts to the h.264/avc standard. ieeexplore.ieee.org, page 1, [8] Ian E. G. Richardson. H.264 / mpeg-4 part 10 white paper: Prediction of inter macroblocks in p-slices. page 3, [9] Martin Fiedler. Seminar paper: Implementation of a basic h.264/avc decoder. page 6, [10] Marta Karczewicz and Ragip Kurceren. The sp- and si-frames desing for h.264/avc. IEEE Transactions on circuits and systems for video technology, 13:8, [11] Ian E. G. Richardson. H.264 and mpeg-4 video compression: Video coding for nextgeneration multimedia. page 306, 2003.

72

73 APPENDIXES

74

75 61 APPENDIX A. A.1. Abbrebiations ISO: International Organization for Standardization ITU-T: International Telecommunication Union - Telecommunication Standardization JVT: Joint Video Team MPEG: Moving Picture Experts Group VCEG: Video Coding Experts Group AVC: Advanced Video Coding xdsl: Digital Subscriber Line UMTS: Universal Mobile Telecommunications System DVB-T: Digital Video Broadcasting HDTV: High Definition TV ISDN: Integrated Services Digital Network QCIF: Quarter Common Interchange Format NAL: Network Abstraction Layer VCL: Video Coding Layer FMO: Flexible Macroblock Ordering CAVLC: Context Adaptative Variable Length Coding CABAC: Coding Adaptative Binary Arithmetic Coding GOP: Group Of Pictures QP: Quantization Parameter RTP: Real Time Protocol RTCP: Real Time Control Protocol PSNR: Peak to Signal-to-Noise Ratio A.2. Table of results for the simulation scenario The results of all the simulations we have made for the chapter Simulation Scenario.

76 Figure A.1: Results for I & P sequences.

77 Figure A.2: Results for I & P & SP sequences.

78 Figure A.3: Results for I & P & SI sequences.