Extraction of Motion Vectors from an MPEG Stream

Transcription

1 Extraction of Motion Vectors from an MPEG Stream Technical report 1999 JOSEPH GILVARRY School of Electronic Engineering Dublin City University

2 Abstract In 1997, a project was started to capture, compress, store, and index up to 24 hours of digital TV broadcasts. The work in this report is to help implement this. In the first chapter of this report, the overall project is introduced and also the motivation behind particular focus of work. The second chapter deals with the theory behind digital video compression. In the third chapter a report is given on how the program to extract the motion vectors from the MPEG stream was developed. It also reports on the further development of the program, so the motion from frame to frame can be calculated. Chapter four explains why knowledge of the motion vectors is not sufficient information to calculate the motion from frame to frame. It gives the extra information that is needed and how all the information is used to calculate the motion from frame to frame. ii

3 Table Of Contents Abstract ii Table Of Contents iii Table of Figures iv Chapter Introduction Chapter Digital Video Compression The MPEG-1 bit stream Description of a frame Bit stream order and display order of frames Description of a macroblock Types of macroblock present in a frame Types of macroblock in an I frame Types of macroblock in a P frame Types of macroblock in a B frame Motion estimation and compensation Encoding the motion vectors Summary Chapter Extraction of the motion vectors Choosing a decoder The Berkeley Decoder The Java Decoder Description of the source code Storage of the Motion Vectors Reordering the bit stream order to the display order Storing the motion vectors Operation of program Alterations made to the decoder Summary Chapter Finding the motion from frame to frame Considerations that have to be taken into account - Frame level Considerations that have to be taken into account - macroblock level Summary Conclusion References Appendix A iii

4 Table of Figures Figure 2.1 The layered structure of the MPEG bit stream...4 Figure 2. 2 P frames use only forward prediction....5 Figure 2.3 B frames use both forward and backward prediction..6 Figure 2.4 A single frame divided up into slices Figure 2.5 Only one set of chrominance components is needed for every four luminance components..7 Figure 2.6 Structure of a macroblock, and the blocks numbering convention.7 Figure 2.7 A forward predicted motion vector 10 Figure 3.1 Converting from bit stream order to Display order 16 Figure 3.2 Diagram of where the motion vectors for the different frames are stored.18 Figure3.3 Flow chart of the operational program 19 Figure 4.1 Motion vectors associated with a moving picture..21 Figure 4.2 Realistic version of vectors associated with a moving picture 23 iv

5 Chapter 1 1. Introduction With the arrival of Digital TV in America and Great Britain recently it is only a short time before its use will be standard. Recent years have also brought huge advances in: Networking - High bandwidth networks not only in the workplace, but reaching many homes also; Data storage - Today we talk only in Gigabytes; Video compression - Modern techniques allow compression rates of up to one in fifty (this topic is discussed in detail in Chapter 2) The combination of these developments will bring the wide spread usage of digital video over the next few years. Following the launch of this new technology will be the launch of many new services, we could see the introduction of the local video server instead of the local video store where connected residents can select a video from a huge multimedia server. A recording of all television broadcasts for the past week may be stored, allowing subscribers to catch up on any missed viewing. Searching through such large archives will see the need of a navigation tool. There is an on going project in DCU at the moment to develop such a tool of which this project is only a part[1]. When complete, the tool will allow the user to pick a category to search through (sport, drama, action, soap). Clicking on a category will display a list of key frames, each frame representing a program. Clicking on one of these frames will display another list of key frames and using this hierarchical approach, the user can narrow the search down to a single shot of video. One of the challenges of the project is to choose a frame to best represent a clip of film. It has been found that the frame after a sequence of frames with a lot of action is sometimes a good representation for that shot. This is one area where the motion vectors may come in useful. To allow navigation, the material has first to be broken up into elements. For video these elements are shots and scenes. A shot is defined as the continuous recording of a single camera, a scene is made up of multiple shots, while a television broadcast consists of a collection of scenes. For studio broadcasts (take for example the news) it is fairly easy to break the program up as the boundaries between shots are hard. However most television programs and films use special techniques to soften the boundaries, this makes them less detectable. There are four different types of boundaries between shots: A cut. This is a hard boundary and occurs when there is a complete change of picture over two consecutive frames. A fade. There are two types of fade, a fade out and a fade in. A fade out occurs when the picture gradually fades to a dot or black screen, while a fade in occurs when the 1

6 picture is gradually displayed from a black screen, both these effects occur over a few frames. A dissolve. This is the simultaneous occurrence of a fade out and a fade in, the two pictures are superimposed on each other. A wipe. This effect is like a virtual line going across the screen clearing one picture as it brings in another, again this occurs over a few frames. There are a lot of techniques (Pixel based difference method, Colour histogram method, Detection of macroblocks and Edge detection[5]) which can reliably detect a cut. However, only Edge detection is any way effective in detecting Fades, Dissolves and Wipes. There is another ongoing project in DCU at the moment that uses edge detection to find shot boundaries. The program takes two consecutive frames, uses special techniques to leave just a black & white outline of any objects in the frames, and then compares the two outlines. If there are a lot differences in them, it concludes a shot cut has occurred. One of the flaws of this method is that it only allows for relatively small movements of the objects from frame to frame. If something large suddenly moves across the screen, it may interpret this as a cut. To illustrate where this may happen, take the example where a journalist is giving a TV report from outside some building, and suddenly a bus goes by in the background. The inclusion of the bus in the frame could confuse the program into thinking a cut has occurred. This is another case where motion vectors could come in useful, as, associated with a lot of movement in a frame is a lot of motion vectors. These motion vectors can be used to compensate for the movement of the bus. Here is a history of the events that lead up to the creation of this project. Develop a system to capture, compress, store and index up to 24 hours of TV broadcasts in digital format. Eight hour recording of television broadcasts in MPEG1 format. This eight hours was broken into twenty minute segments for easier handling. A baseline was created by manually going through the entire recording and labelling where a cut, fade, dissolve and wipe occurred. A note of the frame number and the time the boundary occurred was taken. The results of any program written to find these boundaries can be easily compared to the baseline in order to determine its accuracy. A program was written using edge detection to find the shot boundaries but it was found that a lot of motion in a frame caused the program to falsely detect a cut. The use motion vectors to compensate for the motion should rectify the result. It is hoped that the motion vectors can also be used to enhance the programs performance in detecting fades, dissolves and wipes. Another area where the motion vectors may be used is in the choice of key frame for a shot, [choose a frame after a lot of action?] 2

7 Chapter 2 2. Digital Video Compression In this chapter the techniques used to compress digital video are discussed with a special emphasis on the factors that that need to be considered when finding the motion from one frame to another. Digital video has the advantages of high quality sound and pictures, but its disadvantage is it can t be easily transmitted or stored; it needs to be transmitted at a minimum of 100Mbps which is impractical for to-day s infrastructure. To combat this problem, a lot of work was put into video compression. In 1988 the International Standards Organisation (ISO) set up the Moving Picture Expert Group (MPEG) to standardise this compression. Its first standard, IS (known as MPEG-1) came in five parts: 1. System ( ). This was concerned with the multiplexing and synchronisation of the multiple audio video streams. 2. Video ( ). This dealt with the encoding of the video stream. 3. Audio ( ). This part dealt with the encoding of the audio stream. 4. Compliance testing ( ) 5. Software for MPEG-1 coding ( ) Parts 1, 2 and 3 were approved in November 1992 and parts 4 and 5 in November This project is only concerned with the second part, the Video encoding. A summary of the standard is given in Table 2.1 Table 2.1 Summary of the constrained parameters of MPEG-1[2] Horizontal picture size less than or equal to 768 pels Vertical picture size less than or equal to 576 lines Picture area less than or equal to 396 macroblocks Pel rate less than or equal to 396x25 macroblocks per second Picture rate less than or equal to 30 Hz Motion vector range -64 to 63.5 pels (half pel precision) Input buffer size less than or equal to Kb Bitrate less than or equal to Mbps (constant bitrate) The aim of MPEG-1 was to achieve coding of full motion video at a rate of around 1.5Mbps, this rate was chosen as it would be suitable for transmission over any modern network and also it is nearly the same rate as a CD (1.412Mbps). To allow for greater flexibility and ingenuity in compression techniques, MPEG-1 does not specify a standard for the encoding of video. What it does specify is a standard for the decoding process and the video bit stream. 2.1 The MPEG-1 bit stream The bit stream is in a layered format as shown in Figure 2.1, a brief description of the function of each layer is giver in Table

8 Sequence layer GOP GOP GOP GOP GOP GOP GOP GOP GOP Frame Frame Frame Frame Frame Frame Frame Frame Frame Slice Slice Slice Slice Slice Slice Slice Slice Slice Macro Macro Macro Macro Macro Macro Macro Macro Macro Block Block Block Block Block Block Block Block Block Y0 Y1 Y2 Y3 Cb Cr Block Block Block Block Block Block Figure 2.1 The layered structure of the MPEG bit stream Table 2.2 Function of each layer of the bit stream[2] Layer Sequence layer Group of pictures (GOP) Picture Slice Macroblock Block Function One or more groups of pictures Random access into the sequence Primary coding unit Resynchronisation unit Motion compensation unit DCT unit 4

9 Firstly, each layer is briefly described, and then a more thorough description of the units in the layers is given. 1. The sequence layer contains general information about the video: the vertical and horizontal size of the frames, height/width ratio, picture rate, VBV Buffer size, Intra and non-intra quantizer default tables. 2. Group of pictures (GOP) layer: Pictures are grouped together to support greater flexibility and efficiency in the encoder/decoder [2]. 3. The frame layer (picture layer) is the primary coding unit, it contains information regarding the picture s position in the display order (pictures do not come in the same order as they are displayed), what type of picture it is (Intra, Predicted or Bi-directionally predicted) and the precision and range of any motion vectors present in the frame. 4. The Slice layer is important in the handlingof errors. If the decoder comes across a corrupted slice, it skips it and goes straight to the start of the next slice. 5. The Macroblock layer is the basic coding unit It is within this unit that the motion vectors are stored. Each macroblock may have one or associated with it. 6. The Block layer is the smallest coding unit and it contains information on the coefficients of the pixels Description of a frame As mentioned above there are three types of picture/frame: Intra (I-type). These frames are encoded using only information from itself. Predicted (P-type). These frames are encoded using a past I or P frame as a reference, as illustrated in figure 2.2. This is known as forward prediction. I B B P B B B P Figure 2. 2 P frames use only forward prediction 5

10 Bi-directionally predicted (B-type). These frames are encoded using a past (forward predicted) and a future (backward predicted) I or P frame as a reference, as illustrated in figure 2.3 (a B-type frame is never used as a reference). I B B P B B B P Figure 2.3 B frames use both forward and backward prediction Each frame is divided up into arbitrary sized slices. A slice may contain just one macroblock or all the macroblocks in the frame. As shown in Figure 2.4, a slice is not confined to a single row. Slice 1 Slice 2 Slice 3 Slice 4 Slice 5 Slice 6 Slice 7 Slice 8 Slice 9 Figure 2. 4 A single frame divided up into slices Bit stream order and display order of frames A typical sequence of frames in the display order is shown below. I B B B P B B B B P I B B B I However, this is not the order in which they are transmitted! The P frame numbered five is needed for the decoding of B frames two, three, and four. Therefore five has to be decoded before two, three and four and hence transmitted before them. Similarly P9 is transmitted before B6, B7, B8 and B9 also I15 is transmitted before B12, B13 and B14. The bit stream order is shown below. I P B B B P B B B B I I B B B

11 2.1.3 Description of a macroblock The macroblock is the basic unit in the MPEG stream, it is an area of 16 pixels by 16 pixels and it is at this stage that the first compression takes place. Each pixel has a luminance (Y) component and two chrominance (Cb and Cr) components associated with it. The human eye is much more sensitive to luminance than it is to chrominance. Therefore the luminance components must be encoded at full resolution while the chrominance components can be encoded at quarter resolution without any noticeable loss. This gives compression of one in two already. Figure 2.5 shows this compression. Y Y Y Y Y Y Y Y Cb Cr Cb Cr Cb Cr Cb Cr Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Cb Cr Cb Cr Cb Cr Cb Cr Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Cb Cr Cb Cr Cb Cr Cb Cr Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Cb Cr Cb Cr Cb Cr Cb Cr Y Y Y Y Y Y Y Y Figure 2.5 Only one set of chrominance components is needed for every four luminance components. A block is an 8 by 8 pixel area and is the smallest unit in the MPEG stream. It contains the Discrete Cosine Transform (DCT) coefficients of the luminance and chrominance components [3]. Six blocks are needed to make up a macroblock (16 pixels by 16 pixels), four for the luminance components but only one for each of the two chrominance components due to their compression. Figure 2.6 shows the blocks of a macroblock and their numbering convention Y Cb Cr Figure 2.6 Structure of a macroblock, and the blocks numbering convention 7

12 2.2 Types of macroblock present in a frame In a single frame there may be many different types of macroblock (MB). Tables 2.3, 2.4 and 2.5 show the different types of macroblock that can be present in I, P and B frames respectively Types of macroblock in an I frame In an I frame there are only two types of macroblock, Intra-d uses the default quantizer scale while Intra-q uses a scale defined by the buffer status[2] Types of macroblock in a P frame A P frame uses motion estimation and compensation to reduce the amount of information needed to play the video, this process is described later in the chapter. There are eight different types of macroblock in a P frame, but for the purpose of this project they can be divided up into three categories. 1. Intra. There are no motion vectors present. These macroblocks don t use any reference frame, and are encoded using only information from itself. 2. Predicted. These macroblocks have motion vectors present. 3. Skipped. These macroblocks are the exact same as the macroblock in the previous frame. Table 2.3 Macroblock types in an I frame[2] Type VLC code MB quant Intra-d 1 0 Intra-q 01 1 Table 2.4 Macroblock types in a P frame[2] Type VLC Intra M F Coded pattern Quant pred-mc pred-c 01 1 pred-m Intra-d Pred-mcq Pred-cq Intra-q Skipped 8

13 Table 2.5 Macroblock types in a B frame[2] Type VLC Intra M F M B Coded pattern Quant pred-i pred-ic pred-b pred-bc pred-f pred-fc intra-d pred-icq pred-fcq pred-bcq intra-q skipped Here is the meaning of the abbreviations used in the tables above: VLC - variable length code M F - motion forward M B - motion backward pred - predictive m - motion compensated c - at least one block in the macroblock is coded and transmitted d - default quantizer is used q - quantizer scale is changed i - interpolated. This is a combination of forward prediction and backward prediction. b - backward prediction f - forward prediction Types of macroblock in a B frame A B-frame uses two reference frames for prediction and can have twelve different types of macroblock. This leaves it the most complex, but it gives the highest compression rate. For the purpose of this project, they can be categorised in five groups: 1. Forward predicted. Macroblock in encoded using only a past I or P frame. 2. Backward predicted. Macroblock in encoded using only a future I or P frame. 3. Forward and Backward predicted (Interpolated). macroblock is encoded using both a past and future frame as a reference. The two macroblocks are interpolated to form the predicted macroblock. 4. Intra. No reference frame is used. Macroblock is encoded using information from itself. 5. Skipped. Macroblock is the same as the one in the previous frame 2.3 Motion estimation and compensation MPEG achieves it its high compression rate by the use of motion estimation and compensation. MPEG takes advantage of the fact that from frame to frame there is very little change in the picture (usually only small movements). For this reason macroblock size areas can be 9

14 compared between frames, and instead of encoding the whole macroblock again the difference between the two macroblocks is encoded and transmitted. Figure 2.7 demonstrates how forward motion compensation is achieved (backward compensation is done in the same way except a future frame in the display order is used as the reference frame.) I or P Reference Frame P or B Frame y x Search area y x Figure 2.7 A forward predicted motion vector Macroblock x is the macroblock we wish to encode, macroblock y is its counterpart in the reference frame. A search is done around y to find the best match for x. This search is limited to a finite area, and even if there is a perfectly matching macroblock outside the search area, it will not be used. The displacement between the two macroblocks gives the motion vector associated with x. There are many search algorithms to find the best matching macroblock. A full search gives the best match but is computationally expensive. Alternatives to this are the Logarithmic search, One-at-a-time search, Three-step search and the Hierarchical search [3]. The choice of search is decided by the encoder, with the usual trade-off between time and accuracy. 10

15 2.3.1 Encoding the motion vectors Once the motion vector is found it has to be encoded for transmission. The first step in the encoding process is to find the differential motion vectors (DMV). In a lot of situations, (e.g. a pan) all motion vectors will be nearly the same. Therefore subtracting the motion vector for a macroblock from the previous motion vector in the slice will reduce a lot of the vectors to zero. Note this differential vector is reset to zero if an I-type macroblock is encountered, and also at the end of a slice. The second step is to make sure all differential vectors are within a permitted range. This range is defined by forward_f_code/backward_f_code and is given in table 2.6. If the vectors are outside this range, a modulus is added/subtracted. Finally the differential vectors are variable length coded and transmitted. The variable length codes are given in table 2.7. To illustrate an example suppose the vectors (full pel precision) in a slice are: All vectors lie in the range -32 to 31, therefore a forward_f_code of 2 is used. The differential vectors are: Adding/subtracting the modulus (64 in this case) to any values outside the range gives: The variable length codes for these values are[2]: Value VLC Value VLC The code needed to decode these VLC values is given in MPEG standard Table 2.6 Range of motion vectors and their modulus[2] Forward-f-code or Backward-f-code Half pel precision Full pel precision Modulus 1-8 to to to to to to to to to to to to to to

16 Table 2.7 VLC for the differential motion vectors (DMV) [2] VLC code DMV VLC code DMV Summary In this chapter the MPEG standard is introduced and described, the layered structure of the bit stream was explained and the concept of a motion vector illustrated. The difference between the bit stream order and the display order of the frames was explained and illustrated. The different types of macroblock present in I, P and B frames was given. To find the motion from frame to frame all of these factors have to be considered. 12

17 Chapter 3 3 Extraction of the motion vectors This chapter discusses the steps taken to extract the motion vectors from the MPEG stream. It also describes the alterations made to the source code to allow the calculation of the motion from frame to frame. 3.1 Choosing a decoder The first step was to choose an MPEG1 decoder. The decoder is used to extract and decode the motion vectors. A search of suitable decoders was undertaken, and this resulted in two candidates; the Berkeley decoder and a Java decoder The Berkeley Decoder The Berkeley decoder can be found at: Initially it was thought that this would be the best decoder to use as it was written in C. Speed is an important factor in this project due to the size of the files that have to be processed, and C has a superior processing time to Java. However the source code proved impossible to read. It was not commented and there are pointers pointing to pointers pointing to??? The Java Decoder The Java decoder can be found at: The Java program s speed disadvantage over C was compensated for by its well structured and documented style. There are two versions of the decoder available. The default version stores all the frames as it decodes them. This version is impractical to use, as all the memory is used up after only a few frames are decoded. It has to be able to decode thirty thousand frames! By making a small alteration to the source code, we get the Just-in-time version. This version only stores seven or eight frames at a time, which makes it suitable for our purpose. 3.2 Description of the source code The motion vectors are decoded by the two classes MPEG_video and motion_data. MPEG_video is the main class in the program. It takes in the bit stream and decodes it. A skeleton of the program is given below. 13

18 public class MPEG_video implements Runnable { MPEG_video () { public void run() { mpeg_stream.next_start_code(); do { Parse_sequence_header(); do { Parse_group_of_pictures(); private void Parse_sequence_header() { private void Parse_group_of_pictures() { do { Parse_picture(); private void Parse_picture () { do { Parse_slice(); private void Parse_slice() { do { Parse_macroblock(); private void Parse_Block() { It is clear how the program first takes in the highest level layer, and parses it. The program then extracts the information in a section of that layer, and moves down to the next level. This process is repeated for all the layers. The motion vector information is contained in the macroblock layer. Once this information is known, it is passed to a method in motion_data called compute_motion_vector. To decode the motion vectors, compute_motion_vector uses another method in motion_data called motion_displacement. The code in these methods is given in Appendix A. 14

19 The two components of the vector are right_x and down_x. The conventional direction used for the components is right and down, minus components represent left and up respectively. For this project it was decided to use half pixel precision for the vectors, (recon_right_x and recon_down_x). The vectors may not be pointing to a particular pixel, but it is the true vector for that macroblock. The fact that the vector is not pointing at a pixel should not be an issue. If the motion vectors are used for the selection of the key frame in a shot, there is no need for the vector to be pointing at a pixel. If the vectors are used to compensate for movement in a frame, Edge Detection (the process that will be using the vectors) blows up an area around around each pixel when comparing the two frames[5]. By simply halving the extracted (half pixel precision) vector, and using it for any motion compensation, the need for the extra calculations to get the vector pointing to a pixel will be eliminated. This will enhance the speed of the program. Any inaccuracies in the motion vector will be compensated for by the Edge detection s explosion. Besides Edge detection is not an exact science. 3.3 Storage of the Motion Vectors The motion vectors have to be stored in an order that will allow the motion from frame to frame to be calculated. First, the process of reording the bit stream order to the display order is discused. This is followed by a description of how selective vector storage allows this reordering Reordering the bit stream order to the display order As described in Chapter 2 the frames do not come into the decoder in the same order as they are displayed. To reorder the frames to the display order the following procedure is used (see Figure3): If an I or P frame (lets call it 1 ) comes in it is put in a temporary storage future. I and P frames always come into the decoder before the B frames that reference them. 1 is left in future until another I or P frame ( 5 ) comes in. The arrival of 5 indicates it is 1 s turn in the display order. 1 is taken out of future, put in the display order. 5 is put in future until another I of P frame arrives. All B frames are immediately put in the display order. At the end whatever frame is left in future is taken out and put in the display order. A typical bit stream is shown in figure 3.1, the display order number of each frame is also given. Note this process doesn t use the display order number. It is given to clarify what is happening. 15

20 Bit stream Order Display Order future 1I 1I 5P 1I 5P 2B 2B 5P 3B 3B 5P 4B 4B 5P 10P 5P 10P 6B 6B 10P 7B 7B 10P 8B 8B 10P 9B 9B 10P 11I 10P 11I 15P 11I 15P 12B 12B 15P 13B 13B 15P 14B 14B 15P 15P Figure 3.1 Converting from bit stream order to Display order Storing the motion vectors For ease of handling, it was decided that the motion vectors should be stored in twodimensional arrays. The size of the array corresponds to the frame size (in macroblocks). The position of the entry in the array corresponds to the macroblock s position in the frame. There is a separate array for the two components of the vector, one for the right component and one for the left component. To allow the storage of all the vectors that may be present in a frame, four arrays have to be created. Two arrays are needed for the storage of the forward predicted vectors, and two for backward predicted vectors. To find the motion from one frame to another, a record of the motion vectors in the previous frame has to be kept. This means four more arrays have to be created. Finally the motion vectors in a P frame have to be stored until it is the P frames turn in the display order. As a P frame can only have forward predicted vectors, only two arrays need to be created. The names of all the arrays used in this project are given below: 16

21 Array name Function of array. futureright Store the motion vectors in a P frame until futuredown it is the P frames turn in the display order. presentforwardright presentforwarddown presentbackwardright presentbackwarddown pastforwardright pastforwarddown pastbackwardright pastbackwarddown Stores the motion vectors of the present frame in the display order Stores the motion vectors of the previous frame in the display order Operation of program If an I frame comes into the decoder, all the vectors in future are reset to zero (after the values that were in it are taken out and put in present), as an I frame has no motion vectors. If a P frame comes in all its vectors have to be stored in future (after the values that were in it are taken out and put in present). The problem with the program is that compute_motion_vector (the method that decodes the motion vector) doesn t know what type of frame is in the decoder, or even what type of predicted vector it has to decode. It could be a forward predicted vector in a P or B frame, or a backward predicted vector in a B frame. To over come this problem, an extra variable, Pic_Type, is also passed. Pic_Type determines what type of frame is present in the decoder, Pic_Type = 2 means it is a P frame, and the vectors are put in future. If a B frame comes in, all its vectors have to be stored in present. However present has two types of vector; presentforward and presentbackward. If it is forward predicted vectors that are to be calculated compute_motion_vector is called from the same place as it was for the P frame. This time Pic_Type = 3 (for a B frame), and the vectors are stored in presentforward. If backward predicted vectors are to be calculated, compute_motion_data is called from a different place. The arbitrary value four is passed, to indicate the vectors are to be put in presentbackward. A diagram of where the vectors are stored is given in Figure

22 Frame comes in Is it an I, P or B frame? I B P Reset future Forward or backward prediction All vectors put in future forward backward All vectors put in presentforward All vectors put in presentbackward Figure 3.2 Diagram of where the motion vectors for the different frames are stored Alterations made to the decoder The processes of inputting the motion vectors into the correct arrays and reordering the frames into the display order were incorperted into the decoder. The end result was that the motion vectors for the present frame in display order are in presentforward and presentbackward. While the motion vectors for the previous frame in the display order are in pastforward and pastbackward. A flow chart of the program is given in Figure 3.3. A skeleton of the two files MPEG_video and motion_data, (after the changes were made to then) is given in Appendix A. Also in Appendix A is the new class, Array, that had to be created. 18

23 Frame comes in Put all the vectors from present into past Reset present Is frame I or P Type? Yes No Take Vectors out of future and put in present Reset future Is frame I or P type? I P B No vectors, all vectors in future remain zero All vectors put in future All vectors put in present Figure3.3 Flow chart of the operational program 19

24 To bring in theses changes it was decided it would be best to creat a new class. This was done for a few reasons: 1. MPEG_video.java is a large file. It seemed unsuitable to make it any bigger. 2. Even though MPEG_video is very large, there is a logical flow to it. The bit stream is decoded from top to bottom, introducing new code would only disturb this natural flow and leave the program difficult to read. 3. At some time in the future the MPEG2 standard may be used instead of the MPEG1 standard that is being used at the moment. However most of the code developed for this project may still be relevent. Having the code developed in a single class will leave it easier to make the transition from MPEG1 to MPEG2. Summary There is a program developed which extracts the motion vectors from the bit stream. These vectors are stored in a fashion that allows the motion from frame to frame to be easily calculated. However additional information is needed to calculate this motion. The reasons we need this additional information are explained in the next chapter. Note the source code for the decoder has not been minimised. The code used to calculate the Inverse Discrete Cosine Transform, and also the code used to display the picture can be deleted. 20

25 Chapter 4 4 Finding the motion from frame to frame To find the motion from frame to frame, the motion vectors in the present frame are subtracted from the vectors in the previous frame. However, depending on what type of frame (I, P, or B) is in present and past, not all of the arrays can be used. An explication of this is given below. A vector defines a distance and a direction, it does not define a position. We have to know the vector s inital position (reference point) to find all the motion from frame to frame. Only vectors with the same reference point can be subtracted from each other. To illustrate, lets take the simple example of x moving across a portion of the screen as shown in Figure 4.1 x x x x x I B B B P Figure 4.1 Motion vectors associated with a moving picture. The arrow represents a forward vector and represents a backward vector [Note a forward vector doesn t have to be pointing forward, and a backward vector pointing backward. It is just the nameing convention for whither the reference frame in the past (forward) or future (backward).] The values for the vectors are given below: In the first frame there are no motion vectors In frame 2: forwardright = 2; forwarddown = -3; (2, -3) backwardright = -7; backwarddown = 5; (-7, 4) Frame 3: forward = (4, -6) backward = (-5, 1) Frame 4: forward = (7, -7) backward = (-2, 0) Frame 5: forward = (9, -7) Transition 1 To find the motion in the transition from frame one to frame two, we can only use the forward vector. The backward vector has no reference in the I frame. The motion is just (2, -3) Transition 2 Here, the forward and backward vectors can be used as both forward vectors have the same reference point and both backward vectors have the same reference point. 21

26 presentforward - pastforward = forward motion (4, -6) - (2, -3) = (2, -3) presentbackward - pastbackward = backward motion (-5, -1) - (-7, 4) = (2, -3) To find the total motion avarage the two results motionright = (2+2)/2 = 2 motiondown = (-3+-3)/2 = -3 Total motion = (2, -3) Note in this example the forward motion will always equal the backward motion but this is not usually the case in video. Transition 3 forward (7, -7) - (4, -6) = (3, -1) backward (-2, 0) - (-5, 1) = (3, -1) Total motion (3, -1) Transition 4 Both the forward and backward vectors can be used here. Both forward vectors are referenced to the same point and, as the B frames backward vector is referenced to the P frame. The P frame is said to have a zero backward vector. forward (9,-7) - (7, -7) = (2, 0) backward (0, 0) - (-2, 0) = (2, 0) Total motion (2, 0) The motion for the sequence is: (2,-3), (2, -3), (3, -1), (2, 0) 4.1 Considerations that have to be taken into account - Frame level Table 4.1 shows which types of vector can be subracted depending on what type of frame is in past and present. Table 4.1 Vector types that can be used in the transition from frame to frame past present Vector types that can be subtracted I B or P forward only I I None P B or P forward only P I None B B or P forward and backward B I backward only I Frame to B or P Frame: When going from an I frame to a B or P frame only the forward motion vectors can be used. The P frame will only have forward vectors, the B frames backward vectors can t be used as they have no reference in the I frame. 22

27 I Frame to I frame: There are no vectors present in either frame. P Frame to P or B frame: None of the backward vectors in the B frame have a reference in the P frame. Therefore only forward vectors can be used. P Frame to I Frame: The forward vectors in the P frame do not have a reference in the I frame. No motion can be found. B Frame to B or P Frame: Both forward and backward vectors can be used as both have the same reference point from frame to frame. B Frame to I Frame: Only the backward vectors are referenced in the I frame. 4.2Considerations that have to be taken into account - macroblock level In Chapter 2, all the different types of macroblock that can be present in a frame were described. Each macroblock in a B frame does not have both forward and backward vectors. Some macroblocks will only have either a forward or backward vector. Other macroblocks will have no vector at all, either because it is an Intra macroblock, or it is a skipped macroblock.this complicates the process of finding the motion from frame to frame even further. It is not a simple matter of subtracting all the values in one array from all the values in its corresponding past array. A more accurate representation of x moving accross a portion of the screen may be as shown in Figure 4.2 x x x x x I B B B P Figure 4.2 Realistic version of vectors associated with a moving picture In this example the transition from frame 1 to frame 2 can be calculated as before. If the second transition is calulated as before we get: forward motion: (4, -6) - (2, -3) = (2, -3) backward motion: (-5, 1) - (0, 0) = (-5, 1) Total motion = (-1.5, -1) This result is incorrect. To get the correct result, only the forward motion can be used. Similarly only the backward motion is used for the third transition. The motion for the final transition can not be found 23

28 because there is only a backward vector in frame 4 and only a forward vector in frame 5. Only similar types of vector can be subtracted from each other. Below are further rules to complement the rules that were established in table 4.1 Only if there is a similar type of vector (forward, backward or both) present in both frames can the motion be found. A reference frame is said to have all vectors equal to (0, 0) If there is a skipped macroblock in the present frame, there is zero motion for that transition. If there is a skipped macroblock in the previous frame, the motion for that transition can t be calulated. An exception to this is if there is also a skipped macroblock in the present frame in which case the motion will be zero. If there is an Intra macroblock in either the present or previous frame, the motion for that transition can t be calculated. Summary In this chapter the extra information needed to find the motion from frame to frame is described. A set of rules is established on how to find the motion. Note this set of rules is not rigid. By keeping track of other information, more vectors can be found. For example, if a record of the vector for a macroblock before a skipped macroblock is kept, the motion, in the transition between that skipped macroblock (or the final skipped macroblock in a series of skipped macroblocks) and a non intra macroblock can also be calculated. However, this will only further complicate the program. For a starting point, the rules created in this chapter should be sufficient. If the program does not perform satisfactorily this extra motion can be calculated. 24

29 Conclusion This project set out to extract the motion vectors from an MPEG stream. This information was to be used to calculate the motion of all objects from one frame to another. The first step of the project was to choose an MPEG1 decoder to extract and decode the motion vectors. The choice came down to a Java decoder and a C decoder. Two issues had to be taken into account when choosing the decoder; how fast the decoder could run, and how easily it could be modified. The Java decoder was chosen because although the MPEG bit stream is quite complicated, it is very well structured. Java s superior ability to deal with the complexity of the bit stream in an easy to follow manner outweighed the C decoder s superior processing time. Using the decoder, the motion vectors were extracted and decoded. The decoder was modified to allow the subtraction of all the motion vectors in the present frame (display order) from all the motion vectors in the previous frame (display order). All the modifications were put in a separate class. This means minimal alterations to the decoder s well structured code. The creation of a separate class with all the new code is important because, at some time in the future the MPEG2 standard may be used instead of the MPEG1 standard (the standard we are using at the moment). All the relevant code developed for the MPEG1 standard can be easily taken and used for the MPEG2 standard. On completion of the program, it was realised that in order to find the motion from one frame to another, it is not a simple matter of subtracting all the vectors in the present frame from all the vectors in the previous frame. A set of rules have to be followed. The rules were developed in two stages. First a general set of rules that only take into account what type of frame (I, P or B) the vectors are in were written. Then at a lower level, the macroblock types present in the frames were taken into consideration and a comprehensive set of rules were written. These rules give the true motion from frame to frame. The next step in this project is to incorporated the rules into the program. Finally, to enhance the program s performance, some of the decoder s source code can be deleted. The code which deals with decoding the pixel coefficients is irrelevant. Also, the code used to display the video can be omitted. To conclude, on accomplishing the task presented in this project (to extract the motion vectors from the MPEG stream) it was discovered that more information is needed in order to achieve the ultimate goal of finding the motion of objects from one frame to another. This extra information is identified. Also, a description is given on how to use this information to find the motion from frame to frame. References [1] [2] ISO/IEC , Genève, 1993 [3] K.R. Rao and J.J.Hwang, Techniques & Standards For Image Video & Audio Coding, Prentice Hall PTR, New Jersey, 1996 [4] [5] Aidan Totterdell, An Algorithm for detecting and classifying scene breaks in an MPEG1 video bit stream, Dublin City University,

30 Appendix A Code for the two methods, compute_motion_vector and motion_displacement [4] private int motion_displacement(int motion_code, int PMD, int motion_r) { int dmd, MD; 1); if (x_ward_f == 1 motion_code == 0) { dmd = motion_code; else { dmd = 1 + x_ward_f * (Math.abs(motion_code) - dmd += motion_r; if (motion_code < 0) dmd = -dmd; MD = PMD + dmd; if (MD > max) MD -= range; else if (MD < min) MD += range; return MD; public void compute_motion_vector(int motion_horiz_x_code, int motion_verti_x_code,int motion_horiz_x_r, int motion_verti_x_r) { recon_right_x_prev = recon_right_x = motion_displacement(motion_horiz_x_code, recon_right_x_prev, motion_horiz_x_r); if (Full_pel_x_vector) recon_right_x <<= 1; recon_down_x_prev = recon_down_x = motion_displacement(motion_verti_x_code, recon_down_x_prev, motion_verti_x_r); if (Full_pel_x_vector) recon_down_x <<= 1; right_x = recon_right_x >> 1; down_x = recon_down_x >> 1; right_half_x = (recon_right_x & 0x1)!= 0; down_half_x = (recon_down_x & 0x1)!= 0; right_x_col = recon_right_x >> 2; down_x_col = recon_down_x >> 2; right_half_x_col = (recon_right_x & 0x2)!= 0; down_half_x_col = (recon_down_x & 0x2)!= 0; 26

31 MPEG_video /*This is a skeleton structure of MPEG_video, just to document some of */ /*the things that have been added in. Once the resolution of the video */ /*is known Arrays is called and the size of all the arrays can be set. */ /*If the frame is I or P type the future vectors will become the present */ /*vectors in display order, if the frame is P type any vectors present */ /*in the frame are stored in future until its turn in display order comes */ /*(when another I or P frame comes in) */ /* When compute_motion_vector is called some added information is */ /*passed to it, the macroblockes address (row and column), what type of */ /*frame it is if compute_motion_vector is to calculate forward motion */ /*vectors, if it is to calculate backward motion vectors the arbitrary */ /*value 4 (don't confuse this 4 with a D_type frame) is passed, just */ /*to indicate the vectors are backward. */ import java.io.inputstream; import java.applet.applet; public class MPEG_video implements Runnable { private Array VideoArray = new Array(); MPEG_video () { public void run() { mpeg_stream.next_start_code(); do { Parse_sequence_header(); do { Parse_group_of_pictures(); private void Parse_sequence_header() { Width = mpeg_stream.get_bits(12); Height = mpeg_stream.get_bits(12); mb_width = (Width + 15) / 16; mb_height = (Height + 15) / 16; VideoArray.setDimensions(mb_height, mb_width); private void Parse_group_of_pictures() { frame. new frame do { VideoArray.pastEqualsPresent();//Store vectors of previos VideoArray.resetPresent();();// All Vectors are reset for the Parse_picture(); VideoArray.printArray(1);// Optional private void Parse_picture () { in present if (Pic_Type == P_TYPE Pic_Type == I_TYPE) { VideoArray.futureEqualsPresent(); // Take what is in future and put VideoArray.resetFuture(); // Reset future for new values do { Parse_slice(); 27

32 private void Parse_slice() { do { Parse_macroblock(); private void Parse_macroblock() { if (macro_block_motion_forward) { Forward.compute_motion_vector(motion_horiz_forw_code, motion_verti_forw_code, motion_horiz_forw_r, motion_verti_forw_r, mb_row, mb_column, Pic_Type); if (macro_block_motion_backward) { // motion vector for backward prediction exists b = 4; Backward.compute_motion_vector(motion_horiz_back_code, motion_verti_back_code, motion_horiz_back_r, motion_verti_back_r, mb_row, mb_column, b); motion_data /*This is a skeleton of the class motion_data there is very little added to it. */ /*In the method compute_motion vector some extra information is passed, as was */ /*documented in MPEG_video. All this extra information is passed straight to */ /*Arrays along with the values of the motion vectors (in half pixels) */ public class motion_data { private Array MotionArray = new Array(); //Create instance of The class Array public void init () { public void set_pic_data() { public void reset_prev() { /* The internal method "motion_displacement" computes the difference of the */ /* actual motion vector in respect to the last motion vector. Refer to */ /* ISO to understand tho coding of the motion displacement. */ private int motion_displacement(int motion_code, int PMD, int motion_r) { int dmd, MD; if (x_ward_f == 1 motion_code == 0) { dmd = motion_code; else { 28

33 dmd = 1 + x_ward_f * (Math.abs(motion_code) - 1); dmd += motion_r; if (motion_code < 0) dmd = -dmd; MD = PMD + dmd; if (MD > max) MD -= range; else if (MD < min) MD += range; return MD; /* The method "compute_motion_vector" computes the motion vector according to the */ /* values supplied by the "ScanThread". It uses the method "motion_displacement". */ /* The result are the motion vectors for the luminance and the chrominance blocks.*/ public void compute_motion_vector(int motion_horiz_x_code, int motion_verti_x_code, int motion_horiz_x_r, int motion_verti_x_r, int mr, int mc, int choosearray) { recon_right_x_prev = recon_right_x = motion_displacement(motion_horiz_x_code, recon_right_x_prev, motion_horiz_x_r); if (Full_pel_x_vector) recon_right_x <<= 1; recon_down_x_prev = recon_down_x = motion_displacement(motion_verti_x_code, recon_down_x_prev, motion_verti_x_r); if (Full_pel_x_vector) recon_down_x <<= 1; /* The motion vectors(in half pixels) is sent to Arrays, along with information on*/ /* which array they are to go.*/ MotionArray.fillArray(mr, mc, recon_right_x, recon_down_x, choosearray); public void get_area() { public void copy_area() { public void copy_unchanged() { public void put_area() { public void put_area() { Array /*The class Array is used for the storage of the Motion Vectors*/ /*Two instances of the class Array will be created, one in the class*/ /*MPEG-video called VideoArray. This instance will be used to first*/ /*create the size of the arrays, depending on the reselusion of the video clip.*/ /*This instance will also pass information regarding which arrray*/ /*the Motion Vectors should be in (Past or Present).*/ 29